Your Azure SQL database has public network access disabled. Private endpoint is configured, DNS resolves correctly inside the VNet. Everything is locked down the way your security team wants it. Now you need ADF to move data out of it. The default Azure Integration Runtime can't reach it.
A repeating pattern in Databricks pipelines: you have files landing in ADLS or S3 — from Kafka Capture, from Event Hubs Capture, from an SFTP drop — and you need to pick up every new file as it arrives. The naive implementation is a batch job that lists the directory, filters for
Spark 3.0 is in preview and the feature I'm watching most closely is Adaptive Query Execution. If it ships as documented, it addresses one of the most common performance frustrations in Spark 2.x: the query optimizer making bad choices because it doesn't know your
Column additions to Delta Lake tables are straightforward. You enable schema evolution, write a DataFrame with the new column, and Delta handles the rest. The part that's less documented is what happens to the systems reading that table — and understanding that is what keeps a schema change from
PySpark DataFrames are the native Spark API. They're powerful, they're explicit about what's happening at the execution level, and they're the thing data engineers are usually writing. But there's a class of operations where Python data scientists are already experts
Reading a SQL Server table into Spark via JDBC is straightforward until the table is large. At that point, the naive approach — one JDBC connection, all rows, read sequentially — becomes impractically slow. The fix is parallel JDBC reads using a partition column, but the standard partition column approach has a
Posted in: Microsoft SQL
I’ve had two different clients approach me in the past few weeks asking how to deal with some odd behavior in Biml when working with Oracle sources. I have to be honest, working with Oracle is painful. The good news is there are ways to work around the pain. You...
Posted in: Microsoft SQL
In my last entry, I walked through a vanilla setup for Azure Active Directory (AAD) Connect. Now, what if you need something a little different? Well, let’s talk through some of the features I’ve been successful with turning on and configuring. Return to AAD Connect Whenever you want to make...
Posted in: Microsoft SQL
There are many reasons to integrate your on-premises domain with your Azure Active Directory (AAD) domain. You might want self-service password resets, you might want two-factor authorization, you may want a single identity to manage for each of your users. No matter what the reason you want to inte
Posted in: Microsoft SQL
I’ve worked with a handful of companies helping them integrate their on-premises infrastructure with Azure. About half of them already have an Active Directory domain. If you’re coming from one of those companies, you can skip this blog entry and wait for the next one. But the other half of...
Posted in: Microsoft SQL
Back in college, I had a workstation I named Beast. I named it for the furry genius mutant in the X-men. It was the first multi-processor machine I ever built. By today’s standard, I’m sure my iPhone has more processing power. But, for those days it was an absolute beast!...
I ran my first Azure Data Factory pipeline in 2014. It was a v1 preview, there was no git integration, the connector list was short enough to memorize, and the documentation was aspirational at best. I compared it to SSIS and found it lacking in most dimensions that mattered for