Synapse Pipelines and ADF Pipelines are functionally near-identical, and the Microsoft explanation for why they're different products has not convinced me in the six months since Synapse Analytics went GA. Let me show you what I mean, and then tell you what I think is actually happening. The
I've been containerizing everything in the home lab this year — SQL Server, services, pipelines. The natural next step was getting Great Expectations into a container as well, so validation checks run with the same environment everywhere: my local machine, a CI build agent, a Kubernetes job. No "
Every Databricks workspace starts with the same problem: no guardrails. Developers can create 20-node clusters, leave them running indefinitely, install whatever libraries they want, and access any data they have network access to. The first cloud bill after a team gets comfortable is usually the motivation for adding structure. The
Delta Lake's optimistic concurrency model means most concurrent write conflicts resolve automatically. But "most" isn't "all," and the cases that don't resolve automatically tend to happen at the worst possible time — two high-priority jobs running simultaneously during a critical load
A repeating pattern in Databricks pipelines: you have files landing in ADLS or S3 — from Kafka Capture, from Event Hubs Capture, from an SFTP drop — and you need to pick up every new file as it arrives. The naive implementation is a batch job that lists the directory, filters for
Your Azure SQL database has public network access disabled. Private endpoint is configured, DNS resolves correctly inside the VNet. Everything is locked down the way your security team wants it. Now you need ADF to move data out of it. The default Azure Integration Runtime can't reach it.
Spark 3.0 is in preview and the feature I'm watching most closely is Adaptive Query Execution. If it ships as documented, it addresses one of the most common performance frustrations in Spark 2.x: the query optimizer making bad choices because it doesn't know your
Column additions to Delta Lake tables are straightforward. You enable schema evolution, write a DataFrame with the new column, and Delta handles the rest. The part that's less documented is what happens to the systems reading that table — and understanding that is what keeps a schema change from
PySpark DataFrames are the native Spark API. They're powerful, they're explicit about what's happening at the execution level, and they're the thing data engineers are usually writing. But there's a class of operations where Python data scientists are already experts
Reading a SQL Server table into Spark via JDBC is straightforward until the table is large. At that point, the naive approach — one JDBC connection, all rows, read sequentially — becomes impractically slow. The fix is parallel JDBC reads using a partition column, but the standard partition column approach has a
Posted in: Microsoft SQL
I’ve had two different clients approach me in the past few weeks asking how to deal with some odd behavior in Biml when working with Oracle sources. I have to be honest, working with Oracle is painful. The good news is there are ways to work around the pain. You...
Posted in: Microsoft SQL
In my last entry, I walked through a vanilla setup for Azure Active Directory (AAD) Connect. Now, what if you need something a little different? Well, let’s talk through some of the features I’ve been successful with turning on and configuring. Return to AAD Connect Whenever you want to make...