If you've spent your career in SQL Server, the first time someone says "Spark" you probably picture something on fire. The second time, you hear "it's like Hadoop but faster." Neither of these is useful. Let me give you the version I
Five table properties, applied consistently, turn a Delta table that's "working" into one that performs well over time. Most teams discover them through performance problems — the table was fine for six months and then something changed. The changes I'm describing here prevent most of
Synapse Pipelines and ADF Pipelines are functionally near-identical, and the Microsoft explanation for why they're different products has not convinced me in the six months since Synapse Analytics went GA. Let me show you what I mean, and then tell you what I think is actually happening. The
I've been containerizing everything in the home lab this year — SQL Server, services, pipelines. The natural next step was getting Great Expectations into a container as well, so validation checks run with the same environment everywhere: my local machine, a CI build agent, a Kubernetes job. No "
Every Databricks workspace starts with the same problem: no guardrails. Developers can create 20-node clusters, leave them running indefinitely, install whatever libraries they want, and access any data they have network access to. The first cloud bill after a team gets comfortable is usually the motivation for adding structure. The
Delta Lake's optimistic concurrency model means most concurrent write conflicts resolve automatically. But "most" isn't "all," and the cases that don't resolve automatically tend to happen at the worst possible time — two high-priority jobs running simultaneously during a critical load
A repeating pattern in Databricks pipelines: you have files landing in ADLS or S3 — from Kafka Capture, from Event Hubs Capture, from an SFTP drop — and you need to pick up every new file as it arrives. The naive implementation is a batch job that lists the directory, filters for
Your Azure SQL database has public network access disabled. Private endpoint is configured, DNS resolves correctly inside the VNet. Everything is locked down the way your security team wants it. Now you need ADF to move data out of it. The default Azure Integration Runtime can't reach it.
Spark 3.0 is in preview and the feature I'm watching most closely is Adaptive Query Execution. If it ships as documented, it addresses one of the most common performance frustrations in Spark 2.x: the query optimizer making bad choices because it doesn't know your
Column additions to Delta Lake tables are straightforward. You enable schema evolution, write a DataFrame with the new column, and Delta handles the rest. The part that's less documented is what happens to the systems reading that table — and understanding that is what keeps a schema change from
PySpark DataFrames are the native Spark API. They're powerful, they're explicit about what's happening at the execution level, and they're the thing data engineers are usually writing. But there's a class of operations where Python data scientists are already experts
Reading a SQL Server table into Spark via JDBC is straightforward until the table is large. At that point, the naive approach — one JDBC connection, all rows, read sequentially — becomes impractically slow. The fix is parallel JDBC reads using a partition column, but the standard partition column approach has a