There was a period in ADF's early life when I kept a mental list of sources that weren't supported. It was a meaningful list -- long enough to influence architecture decisions. That period is over. ADF crossed 90 supported connectors in 2020. The blog posts from 2014-
When a Spark job runs slower than expected, the first thing I check is partition count. Not cluster size. Not memory settings. Partitions. Because if your partition count is wrong, adding more machines doesn't help — you're just adding more workers standing around with nothing to do.
You stand up a 10-node Databricks cluster. You open a notebook. You write some pandas code to process a 50GB dataset. It runs slower than it did on your laptop. You file a ticket asking why Databricks is worse than your laptop. I've been that person. Here&
Someone on my team last year asked me how Spark distributes work across a cluster. I started drawing boxes on a whiteboard — driver, executors, partitions, shuffle — and watched eyes glaze over. Then I grabbed the bag of M&Ms from the breakroom and the conversation completely changed. Let me
Before diving into the architecture and the API, you need a cluster to run code against. Databricks Community Edition gives you one for free — no credit card, no time limit on account access, one active cluster at a time. It's not production-grade (it's a single-
The question comes up on every Databricks project: should this pipeline run as a stream or as a batch? Most of the content on this topic presents it as a streaming-vs-batch debate with streaming as the obviously modern choice. The reality is more useful — batch is the right
If you've spent your career in SQL Server, the first time someone says "Spark" you probably picture something on fire. The second time, you hear "it's like Hadoop but faster." Neither of these is useful. Let me give you the version I
Five table properties, applied consistently, turn a Delta table that's "working" into one that performs well over time. Most teams discover them through performance problems — the table was fine for six months and then something changed. The changes I'm describing here prevent most of
Synapse Pipelines and ADF Pipelines are functionally near-identical, and the Microsoft explanation for why they're different products has not convinced me in the six months since Synapse Analytics went GA. Let me show you what I mean, and then tell you what I think is actually happening.
I've been containerizing everything in the home lab this year — SQL Server, services, pipelines. The natural next step was getting Great Expectations into a container as well, so validation checks run with the same environment everywhere: my local machine, a CI build agent, a Kubernetes job. No "
Every Databricks workspace starts with the same problem: no guardrails. Developers can create 20-node clusters, leave them running indefinitely, install whatever libraries they want, and access any data they have network access to. The first cloud bill after a team gets comfortable is usually the motivation for adding structure.
Delta Lake's optimistic concurrency model means most concurrent write conflicts resolve automatically. But "most" isn't "all," and the cases that don't resolve automatically tend to happen at the worst possible time — two high-priority jobs running simultaneously during a critical