Posted in: Microsoft SQL
It’s been more than a year since I contributed anything here. I could say it’s because it’s been a really good time to be in consulting. I could also say, I’ve been finding more and more ways to fill up my time. In the end, the biggest reason is, I...
Data engineering in 2019 had one defining event: Delta Lake going open source in October. Everything else was incremental improvement. It's worth taking a few minutes to document what actually happened this year before the context fades. The Delta Lake Open Source Release This is the story of
Five years. I shipped my first ADF v1 pipeline to production in late 2014. At the time it was called "Azure Data Factory" and it was definitely not what I'd call mature. It had copy jobs, a JSON-based pipeline definition, and an on-premises gateway
When I'm asked how production Databricks pipelines are going, my honest answer is "I look at six things." Not dashboards full of metrics I don't read. Six specific things that tell me quickly whether the pipeline is healthy, whether the data is good, and
Two years ago I wrote a post about moving from SQL Server to Databricks — what skills transfer, where the mental model breaks down, what to learn first. Most of it still holds. But there are things I would change if I were starting that journey today, and things I wish
Please enable cookies. Email Protection You are unable to access this email address toyboxcreations.net The website from which you got to this page is protected by Cloudflare. Email addresses on that page have been hidden in order to keep them from being accessed by malicious bots. You must enable J
Testing Databricks notebooks is awkward. The testing patterns from regular Python projects (pytest, unittest, mocks, fixtures) work against local Python code, but Databricks pipelines have dependencies that don't exist outside a cluster — a running SparkSession, DBFS paths, dbutils, secrets. Most teams deal with this by not testing at
Consumer group lag is the canary in the coal mine for a Kafka pipeline under load. When lag starts rising — slowly, then faster — it usually means one of two things: the source volume increased, or the consumer is doing too much per message to keep up. The first is a
The Databricks CLI is the tool that turns a series of UI clicks into a script. If you're managing a production Databricks workspace — running jobs, deploying notebooks, rotating secrets, managing clusters — doing it through the UI is fine until you need to do it reliably, repeatedly, and without
Getting data from SQL Server into Delta Lake is one of the most common first tasks when a team starts moving to Databricks. The JDBC path works, but there are enough gotchas in the full end-to-end flow that it's worth walking through a complete, production-realistic
I started trying to build a metadata-driven ADF framework in 2015, with ADF v1. It was miserable. V1 didn't have ForEach. It didn't have parameters. The closest you could get was a .NET custom activity that read a config table and emitted dynamic pipeline definitions
I've been writing about getting tSQLt into a CI pipeline since 2013 — the principle being that tests which only run when a developer remembers to run them don't actually protect you. The same argument applies to data quality checks. An expectation suite that lives in a