Posted in: Microsoft SQL
There are many reasons to integrate your on-premises domain with your Azure Active Directory (AAD) domain. You might want self-service password resets, you might want two-factor authorization, you may want a single identity to manage for each of your users. No matter what the reason you want to inte
Posted in: Microsoft SQL
I’ve worked with a handful of companies helping them integrate their on-premises infrastructure with Azure. About half of them already have an Active Directory domain. If you’re coming from one of those companies, you can skip this blog entry and wait for the next one. But the other half of...
Posted in: Microsoft SQL
Back in college, I had a workstation I named Beast. I named it for the furry genius mutant in the X-men. It was the first multi-processor machine I ever built. By today’s standard, I’m sure my iPhone has more processing power. But, for those days it was an absolute beast!...
I ran my first Azure Data Factory pipeline in 2014. It was a v1 preview, there was no git integration, the connector list was short enough to memorize, and the documentation was aspirational at best. I compared it to SSIS and found it lacking in most dimensions that mattered for
Posted in: Microsoft SQL
It’s been more than a year since I contributed anything here. I could say it’s because it’s been a really good time to be in consulting. I could also say, I’ve been finding more and more ways to fill up my time. In the end, the biggest reason is, I...
Data engineering in 2019 had one defining event: Delta Lake going open source in October. Everything else was incremental improvement. It's worth taking a few minutes to document what actually happened this year before the context fades. The Delta Lake Open Source Release This is the story of
Five years. I shipped my first ADF v1 pipeline to production in late 2014. At the time it was called "Azure Data Factory" and it was definitely not what I'd call mature. It had copy jobs, a JSON-based pipeline definition, and an on-premises gateway that mostly
When I'm asked how production Databricks pipelines are going, my honest answer is "I look at six things." Not dashboards full of metrics I don't read. Six specific things that tell me quickly whether the pipeline is healthy, whether the data is good, and
Two years ago I wrote a post about moving from SQL Server to Databricks — what skills transfer, where the mental model breaks down, what to learn first. Most of it still holds. But there are things I would change if I were starting that journey today, and things I wish
Please enable cookies. Email Protection You are unable to access this email address toyboxcreations.net The website from which you got to this page is protected by Cloudflare. Email addresses on that page have been hidden in order to keep them from being accessed by malicious bots. You must enable J
Testing Databricks notebooks is awkward. The testing patterns from regular Python projects (pytest, unittest, mocks, fixtures) work against local Python code, but Databricks pipelines have dependencies that don't exist outside a cluster — a running SparkSession, DBFS paths, dbutils, secrets. Most teams deal with this by not testing at
Consumer group lag is the canary in the coal mine for a Kafka pipeline under load. When lag starts rising — slowly, then faster — it usually means one of two things: the source volume increased, or the consumer is doing too much per message to keep up. The first is a