The field engineers had a problem: they were making risk decisions with data from three different ML models, each running independently, each returning a different score, and no single system aggregating those scores into a decision. One model scored credit risk. Another scored operational risk. A third — newer, still being
The standard approach to incremental file ingestion is messy: track which files you've processed in a table, query the directory listing, compare, load only the new ones, update the tracker. It works, but it's boilerplate that every pipeline ends up reimplementing. Auto Loader is Databricks'
Every pipeline that reads from Kafka eventually encounters a message it can't process. Wrong schema, missing required field, value that fails a business rule, upstream system sent malformed JSON at 2am on a Tuesday. What you do with that message determines whether your pipeline is robust or just
Seven years since ADF was in preview. Time for an honest inventory. I've been running ADF in production for the better part of that timeline. The platform has changed substantially. My view of it has changed substantially. Here's where things actually stand in 2021 -- what'
The appeal of Structured Streaming in Databricks is that it uses the same DataFrame API you already know. The query that runs as a batch job on yesterday's data can run continuously on data as it arrives, with minimal changes. This is the architectural convergence that makes it
Consumer group lag is the number of messages between where your consumer has committed its offset and where the end of the partition currently is. Zero lag means you are caught up — every message that has been published has been consumed. Rising lag means you are falling behind. Lag at
I have described the three-stage streaming architecture in a few posts now — raw ingest, deserialize, merge. This post is the implementation detail: what it actually looks like in Databricks notebooks and Workflows, with Delta Lake as the intermediate store between stages. The Target Architecture Three notebooks, three Databricks jobs,
Databricks gives you a genuine choice between PySpark DataFrames and SQL for almost every operation. Both approaches produce identical Spark execution plans — there's no performance difference. The choice is about readability, team skill set, testability, and context. Here's how I decide. They Produce the Same Plan
Last time, we signed up for Postman, logged into the web portal, and issued a simple GET request to the NHL API for Teams info. This time, we’re going to learn to set some Parameters and configure our client to send credentials to the server for authorizing our access to...
This post started as a tweet From @MMarie, suggesting someone blog about getting started with Postman. I guess I’m lucky I spent so much time with devs who were building application programming interfaces (APIs) on top of databases I was building. I got to know the lingo and technology behind...
The data ingest pipeline was reliable until something upstream changed. A source system would update a row, the pipeline would pick it up on the next run, and whatever was downstream would see the change — but only after the pipeline completed and the Delta table got refreshed. That was fine
A client's security team has disabled public network access on every Azure data resource in production. Azure SQL: private endpoint only. Storage account: no public access. Azure Key Vault: private endpoint required. The network is locked down correctly. Now ADF needs to read from all of them. The