Databricks announced Photon at Data + AI Summit last month, and it's now available in Databricks Runtime 9.1. The pitch: a native vectorized query engine written in C++ that replaces the JVM-based Spark SQL execution for analytical queries. Faster scans, faster aggregations, faster joins — without changing any
Spark Structured Streaming can read from Kafka indefinitely — theoretically. The practical limit is how fast the downstream pipeline can process incoming messages relative to how fast the upstream producers are sending them. When those rates diverge, you have a backpressure problem: more data arriving than the consumer can handle, queues
GPT-3's context window is 4,096 tokens. That number sounds generous until you try to use the model for real pipeline work — and then it becomes the most important constraint in your workflow. Four thousand tokens is roughly 3,000 words. That's enough for a
If you've ever trained a model, gotten good results, then lost track of which hyperparameters you used, you understand why MLflow exists. It's an experiment tracking system — a log of every training run with the parameters, metrics, and artifacts that produced each result. On Databricks, it&
The MLflow Model Registry gives you versioning and stage transitions. The next question is serving: how do you turn a registered model into a REST endpoint that a downstream application can actually call? Databricks has a first-party answer for this now, and it's worth understanding both what
Your Kafka producer is sending orders. Your consumer is writing them to Delta Lake. Everything works in the dev environment where all the producers are your own Python scripts. Then someone onboards a new upstream service, it starts sending messages with a field renamed, a type changed, a required field
ADF has five trigger types in 2021. Most teams use one -- the Schedule Trigger -- for everything. That's fine for simple workloads. It's suboptimal for workloads that would benefit from the other four. Here's the complete map of what each trigger type does and when
A question I've been asked more than once since moving to the Databricks stack: "We're using Delta Lake — doesn't that already handle data quality?" The answer is: partly. Delta handles one category of data quality problems very well and is silent on
You've trained a model. MLflow captured the run — parameters, metrics, the pickled artifact. It lives in the MLflow tracking server, tagged with the experiment name and a run ID that's forty random characters you'll never remember. Now you need to put it in production,
The first time you connect to a new data source, you're flying blind. You have a table name, maybe a schema diagram if you're lucky, and a contact at the source team who can answer questions "when they get a chance." You need to
Microsoft's ADF documentation covers the building blocks thoroughly: how to create a pipeline, how to parameterize an activity, how to use ForEach. What it doesn't give you is a complete, production-ready metadata-driven framework. The documentation shows you the Lego blocks. It does not show
Schema evolution is a persistent source of pain in data engineering. Source systems add columns. Business requirements add metrics. Downstream consumers break when they encounter columns they weren't expecting, or fail to notice that a column they were expecting disappeared. Delta Lake has built-in mechanisms for managing