I've been using ADLS Gen1 as the raw landing zone for analytics workloads for about a year and a half. I've iterated through a few approaches to the ADF-to-ADLS ingestion pattern and I've landed on something that works well enough to write up. The
Airflow DAG files are Python. That sentence should end the conversation about whether they belong in Git, get reviewed in pull requests, and get deployed through a pipeline. They are code. The same discipline that applies to your PySpark jobs applies here. And yet, the standard operating procedure at most
When people build their first streaming pipeline, there is a tacit assumption embedded in the word "streaming": the consumer must be running all the time. Always on. Watching for events. Reacting in real time. The picture is a process that never sleeps. This assumption costs money. Sometimes a
The v1 slice scheduling model was one of the most frequently misunderstood aspects of ADF. It was powerful once you internalized it, but it required internalization — you couldn't just look at it and understand what it was doing. ADF v2 replaces it with an explicit trigger model. Three
I’m pulling the directions for using the interrogator out of the main page into this separate page since I don’t have the project files on hand. You can use the BimlExpress files with these instructions, but you will need to adjust the code slightly. If you run in to troubles,...
I've been building parameterized pipelines in ADF v2 for about eight months now. It's time to do a proper deep dive — how parameterization works, the expression language, the patterns that hold up in production, and the gotchas I've run into. If you've
At most organizations, data engineering and data science are separate teams with separate infrastructure. The engineers run production Spark jobs on EMR or Databricks Jobs. The scientists run experiments in Jupyter notebooks on standalone EC2 instances or SageMaker. They share data through S3 and share nothing else — different Spark versions,
In 2012 I started writing about tSQLt — a framework that brought unit testing discipline to SQL Server stored procedures. The argument was simple: data logic deserves the same testing rigor as application logic. If you'd write a test for a C# method, you should write one for a
The original Spark Streaming API — DStreams — was a competent piece of engineering surrounded by an awkward abstraction. You worked with DStream objects that were not quite RDDs and not quite DataFrames. The Python and Java APIs behaved differently in subtle ways. Windowing operations were powerful but verbose. Most annoyingly, DStreams
There's a tool called dbt — data build tool — that Fishtown Analytics has been developing and using with clients for the last several months. It's not widely known yet. I think it's going to matter, so I want to explain the core idea before it
Three years ago I started writing about Azure Data Factory. At that point ADF v1 was a preview with an interesting connector library, a confusing scheduling model, and no git integration. Two years ago I was in production with it and cataloging the gaps. Last year Microsoft previewed v2 and
AWS announced Glue at re:Invent this week. The pitch: serverless ETL on managed Spark, plus a Data Catalog underneath it that automatically crawls your S3 buckets and makes their schemas queryable via Athena. It's a lot to unpack, and the announcement materials conflate two things that deserve