I’m pulling the directions for using the interrogator out of the main page into this separate page since I don’t have the project files on hand. You can use the BimlExpress files with these instructions, but you will need to adjust the code slightly. If you run in to troubles,...
I've been building parameterized pipelines in ADF v2 for about eight months now. It's time to do a proper deep dive — how parameterization works, the expression language, the patterns that hold up in production, and the gotchas I've run into. If you've
At most organizations, data engineering and data science are separate teams with separate infrastructure. The engineers run production Spark jobs on EMR or Databricks Jobs. The scientists run experiments in Jupyter notebooks on standalone EC2 instances or SageMaker. They share data through S3 and share nothing else — different Spark versions,
In 2012 I started writing about tSQLt — a framework that brought unit testing discipline to SQL Server stored procedures. The argument was simple: data logic deserves the same testing rigor as application logic. If you'd write a test for a C# method, you should write one for a
There's a tool called dbt — data build tool — that Fishtown Analytics has been developing and using with clients for the last several months. It's not widely known yet. I think it's going to matter, so I want to explain the core idea before it
The original Spark Streaming API — DStreams — was a competent piece of engineering surrounded by an awkward abstraction. You worked with DStream objects that were not quite RDDs and not quite DataFrames. The Python and Java APIs behaved differently in subtle ways. Windowing operations were powerful but verbose. Most annoyingly, DStreams
Three years ago I started writing about Azure Data Factory. At that point ADF v1 was a preview with an interesting connector library, a confusing scheduling model, and no git integration. Two years ago I was in production with it and cataloging the gaps. Last year Microsoft previewed v2 and
AWS announced Glue at re:Invent this week. The pitch: serverless ETL on managed Spark, plus a Data Catalog underneath it that automatically crawls your S3 buckets and makes their schemas queryable via Athena. It's a lot to unpack, and the announcement materials conflate two things that deserve
Last month I wrote about Snowflake's architecture — what's genuinely good about it and what warrants scrutiny. I promised a direct comparison with Databricks. Here it is. I'm going to be specific about where the lock-in surfaces are, because this is a decision with
I've been trying to build a proper metadata-driven ingest framework in ADF since 2015. The 2015 attempt with v1 was a partial success: it worked, it was clever, and it had rough edges that required explanation every time someone new touched it. The core problem was the
I've been spending time with Snowflake this quarter, and I want to give it a fair technical read before it becomes the unexamined industry standard. The architecture is genuinely clever. The execution against that architecture is strong. And there are specific questions you should be asking before you
Every large data engineering shop I've worked with in the last ten years has SSIS packages. Some shops have hundreds of them — years of business logic encapsulated in .dtsx files, tested, documented (sometimes), and running reliably on on-premises SQL Server infrastructure. These shops look at Azure and