Posted in: Biml
Biml Solutions and Limited Rights
Handling source system metadata changes automatically are pretty easy with BimlScript. There are several solutions available online check here, here, and here.
Shannon Lowder
The Metadata Layer Problem: What Your Pipelines Should Know About Themselves
A mature data pipeline knows more than how to move data. It knows what data it read, what it produced, how that output relates to what it produced yesterday, and what would break if the upstream source changed its schema. Most pipelines today know almost none of this about themselves
DataType Conversion Utility for Biml
Catherine Wilhelmsen provided the basis for this quick utility. She explains the translation between SSIS, SQL Server and Biml data types. I simply wrapped
ADF Custom Activities: When the Built-In Connectors Still Aren't Enough
ADF has 90+ connectors. Ninety connectors and you still occasionally hit a source that isn't covered, an API with authentication too complex for the HTTP connector, or transformation logic that doesn't fit any built-in activity. For those cases, ADF provides the Custom Activity. The Custom
Don't Fight Your Platform: The Right Way to Combine Databricks and Self-Managed Jobs
The Databricks-skeptic move I see most often is this: run some jobs on Databricks, keep "control" over others with self-managed Spark on EMR or on-prem, and let both read from the same S3 data lake. The reasoning is that you get Databricks' managed runtime
Your First Expectation Suite: Organizing Data Quality Checks in Great Expectations
Running individual expectations against a DataFrame is useful, but it doesn't scale. When you have 20 expectations spread across a data pipeline script, they're going to get lost in the noise. What you need is a way to group expectations into a reusable, named, version-controllable
Posted in: Biml
BimlExpress Directions for the Interrogator
The Biml Interrogator demo was designed to show Biml Developers how to extract useful metadata from sources that do not natively support it. Originally it was
ADF and Azure Data Lake Storage: The Ingestion Architecture I've Settled On
I've been using ADLS Gen1 as the raw landing zone for analytics workloads for about a year and a half. I've iterated through a few approaches to the ADF-to-ADLS ingestion pattern and I've landed on something that works well enough to write
Azure Event Hubs vs. Apache Kafka: After Using Both in Production
I have now shipped production pipelines on both Azure Event Hubs and self-hosted Apache Kafka. The comparison posts I see online are mostly written by people who have done one in depth and read the documentation for the other. Here is a comparison from someone who has actually operated
Infrastructure as Code for Data Pipelines: Managing Airflow Like Software
Airflow DAG files are Python. That sentence should end the conversation about whether they belong in Git, get reviewed in pull requests, and get deployed through a pipeline. They are code. The same discipline that applies to your PySpark jobs applies here. And yet, the standard operating procedure at most
ADF v2 Triggers: Schedule, Tumbling Window, and Event-Based Compared
The v1 slice scheduling model was one of the most frequently misunderstood aspects of ADF. It was powerful once you internalized it, but it required internalization — you couldn't just look at it and understand what it was doing. ADF v2 replaces it with an explicit trigger model. Three
The Real Cost of Keeping a Streaming Consumer Running 24/7 — And When You Don't Have To
When people build their first streaming pipeline, there is a tacit assumption embedded in the word "streaming": the consumer must be running all the time. Always on. Watching for events. Reacting in real time. The picture is a process that never sleeps. This assumption costs money. Sometimes a