Posted in: Microsoft SQL
f you’ve been working with Azure in the last few years, you’ve noticed there is a relatively new feature that’s supposed to make identity management easier with your Azure solutions: Azure Active Directory (AAD). This feature does work with Azure SQL Servers and Azure SQL Databases. But it may not..
Posted in: Biml
One of the questions I get when teaching others how to use Biml is how do you deal with sensitive information like usernames and passwords in your Biml Solution
Posted in: Biml
Handling source system metadata changes automatically are pretty easy with BimlScript. There are several solutions available online check here, here, and here.
A mature data pipeline knows more than how to move data. It knows what data it read, what it produced, how that output relates to what it produced yesterday, and what would break if the upstream source changed its schema. Most pipelines today know almost none of this about themselves
Catherine Wilhelmsen provided the basis for this quick utility. She explains the translation between SSIS, SQL Server and Biml data types. I simply wrapped
ADF has 90+ connectors. Ninety connectors and you still occasionally hit a source that isn't covered, an API with authentication too complex for the HTTP connector, or transformation logic that doesn't fit any built-in activity. For those cases, ADF provides the Custom Activity. The Custom Activity
The Databricks-skeptic move I see most often is this: run some jobs on Databricks, keep "control" over others with self-managed Spark on EMR or on-prem, and let both read from the same S3 data lake. The reasoning is that you get Databricks' managed runtime for the jobs
Running individual expectations against a DataFrame is useful, but it doesn't scale. When you have 20 expectations spread across a data pipeline script, they're going to get lost in the noise. What you need is a way to group expectations into a reusable, named, version-controllable artifact
Posted in: Biml
The Biml Interrogator demo was designed to show Biml Developers how to extract useful metadata from sources that do not natively support it. Originally it was
I have now shipped production pipelines on both Azure Event Hubs and self-hosted Apache Kafka. The comparison posts I see online are mostly written by people who have done one in depth and read the documentation for the other. Here is a comparison from someone who has actually operated both.
I've been using ADLS Gen1 as the raw landing zone for analytics workloads for about a year and a half. I've iterated through a few approaches to the ADF-to-ADLS ingestion pattern and I've landed on something that works well enough to write up. The
Airflow DAG files are Python. That sentence should end the conversation about whether they belong in Git, get reviewed in pull requests, and get deployed through a pipeline. They are code. The same discipline that applies to your PySpark jobs applies here. And yet, the standard operating procedure at most