Posted in: Microsoft SQL
Biml and Oracle Connections
Everyone who knows me, knows I'm a Microsoft data platform professional. I do prefer their solutions over most other solutions. I know there have been a few
Latest
Posted in: Microsoft SQL
SQL Azure and Azure Active Directory: Part Two
After getting AD Password Authentication working with my Azure SQL server, I moved on and set up a local Active Directory domain on my home network. I don’t think this is something every data professional should try and tackle. I count myself extremely fortunate to have made friends with IT...
Posted in: Microsoft SQL
SQL Azure and Azure Active Directory
f you’ve been working with Azure in the last few years, you’ve noticed there is a relatively new feature that’s supposed to make identity management easier with your Azure solutions: Azure Active Directory (AAD). This feature does work with Azure SQL Servers and Azure SQL Databases. But it may not..
Posted in: Biml
SSIS Parameters and Sensitive Information
One of the questions I get when teaching others how to use Biml is how do you deal with sensitive information like usernames and passwords in your Biml Solution
Posted in: Biml
Biml Solutions and Limited Rights
Handling source system metadata changes automatically are pretty easy with BimlScript. There are several solutions available online check here, here, and here.
The Metadata Layer Problem: What Your Pipelines Should Know About Themselves
A mature data pipeline knows more than how to move data. It knows what data it read, what it produced, how that output relates to what it produced yesterday, and what would break if the upstream source changed its schema. Most pipelines today know almost none of this about themselves
DataType Conversion Utility for Biml
Catherine Wilhelmsen provided the basis for this quick utility. She explains the translation between SSIS, SQL Server and Biml data types. I simply wrapped
ADF Custom Activities: When the Built-In Connectors Still Aren't Enough
ADF has 90+ connectors. Ninety connectors and you still occasionally hit a source that isn't covered, an API with authentication too complex for the HTTP connector, or transformation logic that doesn't fit any built-in activity. For those cases, ADF provides the Custom Activity. The Custom Activity
Don't Fight Your Platform: The Right Way to Combine Databricks and Self-Managed Jobs
The Databricks-skeptic move I see most often is this: run some jobs on Databricks, keep "control" over others with self-managed Spark on EMR or on-prem, and let both read from the same S3 data lake. The reasoning is that you get Databricks' managed runtime for the jobs
Your First Expectation Suite: Organizing Data Quality Checks in Great Expectations
Running individual expectations against a DataFrame is useful, but it doesn't scale. When you have 20 expectations spread across a data pipeline script, they're going to get lost in the noise. What you need is a way to group expectations into a reusable, named, version-controllable artifact
Posted in: Biml
BimlExpress Directions for the Interrogator
The Biml Interrogator demo was designed to show Biml Developers how to extract useful metadata from sources that do not natively support it. Originally it was
Azure Event Hubs vs. Apache Kafka: After Using Both in Production
I have now shipped production pipelines on both Azure Event Hubs and self-hosted Apache Kafka. The comparison posts I see online are mostly written by people who have done one in depth and read the documentation for the other. Here is a comparison from someone who has actually operated both.