My data science project this year has me working with NOAA weather data — storm event records going back decades, delivered as CSVs, with all the consistency problems that implies. Field names that changed between years. Magnitude values stored as strings in some vintages and floats in others. Missing state codes.
On a recent project, we had a pipeline that worked perfectly in SQL Server, imported cleanly into Databricks, and produced subtly wrong results for six weeks before anyone noticed. The culprit: a DATETIME column that became a TIMESTAMP in Spark SQL, and a timezone assumption that was implicit in our
Git integration landed. Now let's talk about what you do with it. Because "pipelines in git" is table stakes — what you actually want is a deployment pipeline that takes your ADF changes from development to production the same way you ship application code. Here's
You've spent years reading execution plans. You know what a nested loop join costs. You can spot a missing index from the fat red arrow pointing the wrong direction. When something is slow in SQL Server, you click "Include Actual Execution Plan" and the answer is
In batch ETL, schema changes are annoying but manageable. You run a load, the job fails, you investigate, you fix the mapping, you re-run. The damage is limited: at worst, one load cycle's worth of data is missing or wrong, and you fix it before it reaches the
The scenario: your Kafka consumer is humming along, processing thousands of messages per hour, when it hits one that does not match the expected schema. The JSON has a field with the wrong type. The Avro deserialization throws an exception. Maybe the message is simply empty bytes. What happens next
Azure Event Hubs Kafka protocol support went generally available this year, and it is a bigger deal than most of the announcement posts made it sound. The practical implication: any Kafka producer or consumer can connect to Event Hubs by changing a connection string. No SDK swap. No code changes.
Four years. That's how long I've been asking for git integration in Azure Data Factory. Since 2014, every ADF project I've touched has had the same operational hazard: someone makes an emergency fix in the portal, doesn't document it, and now your
The Biml Interrogator demo was designed to show Biml Developers how to extract useful metadata from sources that do not natively support it. Originally it was designed for delimited text files and Excel files, but it could be used for other sources with a little work. My demos walk you through the d
Posted in: Microsoft SQL
How quickly do you get from the business coming to you with "we need a data warehouse" to delivering that warehouse? If you've sat through any Biml talk, you've
February 2018. Azure Data Factory v2 is officially generally available. After a preview that stretched through 2016 and 2017, the thing is real now. So what does GA actually mean for the people who've been running it in production already? What GA Actually Delivers Three things change with
Posted in: Microsoft SQL
When you build an Alexa skill you’ve got two rich sources of information describing how users are interacting with your skill. The first is the data in the Alexa Skills Kit request object. The request object is the JSON representation of what you said to your Alexa device. Alexa Voice...