This is the question I get most often from clients evaluating their Azure data platform options in 2018: "We're using ADF for data movement. We're using Databricks for transformations. Do we actually need both? Can't Databricks just do everything?" The short answer:
Microsoft announced in September 2018 that Azure Data Lake Analytics is entering a maintenance-only phase, with no major feature work planned. No new regions. No investment in U-SQL tooling. Just: it works, we're not growing it. If you've been following along, you know I've
I have had this exact conversation at least six times with different clients over the past four years: "Can ADF read from SAP?" "Not natively. You need a custom connector, a third-party ETL tool, or a flat file extract from the SAP team." "..." 2018
Posted in: Microsoft SQL
After I finished evaluating the Storm Events Database from NOAA, I was convinced we needed to look for machine recorded events. When you start poking around the
Posted in: Microsoft SQL
This entry picks up the story behind my first data science project predicting hail damage to farms. In this article we identify data quality issues in our first
Posted in: Microsoft SQL
Early on in my new role I was asked to find out how risky it was to offer hail insurance for a given property. If you haven't worked with insurance before
Machine learning models have a talent for finding patterns in garbage data. Give a model training data with a systematic error and it will dutifully learn to predict based on that error. The model validates fine. It performs great in testing. It ships to production. Then the underlying data error
microsoft sql
In my latest professional change I've moved into a role with a bright and shiny title: Data Scientist. I've heard all the hype around Data Science, Machine