Shannon Lowder
Posted in: Microsoft SQL
The Big Cost in Data Science
You hear time and time again how 50 to 80 Percent of Data science projects is spent on data wrangling munging and transformation of raw data into something usable. For me personally. I’ve automated a lot of those steps. I built tools over the last 20 years that help me...
ADF vs. Azure Databricks: Orchestrator or Compute, or Both?
This is the question I get most often from clients evaluating their Azure data platform options in 2018: "We're using ADF for data movement. We're using Databricks for transformations. Do we actually need both? Can't Databricks just do everything?" The short answer:
Writing SQL in Databricks That Your Team Can Actually Maintain
Microsoft Abandons Azure Data Lake Analytics: Why I'm Now Looking at Databricks
Microsoft announced in September 2018 that Azure Data Lake Analytics is entering a maintenance-only phase, with no major feature work planned. No new regions. No investment in U-SQL tooling. Just: it works, we're not growing it. If you've been following along, you know I've
MERGE in Delta Lake: Same Syntax, Different Engine
ADF Connector Update 2018: SAP Finally Shows Up
I have had this exact conversation at least six times with different clients over the past four years: "Can ADF read from SAP?" "Not natively. You need a custom connector, a third-party ETL tool, or a flat file extract from the SAP team." "..." 2018
Delta Lake Partitioning: Unlearning the SQL Server Mindset
Replacing Stored Procedures in Databricks: The Right Way to Think About It
Posted in: Microsoft SQL
NOAA Radar and Severe Weather Data Inventory
After I finished evaluating the Storm Events Database from NOAA, I was convinced we needed to look for machine recorded events. When you start poking around the
Posted in: Microsoft SQL
Data Quaity Issues
This entry picks up the story behind my first data science project predicting hail damage to farms. In this article we identify data quality issues in our first
Posted in: Microsoft SQL
Data Science Project 1: Predicting Hail Damages
Early on in my new role I was asked to find out how risky it was to offer hail insurance for a given property. If you haven't worked with insurance before