Shannon Lowder
Understanding DBFS: What It Is and What It Isn't
Stop Using Notebooks Like SQL Editors
Writing Custom Expectations in Great Expectations
The built-in expectation library covers the common cases well: nulls, ranges, value sets, row counts, regex patterns. But real data has domain-specific rules that a general-purpose library can't anticipate. A storm event can't end before it starts. A geocoded latitude must correspond to
ADF Mapping Data Flows: Code-Free Transformations Built on Spark
The single biggest complaint about ADF since 2014 has been the same: you can copy data, but you can't transform it. Copy Activity moves data from A to B. If you need to derive a column, aggregate rows, join two sources, or apply business logic — you had to
Cluster Autoscaling Isn't Magic: Tuning It for SQL Workloads
Posted in: Microsoft SQL
Data Analysis…can we automate this?
As some of you know, I've made a move from consulting back into a full-time employee for Crop Pro Insurance. There was so much opportunity in this role. First
Delta Lake Isn't Just a Better Data Lake
Posted in: Microsoft SQL
Metadata Model Update
As I began learning Biml, I developed my original metadata model to help automate as much of my BI development as I could. This model still works today, but as
Your SQL Skills Are More Transferable to Databricks Than You Think
Great Expectations and pandas: Validating DataFrames Before They Hit the Database
My data science project this year has me working with NOAA weather data — storm event records going back decades, delivered as CSVs, with all the consistency problems that implies. Field names that changed between years. Magnitude values stored as strings in some vintages and floats in others. Missing state codes.
Data Types in Spark SQL: The Translation Guide SQL Server DBAs Actually Need
On a recent project, we had a pipeline that worked perfectly in SQL Server, imported cleanly into Databricks, and produced subtly wrong results for six weeks before anyone noticed. The culprit: a DATETIME column that became a TIMESTAMP in Spark SQL, and a timezone assumption that was implicit in our