Shannon Lowder

The Hive Metastore for SQL Server DBAs: A Translation Guide

ADF Expression Language: Power, Quirks, and Learning to Stop Fighting It

The ADF expression language is the thing that makes parameterized pipelines actually work. It's the DSL for wiring dynamic behavior into your pipeline configuration: reference parameters, pull values from previous activity outputs, compute dates, build dynamic strings. It runs inside the ADF orchestration layer — not in Spark, not

Window Functions in Spark SQL: What Carries Over from T-SQL

Understanding DBFS: What It Is and What It Isn't

Stop Using Notebooks Like SQL Editors

Writing Custom Expectations in Great Expectations

The built-in expectation library covers the common cases well: nulls, ranges, value sets, row counts, regex patterns. But real data has domain-specific rules that a general-purpose library can't anticipate. A storm event can't end before it starts. A geocoded latitude must correspond to a valid US

ADF Mapping Data Flows: Code-Free Transformations Built on Spark

The single biggest complaint about ADF since 2014 has been the same: you can copy data, but you can't transform it. Copy Activity moves data from A to B. If you need to derive a column, aggregate rows, join two sources, or apply business logic — you had to

Cluster Autoscaling Isn't Magic: Tuning It for SQL Workloads

Delta Lake Isn't Just a Better Data Lake

Your SQL Skills Are More Transferable to Databricks Than You Think