You've been writing GROUP BY aggregations in T-SQL for years. PySpark has all the same aggregate functions — and a few you probably wished SQL Server had. Here's the translation, plus a few patterns that are unique to the distributed world. The Basic Pattern from pyspark.sql
Spark 3.0 shipped in June, and Databricks Runtime 7.x brought it to production workspaces. The headlining feature — Adaptive Query Execution — has been running in my pipelines for a few months now. Here's what it actually changed versus what I expected. What AQE Changed in Practice The
Checkpoint files are the most misunderstood part of a Structured Streaming pipeline. They are not a cache. They are not a nice-to-have. They are the mechanism that makes exactly-once delivery possible. Delete them and you will either duplicate your data, gap your data, or both, depending on where your job
If you want to understand why a Spark job is slow, you need to understand the shuffle. The shuffle is the most expensive operation in Spark, and most of the performance tuning work in any non-trivial Spark application comes back to minimizing, eliminating, or managing shuffles. What a Shuffle Is
New Spark developers often ask why their code runs so fast, until suddenly it doesn't. They'll chain 10 transformations — filter, select, join, groupBy, withColumn — and each one completes immediately. Then they call .show() and wait 45 seconds. What's happening? Lazy evaluation. None of those
If you know SQL Server window functions — ROW_NUMBER, RANK, LEAD, LAG, running totals — you already know the hardest conceptual part of PySpark window functions. The syntax is different but the semantics map directly. Here's the translation. The Import You Need from pyspark.sql import functions as F
The first time I got a Databricks invoice that was significantly higher than expected, I went looking for where the compute was going. The answer: an interactive cluster I'd left running overnight while prototyping. Nobody was using it. It was just there, burning DBUs. Here's how
I have been iterating on the ADF CI/CD pattern since 2018 when git integration shipped. I've rebuilt it for five different clients, hit the same problems each time, and landed on a pattern that I now consider stable. Here it is -- the complete picture, including the
OpenAI just opened their GPT-3 API to a small beta. I got access last week. The first thing I did was paste a data engineering problem into the Playground — describe a pipeline, ask for the Spark code. The results were simultaneously impressive and wrong in ways that taught me more
The operations you've been running in T-SQL for years exist in PySpark. They just look different. This post is the translation table I wish I'd had during my first week with Databricks — every fundamental T-SQL operation mapped to its PySpark equivalent, with actual examples you can
Databricks can read almost anything. CSV, JSON, Parquet, ORC, Avro, Delta, JDBC sources, Kafka topics — the API is consistent across all of them. The question isn't "can Databricks read this" but "should I be storing it in this format." Let me save you some
I've been running ADF Mapping Data Flows in production for over a year now. The marketing copy says "enterprise-scale data transformation powered by Apache Spark." That's accurate, but it doesn't tell you how to size the cluster, what the performance curve looks