New Spark developers often ask why their code runs so fast, until suddenly it doesn't. They'll chain 10 transformations — filter, select, join, groupBy, withColumn — and each one completes immediately. Then they call .show() and wait 45 seconds. What's happening? Lazy evaluation. None of those
If you know SQL Server window functions — ROW_NUMBER, RANK, LEAD, LAG, running totals — you already know the hardest conceptual part of PySpark window functions. The syntax is different but the semantics map directly. Here's the translation. The Import You Need from pyspark.sql import functions as F
The first time I got a Databricks invoice that was significantly higher than expected, I went looking for where the compute was going. The answer: an interactive cluster I'd left running overnight while prototyping. Nobody was using it. It was just there, burning DBUs. Here's how
I have been iterating on the ADF CI/CD pattern since 2018 when git integration shipped. I've rebuilt it for five different clients, hit the same problems each time, and landed on a pattern that I now consider stable. Here it is -- the complete picture, including the parts
OpenAI just opened their GPT-3 API to a small beta. I got access last week. The first thing I did was paste a data engineering problem into the Playground — describe a pipeline, ask for the Spark code. The results were simultaneously impressive and wrong in ways that taught me
The operations you've been running in T-SQL for years exist in PySpark. They just look different. This post is the translation table I wish I'd had during my first week with Databricks — every fundamental T-SQL operation mapped to its PySpark equivalent, with actual examples
Databricks can read almost anything. CSV, JSON, Parquet, ORC, Avro, Delta, JDBC sources, Kafka topics — the API is consistent across all of them. The question isn't "can Databricks read this" but "should I be storing it in this format." Let me save you some
I've been running ADF Mapping Data Flows in production for over a year now. The marketing copy says "enterprise-scale data transformation powered by Apache Spark." That's accurate, but it doesn't tell you how to size the cluster, what the performance curve
The last few years of home lab work have reinforced a pattern I already believed in: anything that controls system behavior belongs in source control. Kubernetes manifests, Terraform configs, Ansible playbooks, SSDT database projects with tSQLt tests. The question isn't whether to version-control it — it's
SQL Server has always had ACID transactions. You write BEGIN TRANSACTION, do your work, COMMIT or ROLLBACK. The database handles the rest: concurrent reads see consistent data, failed writes don't leave partial state, and committed data survives a crash. It's so fundamental to how SQL Server
The first time I saw someone read a Databricks table with a hardcoded ADLS path buried in a production notebook, I had a flashback to SQL Server connection strings stored directly in SSIS packages. The path works today. It stops working when the storage account gets renamed. Nobody knows what
Cloud cost discipline matters more now than it did two years ago. The pandemic has pushed a lot of workloads into the cloud on short timelines, and a lot of those workloads were not designed with cost in mind. Streaming pipelines are a common area of waste — they are easy