Year Two on Databricks: What I Would Tell 2018 Me

Two years ago I wrote a post about moving from SQL Server to Databricks — what skills transfer, where the mental model breaks down, what to learn first. Most of it still holds. But there are things I would change if I were starting that journey today, and things I wish I'd done differently in year one.

What I Got Right

Starting with SQL, not Python. Spark SQL is a real SQL dialect and it covers 80% of what you need for data engineering. The instinct to immediately learn PySpark DataFrames because that's what the documentation focuses on is wrong for SQL professionals. Start where you're strong. You can add Python idioms incrementally as you hit the walls of what SQL can express.

Learning partitioning early. The partitioning decision is probably the highest-leverage architectural choice in Databricks. Getting it right at the start of a project saves enormous pain later. I spent a lot of year one getting this right through trial and error; reading more carefully about partition pruning and data skipping up front would have saved months.

Using Delta from the start. I know some teams started with plain Parquet and planned to migrate to Delta later. That migration is annoying. Delta's overhead is negligible and the benefits — ACID writes, time travel, schema enforcement — are valuable from day one. Start with Delta.

What I'd Do Differently

Set up cluster policies earlier. We let developers create clusters without constraints for the first four months. Cloud bills were... higher than they needed to be. Policies are a 30-minute setup that pays for itself in the first billing cycle.

Build observability into pipelines from the start. We retrofitted MLflow tracking into pipelines that were already in production. It's harder than doing it at the start. Every pipeline should log row counts, run times, and data quality metrics from its first production run.

Establish the bronze-silver-gold pattern before letting teams build ad-hoc. We had six months of everyone writing data wherever made sense to them before we standardized on the medallion architecture. Migrating existing tables to the right layers is more work than building into the right structure from the beginning.

What's Changed Since Year One

The Delta Lake open-source release is the biggest shift. It went from "Databricks feature" to "industry standard format." The ecosystem is catching up fast — Hive, Presto, and other query engines are working on Delta compatibility. The format bet paid off.

Databricks itself has matured: better job scheduling, better cluster management, better monitoring. The rough edges from 2018 are smoother. The platform is worth betting on, but now with much clearer evidence than we had two years ago.

Year three starts here. The fundamentals are in place. What comes next is building on top of them. As always, I'm here to help.