Databricks can read almost anything. CSV, JSON, Parquet, ORC, Avro, Delta, JDBC sources, Kafka topics — the API is consistent across all of them. The question isn't "can Databricks read this" but "should I be storing it in this format." Let me save you some
I've been running ADF Mapping Data Flows in production for over a year now. The marketing copy says "enterprise-scale data transformation powered by Apache Spark." That's accurate, but it doesn't tell you how to size the cluster, what the performance curve looks
The last few years of home lab work have reinforced a pattern I already believed in: anything that controls system behavior belongs in source control. Kubernetes manifests, Terraform configs, Ansible playbooks, SSDT database projects with tSQLt tests. The question isn't whether to version-control it — it's how.
SQL Server has always had ACID transactions. You write BEGIN TRANSACTION, do your work, COMMIT or ROLLBACK. The database handles the rest: concurrent reads see consistent data, failed writes don't leave partial state, and committed data survives a crash. It's so fundamental to how SQL Server
The first time I saw someone read a Databricks table with a hardcoded ADLS path buried in a production notebook, I had a flashback to SQL Server connection strings stored directly in SSIS packages. The path works today. It stops working when the storage account gets renamed. Nobody knows what
Cloud cost discipline matters more now than it did two years ago. The pandemic has pushed a lot of workloads into the cloud on short timelines, and a lot of those workloads were not designed with cost in mind. Streaming pipelines are a common area of waste — they are easy
When a Spark job runs slower than expected, the first thing I check is partition count. Not cluster size. Not memory settings. Partitions. Because if your partition count is wrong, adding more machines doesn't help — you're just adding more workers standing around with nothing to do.
There was a period in ADF's early life when I kept a mental list of sources that weren't supported. It was a meaningful list -- long enough to influence architecture decisions. That period is over. ADF crossed 90 supported connectors in 2020. The blog posts from
You stand up a 10-node Databricks cluster. You open a notebook. You write some pandas code to process a 50GB dataset. It runs slower than it did on your laptop. You file a ticket asking why Databricks is worse than your laptop. I've been that person. Here'
Someone on my team last year asked me how Spark distributes work across a cluster. I started drawing boxes on a whiteboard — driver, executors, partitions, shuffle — and watched eyes glaze over. Then I grabbed the bag of M&Ms from the breakroom and the conversation completely changed. Let me
Before diving into the architecture and the API, you need a cluster to run code against. Databricks Community Edition gives you one for free — no credit card, no time limit on account access, one active cluster at a time. It's not production-grade (it's a single-node cluster
The question comes up on every Databricks project: should this pipeline run as a stream or as a batch? Most of the content on this topic presents it as a streaming-vs-batch debate with streaming as the obviously modern choice. The reality is more useful — batch is the right answer more