One of the quieter but more useful features in Great Expectations is Data Docs — automatically generated HTML documentation that shows you, in a human-readable format, what your expectations are and whether your data passes them. It's not something you build; it's something GE generates from
For most of my career, a "real-time pipeline" meant reading from a SQL Server table on a short schedule — every 5 minutes, every minute, as fast as you could run the query. It worked, but it had edge cases: what if the job ran slower than the
I've been burned by AI code completion enough times to approach every new entrant in this space with genuine skepticism. IntelliSense completes your method names. Kite autocompletes Python idioms based on what's in scope. These are useful but they're not surprising — they're
Delta Lake's transaction log is designed to handle concurrent operations safely — multiple readers can run alongside a writer, and the writer's changes become visible atomically. But what happens when two jobs try to write to the same Delta table at the same time? The answer is
Structured Streaming is how you build real-time pipelines in Databricks. You've probably already seen basic examples — read from Kafka, write to Delta. What those examples almost always skip is watermarking, and that omission tends to cause subtle problems with aggregations over event time. Why Watermarks Exist When
Spark jobs have a startup cost. The cluster needs to provision, the JVMs need to start, the session needs to initialize. For a two-hour batch job, that's noise. For a 45-second unit test, that's the entire runtime budget. Databricks Connect lets you run Spark
It's been a year since ADF git integration shipped. My team has been living with it — daily authoring, PRs, deployments — long enough that the novelty has worn off and the real picture is clear. Here's the honest 12-month assessment. What's Genuinely Good No
Writing a Spark Structured Streaming job that reads from Kafka and writes to Parquet files sounds straightforward until you watch it crash mid-write and leave your output directory in an ambiguous state. Did it write all the records for that micro-batch? Half of them? None? The checkpoint committed,
Most data engineering pipelines start with full loads. You truncate and reload. Every run processes every row from the source. It works, it's simple, and it ages badly — as your source tables grow, your pipeline run time grows with them, until you're running a 4-hour
One of the first things I noticed when I started working seriously with data lakes is that the same raw data would get processed in slightly different ways by different teams, and the "current version" of a customer record or a product attribute was genuinely hard to determine.
Multi-user Databricks workspaces have a problem that SQL Server DBAs will recognize immediately: people will spin up clusters that are too large, leave them running, and run experimental queries on production data when they think no one is looking. The solution in SQL Server is database permissions and resource
SQL Server DBAs have SQL Server Agent. It's deeply integrated with the database, it knows about databases and jobs and schedules, and it's been around long enough that everyone knows how it works. Moving to Databricks means you no longer have SQL Server Agent. The question