Every conference has one announcement that the press covers extensively and one that the practitioners quietly recognize as more significant than the coverage suggests. At DAIS 2024, UniForm was the second kind. The headline version: Delta Lake tables can now be read natively by Iceberg-compatible engines without copying or converting
If you want to understand what Databricks is actually betting on for the next five years, don't look at the model training announcements or the NVIDIA partnership. Look at Unity Catalog. Every major product announcement at DAIS 2024 plugs into UC. Models are registered in UC. Vector stores
Six months is enough time to separate the tool from the hype. I had shipped real work with Copilot in the loop, built and partially rebuilt a context system, and formed opinions about where AI assistance genuinely helps a data engineering workflow and where it introduces overhead that outweighs the
One of the terms that kept surfacing at DAIS 2024 was "compound AI systems." Databricks' research arm has been writing about this concept, and it was woven through several keynote segments: the idea that the AI applications enterprises actually need aren't single model calls, but
Databricks introduced natural language interfaces for data — type a question, get an answer, no SQL required. AI/BI Genie was the showstopper demo: connect it to your lakehouse, ask "what were our top-performing products last quarter," and it generates the query, runs it, and returns a visualization. The
Databricks acquired Tabular at DAIS 2024. If that name doesn't immediately ring a bell: Tabular was founded by Ryan Blue and Daniel Weeks — the original creators of Apache Iceberg, the table format that has been eating into Delta Lake's market position for the last two years.
Day one keynote. Ali Ghodsi takes the stage, runs through the vision, and then Jensen Huang walks out. The room — and I mean this literally — changed temperature. If you want a barometer for how seriously the enterprise data world is taking GPU-accelerated AI right now, watch a room of sixteen
I've been to enough data conferences to have calibrated expectations. The announcements are usually incremental, the keynotes are always polished to the point of being a little hollow, and you get three genuinely useful things out of talking to other practitioners in the hallway. Data + AI Summit 2024
At some point the context file problem stopped being a tooling complaint and started being an engineering problem. That's when things got interesting. I'd spent five months learning where a flat markdown file fails as a memory system for AI-assisted development. The failure modes were clear:
There's a specific frustration that creeps in around month five of working with AI tools: the feeling that you've solved the context problem, when what you've actually done is built an elaborate workaround for it. The context file felt like continuity. It wasn'
The convention file I had been maintaining started as a two-page document. By month four it had grown into something more ambitious — and the growth was telling me something about what I actually needed from an AI-assisted workflow. The markdown-as-memory pattern wasn't a design decision I made deliberately.
I wrote the first post in this series in March 2017, when Great Expectations was a new project and the idea of applying unit-testing discipline to data pipelines was still enough of a novelty that I had to explain why you'd want to do it. Seven years later,