At DAIS 2024, Unity Catalog was extended to govern not just data tables but models, features, vector indexes, and lineage that spans the full machine learning lifecycle. I've spent the last few months integrating this more deeply into real projects, and the expanded UC story is genuinely changing
Three months after DAIS and I've now been in enough post-conference retrospectives to see the same pattern play out. The team that attended Summit comes back energized. They have three to five AI use cases they want to build. They get budget approval for a proof of
The thing nobody says loudly at AI conferences: most GenAI application failures are retrieval failures. The model itself — the LLM doing the generation — is usually not the problem. The problem is that the model received the wrong context, incomplete context, or no relevant context at all, and generated confidently wrong
There's a point where a workaround becomes a problem in its own right. I crossed that line somewhere around month eight, when I noticed that my weekly calendar had a recurring block labeled "context maintenance" that was taking longer than the original task it was supposed
The natural question, given that I'm building LLM-powered data tooling at the moment: can you use a language model to generate a Great Expectations suite? Profile a dataset, describe the domain, and have the model produce a meaningful suite that doesn't require hours of manual
Two months out from DAIS, I've had time to actually use Mosaic AI rather than just watch the demos. The unified branding that Databricks put on this product family at Summit — training, serving, vector search, agent framework, tools catalog — was clearly designed to compete with the fragmented GenAI
Every conference has one announcement that the press covers extensively and one that the practitioners quietly recognize as more significant than the coverage suggests. At DAIS 2024, UniForm was the second kind. The headline version: Delta Lake tables can now be read natively by Iceberg-compatible engines without copying or
If you want to understand what Databricks is actually betting on for the next five years, don't look at the model training announcements or the NVIDIA partnership. Look at Unity Catalog. Every major product announcement at DAIS 2024 plugs into UC. Models are registered in UC. Vector stores
Six months is enough time to separate the tool from the hype. I had shipped real work with Copilot in the loop, built and partially rebuilt a context system, and formed opinions about where AI assistance genuinely helps a data engineering workflow and where it introduces overhead that outweighs the
One of the terms that kept surfacing at DAIS 2024 was "compound AI systems." Databricks' research arm has been writing about this concept, and it was woven through several keynote segments: the idea that the AI applications enterprises actually need aren't single model calls, but
Databricks introduced natural language interfaces for data — type a question, get an answer, no SQL required. AI/BI Genie was the showstopper demo: connect it to your lakehouse, ask "what were our top-performing products last quarter," and it generates the query, runs it, and returns a visualization.
Databricks acquired Tabular at DAIS 2024. If that name doesn't immediately ring a bell: Tabular was founded by Ryan Blue and Daniel Weeks — the original creators of Apache Iceberg, the table format that has been eating into Delta Lake's market position for the last two years.