Six months is enough time to separate the tool from the hype. I had shipped real work with Copilot in the loop, built and partially rebuilt a context system, and formed opinions about where AI assistance genuinely helps a data engineering workflow and where it introduces overhead that outweighs the
One of the terms that kept surfacing at DAIS 2024 was "compound AI systems." Databricks' research arm has been writing about this concept, and it was woven through several keynote segments: the idea that the AI applications enterprises actually need aren't single model calls, but
Databricks introduced natural language interfaces for data — type a question, get an answer, no SQL required. AI/BI Genie was the showstopper demo: connect it to your lakehouse, ask "what were our top-performing products last quarter," and it generates the query, runs it, and returns a visualization. The
Databricks acquired Tabular at DAIS 2024. If that name doesn't immediately ring a bell: Tabular was founded by Ryan Blue and Daniel Weeks — the original creators of Apache Iceberg, the table format that has been eating into Delta Lake's market position for the last two years.
Day one keynote. Ali Ghodsi takes the stage, runs through the vision, and then Jensen Huang walks out. The room — and I mean this literally — changed temperature. If you want a barometer for how seriously the enterprise data world is taking GPU-accelerated AI right now, watch a room of sixteen
I've been to enough data conferences to have calibrated expectations. The announcements are usually incremental, the keynotes are always polished to the point of being a little hollow, and you get three genuinely useful things out of talking to other practitioners in the hallway. Data + AI Summit 2024
At some point the context file problem stopped being a tooling complaint and started being an engineering problem. That's when things got interesting. I'd spent five months learning where a flat markdown file fails as a memory system for AI-assisted development. The failure modes were clear:
There's a specific frustration that creeps in around month five of working with AI tools: the feeling that you've solved the context problem, when what you've actually done is built an elaborate workaround for it. The context file felt like continuity. It wasn'
The convention file I had been maintaining started as a two-page document. By month four it had grown into something more ambitious — and the growth was telling me something about what I actually needed from an AI-assisted workflow. The markdown-as-memory pattern wasn't a design decision I made deliberately.
I wrote the first post in this series in March 2017, when Great Expectations was a new project and the idea of applying unit-testing discipline to data pipelines was still enough of a novelty that I had to explain why you'd want to do it. Seven years later,
Three months with any tool is enough to stop being impressed by what it does well and start noticing what it cannot do. Copilot had earned its place in my workflow by that point. But the shape of its limitations was becoming clearer, and those limitations pointed at a problem
A month into working with GitHub Copilot, the novelty wore off. What replaced it was more useful: a clear-eyed picture of exactly where this tool belongs in a data engineering workflow and where it gets in the way. The first few weeks were about calibration — learning when to trust the