Day one keynote. Ali Ghodsi takes the stage, runs through the vision, and then Jensen Huang walks out. The room — and I mean this literally — changed temperature. If you want a barometer for how seriously the enterprise data world is taking GPU-accelerated AI right now, watch a room of
I've been to enough data conferences to have calibrated expectations. The announcements are usually incremental, the keynotes are always polished to the point of being a little hollow, and you get three genuinely useful things out of talking to other practitioners in the hallway. Data + AI Summit 2024
At some point the context file problem stopped being a tooling complaint and started being an engineering problem. That's when things got interesting. I'd spent five months learning where a flat markdown file fails as a memory system for AI-assisted development. The failure modes were
There's a specific frustration that creeps in around month five of working with AI tools: the feeling that you've solved the context problem, when what you've actually done is built an elaborate workaround for it. The context file felt like continuity. It wasn'
The convention file I had been maintaining started as a two-page document. By month four it had grown into something more ambitious — and the growth was telling me something about what I actually needed from an AI-assisted workflow. The markdown-as-memory pattern wasn't a design
I wrote the first post in this series in March 2017, when Great Expectations was a new project and the idea of applying unit-testing discipline to data pipelines was still enough of a novelty that I had to explain why you'd want to do it. Seven years
Three months with any tool is enough to stop being impressed by what it does well and start noticing what it cannot do. Copilot had earned its place in my workflow by that point. But the shape of its limitations was becoming clearer, and those limitations pointed at a problem
A month into working with GitHub Copilot, the novelty wore off. What replaced it was more useful: a clear-eyed picture of exactly where this tool belongs in a data engineering workflow and where it gets in the way. The first few weeks were about calibration — learning when to trust
There's a moment in data engineering work where you realize you're typing the same structural boilerplate for the fifth time that week — the same null-handling, the same logging setup, the same exception wrapper you copy from the previous file and adjust. It's not
Great Expectations the framework (now branded GX) has always been open source and self-hosted — you manage the project directory, the expectation suite files, the result stores, the Data Docs infrastructure. For teams that want to own that infrastructure, that's fine. For teams that want data quality enforcement
The Arc SSIS. ADF. Fabric. Three names over twenty years. One continuous story about how Microsoft builds and evolves data integration tooling, and about the persistent gap that community ecosystems keep filling because Microsoft keeps leaving the same 20% undone. I've lived through all of it. I want
I want to trace the arc of the last four years, because I think the endpoint is different from where most people in this space are looking. It started with TabNine completing Spark boilerplate. Then GPT-3 as a one-shot thinking tool. Then Copilot making the IDE ambient with