There's a moment in data engineering work where you realize you're typing the same structural boilerplate for the fifth time that week — the same null-handling, the same logging setup, the same exception wrapper you copy from the previous file and adjust. It's not creative
Great Expectations the framework (now branded GX) has always been open source and self-hosted — you manage the project directory, the expectation suite files, the result stores, the Data Docs infrastructure. For teams that want to own that infrastructure, that's fine. For teams that want data quality enforcement without
The Arc SSIS. ADF. Fabric. Three names over twenty years. One continuous story about how Microsoft builds and evolves data integration tooling, and about the persistent gap that community ecosystems keep filling because Microsoft keeps leaving the same 20% undone. I've lived through all of it. I want
I want to trace the arc of the last four years, because I think the endpoint is different from where most people in this space are looking. It started with TabNine completing Spark boilerplate. Then GPT-3 as a one-shot thinking tool. Then Copilot making the IDE ambient with AI. Then
The Moment Before Everything Was Obvious It's easy to look at ADF in 2023 — mature, enterprise-grade, absorbed into Microsoft Fabric — and treat its design decisions as obvious. Of course you make everything JSON. Of course you use managed compute. Of course you separate orchestration from execution. Those decisions
The config-driven pipeline pattern I've been building on since 2015 turns out to be a near-perfect LLM prompt format. Structured, explicit, machine-readable, describes both the data shape and the transformation intent. When I started feeding pipeline configs to GPT-4 alongside source schemas and asking it to generate new
The Question I've Been Asking for Ten Years Every time Microsoft ships a new version of its data integration platform, I ask the same question: has the metadata-driven framework gap finally closed? The pattern — control table, ForEach loop, parameterized pipeline template, one pipeline that handles N tables — has
Data contracts have become a hot topic in the data engineering community over the last couple of years. The idea isn't new — the concept of a formal agreement between a data producer and a data consumer about the shape, semantics, and quality of data has been implicit in
Not "Should I Migrate?" but "When and How?" Fabric Data Factory is in preview. GA is November 2023. Clients are already asking me about migration timelines. The short answer: wait for GA, then migrate deliberately. The long answer is this post. Let me break down what
LangChain has been impossible to avoid for the last six months. Every tutorial on building LLM applications uses it. Every GitHub repo that starts with "AI-powered" depends on it. I've been using it, fighting it, and forming opinions worth sharing. The honest version: LangChain solves real
If you work with CSV files, you know how important it is to have a clear and consistent schema for your data. A schema defines your data’s structure, format, and types and helps you validate, transform, and analyze it. The problem is CSV files have no INFORMATION_SCHEMA or similar metadata...
As a professional, I place a high value on my role as a data engineer, with consulting being a close second. My role requires both technical and interpersonal skills. I’ve come to depend on two AI-based tools to enhance both. Grammarly has helped me be cognizant of my tone and...