The Moment Before Everything Was Obvious It's easy to look at ADF in 2023 — mature, enterprise-grade, absorbed into Microsoft Fabric — and treat its design decisions as obvious. Of course you make everything JSON. Of course you use managed compute. Of course you separate orchestration from execution. Those
The config-driven pipeline pattern I've been building on since 2015 turns out to be a near-perfect LLM prompt format. Structured, explicit, machine-readable, describes both the data shape and the transformation intent. When I started feeding pipeline configs to GPT-4 alongside source schemas and asking
The Question I've Been Asking for Ten Years Every time Microsoft ships a new version of its data integration platform, I ask the same question: has the metadata-driven framework gap finally closed? The pattern — control table, ForEach loop, parameterized pipeline template, one pipeline that handles N tables
Data contracts have become a hot topic in the data engineering community over the last couple of years. The idea isn't new — the concept of a formal agreement between a data producer and a data consumer about the shape, semantics, and quality of data has been implicit in
Not "Should I Migrate?" but "When and How?" Fabric Data Factory is in preview. GA is November 2023. Clients are already asking me about migration timelines. The short answer: wait for GA, then migrate deliberately. The long answer is this post. Let me break down what
LangChain has been impossible to avoid for the last six months. Every tutorial on building LLM applications uses it. Every GitHub repo that starts with "AI-powered" depends on it. I've been using it, fighting it, and forming opinions worth sharing. The honest version: LangChain solves
If you work with CSV files, you know how important it is to have a clear and consistent schema for your data. A schema defines your data’s structure, format, and types and helps you validate, transform, and analyze it. The problem is CSV files have no INFORMATION_SCHEMA or similar metadata...
As a professional, I place a high value on my role as a data engineer, with consulting being a close second. My role requires both technical and interpersonal skills. I’ve come to depend on two AI-based tools to enhance both. Grammarly has helped me be cognizant of my tone and...
A few weeks back, Microsoft announced its newest data offering, Fabric. After spending a little time with marketing materials, documentation, and some hands-on time I’ve learned a few things. The short version is this is just the next step of Synapse. Where Synapse merged Azure Data factory, SQL Dat
The Technical Comparison You Actually Need The Fabric announcement was big on positioning and short on technical specifics. Let me do the comparison properly: standalone Azure Data Factory versus Data Factory in Microsoft Fabric, capability by capability. What carries forward, what changes, and what doesn't exist in Fabric
If the model generated the transformation, can it generate the tests? That's the natural question after a few months of using ChatGPT and GPT-4 for pipeline scaffolding. The answer is yes — with a workflow caveat that turns out to matter a lot. The naive approach: ask for
Build 2023: The Announcement Microsoft Build happened in May 2023. Microsoft Fabric was announced. I predicted it in October 2022 and again in January 2023. I called the name in the October post. I'm not writing this to say I told you so. I'm writing it