After enough metadata-driven pipeline work, you develop strong opinions about what belongs in the config table. Most teams start with the obvious things — source connection info, table names, extract type — and add columns reactively as each new requirement arrives. By the time the table has twenty-five columns and
I ran my first production Mapping Data Flow about two years ago. It was an experiment -- I was watching performance, checking that it handled my edge cases, making sure the Spark execution model didn't do anything surprising at scale. I had a Databricks fallback ready if Data Flows
GitHub Copilot landed in technical preview a couple of months ago. I've had six weeks with it now, and I want to give an honest read rather than a reaction — because the first week reactions I've seen range from "this replaces junior developers" to
After the EDC assessment, the client had a specific problem: the catalog knew about their data assets, but metadata quality was low. Scanner-discovered attributes were incomplete. Business context was missing entirely — no descriptions, no data owners, no sensitivity classifications. The gap wasn't in the tool's
PySpark has over 300 built-in functions in pyspark.sql.functions. For everything those don't cover, you have User-Defined Functions. UDFs are the escape hatch when the built-ins don't fit — but they come with real performance costs. Understanding why helps you know when the
One of the clients this year runs on Informatica's catalog stack — Enterprise Data Catalog for discovery and lineage, Data Quality for profiling and rules, Axon as the business glossary layer. The expectation when I arrived was that these three tools, integrated together, would provide a complete metadata management
I have written about MQTT, Kafka, and Delta Lake separately. This post is the end-to-end: what the full IoT ingestion architecture looks like when you wire them together, what each hop adds, and why the complexity is justified. The scenario: sensors in a manufacturing facility publish readings via
ADF git integration shipped in 2018. I've been using it since the early access period, and I've onboarded more teams to it than I can easily count. Three years of production use, three years of watching teams get confused by the same things, three years of
SQL Server Agent has been the scheduling backbone of SQL-based ETL for twenty years. Multi-step jobs, success/failure routing, email alerts, retry logic, job dependencies — it's a full pipeline orchestrator built into the database engine. Moving to Databricks means finding an equivalent. Here's how
The metadata-driven pipeline generator was producing ADF pipelines and Databricks notebooks from a config table. The missing piece was infrastructure: every new client data source also needed a storage container, a Databricks cluster configuration, a linked service, and sometimes a Key Vault secret reference. Those were still being created
Earlier versions of Great Expectations had a concept called "validation operators" — a way to bundle a validation run with actions like saving results and building Data Docs. The API worked but was awkward to configure and reason about. Checkpoints replace it with something cleaner: a named, reusable configuration
Databricks announced Delta Sharing at Data + AI Summit last month, and I've been running it in a test environment since. If you haven't seen the announcement: Delta Sharing is an open protocol for sharing live data across organizations without copying it. Recipient gets a read endpoint,