A few weeks back, Microsoft announced its newest data offering, Fabric. After spending a little time with marketing materials, documentation, and some hands-on time I’ve learned a few things. The short version is this is just the next step of Synapse. Where Synapse merged Azure Data factory, SQL Dat
The Technical Comparison You Actually Need The Fabric announcement was big on positioning and short on technical specifics. Let me do the comparison properly: standalone Azure Data Factory versus Data Factory in Microsoft Fabric, capability by capability. What carries forward, what changes, and what doesn't exist in Fabric
Build 2023: The Announcement Microsoft Build happened in May 2023. Microsoft Fabric was announced. I predicted it in October 2022 and again in January 2023. I called the name in the October post. I'm not writing this to say I told you so. I'm writing it
If the model generated the transformation, can it generate the tests? That's the natural question after a few months of using ChatGPT and GPT-4 for pipeline scaffolding. The answer is yes — with a workflow caveat that turns out to matter a lot. The naive approach: ask for the
One of the complaints I hear consistently from teams that evaluated Great Expectations a few years ago and haven't revisited it: "the API keeps changing." It's a fair criticism. GE has gone through significant interface changes since the early pandas-wrapper days — the DataContext model,
GPT-4 dropped last month and I've been running it alongside GPT-3.5 on the same data engineering tasks. The performance difference on syntax is negligible — both models generate correct PySpark syntax, both handle standard DataFrame operations, both know the Airflow operator API. On that dimension, the upgrade is
Five Years of "They're Different Tools" Azure Synapse Analytics launched in 2019 with Synapse Pipelines embedded in it. From the beginning, the Microsoft messaging was: "ADF for enterprise data integration teams, Synapse Pipelines for analytics co-location." Two tools for different use cases. Different teams.
Recently a client mentioned he wanted to run spark in a container for testing Databricks code locally. He Was working from the basic docker-compose yaml file provided by bitnami. That way, the containers would run on his local machine. The problem was accessing the cluster from the host machine. I..
If you’ve tried accessing the site over the past couple of weeks, you might have noticed it wasn’t up 100% of the time. That’s because I was upgrading my network infrastructure. Before the migration, I ran the Ubiquiti EdgeRouter 6p. This is a tremendous little quad-core router that ran completely..
Six months of heavy Copilot and ChatGPT use has produced a clear pattern: the model generates code that passes visual inspection and fails automated checks. Not because the model is careless — because the model doesn't know your standards. It knows Python. It knows PySpark. It does not know
When I began work with SQL Server, I used to run a single Windows server with a single instance of SQL Server 2000. I used this instance to test code and configurations separate from my employer’s servers. It helped me verify if a change could help and by how much...
Setting up Delta Sharing in Databricks is straightforward once you understand the diagram provided in the Azure Databricks documentation. Delta sharing is implemented as a part of Databrick’s Unity Catalog. Unity catalog is the official data governance solution for Databricks. You can consider it an