I've been trying to build a proper metadata-driven ingest framework in ADF since 2015. The 2015 attempt with v1 was a partial success: it worked, it was clever, and it had rough edges that required explanation every time someone new touched it. The core problem was the absence
Last month I wrote about Snowflake's architecture — what's genuinely good about it and what warrants scrutiny. I promised a direct comparison with Databricks. Here it is. I'm going to be specific about where the lock-in surfaces are, because this is a decision with a
I've been spending time with Snowflake this quarter, and I want to give it a fair technical read before it becomes the unexamined industry standard. The architecture is genuinely clever. The execution against that architecture is strong. And there are specific questions you should be asking before you
Every large data engineering shop I've worked with in the last ten years has SSIS packages. Some shops have hundreds of them — years of business logic encapsulated in .dtsx files, tested, documented (sometimes), and running reliably on on-premises SQL Server infrastructure. These shops look at Azure and face
I've made this argument before on this blog — in the context of Hive scripts in 2012, in the context of MapReduce jobs in 2013. I'm going to make it again in 2016 because the tools are better now, the excuses are thinner, and the data teams
After the CPU multithreading benchmark that beat SQL Server's parallelism by 3.5×, the natural next question was: what's the ceiling? An 8-core CPU at full utilization was fast. But a GPU has hundreds or thousands of cores. For the right computation, the performance difference is
In 2012, I wrote about treating Azure VM gallery images as release artifacts — versioned, tested, deployable artifacts that embodied infrastructure state rather than manual configuration procedures. In 2016, containers made the same idea faster, lighter, and more widely applicable. The progression from bare metal to VM images to container images
The Integration Runtime is ADF v2's answer to a question v1 never asked explicitly: where does this activity actually execute? In v1, cloud activities ran on Microsoft-managed compute, on-premises activities ran on the DMG server, and nobody made you think about it much. In v2, the IR is
Spark 2.0 shipped in July 2016. The headline features are Structured Streaming and a 2x performance improvement on TPC-DS benchmarks via a rewritten query optimizer. Both are real and worth covering. But the feature I want to focus on is the one getting the least attention in the announcement
Latest Posts No comments yet Click here to cancel reply. Leave a Reply Name: (required): Email: (required): Website: Comment: Note: You can use basic XHTML in your comments. Your email address will never be published. Subscribe to this comment feed via RSS
SQL 402 [posts-by-tag tags = “sql402” number = “100” excerpt = “true” thumbnail = “false” order_by = “date” order = “desc” author = “false” date = “true” content = “false” tag_links = “false”] No comments yet Click here to cancel reply. Leave a Reply Name: (required): Email: (required): Website: Com
Posts No comments yet Click here to cancel reply. Leave a Reply Name: (required): Email: (required): Website: Comment: Note: You can use basic XHTML in your comments. Your email address will never be published. Subscribe to this comment feed via RSS