I want to tell you about a pipeline I inherited. It consumed messages from a Kafka topic and wrote structured records directly into an analytics database — no intermediate storage, no raw landing. The producer was an application team in a different department that had, historically, not told anyone when they
The first time most SQL developers look at Kafka documentation, they hit a wall around partitions. The concept exists in SQL Server too — table partitions, partition functions, partition schemes — but Kafka uses the word to mean something slightly different, and the difference matters when you are trying to figure out
I've had this conversation more times than I can count: "We keep our schema in source control." "What does that mean exactly?" "We have a folder with all our CREATE scripts." And then I look at the folder, and it has one
Windows Azure
Last time I introduced you to Windows Azure and some of the key features: pay as you go, scalability, and ease of use. These are the three key points I make
The most common mistake I saw with early ARM template adoption was writing a separate template for each environment — one for dev, one for staging, one for production. This preserved the "separate documentation per environment" mental model that preceded IaC, just in a JSON format instead of a
Windows Azure
I've been presenting this presentation all year. I've tweaked it a little after each section. Funny thing is I never blogged that content... So I'd like to take
microsoft sql
If you've seen my OnPrem to Azure presentation, then you know one of the two big reservations I had about windows azure was we couldn't create a transactionally
A client called me earlier this year with an interesting problem. They had a building automation system — HVAC, power meters, door sensors — all generating data. They wanted that data in their analytics stack. The problem was that the vendor-supplied monitoring software only showed real-time dashboards and stored nothing past 30
There's a Pig script at one client that is 847 lines long. It ingests raw clickstream data from HDFS, applies session reconstruction logic, calculates funnel conversion rates, joins against a user dimension, filters out internal traffic, handles three different upstream data formats based on a timestamp condition, and
Windows Azure
For the past 3 years...up until TechEd 2013, Microsoft was "All in" when it came to the cloud. Their goal was to see all new features and services come to the
Software developers figured out something operations teams were slow to learn: reproducibility comes from artifacts, not procedures. A compiled binary is reproducible because the build system produces it deterministically from source code. An application deployment is reproducible because the same binary gets applied to each environment. The procedure for deploying
AssertEquals works great for scalar values — a count, a status flag, a calculated amount. But a lot of stored procedure logic produces result sets: multiple rows, multiple columns. Testing that a procedure returned the right set of rows isn't something you can do with a single scalar assertion.