Posted in: Microsoft SQL
In my last blog entry, we covered how to layout folders in your Data Lake Storage account based on a logical design. That’s only half the battle. You also need to set up Access Control Lists (ACLs). Setting up controls via the Azure portal is easy, but not something you...
Posted in: Microsoft SQL
After laying out the structure for our zones, my client quickly asked, is there a way we can automatically stand up this structure each time we bring a new Tenant into our solution. With a smile, I replied it was possible! In the short term, we would use PowerShell to...
Posted in: Microsoft SQL
Over the next few blog posts in this series, I’m going to share with you the story of how a Data Lake project comes together. As I tell this story, I’m going to keep pointing back to traditional ETL work and to Automation techniques. Not all of these will include...
Posted in: Microsoft SQL
Last week you might have seen a tweet about a day in the life of a Data Platform Consultant. To say the least, my days are varied.This day, in particular, I was
ForEach is here. I've been waiting for this activity since I first tried to build a metadata-driven framework in ADF in 2015. Parameterization arrived in v2 last year and gave us half the framework. ForEach gives us the other half: the ability to iterate over a set of
Posted in: Microsoft SQL
As I mentioned in the overview, the largest cost in terms of time is the requesting each web page from the web server and downloading that file to disc. That’s why this file staging loop is a separate step from the parsing and transforming step. This loop can be as...
{"name":"Shannon Lowder","description":"","url":"https:\/\/web.archive.org\/web\/20171005201719\/http:\/\/shannonlowder.com","home":"https:\/\/web.archive.org\/web\/20171005201719\/http:\/\/shannonlowder.com","gmt_offset":-4,"timezone_string":"America\/New_York","namespaces":["oembed\/1.0","akismet\
Posted in: Microsoft SQL
In the last article, I laid out the architecture to dealing with this type of data source. This time, we’re going to get in to the basics of parsing data. A Few New Technologies Before we dive into details, let’s cover the technologies this solution rests on. First, there’s C#....
Posted in: Microsoft SQL
Introduction There's a ton of data on the Internet. Some of that data is really easy to download and extract that data into a relational database. Data that
ADF monitoring has been a persistent complaint of mine since 2014. The v1 monitoring view showed slice status — Ready, Failed, In Progress — and not much else. When a pipeline failed, you got an error message. When a pipeline was slow, you got nothing useful. You learned to tolerate that. ADF
Posted in: Microsoft SQL
Everyone who knows me, knows I'm a Microsoft data platform professional. I do prefer their solutions over most other solutions. I know there have been a few
Posted in: Microsoft SQL
After getting AD Password Authentication working with my Azure SQL server, I moved on and set up a local Active Directory domain on my home network. I don’t think this is something every data professional should try and tackle. I count myself extremely fortunate to have made friends with IT...