Posted in: Microsoft SQL
Over the next few blog posts in this series, I’m going to share with you the story of how a Data Lake project comes together. As I tell this story, I’m going to keep pointing back to traditional ETL work and to Automation techniques. Not all of these will include...
Posted in: Microsoft SQL
Last week you might have seen a tweet about a day in the life of a Data Platform Consultant. To say the least, my days are varied.This day, in particular, I was
ForEach is here. I've been waiting for this activity since I first tried to build a metadata-driven framework in ADF in 2015. Parameterization arrived in v2 last year and gave us half the framework. ForEach gives us the other half: the ability to iterate over a set
Posted in: Microsoft SQL
As I mentioned in the overview, the largest cost in terms of time is the requesting each web page from the web server and downloading that file to disc. That’s why this file staging loop is a separate step from the parsing and transforming step. This loop can be as...
{"name":"Shannon Lowder","description":"","url":"https:\/\/web.archive.org\/web\/20171005201719\/http:\/\/shannonlowder.com","home":"https:\/\/web.archive.org\/web\/20171005201719\/http:\/\/shannonlowder.com","gmt_offset":-4,"timezone_string":"America\/New_York","namespaces":["oembed\/1.0","akismet\
Posted in: Microsoft SQL
In the last article, I laid out the architecture to dealing with this type of data source. This time, we’re going to get in to the basics of parsing data. A Few New Technologies Before we dive into details, let’s cover the technologies this solution rests on. First, there’s C#....
Posted in: Microsoft SQL
Introduction There's a ton of data on the Internet. Some of that data is really easy to download and extract that data into a relational database. Data that
ADF monitoring has been a persistent complaint of mine since 2014. The v1 monitoring view showed slice status — Ready, Failed, In Progress — and not much else. When a pipeline failed, you got an error message. When a pipeline was slow, you got nothing useful. You learned to tolerate that. ADF
Posted in: Microsoft SQL
Everyone who knows me, knows I'm a Microsoft data platform professional. I do prefer their solutions over most other solutions. I know there have been a few
Posted in: Microsoft SQL
After getting AD Password Authentication working with my Azure SQL server, I moved on and set up a local Active Directory domain on my home network. I don’t think this is something every data professional should try and tackle. I count myself extremely fortunate to have made friends with IT...
Posted in: Microsoft SQL
f you’ve been working with Azure in the last few years, you’ve noticed there is a relatively new feature that’s supposed to make identity management easier with your Azure solutions: Azure Active Directory (AAD). This feature does work with Azure SQL Servers and Azure SQL Databases. But it may not..
Posted in: Biml
One of the questions I get when teaching others how to use Biml is how do you deal with sensitive information like usernames and passwords in your Biml Solution