Starting the Year With Eyes Open Every January I write something about where ADF stands. This January feels different. Not because the product changed dramatically in the last twelve months, but because the organizational signals have gotten loud enough that pretending otherwise would be dishonest. 2023 is almost certainly the
Last time, we built a simple transform function in Python, but how do we know if it works? We need to build some tests to find out. I admit data engineering has be late to the practice of test development, but it’s not too hard to adopt. Let’s work through...
Spark engines like Databricks are optimized for dealing with many small-ish files that have already been loaded into your Hadoop-compatible file system. If you want to process data from external sources, you’ll want to extract that data into files and store those in your Azure Data Lake Storage (ADL
If you've adopted dbt for transformation, you've already encountered its built-in tests: not_null, unique, accepted_values, relationships. They're fast to write, they run as part of your dbt build, and for simple constraints they're the right tool. The question I
Last time, we walked through how to perform analysis on Databricks using Visual Studio Code (VSC). This time, we will set up a local solution in VSC that will let us build out our data engineering solutions locally. That way, we don’t have to pay for development and testing time....
You’ve just received a new dataset, and you have to analyze it to prepare for building out the d ta ingestion pipeline. But first, we’ll need to create a cluster to run our analysis. Let’s run through a simple data analysis exercise using Databricks and Visual Studio Code (VSC). Create...
After you have your Databricks workspace, it’s time to set up your IDE. Head over to https://code.visualstudio.com/ to download the version for your operating system. It’s available for Windows, Mac, or Linux. During my most recent Databricks presentation, I was asked to point out that Visual Studio
Now that you’ve had an introduction let’s get started exploring Databricks. Head to https://community.cloud.databricks.com and click the sign up link at the bottom. The community edition is a completely free option. Fill in your contact information. It may help to use a ‘+’ email address to sign up;
Eight Years I started working with Azure Data Factory in 2014. It was in preview. The authoring experience was JSON in a browser. There was no version control. There was no parameterization. There was no CI/CD story. There was a connector list short enough to print on one page.
OpenAI just released ChatGPT and I've spent the last week doing nothing but using it. Not to generate blog posts or marketing copy — to work. Architecture reviews, pipeline design sessions, debugging conversations, schema design tradeoffs. Real data engineering work, with an LLM as a participant rather than a
As a SQL Server user moving into Databricks, reporting and dashboarding can be a more manageable learning task. It all begins by changing to the SQL Persona in Databricks. Click the persona icon from the upper left menu, and choose SQL. Once in the SQL persona, you can begin querying...
Since Databricks is a PaaS option for Spark and Spark is optimized to work on many small files, you might find it odd that you have to get your sources into a file format before you see Databricks shine. The good news is Databricks has partnered with several different data...