After you have your Databricks workspace, it’s time to set up your IDE. Head over to https://code.visualstudio.com/ to download the version for your operating system. It’s available for Windows, Mac, or Linux. During my most recent Databricks presentation, I was asked to point out that Visual Studio
Now that you’ve had an introduction let’s get started exploring Databricks. Head to https://community.cloud.databricks.com and click the sign up link at the bottom. The community edition is a completely free option. Fill in your contact information. It may help to use a ‘+’ email address to sign up;
Eight Years I started working with Azure Data Factory in 2014. It was in preview. The authoring experience was JSON in a browser. There was no version control. There was no parameterization. There was no CI/CD story. There was a connector list short enough to print on one page.
OpenAI just released ChatGPT and I've spent the last week doing nothing but using it. Not to generate blog posts or marketing copy — to work. Architecture reviews, pipeline design sessions, debugging conversations, schema design tradeoffs. Real data engineering work, with an LLM as a participant rather than a
As a SQL Server user moving into Databricks, reporting and dashboarding can be a more manageable learning task. It all begins by changing to the SQL Persona in Databricks. Click the persona icon from the upper left menu, and choose SQL. Once in the SQL persona, you can begin querying...
Since Databricks is a PaaS option for Spark and Spark is optimized to work on many small files, you might find it odd that you have to get your sources into a file format before you see Databricks shine. The good news is Databricks has partnered with several different data...
Last time, I compared SQL Server storage to Databricks storage. This time, let’s compare SQL Server compute to Databricks compute. How SQL Server Processes a Query When you submit your workload to SQL Server, the engine will first parse it to ensure it’s syntactically correct. If it’s not, then it..
Understanding how Databricks works with storage can go a long way toward improving your use of Databricks. Storage In SQL Server, we store our data in two highly structured file types. The first type of file is a data file; these have an extension of MDF or NDF. You get...
I’ve been a Microsoft Data professional for over 20 years. Most of that time I’ve spent in the SQL Server stack, the core query engine, SSIS, SSRS, and a little SSAS. But times changed, and the business problems grew more complex. As they did, I looked at other technologies to...
I have been writing Kafka-based pipelines since 2012. The tooling in 2022 is dramatically better — Delta Lake, Databricks Auto Loader, Unity Catalog, Schema Registry, the Event Hubs Kafka protocol. The architecture principles, on the other hand, have changed remarkably little. The things that made pipelines fail in 2013 still make
Every time I connect to a new data source, the first hour is the same: exploratory queries to understand what's actually in there. How many rows, what columns, what are the null rates, does that "primary key" actually have duplicates, what does a typical row look
On a recent engagement, I was asked to provide best practices. I realized that many of the best practices hadn’t been collected here, so it’s time I fix that. The client was early in their journey of adopting Databricks as their data engine, and a lot of the development they...