SQL Server DBAs know parameterized stored procedures. You pass in a date range, an environment name, a customer ID, and the procedure does the right thing based on those inputs. Logic lives in code, inputs come from outside. It's a clean pattern. Notebooks don't work that
There's a pattern I see on almost every team that's new to Databricks. It looks like this: jdbc_url = "jdbc:sqlserver://prod-sql.mycompany.com:1433;databaseName=Finance" username = "svc_databricks" password = "Pr0duction_P@ssword_2018!" # please don't df
Mapping Data Flows went generally available in late 2018. I've been running them in production for real client workloads since the preview. Six months of real data, real failures, real optimizations. Here's what I've learned. What Works Well The Transformation Set Is Comprehensive After
When you're used to SQL Server, your data lives in a known location: a database, on a server, at a port. When you move to Databricks, your data lives in object storage — S3, ADLS, or GCS — which looks like a filesystem if you squint but behaves differently enough
Your source data is still in SQL Server. That's fine — it doesn't have to move all at once. Databricks can read from SQL Server over JDBC, which means you can start building Spark pipelines against live data before you've migrated a single table. Here&
End of year. Time to take stock. 2018 was the most significant year for Azure Data Factory since the service launched in 2014. Let me walk through what landed, what it means, and what's still unfinished. What Shipped ADF v2 GA — February The GA release locked the v2