Every Delta Lake table has a directory called _delta_log sitting alongside its data files. Most tutorials tell you it exists and leave it at that. Understanding what's actually in it changes how you reason about Delta's behavior — the ACID guarantees, the time travel, and the
When you run an UPDATE or DELETE on a Delta table, the old version of the data doesn't disappear. Delta writes new Parquet files with the changed rows and marks the old files as removed in the transaction log — but the old files stay on disk. That'
When you run an UPDATE or DELETE on a Delta table, the old version of the data doesn't disappear. Delta writes new Parquet files with the changed rows and marks the old files as removed in the transaction log — but the old files stay on disk. That'
One of the first questions I get from SQL Server DBAs when they start using Delta Lake is some version of: "How do I know what happened to this table?" In SQL Server, the answer is usually "it depends" — you might have CDC enabled, or an
One of the first questions I get from SQL Server DBAs when they start using Delta Lake is some version of: "How do I know what happened to this table?" In SQL Server, the answer is usually "it depends" — you might have CDC enabled, or an
SQL Server DBAs know parameterized stored procedures. You pass in a date range, an environment name, a customer ID, and the procedure does the right thing based on those inputs. Logic lives in code, inputs come from outside. It's a clean pattern. Notebooks don't work that
SQL Server DBAs know parameterized stored procedures. You pass in a date range, an environment name, a customer ID, and the procedure does the right thing based on those inputs. Logic lives in code, inputs come from outside. It's a clean pattern. Notebooks don't work that
There's a pattern I see on almost every team that's new to Databricks. It looks like this: jdbc_url = "jdbc:sqlserver://prod-sql.mycompany.com:1433;databaseName=Finance" username = "svc_databricks" password = "Pr0duction_P@ssword_2018!" # please don't df
There's a pattern I see on almost every team that's new to Databricks. It looks like this: jdbc_url = "jdbc:sqlserver://prod-sql.mycompany.com:1433;databaseName=Finance" username = "svc_databricks" password = "Pr0duction_P@ssword_2018!" # please don't df
Mapping Data Flows went generally available in late 2018. I've been running them in production for real client workloads since the preview. Six months of real data, real failures, real optimizations. Here's what I've learned. What Works Well The Transformation Set Is Comprehensive After
When you're used to SQL Server, your data lives in a known location: a database, on a server, at a port. When you move to Databricks, your data lives in object storage — S3, ADLS, or GCS — which looks like a filesystem if you squint but behaves differently enough
When you're used to SQL Server, your data lives in a known location: a database, on a server, at a port. When you move to Databricks, your data lives in object storage — S3, ADLS, or GCS — which looks like a filesystem if you squint but behaves differently enough