The Delta Transaction Log: What Is Inside _delta_log and Why It Matters

Shannon Lowder

Shannon Lowder

— 2 min read

Every Delta Lake table has a directory called _delta_log sitting alongside its data files. Most tutorials tell you it exists and leave it at that. Understanding what's actually in it changes how you reason about Delta's behavior — the ACID guarantees, the time travel, and the concurrent write handling all make a lot more sense once you've seen the structure.

What the Log Looks Like

dbutils.fs.ls("dbfs:/mnt/myproject/orders/_delta_log/")

You'll see something like:

00000000000000000000.json
00000000000000000001.json
...
00000000000000000010.checkpoint.parquet
00000000000000000011.json
...

Each JSON file is one commit. The file number is the version number. Version 0 is table creation. Version 1 is the first write. The checkpoint Parquet file at version 10 is a compact snapshot of everything up to that point.

What's Inside a Commit

Each JSON file is newline-delimited JSON — one action object per line. The main action types:

// Add action: this Parquet file is now part of the table
{"add": {"path": "part-00001-abc123.snappy.parquet", "size": 1048576,
  "stats": "{\"numRecords\":50000,\"minValues\":{...},\"maxValues\":{...}}"}}

// Remove action: this file is no longer part of the table
{"remove": {"path": "part-00001-xyz789.snappy.parquet", "dataChange": true}}

// Metadata action: schema or configuration change
{"metaData": {"schemaString": "{\"type\":\"struct\",\"fields\":[...]}",
  "partitionColumns": ["region"]}}

When you query a Delta table, Spark reads the log from the most recent checkpoint forward and builds a snapshot: files with an add action that haven't been removed are the current table. Everything else is old data eligible for VACUUM.

Checkpoints

After every 10 commits, Delta writes a checkpoint Parquet file summarizing the full current state. Without checkpoints, opening a table with 10,000 commit history would require replaying 10,000 JSON files to reconstruct the current state. With checkpoints, Spark reads the most recent checkpoint and then replays only the JSON files since that checkpoint.

The stats embedded in each add action — min values, max values, null counts per column — are what Delta uses for data skipping. When you run a query with a filter like WHERE order_date = '2019-01-15', Delta reads the file-level statistics from the log and skips files where the max order_date is earlier than that date. No data read required to skip those files.

Why This Design Enables ACID

Writing to a Delta table means writing a new Parquet file to storage AND appending a new JSON entry to _delta_log. Cloud storage systems like S3 and ADLS provide atomic put operations for individual files. If the write fails partway through, the JSON entry either committed or it didn't — there's no partial state for a reader to observe.

A reader that started before the write began sees the snapshot from before that write, because its version of the table doesn't include the new log entry. Readers are always consistent with a specific version, and writers don't block readers. This is fundamentally different from writing to a plain Parquet directory, where a reader can see half the files of an in-progress write.

Reading the Log Directly

You can read the Delta log as data if you want to build custom audit tooling:

log_df = spark.read.json("dbfs:/mnt/myproject/orders/_delta_log/*.json")
display(log_df.filter(log_df.add.isNotNull())
              .select("add.path", "add.size", "add.stats"))

This gives you raw file-level information that DESCRIBE HISTORY doesn't surface — individual file statistics, exact filter conditions used by the writer, and partition values. Useful for debugging data skipping behavior or auditing what data was added in a specific commit.

One final note: don't delete _delta_log. If you delete it, Delta falls back to treating the directory as plain Parquet — which means reading all Parquet files including removed versions, and your results will be wrong in silent, hard-to-diagnose ways. The log is the table. As always, I'm here to help.

Read more