Delta Lake Concurrent Writes: What the Transaction Log Does When Two Jobs Collide

Shannon Lowder

Shannon Lowder

— 2 min read

Delta Lake's transaction log is designed to handle concurrent operations safely — multiple readers can run alongside a writer, and the writer's changes become visible atomically. But what happens when two jobs try to write to the same Delta table at the same time?

The answer is optimistic concurrency control, and understanding it tells you both what Delta can handle and what it can't.

How Concurrent Writes Work

When a write starts, it reads the current version of the table (say, version 42). It does its work, generates new data files, and then tries to commit by adding a new entry to the transaction log at version 43.

If another write committed version 43 in the meantime, the first write's commit fails with a ConcurrentAppendException or ConcurrentModificationException. At this point, Delta checks whether the two writes conflict:

  • If the writes touched different partitions — Delta can often resolve the conflict automatically. Write A touched the "West" partition, write B touched the "East" partition. No overlap. Delta re-attempts the commit at the new version (44). Both writes succeed.
  • If the writes touched the same data — genuine conflict. One write wins, one fails, and the failing job must retry from scratch.

What This Means in Practice

For append-only workflows, concurrent appends to different partitions mostly succeed without user intervention — Delta handles the retry automatically.

# These two jobs can typically run concurrently without conflict
# Job 1: appending West region data
west_df.write.format("delta").mode("append").partitionBy("region").save("/mnt/myproject/orders")

# Job 2: appending East region data (different partition)
east_df.write.format("delta").mode("append").partitionBy("region").save("/mnt/myproject/orders")

For MERGE operations, conflicts are more likely because MERGE reads and writes the same rows. Two concurrent MERGEs on the same table with overlapping keys will cause one to fail and require a retry.

Isolation Levels

Delta Lake supports two isolation levels:

  • WriteSerializable (default) — writes are serializable, reads are snapshot isolated. This means writes behave as if they executed one after another, even if they ran concurrently. Reads see a consistent snapshot and aren't blocked by ongoing writes.
  • Serializable — both reads and writes are serializable. Stronger guarantee, more conflicts, lower throughput.
-- Set isolation level per table
ALTER TABLE orders SET TBLPROPERTIES ('delta.isolationLevel' = 'Serializable')

For most workloads, WriteSerializable is the right choice. Switch to Serializable only if you have correctness requirements that depend on reads seeing a serializable view of concurrent writes — for example, if your reads are feeding into subsequent writes that need to be consistent.

The Takeaway for Pipeline Design

Design your pipelines to partition data in ways that minimize write overlap. If you have hourly jobs writing to date-partitioned Delta tables, they should naturally touch different partitions and concurrent runs will succeed. If you have many jobs writing to the same partition simultaneously, you'll see conflict retries. The fix there is usually to redesign the pipeline to run sequentially or to use a different partitioning scheme. As always, I'm here to help.

Read more