Testing Delta Lake Pipelines: Patterns That Actually Work in Practice

Shannon Lowder

Shannon Lowder

— 2 min read

Testing Databricks notebooks is awkward. The testing patterns from regular Python projects (pytest, unittest, mocks, fixtures) work against local Python code, but Databricks pipelines have dependencies that don't exist outside a cluster — a running SparkSession, DBFS paths, dbutils, secrets. Most teams deal with this by not testing at all, which turns out to be a bad strategy about six months in.

Here are the patterns that actually work, ordered from "easiest to adopt" to "most rigorous."

Pattern 1: Extract Functions From Notebook Cells

The single highest-leverage change is moving transformation logic out of notebook cells and into pure Python functions that don't depend on Spark internals:

# Before: logic buried in a cell, untestable
df = spark.read.format("delta").load("/mnt/orders")
df = df.withColumn("order_category",
    when(col("total_amount") < 100, "small")
    .when(col("total_amount") < 1000, "medium")
    .otherwise("large"))
df = df.filter(col("region").isin(["West", "East"]))

# After: logic in a function that takes and returns a DataFrame
def categorize_and_filter(df, valid_regions):
    return (df
      .withColumn("order_category",
          when(col("total_amount") < 100, "small")
          .when(col("total_amount") < 1000, "medium")
          .otherwise("large"))
      .filter(col("region").isin(valid_regions))
    )

# The notebook cell becomes thin
result_df = categorize_and_filter(
    spark.read.format("delta").load("/mnt/orders"),
    ["West", "East"]
)

Functions that take a DataFrame and return a DataFrame can be tested with small in-memory DataFrames. No DBFS, no cluster dependencies required for the logic test.

Pattern 2: Test With Small In-Memory DataFrames

# In a separate test notebook or a local test file with Databricks Connect
def test_categorize_and_filter():
    test_data = [
        (1, 50.0, "West"),
        (2, 500.0, "East"),
        (3, 5000.0, "North"),  # should be filtered out
        (4, 75.0, "North"),    # should also be filtered out
    ]
    test_df = spark.createDataFrame(test_data, ["order_id", "total_amount", "region"])

    result = categorize_and_filter(test_df, ["West", "East"])

    assert result.count() == 2
    rows = {r.order_id: r.order_category for r in result.collect()}
    assert rows[1] == "small"
    assert rows[2] == "medium"
    print("PASS: categorize_and_filter")

test_categorize_and_filter()

Pattern 3: Delta Table Testing With a Scratch Path

For tests that need to verify write behavior against Delta tables, use a scratch path in DBFS that you clean up after each test:

import uuid

def test_incremental_append():
test_path = f"/tmp/test_{uuid.uuid4().hex}"
try:
sample_df = spark.createDataFrame([(1, "West"), (2, "East")], ["id", "region"])
sample_df.write.format("delta").save(test_path)

new_rows = spark.createDataFrame([(3, "West")], ["id", "region"])
new_rows.write.format("delta").mode("append").save(test_path)

result = spark.read.format("delta").load(test_path)
assert result.count() == 3, f"Expected 3 rows, got {result.count()}"
print("PASS: incremental_append")
finally:
dbutils.fs.rm(test_path, recurse=True)

test_incremental_append()

The try/finally guarantees cleanup even on failure. The random UUID in the path prevents tests from interfering with each other on concurrent runs.

None of these patterns require a testing framework — they run in notebooks and fail loudly on assertion errors. When you're ready to formalize, Databricks Connect lets you run these with pytest from a CI pipeline. Start with the patterns here; the framework comes later when the test suite is worth automating. As always, I'm here to help.

Read more