dbutils.notebook: Orchestrating Databricks From Databricks

Shannon Lowder

Shannon Lowder

— 1 min read

In SQL Server, you call a stored procedure from another stored procedure with EXEC. In Databricks, you call a notebook from another notebook with dbutils.notebook.run(). The concept is the same — modular, reusable units of logic that you orchestrate into a pipeline. The implementation is different enough to matter.

dbutils.notebook.run()

# Call a notebook and pass parameters to it
result = dbutils.notebook.run(
    path="/pipelines/process_region",
    timeout_seconds=3600,
    arguments={
        "region": "West",
        "processing_date": "2019-07-26",
        "environment": "prod"
    }
)

print(f"Notebook returned: {result}")

The called notebook receives the arguments as widget values — accessible via dbutils.widgets.get("region"). The timeout_seconds parameter is a hard deadline; if the notebook doesn't complete within that time, the call raises a NotebookTimedOutError.

Returning Values

The called notebook can return a value using dbutils.notebook.exit():

# In the called notebook (process_region)
region = dbutils.widgets.get("region")
processing_date = dbutils.widgets.get("processing_date")

# ... do the work ...

row_count = result_df.count()

# Exit with a return value (must be a string)
dbutils.notebook.exit(str(row_count))

The return value lands in the result variable in the calling notebook as a string. Parse it if you need a number. Complex return values can be JSON-encoded strings.

Running Notebooks in Parallel

The real power of notebook orchestration is running independent notebooks simultaneously:

from concurrent.futures import ThreadPoolExecutor, as_completed

regions = ["West", "East", "Central", "South"]

def process_region(region):
    result = dbutils.notebook.run(
        "/pipelines/process_region",
        timeout_seconds=3600,
        arguments={"region": region, "processing_date": "2019-07-26"}
    )
    return region, int(result)

with ThreadPoolExecutor(max_workers=4) as executor:
    futures = {executor.submit(process_region, r): r for r in regions}
    for future in as_completed(futures):
        region, count = future.result()
        print(f"{region}: {count} rows processed")

Each dbutils.notebook.run() call runs in a separate notebook context on the cluster. Four parallel calls process four regions simultaneously, then the orchestrating notebook collects the results and continues.

When Notebook Orchestration Is Enough and When It Isn't

For linear pipelines and simple fan-out patterns, notebook orchestration with dbutils.notebook.run() is sufficient. It's fully self-contained within Databricks and requires no additional infrastructure.

Where it falls short: complex DAG dependencies, cross-cluster orchestration, retries with exponential backoff, visibility into long-running pipeline state. For those use cases, Airflow or Databricks Jobs' multi-task configuration is the right move. Start with notebook orchestration for the simple stuff; reach for Airflow when you feel the limits. As always, I'm here to help.

Read more