The Databricks CLI: Scripting Workspace Management and Deployments

Shannon Lowder

Shannon Lowder

— 2 min read

The Databricks CLI is the tool that turns a series of UI clicks into a script. If you're managing a production Databricks workspace — running jobs, deploying notebooks, rotating secrets, managing clusters — doing it through the UI is fine until you need to do it reliably, repeatedly, and without having to be present. That's when you reach for the CLI.

Installation and Configuration

pip install databricks-cli

# Configure for your workspace
databricks configure --token
# Prompts for: Databricks Host, Token

The config is stored in ~/.databrickscfg. You can have multiple profiles for multiple workspaces:

[DEFAULT]
host = https://adb-1234567890.12.azuredatabricks.net
token = dapi1234...

[staging]
host = https://adb-0987654321.34.azuredatabricks.net
token = dapistag...

Pass --profile staging to any CLI command to use a different workspace.

What You'll Actually Use It For

Deploying notebooks:

# Upload a notebook to the workspace
databricks workspace import ./pipelines/process_orders.py \
  /pipelines/process_orders \
  --language PYTHON \
  --overwrite

# Export a notebook for backup or version control
databricks workspace export /pipelines/process_orders \
  ./pipelines/process_orders.py \
  --format SOURCE

Managing secrets:

# List scopes and secrets (names only, never values)
databricks secrets list-scopes
databricks secrets list --scope myproject-prod

# Rotate a secret
databricks secrets put --scope myproject-prod --key sql-server-password
# (prompts for new value in secure editor)

Triggering and monitoring jobs:

# Run a job now (outside its schedule)
databricks jobs run-now --job-id 12345

# List recent runs for a job
databricks runs list --job-id 12345

# Get the status of a specific run
databricks runs get --run-id 67890

Cluster management:

# List running clusters
databricks clusters list

# Start a cluster
databricks clusters start --cluster-id 1234-567890-abc12

# Delete a cluster
databricks clusters permanent-delete --cluster-id 1234-567890-abc12

The Integration That Makes This Useful

The CLI is a prerequisite for any CI/CD pipeline that deploys Databricks assets. A typical deployment script:

#!/bin/bash
# deploy.sh -- run from your CI pipeline

# Deploy notebooks
for notebook in pipelines/*.py; do
    name=$(basename "$notebook" .py)
    databricks workspace import "$notebook" "/production/pipelines/$name" \
      --language PYTHON --overwrite
done

# Rotate the environment-specific parameter
databricks secrets put --scope prod-secrets --key pipeline-version \
  --string-value "$GIT_SHA"

# Trigger a validation run
run_id=$(databricks jobs run-now --job-id 12345 | python3 -c "import json,sys; print(json.load(sys.stdin)['run_id'])")
echo "Validation run started: $run_id"

Without the CLI, this is a series of manual UI steps that someone has to remember and execute in the right order. With the CLI, it's a script that runs the same way every time. As always, I'm here to help.

Read more