Databricks Secrets: Stop Hardcoding Credentials in Your Notebooks

Shannon Lowder

Shannon Lowder

— 2 min read

There's a pattern I see on almost every team that's new to Databricks. It looks like this:

jdbc_url = "jdbc:sqlserver://prod-sql.mycompany.com:1433;databaseName=Finance"
username = "svc_databricks"
password = "Pr0duction_P@ssword_2018!"  # please don't

df = (spark.read.format("jdbc")
  .option("url", jdbc_url)
  .option("user", username)
  .option("password", password)
  .load())

That notebook just became a credentials file. When someone exports it, checks it into Git, shares it in Slack, or runs it with the wrong job configuration, your production password goes with it.

The fix is Databricks secret scopes, and they're about five minutes to set up.

Creating a Secret Scope

You need the Databricks CLI first (pip install databricks-cli, then databricks configure with your workspace URL and a personal access token). Once that's done:

# Create a scope
databricks secrets create-scope --scope myproject-prod

# Add secrets to it
databricks secrets put --scope myproject-prod --key sql-server-password
databricks secrets put --scope myproject-prod --key adls-storage-key
databricks secrets put --scope myproject-prod --key reporting-api-token

The put command drops you into a text editor for the value. The value never appears in your shell history or CLI output after entry.

Using Secrets in a Notebook

password = dbutils.secrets.get(scope="myproject-prod", key="sql-server-password")

# Databricks redacts the value in any output — intentional
print(password)  # prints: [REDACTED]

# Passes correctly to downstream functions
df = (spark.read.format("jdbc")
  .option("url", jdbc_url)
  .option("user", "svc_databricks")
  .option("password", password)
  .load())

The redaction behavior is intentional and slightly annoying until you understand it. Databricks detects the secret value anywhere it would be displayed and replaces it with [REDACTED]. The actual string is passed correctly to functions — only the display is masked. You can't accidentally log your password to notebook output.

Azure Key Vault-Backed Scopes

If your organization already manages secrets in Azure Key Vault, you can back a Databricks scope with Key Vault directly:

databricks secrets create-scope \
  --scope myproject-prod \
  --scope-backend-type AZURE_KEYVAULT \
  --resource-id /subscriptions/SUB_ID/resourceGroups/RG/providers/Microsoft.KeyVault/vaults/myvault \
  --dns-name https://myvault.vault.azure.net/

After that, any secret in Key Vault is accessible via dbutils.secrets.get("myproject-prod", "secret-name") using the Key Vault secret name. Rotation happens in Key Vault, and Databricks picks up the new value automatically — no scope management needed on the Databricks side when you rotate.

Scope Permissions

By default, only the user who created a scope can read its secrets. Grant access to other users or groups:

databricks secrets put-acl \
  --scope myproject-prod \
  --principal data-engineering \
  --permission READ

Valid permissions are READ, WRITE, and MANAGE. Job service principals need at least READ on any scope they reference at runtime.

The pattern in practice: retrieve all credentials once near the top of the notebook. Never assign them to intermediate variables with names you'd put in a log message. Never pass them to print() or display(). Never embed them in f-strings. Databricks would redact them, but the habit matters when the same code runs somewhere else.

Five minutes of setup per scope, and you never have to explain a credential exposure to your security team. Worth it every time. As always, I'm here to help.

Read more