Cluster Policies: Guardrails for Multi-User Databricks Workspaces

Shannon Lowder

Shannon Lowder

— 2 min read

Multi-user Databricks workspaces have a problem that SQL Server DBAs will recognize immediately: people will spin up clusters that are too large, leave them running, and run experimental queries on production data when they think no one is looking. The solution in SQL Server is database permissions and resource governor. In Databricks, it's cluster policies.

What Cluster Policies Do

A cluster policy is a set of rules that applies to cluster creation. Policies can:

  • Restrict which node types are allowed
  • Set maximum cluster sizes
  • Force auto-termination settings
  • Hide options from users (so they can't override your defaults)
  • Set default values that users can change within bounds

Workspace admins create policies and assign them to users or groups. When a user creates a cluster, they choose from the policies they've been assigned. They see only the options the policy allows.

A Practical Policy Example

Here's a policy for a data engineering team — allows reasonable cluster sizes, forces auto-termination, prevents use of GPU nodes:

{
  "autotermination_minutes": {
    "type": "fixed",
    "value": 120,
    "hidden": true
  },
  "num_workers": {
    "type": "range",
    "minValue": 1,
    "maxValue": 8,
    "defaultValue": 2
  },
  "node_type_id": {
    "type": "allowlist",
    "values": [
      "Standard_DS3_v2",
      "Standard_DS4_v2",
      "Standard_DS5_v2"
    ],
    "defaultValue": "Standard_DS3_v2"
  },
  "spark_conf.spark.databricks.cluster.profile": {
    "type": "fixed",
    "value": "serverless",
    "hidden": true
  }
}

The "hidden": true on autotermination_minutes means users don't even see that option in the create cluster UI — the 2-hour auto-termination just silently applies. They get the cluster experience they need without the ability to create a 20-node cluster and leave it running for a week.

Assigning Policies to Groups

Create policies in the Admin Console (Settings → Cluster Policies), then assign them to groups:

# Via the Policies API
PATCH /api/2.0/policies/clusters/{policy_id}/permissions
{
  "access_control_list": [
    {"group_name": "data-engineering", "permission_level": "CAN_USE"},
    {"group_name": "data-science", "permission_level": "CAN_USE"}
  ]
}

Teams can have access to multiple policies — for example, data engineers might have access to a "batch job" policy (larger clusters, job cluster defaults) and an "interactive" policy (smaller clusters, auto-termination enforced).

The Default Policy Gap

By default, workspace users can create clusters with no policy applied — any size, any configuration, no auto-termination required. If you want policy enforcement to be mandatory, remove the "Unrestricted" policy from regular users and only assign your team-specific policies. Admins bypass policies by design, which is intentional for troubleshooting but something to be aware of.

Cluster policies aren't perfect governance — they don't control what data a user accesses, only how big a cluster they can create. For data access control, you need table ACLs or Unity Catalog. But for cloud cost management, policies are the right first line of defense. As always, I'm here to help.

Read more