feat: module and example to create compute policies#226
feat: module and example to create compute policies#226steven-muschler wants to merge 4 commits intodatabricks:mainfrom
Conversation
…te policies using policy families
| policy_key = "personal-vm" | ||
| policy_family_id = "personal-vm" |
There was a problem hiding this comment.
do we need both if they are the same?
There was a problem hiding this comment.
@alexott let me put in a better example for this scenario. The reason these are separate because DLT/SDP does not have a policy family, but requires specific policy elements like cluster_type
| policy_family_id = "power-user" | ||
| group_assignments = ["groupB"] | ||
| service_principal_assignments = [] | ||
| policy_overrides = "{\"autotermination_minutes\":{\"type\":\"fixed\",\"value\":60,\"hidden\": true}}" |
There was a problem hiding this comment.
it's easier to read as jsonencode + object, like here: https://registry.terraform.io/providers/databricks/databricks/latest/docs/resources/cluster_policy#example-usage
There was a problem hiding this comment.
@alexott there's an issue I've run into before on this where the TF does not handle the object typing well. If we set the override type to an optional empty object, TF will type it based on the first one it sees in the variable files if we deploy more than one of the same resource. I'm open to other ideas, but something I've struggled with when working with one of our customers.
| cluster_policy_defaults = ( | ||
| { | ||
| "personal-vm-dev" = merge(local.autotermination, local.required_tags, local.dev_runtimes) | ||
| "personal-vm-prod" = merge(local.autotermination, local.required_tags, local.prod_runtimes) | ||
| "shared-compute-dev" = merge(local.autotermination, local.required_tags, local.dev_runtimes) | ||
| "shared-compute-prod" = merge(local.autotermination, local.required_tags, local.prod_runtimes) | ||
| "power-user-dev" = merge(local.autotermination, local.required_tags, local.dev_runtimes) | ||
| "power-user-prod" = merge(local.autotermination, local.required_tags, local.prod_runtimes) | ||
| "job-cluster-dev" = merge(local.job_cluster_types, local.required_tags, local.dev_runtimes) | ||
| "job-cluster-prod" = merge(local.job_cluster_types, local.required_tags, local.prod_runtimes) | ||
| "sdp-cluster-dev" = merge(local.sdp_cluster_types, local.required_tags, local.dev_runtimes) | ||
| "sdp-cluster-prod" = merge(local.sdp_cluster_types, local.required_tags, local.prod_runtimes) | ||
| } | ||
| ) | ||
| } |
There was a problem hiding this comment.
It would be very useful to provide more information about these built-in policy definitions
There was a problem hiding this comment.
Added a longer description to the module README
| } | ||
|
|
||
| dynamic "access_control" { | ||
| for_each = toset(var.service_principal_assignments) |
There was a problem hiding this comment.
Should we add variable for users as well?
There was a problem hiding this comment.
I can, though I've intentionally not done that as I don't believe that's a great pattern to encourage. Will change though if you feel it's important. LMK. Thanks!
Description for all variables Remove alias as only one provider is used in this example Show example of when policy_key and policy_family_id are different
There was a problem hiding this comment.
Pull request overview
This PR introduces a Terraform module for creating Databricks cluster policies based on policy families, with support for environment-specific configurations and team-level customization. The implementation provides a reusable framework for managing compute policies across different teams and environments.
Changes:
- Added a new Terraform module
cluster-policy-from-policy-familythat creates cluster policies with configurable overrides - Implemented JSON-based policy configurations for different environments (dev/prod), runtimes (standard/ML), and cluster types
- Created an example configuration demonstrating multi-team policy deployment with team-specific overrides
Reviewed changes
Copilot reviewed 18 out of 18 changed files in this pull request and generated 8 comments.
Show a summary per file
| File | Description |
|---|---|
| modules/cluster-policy-from-policy-family/variables.tf | Defines input variables for team, environment, policy version, and override configurations |
| modules/cluster-policy-from-policy-family/provider.tf | Configures required Databricks provider |
| modules/cluster-policy-from-policy-family/outputs.tf | Exposes the created policy ID |
| modules/cluster-policy-from-policy-family/cluster-policies.tf | Main module logic that merges policy configurations and creates resources |
| modules/cluster-policy-from-policy-family/cluster_policy_json/*.json | JSON configuration files for various policy components |
| modules/cluster-policy-from-policy-family/README.md | Module documentation |
| modules/cluster-policy-from-policy-family/Makefile | Build targets for documentation generation |
| examples/cluster-policies/*.tf | Example implementation showing multi-team usage |
| examples/cluster-policies/example.tfvars | Sample variable values for the example |
| examples/cluster-policies/README.md | Example documentation |
| examples/cluster-policies/Makefile | Build targets for example documentation |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
2 participants
Adding in TF module and example that demonstrates how to create compute policies using policy families and allow for team specific overrides