Deploy OpenZiti Controller, Edge Router, and bootstrap identities/policies

[rowan-stein]

User Request

Deploy the OpenZiti overlay network infrastructure required by the OpenZiti Integration architecture document.

Specification

Overview

Add OpenZiti Controller and Edge Router to the Kubernetes cluster via Helm charts, then provision all infrastructure identities, services, and static service policies via the OpenZiti Terraform provider. This is the foundational infrastructure layer that all other OpenZiti-dependent services require.

Changes Required

1. New stacks/ziti Terraform Stack

Create a new Terraform stack stacks/ziti/ that runs after system (Controller is up) and before platform. This stack uses the netfoundry/ziti Terraform provider (~> 1.0) to manage OpenZiti resources.

Provider authentication: Username/password from the Controller's auto-generated admin secret. The admin password must be extracted between stacks in apply.sh.

2. OpenZiti Controller (in stacks/system)

• Chart: openziti/ziti-controller from https://openziti.io/helm-charts, version 2.1.2
• Namespace: ziti
• Prerequisites: cert-manager and trust-manager (Jetstack Helm charts) must be installed first — they are decoupled from the controller chart in v2
• Key config:
  • clientApi.advertisedHost = ziti.agyn.dev (uses local.base_domain)
  • clientApi.advertisedPort = ingress port from k8s stack
  • clientApi.service.type = ClusterIP
  • managementApi.service.enabled = true, type ClusterIP (separate service for security)
  • persistence.enabled = true, 2Gi
• Outputs: The controller creates an admin secret (ziti-controller-admin-secret) with the admin password

3. OpenZiti Edge Router (in stacks/ziti)

• Chart: openziti/ziti-router from https://openziti.io/helm-charts, version 2.1.0
• Create the router entity via ziti_edge_router Terraform resource, store enrollment JWT in K8s Secret, reference from Helm chart via enrollmentJwtFromSecret
• Key config:
  • ctrl.endpoint = ziti-controller-client.ziti.svc:1280
  • edge.advertisedHost = ziti-router.<base_domain>
  • tunnel.mode = host
  • linkListeners.transport.enabled = false (single router, no inter-router mesh)

4. Istio TLS Passthrough (in stacks/routing)

OpenZiti requires TLS passthrough (not SIMPLE) because the Controller/Router must terminate TLS themselves for mTLS with identities.

• Add a new Istio Gateway resource with PASSTHROUGH mode for ziti.<base_domain> and ziti-router.<base_domain>
• Add VirtualService resources routing to ziti-controller-client.ziti.svc and ziti-router-edge.ziti.svc

5. OpenZiti Services (in stacks/ziti)

Service	Role Attributes	Purpose
gateway	["gateway"]	Agents/channels dial to reach Gateway
runner	["runner"]	Orchestrator dials to reach Runners

6. Infrastructure Identities (in stacks/ziti)

Identity	Type	is_admin	Role Attributes	Purpose
gateway	Device	false	["gateway-hosts"]	Binds the gateway service
ziti-management	Device	true	[]	Cert-auth to Management API via Istio
orchestrator	Device	false	["orchestrators"]	Dials runner service

Each identity's enrollment JWT must be stored as a K8s Secret in the platform namespace.

7. Static Service Policies (in stacks/ziti)

Policy	Type	Identity Roles	Service Roles
agents-dial-gateway	Dial	#agents	@gateway (by ID)
channels-dial-gateway	Dial	#channels	@gateway (by ID)
orchestrators-dial-runners	Dial	#orchestrators	@runner (by ID)
gateway-bind	Bind	#gateway-hosts	@gateway (by ID)
runners-bind	Bind	#runners	@runner (by ID)

8. Edge Router Policies (in stacks/ziti)

• Edge Router Policy: #all identities → #all edge routers
• Service Edge Router Policy: #all services → #all edge routers

9. Update apply.sh

Add the ziti stack to the apply order: k8s → system → routing → ziti → data → platform

Extract the admin password between system and ziti stacks:

ZITI_ADMIN_PASSWORD=$(kubectl -n ziti get secret ziti-controller-admin-secret \
  -o go-template='{{index .data "admin-password" | base64decode}}')

Pass it to the ziti stack: -var "ziti_admin_password=${ZITI_ADMIN_PASSWORD}"

Stack Dependencies

stacks/system (cert-manager, trust-manager, ziti-controller Helm)
  → stacks/routing (Istio TLS passthrough for ziti)
    → stacks/ziti (router, services, identities, policies via Terraform provider)
      → stacks/platform (services consume enrollment JWT secrets)

File Structure

stacks/ziti/
├── main.tf           # Router, services, identities, policies
├── variables.tf      # ziti_admin_password, platform_namespace, chart versions
├── providers.tf      # ziti, kubernetes, helm providers
├── versions.tf       # required_providers (netfoundry/ziti ~> 1.0)
├── outputs.tf        # Identity IDs, service IDs, router ID
├── remote_state.tf   # References k8s and system stacks
└── state/

[rowan-stein]

Additional Changes Required (from maintainer review)

1. Remove hardcoded port values

• stacks/ziti/main.tf: ctrl.endpoint hardcodes :2496 — must use local.ingress_port
• stacks/ziti/providers.tf: host hardcodes 127.0.0.1:1281 — must go through Istio ingress (see feat: align k8s stack with agynio k3d provider #2)

2. Remove port-forward, use Istio VirtualService instead

• The ziti provider should use https://ziti-mgmt.${local.base_domain}:${local.ingress_port}/edge/management/v1 — same pattern as ArgoCD/MinIO providers
• Remove all port-forward plumbing from apply.sh (the readiness wait, PID tracking, cleanup)
• The PASSTHROUGH VirtualService for ziti-mgmt.agyn.dev already exists in the routing stack

3. Move Helm charts to ArgoCD applications

• cert-manager, trust-manager, and ziti-controller are currently deployed via helm_release in the system stack
• They should be deployed as argocd_application resources instead (matching the pattern used in data/platform stacks)
• Constraint: Providers must not depend on resources in the same stack. May need an intermediate stack, but try to avoid it.

4. Fix VirtualService port mismatch (bug)

• virtualservice_ziti_controller routes to port 1280 (container port) but the K8s service port is 2496
• virtualservice_ziti_mgmt routes to port 1281 (container port) but the K8s service port is 2496
• These should route to the service port (local.ingress_port)

5. Add comments explaining CoreDNS rewrite rules

• Each rewrite name rule needs a comment explaining why it exists and what in-cluster traffic it enables

6. Remove CoreDNS restart null_resource

• CoreDNS has a reload plugin that auto-detects ConfigMap changes
• Downstream steps have retry loops that absorb the 1-2 min propagation delay
• Remove the null_resource "restart_coredns" and the null provider dependency