From 67358e83d533b84f9b0e2c39c17b75fc2f52752c Mon Sep 17 00:00:00 2001 From: Scot Wells Date: Sat, 25 Jul 2026 16:02:21 -0500 Subject: [PATCH 1/9] feat: Detect upstream/downstream control-plane DNS drift via metrics + alerting MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Adds detection for the failure class behind engineering#346, where a leftover downstream DNSRecordSet kept "reserving" a hostname after its upstream owner was deleted (a multicluster-runtime sharding bug let it linger), so later records for that name were rejected as duplicates and the customer domain was unresolvable for ~24h. There was no signal for the underlying condition — a resource existing at one replication seam with no owner at the seam above it — which is directly observable as a discrepancy between per-object metrics collected from each control plane. The approach uses milo-os/resource-metrics (discovery.mode=milo + collectRootControlPlane) to emit one gauge series per DNSRecordSet/DNSZone from every project (upstream) control plane and the Milo core (downstream) control plane, lifting the replicator's meta.datumapis.com/upstream-* annotations onto labels so the two sides share a join key. Prometheus recording rules then diff them: dns:recordset_downstream_orphan (downstream with no surviving upstream owner — the #346 case) and dns:recordset_downstream_missing (upstream never replicated), with the DNSDownstreamOrphanRecordSet / DNSDownstreamMissingRecordSet / DNSRecordSetNotAccepted alerts. Rules ship as a standard PrometheusRule (victoria-metrics-operator converts it) labeled for the resource-metrics aggregator vmalert; no new operator code is required for the primary signal. config/observability holds the dns-metrics ResourceMetricsPolicy and the dns-drift PrometheusRule. config/dependencies/{milo,resource-metrics} and config/overlays/{replicator-milo,agent-powerdns-milo} bring up the production-accurate Milo topology (mode:milo replicator with 2 active replicas, PowerDNS agent reading the core CP) for the e2e. test/e2e/controlplane-drift is a Chainsaw suite (happy-path / orphan / missing) verified end-to-end against a from-scratch rebuild, wired via new Taskfile tasks (env:milo-all-up, env:milo-bootstrap, env:upstream-milo-up, env:metrics-up, env:observability-up, env:chainsaw-milo). See docs/enhancements/controlplane-drift-detection.md for the full design, the datum-cloud/infra alignment, and the live-validated findings (e.g. the cluster-_ join-key normalization). Co-Authored-By: Claude Opus 4.8 --- Taskfile.yaml | 189 +++++++++++ .../milo/flux-install-infra-crds.yaml | 25 ++ config/dependencies/milo/flux-install.yaml | 54 ++++ config/dependencies/milo/kustomization.yaml | 14 + config/dependencies/milo/namespace.yaml | 10 + config/dependencies/milo/ocirepository.yaml | 16 + .../dependencies/resource-metrics/README.md | 133 ++++++++ .../controller/deployment.yaml | 97 ++++++ .../controller/kustomization.yaml | 36 +++ .../controller/milo-kubeconfig-secret.yaml | 40 +++ .../controller/namespace.yaml | 8 + .../resource-metrics/controller/rbac.yaml | 75 +++++ .../controller/server-config.yaml | 36 +++ .../core-control-plane/crd/kustomization.yaml | 15 + .../crd/resourcemetricspolicies-crd.yaml | 297 ++++++++++++++++++ .../policy/kustomization.yaml | 18 ++ config/observability/dns-drift-rules.yaml | 127 ++++++++ config/observability/dns-metrics-policy.yaml | 108 +++++++ config/observability/kustomization.yaml | 21 ++ config/overlays/agent-powerdns-milo/README.md | 39 +++ .../agent-powerdns-milo/deployment-patch.yaml | 29 ++ .../agent-powerdns-milo/kustomization.yaml | 42 +++ .../milo-kubeconfig-secret.yaml | 42 +++ .../agent-powerdns-milo/server-config.yaml | 46 +++ .../replicator-milo/kustomization.yaml | 48 +++ .../overlays/replicator-milo/patch-milo.yaml | 26 ++ .../replicator-milo/server-config.yaml | 26 ++ .../controlplane-drift-detection.md | 177 +++++++++++ test/e2e/controlplane-drift/.chainsaw.yaml | 18 ++ test/e2e/controlplane-drift/.gitignore | 2 + test/e2e/controlplane-drift/README.md | 117 +++++++ .../controlplane-drift/fixtures/dnszone.yaml | 13 + .../fixtures/dnszoneclass.yaml | 19 ++ .../fixtures/milo-projects.yaml | 33 ++ .../happy-path/chainsaw-test.yaml | 186 +++++++++++ .../missing/chainsaw-test.yaml | 173 ++++++++++ .../orphan/chainsaw-test.yaml | 280 +++++++++++++++++ 37 files changed, 2635 insertions(+) create mode 100644 config/dependencies/milo/flux-install-infra-crds.yaml create mode 100644 config/dependencies/milo/flux-install.yaml create mode 100644 config/dependencies/milo/kustomization.yaml create mode 100644 config/dependencies/milo/namespace.yaml create mode 100644 config/dependencies/milo/ocirepository.yaml create mode 100644 config/dependencies/resource-metrics/README.md create mode 100644 config/dependencies/resource-metrics/controller/deployment.yaml create mode 100644 config/dependencies/resource-metrics/controller/kustomization.yaml create mode 100644 config/dependencies/resource-metrics/controller/milo-kubeconfig-secret.yaml create mode 100644 config/dependencies/resource-metrics/controller/namespace.yaml create mode 100644 config/dependencies/resource-metrics/controller/rbac.yaml create mode 100644 config/dependencies/resource-metrics/controller/server-config.yaml create mode 100644 config/dependencies/resource-metrics/core-control-plane/crd/kustomization.yaml create mode 100644 config/dependencies/resource-metrics/core-control-plane/crd/resourcemetricspolicies-crd.yaml create mode 100644 config/dependencies/resource-metrics/core-control-plane/policy/kustomization.yaml create mode 100644 config/observability/dns-drift-rules.yaml create mode 100644 config/observability/dns-metrics-policy.yaml create mode 100644 config/observability/kustomization.yaml create mode 100644 config/overlays/agent-powerdns-milo/README.md create mode 100644 config/overlays/agent-powerdns-milo/deployment-patch.yaml create mode 100644 config/overlays/agent-powerdns-milo/kustomization.yaml create mode 100644 config/overlays/agent-powerdns-milo/milo-kubeconfig-secret.yaml create mode 100644 config/overlays/agent-powerdns-milo/server-config.yaml create mode 100644 config/overlays/replicator-milo/kustomization.yaml create mode 100644 config/overlays/replicator-milo/patch-milo.yaml create mode 100644 config/overlays/replicator-milo/server-config.yaml create mode 100644 docs/enhancements/controlplane-drift-detection.md create mode 100644 test/e2e/controlplane-drift/.chainsaw.yaml create mode 100644 test/e2e/controlplane-drift/.gitignore create mode 100644 test/e2e/controlplane-drift/README.md create mode 100644 test/e2e/controlplane-drift/fixtures/dnszone.yaml create mode 100644 test/e2e/controlplane-drift/fixtures/dnszoneclass.yaml create mode 100644 test/e2e/controlplane-drift/fixtures/milo-projects.yaml create mode 100644 test/e2e/controlplane-drift/happy-path/chainsaw-test.yaml create mode 100644 test/e2e/controlplane-drift/missing/chainsaw-test.yaml create mode 100644 test/e2e/controlplane-drift/orphan/chainsaw-test.yaml diff --git a/Taskfile.yaml b/Taskfile.yaml index b1efca4..f9b1afe 100644 --- a/Taskfile.yaml +++ b/Taskfile.yaml @@ -27,6 +27,21 @@ vars: # In-cluster-rewritten kubeconfig for the control cluster, mounted into the # replicator (via a Secret) so it can reach control's API as its downstream. CONTROL_INCLUSTER_KUBECONFIG: 'dev/control.incluster.kubeconfig' + # Admin kubeconfig for the Milo core control plane (milo-apiserver), minted + # from the static test-admin-token. Used to install DNS CRDs and create the + # Org + projects on the core CP. Server points at a local port-forward. + MILO_ADMIN_KUBECONFIG: 'dev/milo.admin.kubeconfig' + # dns-operator image used by the replicator on dns-upstream. Loaded into the + # kind node; the overlays reference :latest with imagePullPolicy IfNotPresent. + DNSOP_IMG: 'ghcr.io/datum-cloud/dns-operator:latest' + # Cross-cluster kubeconfig the replicator uses to reach milo-apiserver on + # dns-control, via the envoy gateway NodePort on the control node's IP. + MILO_UPSTREAM_KUBECONFIG: 'dev/milo.upstream.kubeconfig' + MILO_GATEWAY_NODEPORT: '32648' + # resource-metrics controller image (loaded into the dns-control kind node). + # Published image is amd64-only (see milo-os/resource-metrics#13); on arm64 + # build locally: `docker build -t resource-metrics:arm64-dev `. + RM_IMG: 'resource-metrics:arm64-dev' # Dedicated kubeconfig for this environment (not the user's default # ~/.kube/config). go-task's `env:` blocks are implemented as process-wide # os.Setenv calls that leak between sibling task/cmd invocations (a known @@ -317,3 +332,177 @@ tasks: vars: CLUSTER_NAME: "{{.EDGE_CLUSTER_NAME}}" - rm -f {{.RUSTFS_NODEPORT_FILE}} {{.ENV_KUBECONFIG}} {{.CONTROL_INCLUSTER_KUBECONFIG}} + + # ---- Milo control-plane (drift-detection / Phase B) -------------------- + # Installs the Milo apiserver + controller-manager into the control cluster + # via Flux, so the replicator can run discovery.mode=milo and resource-metrics + # can collect project + core control planes. See config/dependencies/milo and + # docs/enhancements/controlplane-drift-detection.md. + env:milo-up: + desc: "Install Milo (apiserver + controller-manager) into the control cluster via Flux" + silent: true + cmds: + - echo "➡️ Installing Milo into kind-{{.CONTROL_CLUSTER_NAME}}..." + - KUBECONFIG={{.ENV_KUBECONFIG}} kubectl --context kind-{{.CONTROL_CLUSTER_NAME}} apply -k config/dependencies/milo + - echo "⏳ waiting for Flux OCIRepository 'milo' to reconcile..." + - KUBECONFIG={{.ENV_KUBECONFIG}} kubectl --context kind-{{.CONTROL_CLUSTER_NAME}} -n flux-system wait ocirepository/milo --for=condition=Ready --timeout=180s + - echo "⏳ waiting for Flux Kustomization 'milo-infra-crds'..." + - KUBECONFIG={{.ENV_KUBECONFIG}} kubectl --context kind-{{.CONTROL_CLUSTER_NAME}} -n flux-system wait kustomization/milo-infra-crds --for=condition=Ready --timeout=300s + - echo "⏳ waiting for Flux Kustomization 'milo' (cold start applies CRDs + deploys apiserver, up to 10m)..." + - KUBECONFIG={{.ENV_KUBECONFIG}} kubectl --context kind-{{.CONTROL_CLUSTER_NAME}} -n flux-system wait kustomization/milo --for=condition=Ready --timeout=600s + - echo "✅ Milo installed. milo-system objects:" + - KUBECONFIG={{.ENV_KUBECONFIG}} kubectl --context kind-{{.CONTROL_CLUSTER_NAME}} -n milo-system get pods,svc + + env:derisk-control-milo: + desc: "De-risk: bring up ONLY the control cluster + Milo (no agent/rustfs) to inspect the live milo-apiserver" + cmds: + - task: env:test-infra-cluster-up + vars: + CLUSTER_NAME: "{{.CONTROL_CLUSTER_NAME}}" + - task: env:milo-up + + # internal: mint an admin kubeconfig for the Milo core CP via a transient + # port-forward, then run CMD (a kubectl invocation with KUBECONFIG=$mk). + # milo-apiserver's NodePort is not host-mapped, and on macOS the host cannot + # reach the kind docker-network IP, so a port-forward is the portable way to + # drive the core CP from the host (macOS dev + Linux CI). + env:with-milo-admin: + internal: true + silent: true + cmds: + - | + set -euo pipefail + export KUBECONFIG={{.ENV_KUBECONFIG}} + kubectl --context kind-{{.CONTROL_CLUSTER_NAME}} -n milo-system port-forward svc/milo-apiserver 16443:6443 >/tmp/milo-pf.log 2>&1 & + pf=$!; trap 'kill $pf 2>/dev/null || true' EXIT + for i in $(seq 1 30); do + curl -sk -o /dev/null -m 2 https://127.0.0.1:16443/livez && break || sleep 1 + done + mk={{.MILO_ADMIN_KUBECONFIG}} + kubectl config --kubeconfig="$mk" set-cluster milo --server=https://127.0.0.1:16443 --insecure-skip-tls-verify=true >/dev/null + kubectl config --kubeconfig="$mk" set-credentials admin --token=test-admin-token >/dev/null + kubectl config --kubeconfig="$mk" set-context milo --cluster=milo --user=admin >/dev/null + kubectl config --kubeconfig="$mk" use-context milo >/dev/null + KUBECONFIG="$mk" {{.CMD}} + + env:observability-up: + desc: "Install the observability stack (Victoria Metrics + OTel + Prometheus CRDs) into the control cluster" + cmds: + - KUBECONFIG={{.ENV_KUBECONFIG}} TASK_X_REMOTE_TASKFILES=1 task --yes test-infra:install-observability + + env:metrics-up: + desc: "Deploy resource-metrics (mode:milo + collectRootControlPlane) + the dns-metrics policy + drift rules" + cmds: + - kind load docker-image {{.RM_IMG}} --name {{.CONTROL_CLUSTER_NAME}} + # ResourceMetricsPolicy CRD + the dns-metrics policy live on the Milo core CP + - task: env:with-milo-admin + vars: + CMD: kubectl apply -k config/dependencies/resource-metrics/core-control-plane/crd + - task: env:with-milo-admin + vars: + CMD: kubectl wait --for=condition=Established crd/resourcemetricspolicies.resourcemetrics.miloapis.com --timeout=60s + - task: env:with-milo-admin + vars: + CMD: kubectl apply -f config/observability/dns-metrics-policy.yaml + # Controller runs on the control kind cluster; drift rules load into VM + - KUBECONFIG={{.ENV_KUBECONFIG}} kubectl --context kind-{{.CONTROL_CLUSTER_NAME}} apply -k config/dependencies/resource-metrics/controller + - KUBECONFIG={{.ENV_KUBECONFIG}} kubectl --context kind-{{.CONTROL_CLUSTER_NAME}} -n resource-metrics-system rollout status deploy/resource-metrics-controller-manager --timeout=150s + - KUBECONFIG={{.ENV_KUBECONFIG}} kubectl --context kind-{{.CONTROL_CLUSTER_NAME}} -n telemetry-system apply -f config/observability/dns-drift-rules.yaml + - echo "✅ resource-metrics + dns-metrics policy + drift rules deployed" + + env:milo-all-up: + desc: "Full clean bring-up: control+Milo, CRDs+projects, observability, resource-metrics, upstream replicator" + cmds: + - task: env:derisk-control-milo + - task: env:milo-bootstrap + - task: env:observability-up + - task: env:metrics-up + - task: env:upstream-milo-up + + env:chainsaw-milo: + desc: "Run the control-plane drift Chainsaw suite (happy-path/orphan/missing) against the running Milo env" + cmds: + - | + set -euo pipefail + root="$(pwd)" + cd test/e2e/controlplane-drift + # infra = dns-control kind (VM + OTel); replicator = dns-upstream kind. + # Both kind APIs are host-reachable. + kind get kubeconfig --name {{.CONTROL_CLUSTER_NAME}} > kubeconfig-infra + kind get kubeconfig --name {{.UPSTREAM_CLUSTER_NAME}} > kubeconfig-replicator + # alpha (project CP) + core (Milo core CP): reach milo-apiserver via a + # localhost port-forward — portable across macOS + Linux, since the + # gateway NodePort is not host-mapped and macOS can't reach the kind + # docker-network IP directly. + KUBECONFIG="$root/{{.ENV_KUBECONFIG}}" kubectl --context kind-{{.CONTROL_CLUSTER_NAME}} \ + -n milo-system port-forward svc/milo-apiserver 16443:6443 >/tmp/milo-pf-chainsaw.log 2>&1 & + pf=$!; trap 'kill $pf 2>/dev/null || true' EXIT + for i in $(seq 1 30); do curl -sk -o /dev/null -m2 https://127.0.0.1:16443/livez && break || sleep 1; done + for name in core alpha; do + kubectl config --kubeconfig=kubeconfig-$name set-credentials a --token=test-admin-token >/dev/null + kubectl config --kubeconfig=kubeconfig-$name set-context $name --cluster=$name --user=a >/dev/null + kubectl config --kubeconfig=kubeconfig-$name use-context $name >/dev/null + done + kubectl config --kubeconfig=kubeconfig-core set-cluster core \ + --server=https://127.0.0.1:16443 --insecure-skip-tls-verify=true >/dev/null + kubectl config --kubeconfig=kubeconfig-alpha set-cluster alpha \ + --server=https://127.0.0.1:16443/apis/resourcemanager.miloapis.com/v1alpha1/projects/alpha/control-plane \ + --insecure-skip-tls-verify=true >/dev/null + KUBECONFIG=kubeconfig-infra chainsaw test --config .chainsaw.yaml {{.CHAINSAW_SUITE | default "."}} + + env:upstream-milo-up: + desc: "Create dns-upstream and deploy the replicator (discovery.mode=milo, 2 replicas) pointed at Milo on dns-control" + cmds: + - task: env:test-infra-cluster-up + vars: + CLUSTER_NAME: "{{.UPSTREAM_CLUSTER_NAME}}" + - kind load docker-image {{.DNSOP_IMG}} --name {{.UPSTREAM_CLUSTER_NAME}} + # Mint the cross-cluster milo kubeconfig (control node IP + gateway + # NodePort) and stash it as the milo-kubeconfig Secret the replicator-milo + # overlay mounts at /milo. NODE_IP via python to avoid go-task/docker + # template brace conflicts. + - | + set -euo pipefail + export KUBECONFIG={{.ENV_KUBECONFIG}} + NODE_IP=$(docker inspect {{.CONTROL_CLUSTER_NAME}}-control-plane | python3 -c "import json,sys; n=json.load(sys.stdin)[0]['NetworkSettings']['Networks']; print(next(iter(n.values()))['IPAddress'])") + # Derive the envoy gateway NodePort dynamically — test-infra's + # fix/remove-hardcoded-nodeports branch assigns it per-cluster, so it is + # NOT stable across rebuilds. Look up the nodePort mapped to gateway + # port 8443 (the HTTPS listener that fronts milo-apiserver). + NODEPORT=$(kubectl --context kind-{{.CONTROL_CLUSTER_NAME}} -n envoy-gateway-system get svc \ + -o jsonpath='{range .items[*]}{range .spec.ports[?(@.port==8443)]}{.nodePort}{end}{end}') + test -n "$NODEPORT" || { echo "could not resolve envoy gateway 8443 NodePort" >&2; exit 1; } + echo "➡️ milo gateway endpoint for replicator: https://${NODE_IP}:${NODEPORT}" + mk={{.MILO_UPSTREAM_KUBECONFIG}} + kubectl config --kubeconfig="$mk" set-cluster milo --server="https://${NODE_IP}:${NODEPORT}" --insecure-skip-tls-verify=true >/dev/null + kubectl config --kubeconfig="$mk" set-credentials admin --token=test-admin-token >/dev/null + kubectl config --kubeconfig="$mk" set-context milo --cluster=milo --user=admin >/dev/null + kubectl config --kubeconfig="$mk" use-context milo >/dev/null + kubectl --context kind-{{.UPSTREAM_CLUSTER_NAME}} create ns dns-replicator-system --dry-run=client -o yaml | kubectl --context kind-{{.UPSTREAM_CLUSTER_NAME}} apply -f - + kubectl --context kind-{{.UPSTREAM_CLUSTER_NAME}} -n dns-replicator-system create secret generic milo-kubeconfig --from-file=kubeconfig="$mk" --dry-run=client -o yaml | kubectl --context kind-{{.UPSTREAM_CLUSTER_NAME}} apply -f - + - KUBECONFIG={{.ENV_KUBECONFIG}} CONTEXT=kind-{{.UPSTREAM_CLUSTER_NAME}} KUSTOMIZE_DIR=config/overlays/replicator-milo make kustomize-apply + - KUBECONFIG={{.ENV_KUBECONFIG}} kubectl --context kind-{{.UPSTREAM_CLUSTER_NAME}} -n dns-replicator-system rollout status deploy/dns-operator-controller-manager --timeout=180s + - echo "✅ replicator up on kind-{{.UPSTREAM_CLUSTER_NAME}} (mode=milo, 2 replicas)" + + env:milo-bootstrap: + desc: "Install DNS + networking CRDs into the Milo core CP and create the drift Org + projects (alpha, beta)" + cmds: + - task: env:with-milo-admin + vars: + CMD: kubectl apply -k config/crd + # The replicator (dnszone/dnszonediscovery controllers) indexes Domain + # (networking.datumapis.com); without its CRD, project-CP engagement fails + # on cache-index setup. Generate the networking CRDs and install them into + # the core CP (shared to all project CPs). The make target also applies to + # the local kind API (harmless) and writes dev/crds/network-services. + - KUBECONFIG={{.ENV_KUBECONFIG}} make install-networking-crds CONTEXT=kind-{{.CONTROL_CLUSTER_NAME}} + - task: env:with-milo-admin + vars: + CMD: kubectl apply -f dev/crds/network-services + - task: env:with-milo-admin + vars: + CMD: kubectl apply -f test/e2e/controlplane-drift/fixtures/milo-projects.yaml + - task: env:with-milo-admin + vars: + CMD: kubectl wait --for=condition=Ready project/alpha project/beta --timeout=120s + - echo "✅ Milo core CP has DNS + networking CRDs; projects alpha, beta are Ready" diff --git a/config/dependencies/milo/flux-install-infra-crds.yaml b/config/dependencies/milo/flux-install-infra-crds.yaml new file mode 100644 index 0000000..d573c49 --- /dev/null +++ b/config/dependencies/milo/flux-install-infra-crds.yaml @@ -0,0 +1,25 @@ +apiVersion: kustomize.toolkit.fluxcd.io/v1 +kind: Kustomization +metadata: + name: milo-infra-crds + namespace: flux-system +# milo's overlays/test-infra does NOT install the ProjectControlPlane CRD, but +# milo-controller-manager runs with --control-plane-scope=core and watches +# ProjectControlPlane. Without this CRD, the project reconciler loops on +# "no matches for kind ProjectControlPlane", Projects never go Ready, and the +# drift chainsaw suite times out. Install the CRD separately from the same OCI +# bundle so it tracks the pinned milo tag. Remove once milo's test-infra +# overlay ships the infrastructure-group CRDs. +spec: + interval: 10m + retryInterval: 1m + timeout: 2m + prune: true + wait: true + sourceRef: + kind: OCIRepository + name: milo + # The published bundle flattens the config tree: milo's + # config/crd/bases/infrastructure is at crd/bases/infrastructure here. + path: crd/bases/infrastructure + dependsOn: [] diff --git a/config/dependencies/milo/flux-install.yaml b/config/dependencies/milo/flux-install.yaml new file mode 100644 index 0000000..19799c2 --- /dev/null +++ b/config/dependencies/milo/flux-install.yaml @@ -0,0 +1,54 @@ +apiVersion: kustomize.toolkit.fluxcd.io/v1 +kind: Kustomization +metadata: + name: milo + namespace: flux-system +spec: + interval: 10m + retryInterval: 1m + timeout: 5m + prune: true + wait: true + targetNamespace: milo-system + # ProjectControlPlane CRD (from milo-infra-crds) must exist before the milo + # overlay starts, or milo-controller-manager crash-loops on the missing CRD. + dependsOn: + - name: milo-infra-crds + sourceRef: + kind: OCIRepository + name: milo + path: overlays/test-infra + # The upstream test-infra overlay assumes locally kind-loaded `dev` images; + # CI pulls from the registry, so override the tag to the OCI bundle pin. + # Keep in sync with ocirepository.yaml. + images: + - name: ghcr.io/datum-cloud/milo + newTag: v0.0.0-main + # Strip argo-system (argo-events HelmRelease + argo HelmRepository): the DNS + # drift e2e does not exercise event-driven flows, and argo-events pulls in a + # NATS JetStream dependency that bloats spin-up. Remove via $patch: delete. + patches: + - target: + group: helm.toolkit.fluxcd.io + version: v2 + kind: HelmRelease + name: argo-events + patch: | + apiVersion: helm.toolkit.fluxcd.io/v2 + kind: HelmRelease + metadata: + name: argo-events + namespace: milo-system + $patch: delete + - target: + group: source.toolkit.fluxcd.io + version: v1 + kind: HelmRepository + name: argo + patch: | + apiVersion: source.toolkit.fluxcd.io/v1 + kind: HelmRepository + metadata: + name: argo + namespace: milo-system + $patch: delete diff --git a/config/dependencies/milo/kustomization.yaml b/config/dependencies/milo/kustomization.yaml new file mode 100644 index 0000000..c0f6171 --- /dev/null +++ b/config/dependencies/milo/kustomization.yaml @@ -0,0 +1,14 @@ +# Deploys the Milo apiserver + controller-manager into the control cluster via +# Flux, so the DNS drift e2e can run the replicator and resource-metrics in the +# production `discovery.mode: milo` topology (project control planes for the +# upstream side, the Milo core control plane as the downstream). Lifted from +# milo-os/resource-metrics config/dependencies/milo. +# +# See docs/enhancements/controlplane-drift-detection.md. +apiVersion: kustomize.config.k8s.io/v1beta1 +kind: Kustomization +resources: + - namespace.yaml + - ocirepository.yaml + - flux-install-infra-crds.yaml + - flux-install.yaml diff --git a/config/dependencies/milo/namespace.yaml b/config/dependencies/milo/namespace.yaml new file mode 100644 index 0000000..ac4a1e9 --- /dev/null +++ b/config/dependencies/milo/namespace.yaml @@ -0,0 +1,10 @@ +apiVersion: v1 +kind: Namespace +metadata: + name: milo-system + labels: + app.kubernetes.io/name: milo + app.kubernetes.io/managed-by: kustomize +# milo's overlays/test-infra targets milo-system (via spec.namespace). Flux does +# not auto-create a Kustomization's target namespace, so pre-create it here +# alongside the OCIRepository + Flux Kustomizations. diff --git a/config/dependencies/milo/ocirepository.yaml b/config/dependencies/milo/ocirepository.yaml new file mode 100644 index 0000000..c452789 --- /dev/null +++ b/config/dependencies/milo/ocirepository.yaml @@ -0,0 +1,16 @@ +apiVersion: source.toolkit.fluxcd.io/v1beta2 +kind: OCIRepository +metadata: + name: milo + namespace: flux-system +spec: + interval: 5m + url: oci://ghcr.io/datum-cloud/milo-kustomize + # milo publishes this bundle from milo/.github/workflows/build-apiserver.yaml. + # Tags: semver releases (v0.24.3) and per-merge v0.0.0- / + # v0.0.0-main-. Pin to a reproducible tag; keep in sync with the + # image newTag override in flux-install.yaml. Tracking the v0.0.0-main + # floating tag for now because the latest semver predates the multi-arch + # publish and ImagePullBackOffs on arm64 kind clusters. Bump deliberately. + ref: + tag: v0.0.0-main diff --git a/config/dependencies/resource-metrics/README.md b/config/dependencies/resource-metrics/README.md new file mode 100644 index 0000000..0bbde2a --- /dev/null +++ b/config/dependencies/resource-metrics/README.md @@ -0,0 +1,133 @@ +# resource-metrics dependency (DNS control-plane drift detection) + +Deploys the [milo-os/resource-metrics](https://github.com/milo-os/resource-metrics) +controller for the DNS control-plane drift-detection e2e, plus the pieces that +must live on the Milo core control plane. + +`resource-metrics` watches every project (UPSTREAM) control plane served by +`milo-apiserver` and — with `discovery.collectRootControlPlane: true` — the +Milo core/root (DOWNSTREAM) control plane as well. It evaluates the +`dns-metrics` `ResourceMetricsPolicy` and pushes one gauge series per matching +object over OTLP to the test-infra OTel collector, which forwards to Victoria +Metrics. Recording rules then diff the upstream (source of truth) and +downstream (replicated copy) series to detect drift. See +`docs/enhancements/controlplane-drift-detection.md`. + +## Topology (validated on the dns-control kind cluster) + +- **dns-control** hosts the Milo core control plane (the DOWNSTREAM / "root"), + the PowerDNS agent, RustFS, the observability stack, AND this + resource-metrics controller. +- Project control planes **alpha/beta** (served by `milo-apiserver`) are the + UPSTREAM. +- **dns-upstream** hosts the replicator; **dns-edge** hosts PowerDNS. + +Because resource-metrics runs on dns-control alongside `milo-apiserver`, it +reaches the core CP over the in-cluster Service +(`https://milo-apiserver.milo-system.svc.cluster.local:6443`, self-signed → +`insecure-skip-tls-verify`) using the static `test-admin-token` +(`system:masters`). + +## Two deployment slices, two different API servers + +This directory is split by **which control plane / kubeconfig each slice +targets**. They are applied separately and never combined into a single +`kubectl apply`. + +### `controller/` — applied to the KIND (dns-control) context + +The controller Deployment, its namespace, RBAC, the `milo-kubeconfig` Secret, +and the `mode: milo` server-config. Apply with the **kind kubeconfig/context** +for dns-control. The controller itself then talks to the Milo core CP through +the mounted `milo-kubeconfig` Secret. + +| File | Purpose | +| --- | --- | +| `namespace.yaml` | `resource-metrics-system` namespace. | +| `rbac.yaml` | ServiceAccount + ClusterRole + ClusterRoleBinding (vendored from the operator's `controller_rbac` component). | +| `milo-kubeconfig-secret.yaml` | Kubeconfig Secret → in-cluster `milo-apiserver` + `test-admin-token`, `insecure-skip-tls-verify`. Modeled on the operator's `overlays/test-infra/milo-kubeconfig-secret.yaml`. **Test-only** credential. | +| `server-config.yaml` | `ResourceMetricsOperator` config: `discovery.mode: milo`, `discoveryKubeconfigPath`/`projectKubeconfigPath` → `/etc/milo/kubeconfig`, and `discovery.collectRootControlPlane: true` (so the root CP is collected as cluster `root`). Also the OTLP endpoint. Rendered into the `resource-metrics-service-config` ConfigMap. | +| `deployment.yaml` | Controller Deployment (base manager + test-infra patch folded in: `KUBECONFIG` env, milo-kubeconfig mount at `/etc/milo`, `--server-config=/etc/resource-metrics/server.yaml`, `imagePullPolicy: IfNotPresent`, no `--leader-elect`). | +| `kustomization.yaml` | Ties the above together; `images:` override for the controller image; `configMapGenerator` for the server-config. | + +```sh +# Uses the dns-control kind context. +kustomize build config/dependencies/resource-metrics/controller \ + | kubectl --context kind-dns-control apply -f - +``` + +> [!NOTE] +> Override the controller image before applying if you are not using +> `ghcr.io/milo-os/resource-metrics:latest`, e.g. +> `kustomize edit set image ghcr.io/milo-os/resource-metrics=ghcr.io/milo-os/resource-metrics:` +> or load a `dev` image into kind and set `newTag: dev` in `kustomization.yaml`. + +### `core-control-plane/` — applied with a MILO kubeconfig + +These target the **Milo core control plane** (`milo-apiserver`), NOT the kind +apiserver. Apply them with a milo kubeconfig — the same in-cluster +endpoint + `test-admin-token` used by the controller, or an equivalent +admin kubeconfig. This mirrors milo-os/infra +`apps/resource-metrics-system/base/milo-control-plane.yaml`, which applies the +operator's `crd` path onto the Milo CP. + +| Path | Purpose | +| --- | --- | +| `core-control-plane/crd/` | The `ResourceMetricsPolicy` CRD (`resourcemetrics.miloapis.com`), vendored from the operator's `config/crd/bases`. Must be **Established first**. | +| `core-control-plane/policy/` | Applies the `dns-metrics` `ResourceMetricsPolicy`. It **references** `config/observability/dns-metrics-policy.yaml` (single source of truth) rather than duplicating it — so the build needs `--load-restrictor LoadRestrictionsNone`. | + +```sh +# Point kubectl at the Milo core control plane. For example, extract the +# controller's kubeconfig from the Secret, or use any admin kubeconfig for +# milo-apiserver. Example using the same Secret the controller mounts: +kubectl --context kind-dns-control -n resource-metrics-system \ + get secret milo-kubeconfig -o jsonpath='{.data.kubeconfig}' \ + | base64 -d > /tmp/milo.kubeconfig + +# 1) Install the CRD and wait for it to be Established. +kustomize build config/dependencies/resource-metrics/core-control-plane/crd \ + | kubectl --kubeconfig /tmp/milo.kubeconfig apply -f - +kubectl --kubeconfig /tmp/milo.kubeconfig wait --for=condition=Established \ + crd/resourcemetricspolicies.resourcemetrics.miloapis.com --timeout=60s + +# 2) Apply the dns-metrics policy. +kustomize build --load-restrictor LoadRestrictionsNone \ + config/dependencies/resource-metrics/core-control-plane/policy \ + | kubectl --kubeconfig /tmp/milo.kubeconfig apply -f - +``` + +## OTel endpoint assumption (confirm live) + +`controller/server-config.yaml` sets: + +``` +otel.endpoint: otel-collector-collector.telemetry-system.svc.cluster.local:4317 +``` + +Rationale: test-infra's `install-observability` task applies an +`OpenTelemetryCollector` CR named `otel-collector` in namespace +`telemetry-system` +(`.test-infra/components/observability/otel-collector/opentelemetry-collector.yaml`). +The OpenTelemetry Operator renders a Service named `-collector` +(`otel-collector-collector`), and the CR's `receivers.otlp.protocols.grpc` +listens on `:4317`. The task even waits on +`daemonset/otel-collector-collector` in `telemetry-system`, confirming the +name. + +> [!NOTE] +> This differs from milo-os/infra, whose resource-metrics ships its own +> `metrics-collector` CR and points at +> `metrics-collector-collector.resource-metrics-system...:4317`. We reuse the +> shared test-infra collector in `telemetry-system` instead of deploying a +> second collector. If the collector CR name or namespace changes, update the +> endpoint. The collector CR is a **daemonset**; the OTel Operator still +> renders the `-collector` Service used above. + +## Assumptions needing live confirmation + +- **OTel endpoint** — as above; confirm `otel-collector-collector` exists in + `telemetry-system` and serves gRPC on 4317 after `install-observability`. +- **Controller image tag** — defaults to `:latest`; pin to whatever tag is + published/loaded for the e2e run. +- **`test-admin-token`** — must match milo-apiserver's `tokens.csv` + (`milo-apiserver-auth-tokens` in `milo-system`). diff --git a/config/dependencies/resource-metrics/controller/deployment.yaml b/config/dependencies/resource-metrics/controller/deployment.yaml new file mode 100644 index 0000000..863ed23 --- /dev/null +++ b/config/dependencies/resource-metrics/controller/deployment.yaml @@ -0,0 +1,97 @@ +# resource-metrics controller Deployment for the DNS drift-detection e2e on the +# dns-control kind cluster. Vendored from milo-os/resource-metrics +# config/manager/manager.yaml with the config/overlays/test-infra +# deployment-patch folded in (this repo can't reference the operator's relative +# kustomize bases, so the merged result is inlined here). +# +# Key test-infra wiring: +# * KUBECONFIG env -> /etc/milo/kubeconfig so ctrl.GetConfigOrDie() (the local +# cluster handed to the multicluster manager) targets milo-apiserver, not the +# kind apiserver. The controller's real control plane (CRD, policies, leader +# lease) lives on milo. +# * --server-config=/etc/resource-metrics/server.yaml, backed by the +# resource-metrics-service-config ConfigMap (see kustomization.yaml). +# * milo-kubeconfig Secret mounted at /etc/milo for discovery + per-project +# clients (matches discovery{,project}KubeconfigPath in server-config.yaml). +# * --leader-elect omitted: single replica; the lease would otherwise need a +# namespace on milo that controller-runtime can't default to. +# * imagePullPolicy IfNotPresent so a kind-loaded image is used when present. +apiVersion: apps/v1 +kind: Deployment +metadata: + name: resource-metrics-controller-manager + namespace: resource-metrics-system + labels: + control-plane: controller-manager + app.kubernetes.io/name: resource-metrics + app.kubernetes.io/managed-by: kustomize +spec: + replicas: 1 + selector: + matchLabels: + control-plane: controller-manager + app.kubernetes.io/name: resource-metrics + template: + metadata: + annotations: + kubectl.kubernetes.io/default-container: manager + labels: + control-plane: controller-manager + app.kubernetes.io/name: resource-metrics + spec: + securityContext: + runAsNonRoot: true + seccompProfile: + type: RuntimeDefault + serviceAccountName: resource-metrics-controller-manager + terminationGracePeriodSeconds: 10 + containers: + - name: manager + command: + - /manager + args: + - --health-probe-bind-address=:8081 + - --server-config=/etc/resource-metrics/server.yaml + image: ghcr.io/milo-os/resource-metrics:latest + imagePullPolicy: IfNotPresent + env: + - name: KUBECONFIG + value: /etc/milo/kubeconfig + securityContext: + allowPrivilegeEscalation: false + capabilities: + drop: + - "ALL" + livenessProbe: + httpGet: + path: /healthz + port: 8081 + initialDelaySeconds: 15 + periodSeconds: 20 + readinessProbe: + httpGet: + path: /readyz + port: 8081 + initialDelaySeconds: 5 + periodSeconds: 10 + resources: + limits: + cpu: 1 + memory: 2Gi + requests: + cpu: 500m + memory: 512Mi + volumeMounts: + - name: service-config + mountPath: /etc/resource-metrics + readOnly: true + - name: milo-kubeconfig + mountPath: /etc/milo + readOnly: true + volumes: + - name: service-config + configMap: + name: resource-metrics-service-config + - name: milo-kubeconfig + secret: + secretName: milo-kubeconfig diff --git a/config/dependencies/resource-metrics/controller/kustomization.yaml b/config/dependencies/resource-metrics/controller/kustomization.yaml new file mode 100644 index 0000000..a970901 --- /dev/null +++ b/config/dependencies/resource-metrics/controller/kustomization.yaml @@ -0,0 +1,36 @@ +# Deploys the resource-metrics controller into namespace resource-metrics-system +# on the dns-control kind cluster (the local/kind kubectl context). +# +# Apply with the KIND kubeconfig/context — NOT a milo kubeconfig. The controller +# then reaches the milo core control plane via the mounted milo-kubeconfig +# Secret. The resource-metrics CRD + dns-metrics ResourceMetricsPolicy live on +# the milo core CP and are installed separately (see ../core-control-plane). +apiVersion: kustomize.config.k8s.io/v1beta1 +kind: Kustomization + +namespace: resource-metrics-system + +resources: + - namespace.yaml + - rbac.yaml + - milo-kubeconfig-secret.yaml + - deployment.yaml + +# Pin/override the controller image. The upstream default tag is `latest`; +# override newTag here (or via `kustomize edit set image`) to the tag published +# for this e2e, or load a `dev` image into kind and set newTag: dev. +images: + - name: ghcr.io/milo-os/resource-metrics + newName: resource-metrics + newTag: arm64-dev + +# server-config.yaml -> mounted at /etc/resource-metrics/server.yaml and passed +# to --server-config. disableNameSuffixHash keeps the ConfigMap name stable so +# the Deployment volume reference resolves without a kustomize name reference. +configMapGenerator: + - name: resource-metrics-service-config + behavior: create + files: + - server.yaml=server-config.yaml + options: + disableNameSuffixHash: true diff --git a/config/dependencies/resource-metrics/controller/milo-kubeconfig-secret.yaml b/config/dependencies/resource-metrics/controller/milo-kubeconfig-secret.yaml new file mode 100644 index 0000000..8ffb41e --- /dev/null +++ b/config/dependencies/resource-metrics/controller/milo-kubeconfig-secret.yaml @@ -0,0 +1,40 @@ +# Kubeconfig the resource-metrics controller uses to reach the milo core +# control plane (the DOWNSTREAM / "root" CP) and to discover the project +# (UPSTREAM) control planes served by milo-apiserver. Modeled on +# milo-os/resource-metrics config/overlays/test-infra/milo-kubeconfig-secret.yaml. +# +# resource-metrics runs ON dns-control alongside milo-apiserver, so it uses the +# in-cluster Service endpoint directly (bypassing the Envoy gateway) — the pod +# serves a self-signed cert on :6443, so TLS verification is skipped. +# +# The token is the static `test-admin-token` wired into milo-apiserver's +# token-auth-file via secret milo-apiserver-auth-tokens (key tokens.csv) in +# namespace milo-system; it grants system:masters in the test environment. +# +# TEST-ONLY: production must use a least-privilege token or mTLS and source +# this Secret from a sealed/external secret store. +apiVersion: v1 +kind: Secret +metadata: + name: milo-kubeconfig + namespace: resource-metrics-system +type: Opaque +stringData: + kubeconfig: | + apiVersion: v1 + kind: Config + clusters: + - name: milo + cluster: + server: https://milo-apiserver.milo-system.svc.cluster.local:6443 + insecure-skip-tls-verify: true + users: + - name: milo-admin + user: + token: test-admin-token + contexts: + - name: milo + context: + cluster: milo + user: milo-admin + current-context: milo diff --git a/config/dependencies/resource-metrics/controller/namespace.yaml b/config/dependencies/resource-metrics/controller/namespace.yaml new file mode 100644 index 0000000..863bf48 --- /dev/null +++ b/config/dependencies/resource-metrics/controller/namespace.yaml @@ -0,0 +1,8 @@ +apiVersion: v1 +kind: Namespace +metadata: + name: resource-metrics-system + labels: + control-plane: controller-manager + app.kubernetes.io/name: resource-metrics + app.kubernetes.io/managed-by: kustomize diff --git a/config/dependencies/resource-metrics/controller/rbac.yaml b/config/dependencies/resource-metrics/controller/rbac.yaml new file mode 100644 index 0000000..78fc04b --- /dev/null +++ b/config/dependencies/resource-metrics/controller/rbac.yaml @@ -0,0 +1,75 @@ +# Cluster-scoped RBAC for the resource-metrics controller, vendored from +# milo-os/resource-metrics config/components/controller_rbac (service_account + +# role + role_binding). These grants apply on whatever API server the +# ServiceAccount authenticates against, but note: in this deployment the +# controller talks to the milo core control plane via the mounted +# milo-kubeconfig (system:masters test token), so the effective permissions +# there come from that token, not this ServiceAccount. This RBAC is retained +# for correctness/parity with the upstream bundle and for any in-kind API +# access (health/leader machinery). +--- +apiVersion: v1 +kind: ServiceAccount +metadata: + name: resource-metrics-controller-manager + namespace: resource-metrics-system + labels: + app.kubernetes.io/name: resource-metrics + app.kubernetes.io/managed-by: kustomize +--- +apiVersion: rbac.authorization.k8s.io/v1 +kind: ClusterRole +metadata: + name: resource-metrics-manager-role + labels: + app.kubernetes.io/name: resource-metrics + app.kubernetes.io/managed-by: kustomize +rules: +- apiGroups: + - authorization.k8s.io + resources: + - selfsubjectaccessreviews + verbs: + - create +- apiGroups: + - resourcemetrics.miloapis.com + resources: + - resourcemetricspolicies + verbs: + - create + - delete + - get + - list + - patch + - update + - watch +- apiGroups: + - resourcemetrics.miloapis.com + resources: + - resourcemetricspolicies/finalizers + verbs: + - update +- apiGroups: + - resourcemetrics.miloapis.com + resources: + - resourcemetricspolicies/status + verbs: + - get + - patch + - update +--- +apiVersion: rbac.authorization.k8s.io/v1 +kind: ClusterRoleBinding +metadata: + name: resource-metrics-manager-rolebinding + labels: + app.kubernetes.io/name: resource-metrics + app.kubernetes.io/managed-by: kustomize +roleRef: + apiGroup: rbac.authorization.k8s.io + kind: ClusterRole + name: resource-metrics-manager-role +subjects: +- kind: ServiceAccount + name: resource-metrics-controller-manager + namespace: resource-metrics-system diff --git a/config/dependencies/resource-metrics/controller/server-config.yaml b/config/dependencies/resource-metrics/controller/server-config.yaml new file mode 100644 index 0000000..13d32f0 --- /dev/null +++ b/config/dependencies/resource-metrics/controller/server-config.yaml @@ -0,0 +1,36 @@ +# Operator configuration consumed by the resource-metrics manager via the +# --server-config flag. Substituted into the `resource-metrics-service-config` +# ConfigMap by configMapGenerator (keyed server.yaml) in kustomization.yaml. +# +# apiVersion/kind and field names come from the operator's internal/config +# package (ResourceMetricsOperator, DiscoveryConfig, OtelConfig). +apiVersion: apiserver.config.miloapis.com/v1alpha1 +kind: ResourceMetricsOperator +discovery: + # milo mode: discover project (upstream) control planes through the milo + # multi-cluster provider, using the mounted milo kubeconfig for both the + # discovery client and per-project client construction. Without these paths + # the controller falls back to the in-cluster ServiceAccount (the kind + # apiserver), which does not serve resourcemanager.miloapis.com — so Project + # discovery would find nothing. + mode: milo + internalServiceDiscovery: false + discoveryKubeconfigPath: /etc/milo/kubeconfig + projectKubeconfigPath: /etc/milo/kubeconfig + # CRITICAL for drift detection: also collect the milo core/root control plane + # (the DOWNSTREAM, where the replicator writes the replicated copies) as + # cluster name "root". The upstream project CPs supply + # dns_recordset_upstream_info; the root CP supplies + # dns_recordset_downstream_info. Recording rules diff the two. + collectRootControlPlane: true +otel: + # OTLP gRPC endpoint of the OTel collector deployed by test-infra's + # `install-observability` task. That task applies an OpenTelemetryCollector + # CR named `otel-collector` in namespace `telemetry-system`; the OTel + # Operator renders a Service `-collector` (otel-collector-collector) + # and the CR's receivers.otlp.protocols.grpc listens on :4317. + # See README.md — confirm live if the collector name/namespace changes. + endpoint: otel-collector-collector.telemetry-system.svc.cluster.local:4317 + insecure: true + collectionInterval: 5s + defaultMetricPrefix: "" diff --git a/config/dependencies/resource-metrics/core-control-plane/crd/kustomization.yaml b/config/dependencies/resource-metrics/core-control-plane/crd/kustomization.yaml new file mode 100644 index 0000000..6b1821c --- /dev/null +++ b/config/dependencies/resource-metrics/core-control-plane/crd/kustomization.yaml @@ -0,0 +1,15 @@ +# Installs the resource-metrics CRD (ResourceMetricsPolicy, +# resourcemetrics.miloapis.com) INTO the Milo core control plane. +# +# TARGET: milo-apiserver (the core/downstream CP) — NOT the kind apiserver. +# Apply with a milo kubeconfig (see ../../README.md). Mirrors the intent of +# milo-os/infra apps/resource-metrics-system/base/milo-control-plane.yaml +# (which applies the operator's `crd` path onto the Milo CP via Flux). +# +# Must be Established before the dns-metrics ResourceMetricsPolicy in ../policy +# is applied. +apiVersion: kustomize.config.k8s.io/v1beta1 +kind: Kustomization + +resources: + - resourcemetricspolicies-crd.yaml diff --git a/config/dependencies/resource-metrics/core-control-plane/crd/resourcemetricspolicies-crd.yaml b/config/dependencies/resource-metrics/core-control-plane/crd/resourcemetricspolicies-crd.yaml new file mode 100644 index 0000000..86e1d5d --- /dev/null +++ b/config/dependencies/resource-metrics/core-control-plane/crd/resourcemetricspolicies-crd.yaml @@ -0,0 +1,297 @@ +--- +apiVersion: apiextensions.k8s.io/v1 +kind: CustomResourceDefinition +metadata: + annotations: + controller-gen.kubebuilder.io/version: v0.17.2 + discovery.miloapis.com/parent-contexts: Platform + name: resourcemetricspolicies.resourcemetrics.miloapis.com +spec: + group: resourcemetrics.miloapis.com + names: + kind: ResourceMetricsPolicy + listKind: ResourceMetricsPolicyList + plural: resourcemetricspolicies + singular: resourcemetricspolicy + scope: Cluster + versions: + - additionalPrinterColumns: + - jsonPath: .metadata.creationTimestamp + name: Age + type: date + name: v1alpha1 + schema: + openAPIV3Schema: + description: ResourceMetricsPolicy is the Schema for the resourcemetricspolicies + API. + properties: + apiVersion: + description: |- + APIVersion defines the versioned schema of this representation of an object. + Servers should convert recognized schemas to the latest internal value, and + may reject unrecognized values. + More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources + type: string + kind: + description: |- + Kind is a string value representing the REST resource this object represents. + Servers may infer this from the endpoint the client submits requests to. + Cannot be updated. + In CamelCase. + More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds + type: string + metadata: + type: object + spec: + description: ResourceMetricsPolicySpec defines the desired state of ResourceMetricsPolicy. + properties: + generators: + description: Generators defines the set of resource metric generators. + items: + description: GeneratorSpec defines a single resource metric generator. + properties: + families: + description: Families defines the metric families to emit for + each resource instance. + items: + description: MetricFamilySpec defines a Prometheus metric + family emitted per resource instance. + properties: + help: + description: Help is the help string for this metric family. + type: string + metrics: + description: |- + Metrics defines how to produce individual metric series from each resource. + Metrics in a family have no stable identity, so this list is treated as + atomic for server-side apply. + items: + description: MetricSpec defines a single metric series + within a family. + properties: + forEach: + description: |- + ForEach, when set, is a CEL expression that must evaluate to a list. + The metric is emitted once per element of that list. Within Value and + each label's Value expression, the variable "item" is bound to the + current list element (type dyn). When ForEach is absent, the metric + is emitted once per object (existing behaviour). + maxLength: 4096 + type: string + labels: + description: Labels defines the labels to attach + to this metric series. + items: + description: LabelSpec defines a single label + on a metric series. + properties: + name: + description: Name is the label name. Must + match the Prometheus label name syntax. + maxLength: 253 + pattern: ^[a-zA-Z_][a-zA-Z0-9_]*$ + type: string + value: + description: Value is a CEL expression evaluated + against the resource object. + type: string + required: + - name + - value + type: object + type: array + value: + description: |- + Value is a CEL expression evaluated against the resource object + that produces the metric value. Defaults to 1 if omitted. + type: string + type: object + minItems: 1 + type: array + x-kubernetes-list-type: atomic + name: + description: Name is the base metric name (e.g. "workload_info"). + maxLength: 253 + pattern: ^[a-zA-Z_:][a-zA-Z0-9_:]*$ + type: string + type: + default: gauge + description: Type is the Prometheus metric type. Only + "gauge" is supported in v1. + enum: + - gauge + type: string + required: + - name + type: object + type: array + x-kubernetes-list-map-keys: + - name + x-kubernetes-list-type: map + name: + description: Name is a unique name for this generator. + maxLength: 253 + pattern: ^[a-z][a-z0-9-]*$ + type: string + resource: + description: Resource identifies the Kubernetes API resource + to monitor. + properties: + group: + description: |- + Group is the API group of the resource (e.g. "compute.miloapis.com"). + Empty string targets core resources (configmaps, pods, namespaces, …). + type: string + resource: + description: Resource is the plural resource name (e.g. + "workloads"). + minLength: 1 + type: string + version: + description: Version is the API version (e.g. "v1alpha1"). + minLength: 1 + type: string + required: + - group + - resource + - version + type: object + required: + - name + - resource + type: object + minItems: 1 + type: array + x-kubernetes-list-map-keys: + - name + x-kubernetes-list-type: map + metricNamePrefix: + description: |- + MetricNamePrefix, when set, overrides the controller's + --default-metric-prefix flag for metrics emitted by this policy. + Must start with a letter, underscore, or colon and contain only + letters, digits, underscores, and colons. + maxLength: 32 + pattern: ^[a-zA-Z_:][a-zA-Z0-9_:]*$ + type: string + required: + - generators + type: object + status: + description: ResourceMetricsPolicyStatus defines the observed state of + ResourceMetricsPolicy. + properties: + activeGenerators: + description: |- + ActiveGenerators is the number of generators currently compiled and + actively emitting metrics for this policy. + format: int32 + type: integer + compilationFailures: + description: |- + CompilationFailures is the number of generators that failed to compile + (typically due to invalid CEL). + format: int32 + type: integer + conditions: + description: Conditions represent the latest observations of the resource's + state. + items: + description: Condition contains details for one aspect of the current + state of this API Resource. + properties: + lastTransitionTime: + description: |- + lastTransitionTime is the last time the condition transitioned from one status to another. + This should be when the underlying condition changed. If that is not known, then using the time when the API field changed is acceptable. + format: date-time + type: string + message: + description: |- + message is a human readable message indicating details about the transition. + This may be an empty string. + maxLength: 32768 + type: string + observedGeneration: + description: |- + observedGeneration represents the .metadata.generation that the condition was set based upon. + For instance, if .metadata.generation is currently 12, but the .status.conditions[x].observedGeneration is 9, the condition is out of date + with respect to the current state of the instance. + format: int64 + minimum: 0 + type: integer + reason: + description: |- + reason contains a programmatic identifier indicating the reason for the condition's last transition. + Producers of specific condition types may define expected values and meanings for this field, + and whether the values are considered a guaranteed API. + The value should be a CamelCase string. + This field may not be empty. + maxLength: 1024 + minLength: 1 + pattern: ^[A-Za-z]([A-Za-z0-9_,:]*[A-Za-z0-9_])?$ + type: string + status: + description: status of the condition, one of True, False, Unknown. + enum: + - "True" + - "False" + - Unknown + type: string + type: + description: type of condition in CamelCase or in foo.example.com/CamelCase. + maxLength: 316 + pattern: ^([a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*/)?(([A-Za-z0-9][-A-Za-z0-9_.]*)?[A-Za-z0-9])$ + type: string + required: + - lastTransitionTime + - message + - reason + - status + - type + type: object + type: array + x-kubernetes-list-map-keys: + - type + x-kubernetes-list-type: map + missingPermissions: + description: |- + MissingPermissions lists the GVRs for which the controller lacks the + RBAC permissions required to watch or list on at least one engaged + project control plane. + items: + description: GVRRef identifies a Kubernetes API resource by group, + version, and plural name. + properties: + group: + description: |- + Group is the API group of the resource. + Empty string targets core resources (configmaps, pods, namespaces, …). + type: string + resource: + description: Resource is the plural resource name. + minLength: 1 + type: string + version: + description: Version is the API version. + minLength: 1 + type: string + required: + - group + - resource + - version + type: object + type: array + x-kubernetes-list-type: atomic + observedGeneration: + description: |- + ObservedGeneration reflects the .metadata.generation the controller + has most recently acted upon. + format: int64 + type: integer + type: object + type: object + served: true + storage: true + subresources: + status: {} diff --git a/config/dependencies/resource-metrics/core-control-plane/policy/kustomization.yaml b/config/dependencies/resource-metrics/core-control-plane/policy/kustomization.yaml new file mode 100644 index 0000000..4fc681b --- /dev/null +++ b/config/dependencies/resource-metrics/core-control-plane/policy/kustomization.yaml @@ -0,0 +1,18 @@ +# Applies the dns-metrics ResourceMetricsPolicy onto the Milo core control +# plane. The policy itself is authored (and owned) elsewhere at +# config/observability/dns-metrics-policy.yaml — it is referenced here, not +# duplicated, so there is a single source of truth. +# +# TARGET: milo-apiserver (the core/downstream CP) — NOT the kind apiserver. +# Apply with a milo kubeconfig (see ../../README.md), AFTER the CRD in ../crd +# is Established. resource-metrics watches this policy on the core CP and, with +# discovery.collectRootControlPlane=true, emits the upstream/downstream series +# the drift recording rules diff. +# +# Referencing a path outside this kustomization root requires +# `--load-restrictor LoadRestrictionsNone` at build time. +apiVersion: kustomize.config.k8s.io/v1beta1 +kind: Kustomization + +resources: + - ../../../../observability/dns-metrics-policy.yaml diff --git a/config/observability/dns-drift-rules.yaml b/config/observability/dns-drift-rules.yaml new file mode 100644 index 0000000..49d8155 --- /dev/null +++ b/config/observability/dns-drift-rules.yaml @@ -0,0 +1,127 @@ +# Recording + alerting rules that detect drift between the upstream (project) +# and downstream (root) control planes, using the series emitted by the +# dns-metrics ResourceMetricsPolicy (see dns-metrics-policy.yaml). +# +# Join key: (upstream_cluster, upstream_namespace, upstream_name). +# +# A single dns-metrics policy is applied to every control plane, so BOTH the +# upstream and downstream generators emit on every CP. resource-metrics tags +# each series with the source CP type via milo.control_plane.type (promoted to +# the Prometheus label milo_control_plane_type): "root" on the core (downstream) +# CP, "project" on project (upstream) CPs. We partition on that label so the +# downstream generator's (empty-label) series on project CPs and the upstream +# generator's series on the core CP don't pollute the diff. +# +# Join normalization: the downstream series carries upstream_cluster="cluster- +# " (the replicator prefixes it; is the Milo cluster key, which for +# a cluster-scoped Project is the bare project name). The upstream series is +# tagged with milo_project_name = the project name (resource-metrics strips the +# leading "/" the Milo provider uses). We add the "cluster-" prefix to the +# upstream side so both align. Label names/values confirmed against +# resource-metrics v-current (milo.project.name / milo.control_plane.type, +# root value "root"); re-verify if resource-metrics changes its attribute keys. +# +# See docs/enhancements/controlplane-drift-detection.md for the full design. +# Standard Prometheus-operator PrometheusRule. datum-cloud/infra runs the +# victoria-metrics-operator, which converts PrometheusRule -> VMRule and carries +# labels through, so no VM-specific CRD is needed. The +# telemetry.miloapis.com/resource-metrics-aggregator label routes these rules to +# the dedicated `vmalert-datum-resource-metrics-aggregator` (which has the +# resource-metrics datasource); the general vmalert selects the complement (that +# label DoesNotExist). See infra .../victoria-metrics/base/vmalert.yaml. +apiVersion: monitoring.coreos.com/v1 +kind: PrometheusRule +metadata: + name: dns-controlplane-drift + labels: + managed-by: flux + app.kubernetes.io/part-of: dns-operator + # Route to the resource-metrics aggregator vmalert (label presence is the + # selector; value is not significant). + telemetry.miloapis.com/resource-metrics-aggregator: "true" +spec: + groups: + - name: dns.drift.recording + interval: 30s + rules: + # Both sides are normalized to a bare `proj` (project name) join key. + # Upstream: milo_project_name is already the bare name ("alpha"). + # Downstream: the replicator stamps upstream-cluster-name as + # "cluster-" + replace(, "/", "_"), and the milo + # provider keys a project cluster as "/alpha", so the annotation is + # "cluster-_alpha". Strip the "cluster-_?" prefix to recover "alpha". + # (Verified live: downstream upstream_cluster="cluster-_alpha" vs + # upstream milo_project_name="alpha".) + - record: dns:recordset_upstream:normalized + expr: | + label_replace( + dns_recordset_upstream_info{milo_control_plane_type="project"}, + "proj", "$1", "milo_project_name", "(.*)" + ) + + # Downstream view: core CP only (where replicated objects live), with + # the cluster annotation normalized to the bare project name. + - record: dns:recordset_downstream:scoped + expr: | + label_replace( + dns_recordset_downstream_info{milo_control_plane_type="root"}, + "proj", "$1", "upstream_cluster", "cluster-_?(.*)" + ) + + # ORPHANS: exists downstream, no surviving upstream owner. + # This is the failure mode from engineering#346 — a replicated record + # left "reserving" a name after its upstream owner was deleted. + - record: dns:recordset_downstream_orphan + expr: | + dns:recordset_downstream:scoped + unless on(proj, upstream_namespace, upstream_name) + dns:recordset_upstream:normalized + + # MISSING: exists upstream, never replicated / replication stalled. + - record: dns:recordset_downstream_missing + expr: | + dns:recordset_upstream:normalized + unless on(proj, upstream_namespace, upstream_name) + dns:recordset_downstream:scoped + + - name: dns.drift.alerts + rules: + - alert: DNSDownstreamOrphanRecordSet + # for: rides out normal replication + OTLP push/staleness lag so we + # only page on drift that actually persists. + expr: dns:recordset_downstream_orphan > 0 + for: 10m + labels: + severity: warning + annotations: + summary: "Orphaned downstream DNSRecordSet (no upstream owner)" + description: >- + DNSRecordSet {{ $labels.upstream_name }} in namespace + {{ $labels.upstream_namespace }} (cluster {{ $labels.upstream_cluster }}) + exists on the downstream control plane with no matching upstream + owner. This is the failure mode from engineering#346 and can block + new records for the same name. Runbook: docs/enhancements/controlplane-drift-detection.md + + - alert: DNSDownstreamMissingRecordSet + expr: dns:recordset_downstream_missing > 0 + for: 10m + labels: + severity: warning + annotations: + summary: "Upstream DNSRecordSet not replicated downstream" + description: >- + DNSRecordSet {{ $labels.upstream_name }} in namespace + {{ $labels.upstream_namespace }} (cluster {{ $labels.upstream_cluster }}) + exists upstream but has no downstream copy — replication may be + stalled or a replica may not own this cluster. + + - alert: DNSRecordSetNotAccepted + expr: dns_recordset_upstream_info{accepted="false", milo_control_plane_type="project"} == 1 + for: 15m + labels: + severity: warning + annotations: + summary: "DNSRecordSet stuck not-Accepted for 15m" + description: >- + DNSRecordSet {{ $labels.upstream_name }} in namespace + {{ $labels.upstream_namespace }} has not reached Accepted=True. diff --git a/config/observability/dns-metrics-policy.yaml b/config/observability/dns-metrics-policy.yaml new file mode 100644 index 0000000..e812bdc --- /dev/null +++ b/config/observability/dns-metrics-policy.yaml @@ -0,0 +1,108 @@ +# ResourceMetricsPolicy consumed by the milo-os/resource-metrics controller. +# +# resource-metrics watches every project (upstream) control plane via the Milo +# multi-cluster provider, and — with discovery.collectRootControlPlane=true — +# the root/core (downstream) control plane as well. It emits one gauge series +# per matching object on whichever control plane the object lives on, pushing +# over OTLP to the platform OTel collector -> Victoria Metrics. +# +# The two generators below emit structurally different series on the two sides +# of the replication boundary so that recording rules can diff them: +# +# dns_recordset_upstream_info <- source of truth, on project control planes +# dns_recordset_downstream_info <- replicated copy, on the root control plane +# +# The downstream series lifts the replicator's `meta.datumapis.com/upstream-*` +# annotations onto labels so the two sides share a join key +# (upstream_cluster, upstream_namespace, upstream_name). +# +# See docs/enhancements/controlplane-drift-detection.md for the full design. +apiVersion: resourcemetrics.miloapis.com/v1alpha1 +kind: ResourceMetricsPolicy +metadata: + name: dns-metrics +spec: + generators: + # ---- Upstream (project control planes): desired state ----------------- + - name: dnsrecordset-upstream-info + resource: + group: dns.networking.miloapis.com + version: v1alpha1 + resource: dnsrecordsets + families: + - name: dns_recordset_upstream_info + help: "One series per DNSRecordSet on a source (upstream) control plane." + type: gauge + metrics: + - value: "1.0" + labels: + - name: upstream_namespace + value: "object.metadata.namespace" + - name: upstream_name + value: "object.metadata.name" + - name: accepted + # Null-safe: a record with no status yet (no reconciler) has no + # status.conditions; guarding with has() avoids a CEL eval error + # that would make resource-metrics silently drop the whole series. + value: "has(object.status) && has(object.status.conditions) && object.status.conditions.exists(c, c.type == 'Accepted' && c.status == 'True') ? 'true' : 'false'" + + - name: dnszone-upstream-info + resource: + group: dns.networking.miloapis.com + version: v1alpha1 + resource: dnszones + families: + - name: dns_zone_upstream_info + help: "One series per DNSZone on a source (upstream) control plane." + type: gauge + metrics: + - value: "1.0" + labels: + - name: upstream_namespace + value: "object.metadata.namespace" + - name: upstream_name + value: "object.metadata.name" + + # ---- Downstream (root control plane): replicated / actual state ------- + # Join labels come from the annotations the replicator stamps in + # internal/downstreamclient/mappednamespace.go. NOTE: the namespace is + # remapped downstream, so metadata.namespace is NOT the upstream namespace; + # the real one is in the upstream-namespace annotation. The cluster-name + # annotation is prefixed "cluster-" — the recording rules normalize for it. + - name: dnsrecordset-downstream-info + resource: + group: dns.networking.miloapis.com + version: v1alpha1 + resource: dnsrecordsets + families: + - name: dns_recordset_downstream_info + help: "One series per replicated DNSRecordSet on the downstream (root) control plane." + type: gauge + metrics: + - value: "1.0" + labels: + - name: upstream_cluster + value: "object.metadata.annotations['meta.datumapis.com/upstream-cluster-name']" + - name: upstream_namespace + value: "object.metadata.annotations['meta.datumapis.com/upstream-namespace']" + - name: upstream_name + value: "object.metadata.annotations['meta.datumapis.com/upstream-name']" + + - name: dnszone-downstream-info + resource: + group: dns.networking.miloapis.com + version: v1alpha1 + resource: dnszones + families: + - name: dns_zone_downstream_info + help: "One series per replicated DNSZone on the downstream (root) control plane." + type: gauge + metrics: + - value: "1.0" + labels: + - name: upstream_cluster + value: "object.metadata.annotations['meta.datumapis.com/upstream-cluster-name']" + - name: upstream_namespace + value: "object.metadata.annotations['meta.datumapis.com/upstream-namespace']" + - name: upstream_name + value: "object.metadata.annotations['meta.datumapis.com/upstream-name']" diff --git a/config/observability/kustomization.yaml b/config/observability/kustomization.yaml new file mode 100644 index 0000000..23eb333 --- /dev/null +++ b/config/observability/kustomization.yaml @@ -0,0 +1,21 @@ +# Observability configuration for detecting upstream/downstream control-plane +# drift in the DNS service. +# +# dns-metrics-policy.yaml ResourceMetricsPolicy (milo-os/resource-metrics) +# dns-drift-rules.yaml recording + alerting rules (PrometheusRule) +# +# These are platform/infra objects. Per datum-cloud/infra conventions: +# - The ResourceMetricsPolicy is applied to the Milo core control plane (where +# the resource-metrics CRDs are installed via milo-configuration-kubeconfig), +# NOT a plain cluster. resource-metrics must run with +# discovery.collectRootControlPlane: true (NOT yet enabled in infra) for the +# downstream/core-CP series this depends on. +# - The PrometheusRule (auto-converted to VMRule by victoria-metrics-operator) +# is picked up by the resource-metrics aggregator vmalert via its label. +# Kept in this repo co-located with the operator they observe; deployment wiring +# lives in the infra repo (or the e2e overlay). +apiVersion: kustomize.config.k8s.io/v1beta1 +kind: Kustomization +resources: + - dns-metrics-policy.yaml + - dns-drift-rules.yaml diff --git a/config/overlays/agent-powerdns-milo/README.md b/config/overlays/agent-powerdns-milo/README.md new file mode 100644 index 0000000..611e8b7 --- /dev/null +++ b/config/overlays/agent-powerdns-milo/README.md @@ -0,0 +1,39 @@ +# agent-powerdns-milo + +PowerDNS agent overlay that points the DNS agent at the **Milo core control plane** (`milo-apiserver`, root scope) as its DNS read (and status-write) source, for the DNS drift-detection e2e. + +In this environment the DNS CRDs and `DNSRecordSet`s live on the Milo core control plane, **not** on the local kind API. The PowerDNS agent (`--role=downstream`) must read `DNSRecordSet`/`DNSZone` objects from the Milo core CP and program PowerDNS from them. + +## What changed vs. `agent-powerdns-federated` + +This overlay bases on `../agent-powerdns-federated` and leaves the PowerDNS, Lightningstream, and RustFS/S3 (`s3-credentials`) wiring completely intact. It changes only the agent's API target: + +1. **Replaces the agent server-config** (`configMapGenerator` `behavior: replace` on `agent-server-config`) with a milo-targeted `server-config.yaml`. +2. **Mounts a `milo-kubeconfig` Secret at `/milo`** on the `manager` container of the `pdns-auth` StatefulSet and sets `KUBECONFIG=/milo/kubeconfig` (`deployment-patch.yaml`). +3. **Ships the `milo-kubeconfig` Secret** (`milo-kubeconfig-secret.yaml`) pointing at the in-cluster `milo-apiserver` endpoint with the static `test-admin-token` and `insecure-skip-tls-verify`. + +`disableNameSuffixHash: true` is kept so the StatefulSet's existing `agent-server-config` volume reference stays valid after the replace. + +## Which field retargets the read source to the core CP + +> [!IMPORTANT] +> For `--role=downstream`, it is **not** a server-config field — it is the `KUBECONFIG` env var. + +`cmd/main.go`'s `case "downstream":` branch builds its manager and all three controllers (`DNSZone`, `DNSRecordSet`, `DNSRecordSetPowerDNS`) from `ctrl.GetConfigOrDie()` and `mgr.GetClient()`. It **never** consults `discovery.*` or `downstreamResourceManagement.kubeconfigPath` — those fields are read only by the `case "replicator":` branch (`serverConfig.DownstreamResourceManagement.RestConfig()` / `initializeClusterDiscovery`). So `discovery.mode: single` plus `downstreamResourceManagement.kubeconfigPath` cannot retarget a downstream agent's read source on their own. + +The retarget is done by `KUBECONFIG=/milo/kubeconfig` in `deployment-patch.yaml`. controller-runtime's `ctrl.GetConfig()` honors `KUBECONFIG` before falling back to the in-cluster config, and no `--kubeconfig` flag is registered on the binary — so both the reads (DNSRecordSet/DNSZone informers) and the writes (status updates via `mgr.GetClient()`) resolve to the mounted Milo core-CP kubeconfig instead of the in-cluster kind API. + +The milo-targeted values in `server-config.yaml` (`discovery` paths + `downstreamResourceManagement.kubeconfigPath` all set to `/milo/kubeconfig`) are kept for consistency and remain correct if this agent is ever run as `--role=replicator`; they are **inert** under `--role=downstream`. + +## Assumptions needing live confirmation + +- The Milo core CP is reachable at `https://milo-apiserver.milo-system.svc.cluster.local:6443` from the `dns-agent-system`/`dns-control` cluster and serves `dns.networking.miloapis.com/v1alpha1` `DNSRecordSet`/`DNSZone`/`DNSZoneClass` at its **root** endpoint (no aggregation path). Validated live per the task grounding; re-confirm if the endpoint or scope changes. +- `test-admin-token` (system:masters) is present in secret `milo-apiserver-auth-tokens` (`tokens.csv`) in ns `milo-system` and accepted by milo-apiserver's token-auth-file. +- The agent namespace is `dns-agent-system` (inherited from the federated overlay); the `milo-kubeconfig` Secret is created there. +- `KUBECONFIG` retargeting assumes the downstream binary registers no `--kubeconfig` flag (confirmed in `cmd/main.go`). If the code later adds one or stops using `ctrl.GetConfigOrDie()` in the downstream branch, this approach must be revisited (the minimal code alternative would be to have the downstream branch build its cluster from `serverConfig.DownstreamResourceManagement.RestConfig()`). + +## Validate + +``` +kustomize build --load-restrictor LoadRestrictionsNone config/overlays/agent-powerdns-milo +``` diff --git a/config/overlays/agent-powerdns-milo/deployment-patch.yaml b/config/overlays/agent-powerdns-milo/deployment-patch.yaml new file mode 100644 index 0000000..db4e060 --- /dev/null +++ b/config/overlays/agent-powerdns-milo/deployment-patch.yaml @@ -0,0 +1,29 @@ +# Retarget the PowerDNS agent's read/write source at the Milo core control plane. +# +# For --role=downstream (cmd/main.go), the manager and all controllers are built +# from ctrl.GetConfigOrDie() + mgr.GetClient(); the server-config discovery/ +# downstream kubeconfig fields are NOT consulted in that branch. controller- +# runtime's ctrl.GetConfig() honors the KUBECONFIG env var before falling back to +# in-cluster config (no --kubeconfig flag is registered), so pointing KUBECONFIG +# at the mounted milo kubeconfig makes both the DNSRecordSet/DNSZone reads and the +# status writes resolve to the Milo core CP instead of the local kind API. +apiVersion: apps/v1 +kind: StatefulSet +metadata: + name: pdns-auth +spec: + template: + spec: + containers: + - name: manager + env: + - name: KUBECONFIG + value: /milo/kubeconfig + volumeMounts: + - name: milo-kubeconfig + mountPath: /milo + readOnly: true + volumes: + - name: milo-kubeconfig + secret: + secretName: milo-kubeconfig diff --git a/config/overlays/agent-powerdns-milo/kustomization.yaml b/config/overlays/agent-powerdns-milo/kustomization.yaml new file mode 100644 index 0000000..4a30649 --- /dev/null +++ b/config/overlays/agent-powerdns-milo/kustomization.yaml @@ -0,0 +1,42 @@ +# PowerDNS agent overlay that points the agent at the Milo core control plane as +# its DNS read (and status-write) source, for the DNS drift-detection e2e. +# +# Bases on ../agent-powerdns-federated (PR #60: PowerDNS agent + Lightningstream +# + RustFS/S3 sync, deployed on the control cluster) and changes ONLY the agent's +# API target — the PowerDNS, Lightningstream, and s3-credentials wiring are left +# intact and inherited unchanged. +# +# What this overlay adds vs. agent-powerdns-federated: +# 1. Replaces the agent server-config (configMapGenerator behavior:replace) with +# a milo-targeted one. +# 2. Mounts a milo-kubeconfig Secret at /milo on the manager container and sets +# KUBECONFIG=/milo/kubeconfig (deployment-patch.yaml) — this is what actually +# retargets the downstream agent's read/write to the Milo core CP (see the +# note in server-config.yaml / deployment-patch.yaml). +# 3. Ships the milo-kubeconfig Secret (in-cluster milo-apiserver + test-admin-token). +apiVersion: kustomize.config.k8s.io/v1beta1 +kind: Kustomization + +namespace: dns-agent-system + +resources: + - ../agent-powerdns-federated + - milo-kubeconfig-secret.yaml + +# Keep the deterministic name (base disables the name-suffix hash so the +# StatefulSet's `agent-server-config` volume ref stays valid). +generatorOptions: + disableNameSuffixHash: true + +# Replace the base agent server-config with the milo-targeted one. +configMapGenerator: + - name: agent-server-config + behavior: replace + files: + - server-config.yaml + +patches: + - path: deployment-patch.yaml + target: + kind: StatefulSet + name: pdns-auth diff --git a/config/overlays/agent-powerdns-milo/milo-kubeconfig-secret.yaml b/config/overlays/agent-powerdns-milo/milo-kubeconfig-secret.yaml new file mode 100644 index 0000000..404c023 --- /dev/null +++ b/config/overlays/agent-powerdns-milo/milo-kubeconfig-secret.yaml @@ -0,0 +1,42 @@ +# Kubeconfig the PowerDNS agent uses to talk to the Milo *core* control plane +# (milo-apiserver root scope), where the DNS CRDs and DNSRecordSets live in this +# drift-detection e2e. The base agent config defaults to the in-cluster +# ServiceAccount kubeconfig (the local kind apiserver), which does not host the +# DNSRecordSets the agent must program into PowerDNS — so without this Secret the +# agent reads an empty/wrong API. +# +# - server: in-cluster Service for milo-apiserver. The pod listens on port 6443 +# with a self-signed cert, so we skip TLS verification. This targets the +# milo-apiserver ROOT endpoint directly (no aggregation path), which serves +# dns.networking.miloapis.com/v1alpha1 DNSRecordSet/DNSZone/DNSZoneClass. +# - token: the static `test-admin-token` wired into milo's test-infra overlay +# (token-auth-file, secret milo-apiserver-auth-tokens key tokens.csv, ns +# milo-system). It grants system:masters in the test environment. +# +# Test-overlay-only construct; production deployments should use a least-privilege +# token (or mTLS) sourced from a sealed-secret / external secret store. +apiVersion: v1 +kind: Secret +metadata: + name: milo-kubeconfig + namespace: dns-agent-system +type: Opaque +stringData: + kubeconfig: | + apiVersion: v1 + kind: Config + clusters: + - name: milo + cluster: + server: https://milo-apiserver.milo-system.svc.cluster.local:6443 + insecure-skip-tls-verify: true + users: + - name: milo-admin + user: + token: test-admin-token + contexts: + - name: milo + context: + cluster: milo + user: milo-admin + current-context: milo diff --git a/config/overlays/agent-powerdns-milo/server-config.yaml b/config/overlays/agent-powerdns-milo/server-config.yaml new file mode 100644 index 0000000..1243ff9 --- /dev/null +++ b/config/overlays/agent-powerdns-milo/server-config.yaml @@ -0,0 +1,46 @@ +apiVersion: dns.networking.miloapis.com/v1alpha1 +kind: DNSOperator +# Milo-targeted PowerDNS agent config for the DNS drift-detection e2e. +# +# In this environment the DNS CRDs (DNSZone/DNSRecordSet/DNSZoneClass in +# dns.networking.miloapis.com/v1alpha1) live on the Milo *core* control plane +# (milo-apiserver root scope), NOT on the local kind API. The PowerDNS agent +# runs --role=downstream, so it must READ DNSRecordSets/DNSZones from the Milo +# core CP and program PowerDNS. +# +# IMPORTANT — how the read/write source is actually retargeted for role=downstream: +# cmd/main.go's `case "downstream":` branch builds its manager and *all* three +# controllers (DNSZone, DNSRecordSet, DNSRecordSetPowerDNS) from +# ctrl.GetConfigOrDie() and mgr.GetClient(). It does NOT consult +# discovery.* or downstreamResourceManagement.kubeconfigPath at all — those +# fields are only read by the `case "replicator":` branch. So for a downstream +# agent, mode:single + downstreamResourceManagement.kubeconfigPath CANNOT +# retarget the read source on their own. +# +# The actual retarget is done in kustomization/deployment-patch.yaml by setting +# the KUBECONFIG env var to /milo/kubeconfig on the manager container: +# ctrl.GetConfig() honors KUBECONFIG (no --kubeconfig flag is registered), so +# both the read (DNSRecordSet/DNSZone informers) and the write (status updates) +# resolve to the mounted Milo core-CP kubeconfig instead of the in-cluster kind +# API. +# +# The fields below are kept milo-targeted for consistency and to remain correct +# if this agent is ever run as --role=replicator; they are inert for +# --role=downstream. +discovery: + mode: single + internalServiceDiscovery: false + # Inert under --role=downstream (single mode uses ctrl.GetConfigOrDie()); set + # to the mounted milo kubeconfig so it is correct if run as replicator. + discoveryKubeconfigPath: "/milo/kubeconfig" + projectKubeconfigPath: "/milo/kubeconfig" +downstreamResourceManagement: + # Inert under --role=downstream (see note above); the KUBECONFIG env var in the + # deployment patch is what points the downstream client at the Milo core CP. + kubeconfigPath: "/milo/kubeconfig" + +controllers: + dnsRecordSetPowerDNS: + # maxConcurrentReconciles: 4 + # rateLimiterBaseDelay: 1s + # rateLimiterMaxDelay: 30s diff --git a/config/overlays/replicator-milo/kustomization.yaml b/config/overlays/replicator-milo/kustomization.yaml new file mode 100644 index 0000000..272bd5d --- /dev/null +++ b/config/overlays/replicator-milo/kustomization.yaml @@ -0,0 +1,48 @@ +# Replicator overlay for the production-accurate Milo drift-detection e2e. +# Mirrors config/overlays/replicator (namespace + webhook-cert patches) but runs +# discovery.mode=milo with 2 active replicas and a milo-apiserver kubeconfig +# instead of the single-mode downstream-kubeconfig Secret. +apiVersion: kustomize.config.k8s.io/v1beta1 +kind: Kustomization + +namespace: dns-replicator-system + +resources: + - ../../default + +# Replace the base single-mode server-config with the milo-mode one. +configMapGenerator: + - name: server-config + behavior: replace + files: + - server-config.yaml + +patches: + - path: patch-milo.yaml + target: + kind: Deployment + name: controller-manager + - path: ../replicator/patch-namespace.yaml + target: + kind: ServiceAccount + name: controller-manager + - path: ../replicator/patch-namespace.yaml + target: + kind: Service + name: controller-manager-metrics-service + - path: ../replicator/patch-namespace.yaml + target: + kind: ClusterRoleBinding + name: manager-rolebinding + - path: ../replicator/patch-namespace.yaml + target: + kind: ClusterRoleBinding + name: dns-operator-metrics-auth-rolebinding + - path: ../replicator/patch-namespace.yaml + target: + kind: RoleBinding + name: leader-election-rolebinding + - path: ../replicator/patch-mwc-ca-injection.yaml + target: + kind: MutatingWebhookConfiguration + name: mutating-webhook-configuration diff --git a/config/overlays/replicator-milo/patch-milo.yaml b/config/overlays/replicator-milo/patch-milo.yaml new file mode 100644 index 0000000..8986f17 --- /dev/null +++ b/config/overlays/replicator-milo/patch-milo.yaml @@ -0,0 +1,26 @@ +# Milo-mode replicator: 2 active replicas (leader election OFF so both replicas +# reconcile — this is what exercises the multicluster per-cluster ownership that +# regressed in engineering#346), plus the milo-apiserver kubeconfig mount. +apiVersion: apps/v1 +kind: Deployment +metadata: + name: controller-manager +spec: + replicas: 2 + template: + spec: + containers: + - name: manager + # Replaces the base args wholesale — note NO --leader-elect. + args: + - --role=replicator + - --health-probe-bind-address=:8081 + - --server-config=/config/server-config.yaml + volumeMounts: + - name: milo-kubeconfig + mountPath: /milo + readOnly: true + volumes: + - name: milo-kubeconfig + secret: + secretName: milo-kubeconfig diff --git a/config/overlays/replicator-milo/server-config.yaml b/config/overlays/replicator-milo/server-config.yaml new file mode 100644 index 0000000..80e6637 --- /dev/null +++ b/config/overlays/replicator-milo/server-config.yaml @@ -0,0 +1,26 @@ +apiVersion: dns.networking.miloapis.com/v1alpha1 +kind: DNSOperator +# Production-accurate Milo topology for the drift-detection e2e (Phase B). +# The replicator discovers project (upstream) control planes through +# milo-apiserver and writes replicated resources to the Milo core (downstream) +# control plane. All three paths target the same milo-apiserver kubeconfig: +# - discovery: lists ProjectControlPlane objects +# - project: template rest-config the provider rewrites to per-project +# aggregation paths +# - downstream: the core control plane where replicated DNSRecordSets land +# VERIFY against the live env that the core control plane is reachable via the +# same milo-apiserver endpoint (it is the root scope); if milo serves the core +# CP at a distinct path, split downstreamResourceManagement.kubeconfigPath out. +discovery: + mode: milo + internalServiceDiscovery: false + discoveryKubeconfigPath: "/milo/kubeconfig" + projectKubeconfigPath: "/milo/kubeconfig" +downstreamResourceManagement: + kubeconfigPath: "/milo/kubeconfig" + +controllers: + dnsRecordSetPowerDNS: + # maxConcurrentReconciles: 4 + # rateLimiterBaseDelay: 1s + # rateLimiterMaxDelay: 30s diff --git a/docs/enhancements/controlplane-drift-detection.md b/docs/enhancements/controlplane-drift-detection.md new file mode 100644 index 0000000..9a688c7 --- /dev/null +++ b/docs/enhancements/controlplane-drift-detection.md @@ -0,0 +1,177 @@ +# Enhancement: Control-Plane Drift Detection via Resource Metrics + +**Status**: Proposed +**Author**: Engineering +**Created**: 2026-07-24 + +## Summary + +Detect when the upstream (project) and downstream (root) control planes get out of sync — the failure class behind [engineering#346](https://github.com/datum-cloud/engineering/issues/346) — using the [`milo-os/resource-metrics`](https://github.com/datum-cloud/resource-metrics) controller to emit per-object state metrics from every control plane, plus Prometheus recording/alerting rules that diff the two sides. No new instrumentation in the DNS operator is required for the primary signal. + +## Motivation + +In #346 a `multicluster-runtime` sharding bug caused every replicator replica to reconcile events for clusters it did not own. Multiple replicas programmed DNS downstream, and on AI-Edge deletion a leftover internal record kept "reserving" `www.ab.dk`, so every later record for that name was rejected as a duplicate (PowerDNS 422). The customer domain was unresolvable for ~24 hours and it was found by hand. + +We had no signal for the underlying condition: **a resource exists at one replication seam with no owner at the seam above it.** That condition is directly observable as a discrepancy between per-object metrics collected from each control plane. + +## Goals + +- Alert when a downstream `DNSRecordSet`/`DNSZone` has no surviving upstream owner (orphan — the #346 leftover). +- Alert when an upstream object is not replicated downstream (missing — stalled replication / shard-ownership gaps). +- Alert when an upstream object is stuck not-`Accepted`. +- Prove the whole path (emit → diff → alert) in e2e, building on the three-cluster harness from [PR #60](https://github.com/datum-cloud/dns-operator/pull/60). + +## Non-Goals + +- Downstream ↔ PowerDNS drift (PowerDNS is not a control plane; see Limitations). +- Detecting the shard-ownership root cause directly (runtime behavior, not resource state). The rules catch the *effect* (orphan), which is what pages. + +--- + +## Background: the replication topology + +``` +Project CP (upstream) Root CP (downstream) PowerDNS + DNSRecordSet / DNSZone ---> DNSRecordSet / DNSZone ---> RRsets + ^ replicator (mode: milo, per-project shards) ^ PowerDNS controller +``` + +- The **replicator** (`internal/controller/dnsrecordset_replicator_controller.go`) uses `multicluster-runtime` with the Milo provider (`cmd/main.go` `initializeClusterDiscovery`, `discovery.mode: milo`) to watch every project control plane and copy DNS resources to the root control plane. +- Each downstream object is stamped with `meta.datumapis.com/upstream-{cluster-name,group,kind,name,namespace}` (`internal/downstreamclient/mappednamespace.go:117-121`). The downstream **namespace is remapped**, so the real upstream namespace lives only in the annotation. The cluster-name annotation value is prefixed `cluster-`. +- The **PowerDNS controller** (`internal/controller/dnsrecordset_powerdns_controller.go`, `internal/pdns/client.go`) programs RRsets and enforces name ownership; a conflicting owner is the 422 the customer hit. + +## Design + +### Collection: resource-metrics + +`resource-metrics` runs one controller centrally, discovers project control planes via the Milo provider (`discovery.mode: milo`), and — with `discovery.collectRootControlPlane: true` — also watches the root control plane. It evaluates CEL-defined gauge families per object and pushes OTLP → OTel collector → Victoria Metrics. Metric/label definitions live in a cluster-scoped `ResourceMetricsPolicy`. + +**Confirmed feasibility:** the root/downstream control plane is reachable via `collectRootControlPlane: true` (`resource-metrics internal/config/config.go`). The only open confirmation is that the DNS "downstream" CP *is* the Milo root CP; if it is a separate infra CP the Milo provider does not engage, discovery must be extended to include it. + +The policy (`config/observability/dns-metrics-policy.yaml`) defines two series: + +| Series | Emitted on | Join labels | +|---|---|---| +| `dns_recordset_upstream_info` | project CPs | `upstream_namespace`, `upstream_name` + Milo project label | +| `dns_recordset_downstream_info` | root CP | `upstream_cluster`, `upstream_namespace`, `upstream_name` (from annotations) | + +The downstream generator lifts the `upstream-*` annotations onto labels — this is the join key and the reason no operator code is needed. + +### Detection: recording + alerting rules + +`config/observability/dns-drift-prometheusrule.yaml`: + +- `dns:recordset_downstream_orphan` = `downstream unless on(join keys) upstream` → orphan (the #346 case). +- `dns:recordset_downstream_missing` = `upstream unless on(join keys) downstream` → not replicated. +- Alerts fire with `for: 10m` to ride out normal replication + OTLP push/staleness lag (avoids flapping during healthy eventual-consistency windows). + +**Label normalization (verified live):** the two sides label the source project differently, so the rules normalize both to a bare `proj` join key: +- Downstream `upstream_cluster` = `cluster-` + `replace(, "/", "_")`. The Milo provider keys a project cluster as `/alpha`, so the annotation is **`cluster-_alpha`** (note the underscore). +- Upstream is tagged by `resource-metrics` as `milo_project_name` = the bare name (`alpha`), and `milo_control_plane_type` = `project` vs `root`. + +The recording rules strip `cluster-_?` from the downstream label and use `milo_project_name` directly for the upstream, joining on `(proj, upstream_namespace, upstream_name)`. (An earlier assumption that both sides read `cluster-` was wrong — caught by live e2e, where the matched pair was falsely flagged as both orphan and missing until the normalization was fixed.) + +### Limitations / complementary work + +- **PowerDNS seam is invisible.** A leak purely downstream→PDNS (downstream object deleted, RRset leaked) won't show here. Keep a `dns_pdns_apply_errors_total{reason="conflict"}` counter in `internal/pdns` for the direct 422 symptom, and consider a PDNS RRset exporter that diffs against downstream objects. +- **Root cause (shard ownership) not directly observed.** Optionally add a `dns_replicator_cluster_reconcile_total{pod,cluster}` metric to alert when >1 pod reconciles one cluster — an early warning ahead of any drift. + +## E2E plan: Milo-based, build on PR #60 + +[PR #60](https://github.com/datum-cloud/dns-operator/pull/60) (`feat/dns-federation-test-env`) already stands up the topology and harness we need. It brings up **three** `datum-cloud/test-infra` kind clusters via a `Taskfile.yaml` (remote test-infra include, the same pattern `milo-os/activity` and `resource-metrics` use): + +- **`dns-upstream`** — runs the **replicator** (`config/overlays/replicator`), pushes DNSZone/DNSRecordSet to control. +- **`dns-control`** — dns-operator agent + PowerDNS + RustFS; plays the **downstream** role for the replicator and the Lightningstream source for edge. +- **`dns-edge`** — PowerDNS only; receives via Lightningstream. + +The `dns-upstream → dns-control` hop **is the #346 seam.** #60 also already wires multi-cluster Chainsaw: `env:chainsaw-prepare-kubeconfigs` exports `kubeconfig-{upstream,control,edge}`, and `env:chainsaw` runs every suite under `test/e2e/`. So drift detection is almost purely additive on top of #60. + +### What #60 gives us for free + +- Task + remote `test-infra` harness, three-cluster bring-up/tear-down (`env:up` / `env:stack-up` / `env:down`). +- Cross-cluster addressing pattern (NodePort on the control-plane container) already used for RustFS and the replicator's downstream kubeconfig — reuse it to point OTLP at the collector. +- The pinned test-infra ref already ships `install-observability` — **Victoria Metrics + OTel Collector + the Prometheus-operator CRDs including `PrometheusRule`** (`prometheusrules.monitoring.coreos.com`). It's marked optional and #60 doesn't invoke it yet; we just call it. +- Multi-cluster Chainsaw suites addressing clusters by name, plus a `test/e2e/federation` suite to model the new one on. + +### New work (additive to #60) + +1. **`config/dependencies/resource-metrics/`** — kustomize/Flux to deploy the `resource-metrics` controller in `discovery.mode: single` (default mode; watches its own cluster's API). Instantiate it on **both** `dns-upstream` and `dns-control`, each pushing OTLP to the OTel collector running on `dns-control` (cross-cluster via NodePort, same as RustFS). In single mode there is no auto project label, so set a static `upstream_cluster` resource attribute per instance so the join keys line up. +2. **Taskfile additions:** `env:observability-up` (call `test-infra:install-observability` on `dns-control`), `env:metrics-up` (deploy the two resource-metrics instances + `kubectl apply -k config/observability`). Fold both into `env:stack-up` behind a flag so the default federation flow stays lean. +3. **New Chainsaw suite `test/e2e/controlplane-drift/`** (added to `env:chainsaw`, addressing `upstream` + `control` by the already-exported kubeconfigs, querying VM's HTTP API): + - **Happy path:** create a DNSRecordSet on `dns-upstream` → replicator copies to `dns-control` → assert both `*_upstream_info` and `*_downstream_info` series exist in VM and `dns:recordset_downstream_orphan == 0`. + - **Orphan / #346 regression:** create the record, let it replicate, then delete the upstream object while preventing GC cascade (scale the replicator to 0, or drop the downstream owner/finalizer) → poll VM until `dns:recordset_downstream_orphan > 0` and assert `DNSDownstreamOrphanRecordSet` enters `firing`; restore and assert it clears. + - **Missing:** scale the replicator to 0, create an upstream object → assert `dns:recordset_downstream_missing > 0` / `DNSDownstreamMissingRecordSet` fires. +4. **CI:** extend #60's `e2e.yml` to run `env:observability-up` + `env:metrics-up` before the drift suite; keep it a separate job/flag so the base federation e2e stays fast. + +### Milo topology (production-accurate) + +We run the **real production path** so the e2e reproduces the #346 root cause, not just an injected orphan. `milo-apiserver` runs on the control cluster; the downstream is the Milo **core control plane**; a single `resource-metrics` in `mode: milo` + `collectRootControlPlane: true` collects both sides — exactly as production would. + +| Cluster (#60) | Role in Phase B | +|---|---| +| `dns-control` | Hosts `milo-apiserver` + `milo-controller-manager` (`--control-plane-scope=core`). Serves the **Milo core CP (downstream)** and **≥2 project CPs (upstream)** via aggregation. Also runs the PowerDNS agent (reading DNSRecordSets from the **core CP**), RustFS, the observability stack (OTel + VM), and the single `resource-metrics` controller. | +| `dns-upstream` | Hosts the **replicator** process only, `discovery.mode: milo`, **2 replicas, leader election off**, discovery + downstream both pointed at `milo-apiserver` on control. (The name is now a slight misnomer — upstream *data* lives in Milo project CPs, not this kind cluster.) | +| `dns-edge` | Unchanged from #60: PowerDNS + Lightningstream from control. | + +This rewires #60's control cluster in two ways: the **replicator's downstream target** and the **PowerDNS agent's read source** both move from the plain `dns-control` kind API to the Milo core CP (served by `milo-apiserver`). + +### Milo deployment — reuse resource-metrics' proven setup + +Lifted from `resource-metrics` (verified in its e2e): +- `config/dependencies/milo/` — Flux `OCIRepository` on `oci://ghcr.io/datum-cloud/milo-kustomize` (pin a tag) → `overlays/test-infra`, plus the separate `milo-infra-crds` Kustomization that installs the `ProjectControlPlane` CRD (the core controller crash-loops without it). Added to this repo as drafts. +- A `milo-kubeconfig` Secret targeting `https://milo-apiserver.milo-system.svc.cluster.local:6443` with the static `test-admin-token` from milo's test-infra overlay (`system:masters` in test only). +- `resource-metrics` server-config: `discovery.mode: milo`, `discoveryKubeconfigPath`/`projectKubeconfigPath` → the milo kubeconfig, `collectRootControlPlane: true`. +- `dns-operator` replicator server-config: `discovery.mode: milo`, same discovery/project kubeconfig, downstream client pointed at the Milo core CP. + +### Generator scoping — one policy, two control-plane roles + +The single `dns-metrics` policy is applied to all CPs, so **both** the upstream and downstream generators run on every control plane. They are separated by the `milo.project.name` label the Milo provider adds — `"root"` on the core CP, the project name on project CPs. The recording rules therefore filter: +- upstream view: `dns_recordset_upstream_info{milo_project_name!="root"}` +- downstream view: `dns_recordset_downstream_info{milo_project_name="root"}` + +Without this filter, the downstream generator's series on project CPs (empty `upstream_*` labels, annotations absent) and the upstream generator's series on the core CP would pollute the diff. **Verify** the exact promoted label name (`milo_project_name` vs `milo.project.name`) against a real series and adjust both the rules and the filter. + +### Chainsaw suite `test/e2e/controlplane-drift/` + +Added to `env:chainsaw`, addressing Milo project CPs (via aggregation-path kubeconfigs, the pattern `resource-metrics` uses) and querying VM's HTTP API: +- **Happy path:** create a DNSRecordSet on project CP `alpha` → replicator copies it to the core CP → assert both `*_upstream_info{milo_project_name="alpha"}` and `*_downstream_info{milo_project_name="root"}` exist and `dns:recordset_downstream_orphan == 0`. +- **Orphan / #346 regression (injected):** delete the upstream object while preventing GC cascade (scale replicator to 0, or drop the downstream owner/finalizer) → poll VM until `dns:recordset_downstream_orphan > 0` and assert `DNSDownstreamOrphanRecordSet` fires; restore and assert it clears. +- **Missing:** scale replicator to 0, create an upstream object → assert `dns:recordset_downstream_missing > 0`. +- **Genuine sharding repro (the reason for Phase B):** pin the **pre-fix** `multicluster-runtime` (before [datum-cloud/network-services-operator#320](https://github.com/datum-cloud/network-services-operator/pull/320) / [kubernetes-sigs/multicluster-runtime#173](https://github.com/kubernetes-sigs/multicluster-runtime/pull/173)), run **2 replicas across ≥2 project CPs**, exercise a create/delete cycle, and assert the drift alert fires from the real bug — every replica reconciling clusters it doesn't own — not an injected orphan. Flip to the fixed fork and assert it stays clean. + +### Taskfile additions (on #60) + +- `env:milo-up` — `kubectl apply -k config/dependencies/milo` on control, wait for the Flux Kustomizations, mint the `milo-kubeconfig`, create Org + ≥2 Projects (project CPs). +- `env:observability-up` — `test-infra:install-observability` on control (VM + OTel + PrometheusRule CRD). +- `env:metrics-up` — deploy the single `resource-metrics` (mode:milo, collectRootControlPlane) + `kubectl apply -k config/observability`. +- Rework `env:upstream-up` → deploy the replicator with the `mode: milo` overlay, 2 replicas, pointed at milo on control. +- Rework `env:control-up` → agent reads from the Milo core CP. +- Fold into `env:stack-up`; extend #60's `e2e.yml` to run the drift suite as a separate job. + +> Phase B is environment-heavy (Flux reconcile timing, milo token auth, cross-cluster addressing, 2-replica sharding). Expect live-cluster iteration to shake out the wiring; the config drafts here are the starting point, not turnkey. + +## Sequencing + +1. Land `config/observability/*` (policy + rules) — done as drafts in this change. +2. Add `config/dependencies/milo/`, the replicator `mode: milo` overlay + server-config, the `resource-metrics` deploy overlay, and the Taskfile `env:milo-up`/`env:observability-up`/`env:metrics-up` additions on top of PR #60. +3. Add the `test/e2e/controlplane-drift/` suite (happy / orphan / missing / genuine-sharding-repro) and wire it into `env:chainsaw` + CI. +4. Confirm the production downstream CP == Milo core CP; enable `collectRootControlPlane` in the prod `resource-metrics` config; validate series parity in staging for one retention window before wiring alerts to paging. +5. Complementary operator-side signals: PDNS conflict counter, replica-ownership metric. + +## Verified against datum-cloud/infra + +Reading the real staging/production deployment (`datum-cloud/infra`) resolved the assumptions that code alone couldn't: + +- **The Milo core control plane serves `DNSRecordSet`s.** `apps/dns-operator/control-plane/base/core-control-plane-resources.yaml` installs the DNS CRDs *into the core control plane* (via the `milo-configuration` cert, `system:control@services.miloapis.com`). So replicated records land on the core CP and `resource-metrics`' root collection can see them — the previously live-only caveat, now confirmed. +- **The replicator already runs `mode: milo` in prod** (`apps/dns-operator/control-plane/staging/config.yaml`) with discovery + project kubeconfigs, and `downstreamResourceManagement` **commented out** so it writes downstream via the pod's in-cluster SA — i.e. to the core control plane it runs on. Replicator (`control-plane/`) and agent (`downstream/`) both deploy to the same infra cluster (`datum-dns-system`). +- **`resource-metrics` is deployed but `collectRootControlPlane` is NOT set** (`apps/resource-metrics-system/{staging,production}/config.yaml` have only `mode: milo`). So the downstream/core-CP series this design needs are **not currently emitted** — enabling `collectRootControlPlane: true` is a required, net-new infra change, not just new rules. +- **Policies target the Milo core CP** — `resource-metrics` installs its CRDs into the core control plane (`apps/resource-metrics-system/base/milo-control-plane.yaml`), so the `dns-metrics` `ResourceMetricsPolicy` is applied there, not to a plain cluster. +- **Alert/recording rules stay a standard `PrometheusRule`.** The victoria-metrics-operator auto-converts `PrometheusRule` → `VMRule` (carrying labels through), so no VM-specific CRD is needed. What matters is the label `telemetry.miloapis.com/resource-metrics-aggregator`, which routes the rules to the dedicated `vmalert-datum-resource-metrics-aggregator` (the general vmalert selects the complement — that label `DoesNotExist`). `config/observability/dns-drift-rules.yaml` reflects this. + +Implication for the e2e topology: prod co-locates the replicator on the core/Milo cluster (no separate upstream cluster; "upstream" = Milo project CPs). The e2e should therefore run the replicator on `dns-control` alongside Milo (downstream via in-cluster SA), rather than on a separate `dns-upstream` kind cluster — simpler and more faithful than the #60-derived split. + +## Open questions + +- Enabling `collectRootControlPlane: true` in the infra `resource-metrics` config: confirm it doesn't balloon cardinality (it adds the whole core CP's objects) and that the root series carry `milo_control_plane_type="root"` as expected. +- Victoria Metrics staleness window vs. the `for: 10m` alert delay — tune together so orphans are neither missed nor flapped. +- Prod auth for the policy/agent paths uses cert-based `milo-configuration-kubeconfig` (not the e2e's static `test-admin-token`); keep the e2e's test-only creds out of any shared overlay. diff --git a/test/e2e/controlplane-drift/.chainsaw.yaml b/test/e2e/controlplane-drift/.chainsaw.yaml new file mode 100644 index 0000000..05691e6 --- /dev/null +++ b/test/e2e/controlplane-drift/.chainsaw.yaml @@ -0,0 +1,18 @@ +# Runner config for the control-plane drift suite. The scenarios share one +# replicator Deployment (scaled up/down) and one Victoria Metrics instance, so +# they MUST run sequentially. Delete/cleanup budgets are generous because +# deleting an upstream DNSRecordSet waits on the replicator removing its +# finalizer + cascading to the downstream copy. +apiVersion: chainsaw.kyverno.io/v1alpha1 +kind: Configuration +metadata: + name: controlplane-drift +spec: + parallel: 1 + timeouts: + apply: 60s + assert: 120s + error: 60s + delete: 150s + cleanup: 150s + exec: 600s diff --git a/test/e2e/controlplane-drift/.gitignore b/test/e2e/controlplane-drift/.gitignore new file mode 100644 index 0000000..5dafaf0 --- /dev/null +++ b/test/e2e/controlplane-drift/.gitignore @@ -0,0 +1,2 @@ +# Generated per-run by the Taskfile (env:chainsaw-milo) — never commit. +kubeconfig-* diff --git a/test/e2e/controlplane-drift/README.md b/test/e2e/controlplane-drift/README.md new file mode 100644 index 0000000..1a899ed --- /dev/null +++ b/test/e2e/controlplane-drift/README.md @@ -0,0 +1,117 @@ +# control-plane drift e2e + +Chainsaw scenarios that prove upstream/downstream control-plane DNS drift +detection: the resource-metrics collectors emit per-object series from every +control plane, the `dns-controlplane-drift` recording/alerting rules diff the +two sides, and these tests assert the resulting series land in Victoria Metrics. + +See `docs/enhancements/controlplane-drift-detection.md` for the full design and +`config/observability/{dns-metrics-policy,dns-drift-rules}.yaml` for the policy +and rules under test. + +## Scenarios + +| Dir | Proves | +|---|---| +| `happy-path/` | Record created on project CP `alpha` replicates to the core CP; both `dns_recordset_upstream_info{milo_project_name="alpha"}` and `dns_recordset_downstream_info{milo_control_plane_type="root"}` exist in VM and `dns:recordset_downstream_orphan == 0`. | +| `orphan/` | engineering#346 regression. Replicate a record, scale the replicator to 0, delete the upstream object → downstream copy is orphaned → `dns:recordset_downstream_orphan > 0` and `DNSDownstreamOrphanRecordSet` becomes active. Restore the replicator → orphan clears. | +| `missing/` | Scale the replicator to 0, create an upstream record → never replicated → `dns:recordset_downstream_missing > 0`. | + +## Named clusters the Taskfile must provide + +Each `chainsaw-test.yaml` declares its clusters inline (matching the +`federation/` and `zones-and-records/` suites) and expects these kubeconfig +files, one directory up (`test/e2e/`), exactly as the existing suites consume +`kubeconfig-{control,upstream,edge,downstream}`: + +| Chainsaw cluster name | Kubeconfig file | Role / server URL | +|---|---|---| +| `alpha` | `test/e2e/kubeconfig-alpha` | Project CP **alpha** (UPSTREAM). Aggregation path: `/apis/resourcemanager.miloapis.com/v1alpha1/projects/alpha/control-plane`. | +| `core` | `test/e2e/kubeconfig-core` | Milo **core** control plane (DOWNSTREAM), `` root. Replicated copies land here. | +| `infra` | `test/e2e/kubeconfig-infra` | The `dns-control` kind cluster that hosts Victoria Metrics + the OTel collector **and** the replicator `Deployment`. Used to run in-cluster `curl` against VM and to `kubectl scale` the replicator. | + +Notes for wiring: + +- `fixtures/milo-projects.yaml` (already present, do not edit) creates the Org + + projects `alpha`/`beta` on the core CP; the Taskfile should apply it and mint + the `alpha` aggregation-path kubeconfig the same way `resource-metrics` does + (clone the milo kubeconfig, rewrite the `server:` URL to the project's + `.../projects/alpha/control-plane` path). +- `kubeconfig-beta` (project CP `beta`) is created by the fixture but is **not** + consumed by the current scenarios; provide it only if/when a multi-project + sharding scenario is added. +- `alpha`, `core`, and `infra` may all be served by the same `dns-control` kind + cluster (milo-apiserver serves the aggregation views); the three kubeconfigs + differ only in their `server:` URL and token. Admin token: `test-admin-token`. +- The replicator is scaled via `kubectl -n dns-replicator-system scale + deployment --all --replicas=0|1` — namespace `dns-replicator-system`, matched + by `--all` rather than a hard-coded deployment name. Confirm that namespace is + correct for the drift topology (the design doc co-locates the replicator on + the core cluster; adjust the `infra` kubeconfig / namespace if it differs). + +## Victoria Metrics query endpoint + +Scenarios query VM over its HTTP API from a one-shot `curl` pod running inside +the `infra` cluster (the `kubectl run --rm ... curlimages/curl` pattern from the +resource-metrics suite). Default endpoint: + +``` +http://vmsingle-telemetry-system-vm.telemetry-system.svc.cluster.local:8428/api/v1/query +``` + +Override per run by exporting `VM_QUERY_URL` (the scripts honour it). If the +observability stack uses a `vmselect`/cluster VM instead of `vmsingle`, point +`VM_QUERY_URL` at that `.../api/v1/query`. The endpoint must serve BOTH raw +series (`dns_recordset_*_info`) and the recording-rule / `ALERTS` series that +`vmalert` remote-writes back — i.e. `vmalert` must be configured against this +VM as its remote-write target (the `telemetry.miloapis.com/resource-metrics-aggregator` +`vmalert` in datum-cloud/infra). + +## Timing / `for:` considerations + +- **Recording rules are the primary gate.** `dns:recordset_downstream_orphan` + and `dns:recordset_downstream_missing` have no `for:` and are evaluated every + 30s, so the tests assert on those series (`> 0`) rather than on the alert's + full delay. This is fast and deterministic. +- **The alerts carry `for: 10m`.** After the recording rule fires, the alert + sits in `pending` for 10m before `firing`. Waiting the full period would blow + the e2e budget, so `orphan/` accepts `ALERTS{...alertstate="pending"|"firing"}` + (the alert is active on the correct series). To assert `firing` specifically, + deploy the drift rules with a shortened `for:` in a test-only overlay (e.g. + `for: 30s`); that is an infra/Taskfile change outside this suite. +- **Orphan detection lags by VM's staleness window.** The orphan only + materializes once the deleted upstream series ages out of VM's staleness + window (default ~5m) so the recording rule's `unless` no longer cancels the + downstream vector. `orphan/` therefore polls for up to ~9m (`timeout: 540s`). + Shortening VM's staleness window in the test-infra observability stack would + let this budget drop. +- **Pipeline latency.** watch → collect (OTel interval ~5s) → OTLP → remote-write + → VM ingest adds seconds; the happy-path VM polls allow a few minutes. +- **Run sequentially.** All scenarios share the `alpha`/`core`/`infra` clusters + and the `orphan`/`missing` scenarios scale the replicator globally, so run the + suite with `parallel: 1` (chainsaw's default when unset, or set it in the + Taskfile's chainsaw invocation). + +## Assumptions needing live confirmation + +- **Promoted label names/values** — `milo_project_name` (value `alpha`, and + `root` for the core CP) and `milo_control_plane_type` (`project` / `root`) are + taken from the resource-metrics OTLP attributes (`milo.project.name`, + `milo.control_plane.type`) after prometheus-remote-write dot→underscore + normalization. Confirm against a real series and adjust the queries + rules if + the aggregator promotes different keys. +- **Downstream join labels** — the downstream series lifts + `meta.datumapis.com/upstream-{cluster-name,namespace,name}` onto + `upstream_{cluster,namespace,name}`. The tests key on `upstream_name` + (= the upstream DNSRecordSet's `metadata.name`); verify the replicator stamps + the object name (not a derived value) into the `upstream-name` annotation. +- **Replicator namespace / deployment** — assumed `dns-replicator-system` on the + `infra` cluster. Confirm and adjust if the drift topology runs the replicator + elsewhere. +- **`vmalert` wiring** — assumes the drift `PrometheusRule`/`VMRule` is evaluated + and its recording results + `ALERTS` are queryable at `VM_QUERY_URL`. If the + observability stack doesn't remote-write vmalert results back to the same VM, + point the queries at the vmalert datasource instead. +- **Whether the upstream DNSRecordSet reaches `Accepted` on the project CP** is + not asserted (the condition-setting controller for that topology is + unconfirmed); the tests assert object existence + the VM series instead. diff --git a/test/e2e/controlplane-drift/fixtures/dnszone.yaml b/test/e2e/controlplane-drift/fixtures/dnszone.yaml new file mode 100644 index 0000000..ac0d2e3 --- /dev/null +++ b/test/e2e/controlplane-drift/fixtures/dnszone.yaml @@ -0,0 +1,13 @@ +# Shared DNSZone for the control-plane drift e2e. Namespaced; created on the +# upstream project CP (e.g. cluster: alpha) in the `default` namespace. The +# replicator copies it to the downstream (core) CP, remapping the namespace and +# stamping the meta.datumapis.com/upstream-* annotations that resource-metrics +# lifts onto the dns_zone_downstream_info series. +apiVersion: dns.networking.miloapis.com/v1alpha1 +kind: DNSZone +metadata: + name: drift-example + namespace: default +spec: + domainName: drift.example + dnsZoneClassName: drift-powerdns diff --git a/test/e2e/controlplane-drift/fixtures/dnszoneclass.yaml b/test/e2e/controlplane-drift/fixtures/dnszoneclass.yaml new file mode 100644 index 0000000..acac424 --- /dev/null +++ b/test/e2e/controlplane-drift/fixtures/dnszoneclass.yaml @@ -0,0 +1,19 @@ +# Shared DNSZoneClass for the control-plane drift e2e. DNSZoneClass is +# cluster-scoped, so it must be created on whichever control plane the +# DNSZone/DNSRecordSet objects live on (the project CPs / "upstream" side). +# Each scenario applies this to its upstream project CP (e.g. cluster: alpha) +# before creating a DNSZone that references it by name. +apiVersion: dns.networking.miloapis.com/v1alpha1 +kind: DNSZoneClass +metadata: + name: drift-powerdns +spec: + controllerName: powerdns + nameServerPolicy: + mode: Static + static: + servers: + - ns1.drift.example. + - ns2.drift.example. + defaults: + defaultTTL: 300 diff --git a/test/e2e/controlplane-drift/fixtures/milo-projects.yaml b/test/e2e/controlplane-drift/fixtures/milo-projects.yaml new file mode 100644 index 0000000..706af5d --- /dev/null +++ b/test/e2e/controlplane-drift/fixtures/milo-projects.yaml @@ -0,0 +1,33 @@ +# Org + two projects for the control-plane drift e2e. Applied to the Milo core +# control plane (milo-apiserver). Milo's project controller creates an isolated +# virtual control plane per Project, reachable at +# {milo}/apis/resourcemanager.miloapis.com/v1alpha1/projects//control-plane +# and shares the core-registered DNS CRDs. These two projects are the "upstream" +# control planes the replicator discovers in discovery.mode=milo; the core CP is +# the "downstream". See docs/enhancements/controlplane-drift-detection.md. +apiVersion: resourcemanager.miloapis.com/v1alpha1 +kind: Organization +metadata: + name: drift-org + annotations: + kubernetes.io/display-name: "DNS Drift E2E Org" +spec: + type: Standard +--- +apiVersion: resourcemanager.miloapis.com/v1alpha1 +kind: Project +metadata: + name: alpha +spec: + ownerRef: + kind: Organization + name: drift-org +--- +apiVersion: resourcemanager.miloapis.com/v1alpha1 +kind: Project +metadata: + name: beta +spec: + ownerRef: + kind: Organization + name: drift-org diff --git a/test/e2e/controlplane-drift/happy-path/chainsaw-test.yaml b/test/e2e/controlplane-drift/happy-path/chainsaw-test.yaml new file mode 100644 index 0000000..d0b15db --- /dev/null +++ b/test/e2e/controlplane-drift/happy-path/chainsaw-test.yaml @@ -0,0 +1,186 @@ +# yaml-language-server: $schema=https://raw.githubusercontent.com/kyverno/chainsaw/main/.schemas/json/test-chainsaw-v1alpha1.json +# +# Happy path: create a DNSZone(class) + DNSRecordSet on project CP `alpha` +# (upstream), wait for the replicated copy to land on the core CP (downstream), +# then assert Victoria Metrics carries BOTH sides of the join and that the +# orphan recording rule is clean. +# +# Clusters (kubeconfigs provided by the Taskfile — see ../README.md): +# alpha -> project CP alpha (UPSTREAM) ../kubeconfig-alpha +# core -> Milo core control plane (DOWNSTREAM) ../kubeconfig-core +# infra -> kind cluster hosting Victoria Metrics + OTel ../kubeconfig-infra +# (used only to run in-cluster curl against VM) +apiVersion: chainsaw.kyverno.io/v1alpha1 +kind: Test +metadata: + name: controlplane-drift-happy-path +spec: + # The VM-poll scripts loop for several minutes to ride out the + # watch -> collect -> OTLP -> remote-write -> ingest pipeline latency. + timeouts: + exec: 600s + clusters: + alpha: + kubeconfig: ../kubeconfig-alpha + core: + kubeconfig: ../kubeconfig-core + infra: + kubeconfig: ../kubeconfig-infra + cluster: alpha + steps: + - name: Prereq - DNSZoneClass + DNSZone on upstream project CP alpha + try: + - apply: + cluster: alpha + file: ../fixtures/dnszoneclass.yaml + - apply: + cluster: alpha + file: ../fixtures/dnszone.yaml + - assert: + cluster: alpha + resource: + apiVersion: dns.networking.miloapis.com/v1alpha1 + kind: DNSZone + metadata: + name: drift-example + namespace: default + + - name: Create DNSRecordSet on upstream project CP alpha + try: + - create: + cluster: alpha + resource: + apiVersion: dns.networking.miloapis.com/v1alpha1 + kind: DNSRecordSet + metadata: + name: www-happy + namespace: default + spec: + dnsZoneRef: + name: drift-example + recordType: A + records: + - name: www + ttl: 60 + a: + content: 192.0.2.10 + - assert: + cluster: alpha + resource: + apiVersion: dns.networking.miloapis.com/v1alpha1 + kind: DNSRecordSet + metadata: + name: www-happy + namespace: default + + - name: Wait for the replicated copy to land on the downstream core CP + try: + - script: + cluster: core + timeout: 300s + content: | + set -eu + # The downstream namespace is remapped (ns-) and the real + # upstream identity lives only in annotations, so match on the + # upstream-name annotation across all namespaces rather than by + # name/namespace. + want="www-happy" + attempts=30 + i=0 + while [ "$i" -lt "$attempts" ]; do + i=$((i+1)) + got=$(kubectl get dnsrecordset -A \ + -o jsonpath='{range .items[*]}{.metadata.annotations.meta\.datumapis\.com/upstream-name}{"\n"}{end}' \ + 2>/dev/null || true) + echo "[attempt $i/$attempts] downstream upstream-name annotations: ${got}" + if printf '%s\n' "$got" | grep -qx "$want"; then + echo "replicated copy present on core CP" + exit 0 + fi + sleep 10 + done + echo "replicated DNSRecordSet ($want) never appeared on core CP" >&2 + exit 1 + + - name: Assert upstream + downstream series exist in Victoria Metrics + try: + - script: + cluster: infra + timeout: 300s + content: | + set -eu + vm_url="${VM_QUERY_URL:-http://vmsingle-telemetry-system-vm.telemetry-system.svc.cluster.local:8428/api/v1/query}" + + # Poll VM for a PromQL query until it returns a non-empty result. + poll() { + query="$1"; label="$2" + attempt=0 + until [ "$attempt" -ge 24 ]; do + attempt=$((attempt+1)) + echo "[$label attempt $attempt] query: $query" + out=$(kubectl run e2e-vmq-happy-$label-$attempt \ + --rm -i --restart=Never --quiet \ + --image=curlimages/curl:8.10.1 \ + --timeout=20s \ + -- curl -fsS --max-time 5 \ + --data-urlencode "query=${query}" "${vm_url}" \ + || true) + json=$(printf '%s' "$out" | sed '/pod ".*" deleted/d') + echo "response: $json" + if printf '%s' "$json" | grep -Eq '"result":\s*\[\s*\{'; then + echo "[$label] series found" + return 0 + fi + sleep 10 + done + echo "[$label] series never appeared: $query" >&2 + return 1 + } + + poll 'dns_recordset_upstream_info{upstream_name="www-happy",milo_project_name="alpha"}' upstream + poll 'dns_recordset_downstream_info{upstream_name="www-happy",milo_control_plane_type="root"}' downstream + + - name: Assert the orphan recording rule is clean (== 0 / empty) + try: + - script: + cluster: infra + timeout: 120s + content: | + set -eu + vm_url="${VM_QUERY_URL:-http://vmsingle-telemetry-system-vm.telemetry-system.svc.cluster.local:8428/api/v1/query}" + # The orphan recording rule only PRODUCES a series when an orphan + # exists (`unless` yields the leftover downstream vectors). A clean + # state is therefore an empty result. Scope to our record so an + # unrelated pre-existing orphan can't fail the happy path. + query='dns:recordset_downstream_orphan{upstream_name="www-happy"}' + out=$(kubectl run e2e-vmq-happy-orphan \ + --rm -i --restart=Never --quiet \ + --image=curlimages/curl:8.10.1 \ + --timeout=20s \ + -- curl -fsS --max-time 5 \ + --data-urlencode "query=${query}" "${vm_url}" \ + || true) + json=$(printf '%s' "$out" | sed '/pod ".*" deleted/d') + echo "response: $json" + if printf '%s' "$json" | grep -Eq '"result":\s*\[\s*\{'; then + echo "unexpected orphan series for www-happy — drift detected on happy path" >&2 + exit 1 + fi + echo "orphan recording rule clean (no series for www-happy)" + + - name: Teardown + try: + - delete: + cluster: alpha + ref: + apiVersion: dns.networking.miloapis.com/v1alpha1 + kind: DNSRecordSet + namespace: default + name: www-happy + - delete: + cluster: alpha + ref: + apiVersion: dns.networking.miloapis.com/v1alpha1 + kind: DNSZone + namespace: default + name: drift-example diff --git a/test/e2e/controlplane-drift/missing/chainsaw-test.yaml b/test/e2e/controlplane-drift/missing/chainsaw-test.yaml new file mode 100644 index 0000000..014e7be --- /dev/null +++ b/test/e2e/controlplane-drift/missing/chainsaw-test.yaml @@ -0,0 +1,173 @@ +# yaml-language-server: $schema=https://raw.githubusercontent.com/kyverno/chainsaw/main/.schemas/json/test-chainsaw-v1alpha1.json +# +# Missing: an upstream DNSRecordSet that was never replicated downstream +# (stalled replication / shard-ownership gap). We reproduce it by scaling the +# replicator to 0 FIRST, then creating an upstream object — no downstream copy +# is ever made — and assert the missing recording rule fires. +# +# Clusters (kubeconfigs provided by the Taskfile — see ../README.md): +# alpha -> project CP alpha (UPSTREAM) ../kubeconfig-alpha +# core -> Milo core control plane (DOWNSTREAM) ../kubeconfig-core +# infra -> kind cluster hosting VM + OTel + the ../kubeconfig-infra +# replicator Deployment (ns dns-replicator-system) +apiVersion: chainsaw.kyverno.io/v1alpha1 +kind: Test +metadata: + name: controlplane-drift-missing +spec: + timeouts: + exec: 600s + clusters: + alpha: + kubeconfig: ../kubeconfig-alpha + core: + kubeconfig: ../kubeconfig-core + infra: + kubeconfig: ../kubeconfig-infra + replicator: + kubeconfig: ../kubeconfig-replicator + cluster: alpha + steps: + - name: Prereq - DNSZoneClass + DNSZone on upstream project CP alpha + try: + - apply: + cluster: alpha + file: ../fixtures/dnszoneclass.yaml + - apply: + cluster: alpha + file: ../fixtures/dnszone.yaml + - assert: + cluster: alpha + resource: + apiVersion: dns.networking.miloapis.com/v1alpha1 + kind: DNSZone + metadata: + name: drift-example + namespace: default + + - name: Scale the replicator to 0 so nothing gets replicated + try: + - script: + cluster: replicator + timeout: 120s + content: | + set -eu + kubectl -n dns-replicator-system scale deployment/dns-operator-controller-manager --replicas=0 + kubectl -n dns-replicator-system rollout status deployment/dns-operator-controller-manager --timeout=90s || true + kubectl -n dns-replicator-system get pods + + - name: Create an upstream DNSRecordSet that will never replicate + try: + - create: + cluster: alpha + resource: + apiVersion: dns.networking.miloapis.com/v1alpha1 + kind: DNSRecordSet + metadata: + name: www-missing + namespace: default + spec: + dnsZoneRef: + name: drift-example + recordType: A + records: + - name: www-missing + ttl: 60 + a: + content: 192.0.2.30 + - assert: + cluster: alpha + resource: + apiVersion: dns.networking.miloapis.com/v1alpha1 + kind: DNSRecordSet + metadata: + name: www-missing + namespace: default + + - name: Confirm no downstream copy exists on the core CP + try: + - script: + cluster: core + timeout: 60s + content: | + set -eu + got=$(kubectl get dnsrecordset -A \ + -o jsonpath='{range .items[*]}{.metadata.annotations.meta\.datumapis\.com/upstream-name}{"\n"}{end}' \ + 2>/dev/null || true) + echo "downstream upstream-name annotations: ${got}" + if printf '%s\n' "$got" | grep -qx "www-missing"; then + echo "unexpected: www-missing was replicated despite replicator scaled to 0" >&2 + exit 1 + fi + echo "confirmed: no downstream copy for www-missing" + + - name: Poll VM until the missing recording rule fires + try: + - script: + cluster: infra + timeout: 300s + content: | + set -eu + vm_url="${VM_QUERY_URL:-http://vmsingle-telemetry-system-vm.telemetry-system.svc.cluster.local:8428/api/v1/query}" + query='dns:recordset_downstream_missing{upstream_name="www-missing"} > 0' + attempt=0 + until [ "$attempt" -ge 30 ]; do + attempt=$((attempt+1)) + echo "[missing attempt $attempt] query: $query" + out=$(kubectl run e2e-vmq-missing-$attempt \ + --rm -i --restart=Never --quiet \ + --image=curlimages/curl:8.10.1 \ + --timeout=20s \ + -- curl -fsS --max-time 5 \ + --data-urlencode "query=${query}" "${vm_url}" \ + || true) + json=$(printf '%s' "$out" | sed '/pod ".*" deleted/d') + echo "response: $json" + if printf '%s' "$json" | grep -Eq '"result":\s*\[\s*\{'; then + echo "missing detected by recording rule" + exit 0 + fi + sleep 10 + done + echo "dns:recordset_downstream_missing never went > 0 for www-missing" >&2 + exit 1 + + - name: Teardown - restore the replicator and clean up + try: + - script: + cluster: replicator + timeout: 180s + content: | + set -eu + # Restore the replicator BEFORE deleting the upstream object so the + # (now-replicated) downstream copy gets cascade-deleted normally and + # no orphan is left behind for the next scenario. + kubectl -n dns-replicator-system scale deployment/dns-operator-controller-manager --replicas=1 + kubectl -n dns-replicator-system rollout status deployment/dns-operator-controller-manager --timeout=120s + # rollout status only waits for pod readiness — wait for the + # multicluster provider to actually re-engage the project CPs before + # deleting upstream objects, otherwise the replicator can't release + # finalizers / cascade-delete the downstream copies and the deletes + # below time out. + for i in $(seq 1 20); do + if kubectl -n dns-replicator-system logs -l control-plane=controller-manager --since=90s 2>/dev/null \ + | grep -q "Successfully registered and engaged"; then + echo "replicator re-engaged project CPs"; break + fi + sleep 3 + done + sleep 10 + - delete: + cluster: alpha + ref: + apiVersion: dns.networking.miloapis.com/v1alpha1 + kind: DNSRecordSet + namespace: default + name: www-missing + - delete: + cluster: alpha + ref: + apiVersion: dns.networking.miloapis.com/v1alpha1 + kind: DNSZone + namespace: default + name: drift-example diff --git a/test/e2e/controlplane-drift/orphan/chainsaw-test.yaml b/test/e2e/controlplane-drift/orphan/chainsaw-test.yaml new file mode 100644 index 0000000..2e7019c --- /dev/null +++ b/test/e2e/controlplane-drift/orphan/chainsaw-test.yaml @@ -0,0 +1,280 @@ +# yaml-language-server: $schema=https://raw.githubusercontent.com/kyverno/chainsaw/main/.schemas/json/test-chainsaw-v1alpha1.json +# +# Orphan / engineering#346 regression: a DNSRecordSet exists on the downstream +# (core) CP with no surviving upstream owner. We reproduce it by replicating a +# record, scaling the replicator to 0 (so it can't cascade-delete the downstream +# copy), then deleting the upstream object. The downstream copy is left behind => +# orphan. We assert the orphan recording rule fires and the alert becomes active, +# then restore the replicator and assert it clears. +# +# Clusters (kubeconfigs provided by the Taskfile — see ../README.md): +# alpha -> project CP alpha (UPSTREAM) ../kubeconfig-alpha +# core -> Milo core control plane (DOWNSTREAM) ../kubeconfig-core +# infra -> kind cluster hosting VM + OTel + the ../kubeconfig-infra +# replicator Deployment (ns dns-replicator-system) +apiVersion: chainsaw.kyverno.io/v1alpha1 +kind: Test +metadata: + name: controlplane-drift-orphan +spec: + timeouts: + exec: 600s + clusters: + alpha: + kubeconfig: ../kubeconfig-alpha + core: + kubeconfig: ../kubeconfig-core + infra: + kubeconfig: ../kubeconfig-infra + replicator: + kubeconfig: ../kubeconfig-replicator + cluster: alpha + steps: + - name: Prereq - DNSZoneClass + DNSZone on upstream project CP alpha + try: + - apply: + cluster: alpha + file: ../fixtures/dnszoneclass.yaml + - apply: + cluster: alpha + file: ../fixtures/dnszone.yaml + - assert: + cluster: alpha + resource: + apiVersion: dns.networking.miloapis.com/v1alpha1 + kind: DNSZone + metadata: + name: drift-example + namespace: default + + - name: Create DNSRecordSet on upstream and wait for downstream replica + try: + - create: + cluster: alpha + resource: + apiVersion: dns.networking.miloapis.com/v1alpha1 + kind: DNSRecordSet + metadata: + name: www-orphan + namespace: default + spec: + dnsZoneRef: + name: drift-example + recordType: A + records: + - name: www-orphan + ttl: 60 + a: + content: 192.0.2.20 + - script: + cluster: core + timeout: 300s + content: | + set -eu + want="www-orphan" + attempts=30 + i=0 + while [ "$i" -lt "$attempts" ]; do + i=$((i+1)) + got=$(kubectl get dnsrecordset -A \ + -o jsonpath='{range .items[*]}{.metadata.annotations.meta\.datumapis\.com/upstream-name}{"\n"}{end}' \ + 2>/dev/null || true) + echo "[attempt $i/$attempts] downstream upstream-name annotations: ${got}" + if printf '%s\n' "$got" | grep -qx "$want"; then + echo "replicated copy present on core CP" + exit 0 + fi + sleep 10 + done + echo "replicated DNSRecordSet ($want) never appeared on core CP" >&2 + exit 1 + + - name: Scale the replicator to 0 to prevent GC cascade + try: + - script: + cluster: replicator + timeout: 120s + content: | + set -eu + # Scaling every deployment in the replicator namespace avoids + # coupling the test to a specific deployment name. With no + # replicator running, deleting the upstream object cannot cascade + # to the downstream copy (they live on different control planes; + # only the replicator would remove it) — that leftover is the + # orphan. + kubectl -n dns-replicator-system scale deployment/dns-operator-controller-manager --replicas=0 + kubectl -n dns-replicator-system rollout status deployment/dns-operator-controller-manager --timeout=90s || true + kubectl -n dns-replicator-system get pods + + - name: Delete the upstream object, leaving the downstream copy orphaned + try: + - script: + cluster: alpha + timeout: 60s + content: | + set -eu + # The replicator is scaled to 0, so it will NOT remove its + # finalizer — a plain delete would hang. Strip the finalizer so the + # upstream fully deletes, leaving the downstream copy as a true + # orphan (the engineering#346 end state). + kubectl -n default patch dnsrecordset www-orphan \ + --type=merge -p '{"metadata":{"finalizers":[]}}' || true + kubectl -n default delete dnsrecordset www-orphan --wait=false || true + - script: + cluster: core + timeout: 60s + content: | + set -eu + # Confirm the downstream copy is still present (orphaned). + got=$(kubectl get dnsrecordset -A \ + -o jsonpath='{range .items[*]}{.metadata.annotations.meta\.datumapis\.com/upstream-name}{"\n"}{end}' \ + 2>/dev/null || true) + echo "downstream upstream-name annotations after upstream delete: ${got}" + printf '%s\n' "$got" | grep -qx "www-orphan" + + - name: Poll VM until the orphan recording rule fires + try: + - script: + cluster: infra + # Budget generously: the orphan only materializes once the deleted + # upstream series ages out of VM's staleness window (default ~5m) + # so `unless` no longer cancels the downstream vector. + timeout: 540s + content: | + set -eu + vm_url="${VM_QUERY_URL:-http://vmsingle-telemetry-system-vm.telemetry-system.svc.cluster.local:8428/api/v1/query}" + query='dns:recordset_downstream_orphan{upstream_name="www-orphan"} > 0' + attempt=0 + until [ "$attempt" -ge 48 ]; do + attempt=$((attempt+1)) + echo "[orphan attempt $attempt] query: $query" + out=$(kubectl run e2e-vmq-orphan-$attempt \ + --rm -i --restart=Never --quiet \ + --image=curlimages/curl:8.10.1 \ + --timeout=20s \ + -- curl -fsS --max-time 5 \ + --data-urlencode "query=${query}" "${vm_url}" \ + || true) + json=$(printf '%s' "$out" | sed '/pod ".*" deleted/d') + echo "response: $json" + if printf '%s' "$json" | grep -Eq '"result":\s*\[\s*\{'; then + echo "orphan detected by recording rule" + exit 0 + fi + sleep 10 + done + echo "dns:recordset_downstream_orphan never went > 0 for www-orphan" >&2 + exit 1 + + - name: Assert the DNSDownstreamOrphanRecordSet alert is active + try: + - script: + cluster: infra + timeout: 180s + content: | + set -eu + vm_url="${VM_QUERY_URL:-http://vmsingle-telemetry-system-vm.telemetry-system.svc.cluster.local:8428/api/v1/query}" + # The alert carries `for: 10m`, so it sits in `pending` for 10m + # before `firing`. Waiting the full period would blow the e2e + # budget, so we accept EITHER pending or firing here (the alert is + # active on the correct series). To assert `firing` specifically, + # deploy the rules with a shortened `for:` in a test overlay — see + # ../README.md. + query='ALERTS{alertname="DNSDownstreamOrphanRecordSet",upstream_name="www-orphan"}' + attempt=0 + until [ "$attempt" -ge 18 ]; do + attempt=$((attempt+1)) + echo "[alert attempt $attempt] query: $query" + out=$(kubectl run e2e-vmq-orphan-alert-$attempt \ + --rm -i --restart=Never --quiet \ + --image=curlimages/curl:8.10.1 \ + --timeout=20s \ + -- curl -fsS --max-time 5 \ + --data-urlencode "query=${query}" "${vm_url}" \ + || true) + json=$(printf '%s' "$out" | sed '/pod ".*" deleted/d') + echo "response: $json" + if printf '%s' "$json" | grep -Eq '"alertstate":"(pending|firing)"'; then + echo "DNSDownstreamOrphanRecordSet alert is active" + exit 0 + fi + sleep 10 + done + echo "DNSDownstreamOrphanRecordSet never became active for www-orphan" >&2 + exit 1 + + - name: Remediate (remove the leftover) and assert the orphan clears + try: + - script: + cluster: replicator + timeout: 180s + content: | + set -eu + # Restore the replicator for environment health. NOTE: it does NOT + # auto-GC the orphan — with the upstream owner gone there is no + # event to drive a reconcile of the leftover, so (exactly as in + # engineering#346) the downstream leftover PERSISTS until it is + # explicitly removed. That persistence is the whole reason this + # detection matters. + kubectl -n dns-replicator-system scale deployment/dns-operator-controller-manager --replicas=1 + kubectl -n dns-replicator-system rollout status deployment/dns-operator-controller-manager --timeout=120s + - script: + cluster: core + timeout: 120s + content: | + set -eu + # Remediation: remove the orphaned downstream leftover (the manual + # fix that resolved #346). Locate it by its upstream-name annotation + # and strip the finalizer so it fully deletes. + row=$(kubectl get dnsrecordset -A \ + -o jsonpath='{range .items[*]}{.metadata.namespace}{" "}{.metadata.name}{" "}{.metadata.annotations.meta\.datumapis\.com/upstream-name}{"\n"}{end}' \ + | awk '$3=="www-orphan"{print $1" "$2; exit}') + if [ -n "$row" ]; then + set -- $row + kubectl -n "$1" patch dnsrecordset "$2" --type=merge -p '{"metadata":{"finalizers":[]}}' || true + kubectl -n "$1" delete dnsrecordset "$2" --wait=false || true + echo "removed orphaned leftover $1/$2" + else + echo "no leftover found (already removed)" + fi + - script: + cluster: core + timeout: 90s + content: | + set -eu + # Assert remediation at the source of truth: the orphaned downstream + # object is gone from the core CP. We deliberately do NOT poll the + # VM recording rule here — its series only clears once the removed + # object's samples age out of VM's ~5m staleness window, which makes + # a rule-based clear assertion slow and flaky. Once the object is + # gone, the rule is guaranteed to clear after staleness. + for i in $(seq 1 12); do + got=$(kubectl get dnsrecordset -A \ + -o jsonpath='{range .items[*]}{.metadata.annotations.meta\.datumapis\.com/upstream-name}{"\n"}{end}' \ + 2>/dev/null || true) + if ! printf '%s\n' "$got" | grep -qx "www-orphan"; then + echo "orphaned leftover removed from core CP — drift resolved at source" + exit 0 + fi + echo "[clear attempt $i] leftover still present; waiting" + sleep 5 + done + echo "orphaned downstream leftover was not removed from the core CP" >&2 + exit 1 + + - name: Teardown + try: + # Ensure the replicator is back up even if an earlier step failed. + - script: + cluster: replicator + timeout: 120s + content: | + set -eu + kubectl -n dns-replicator-system scale deployment/dns-operator-controller-manager --replicas=1 || true + - delete: + cluster: alpha + ref: + apiVersion: dns.networking.miloapis.com/v1alpha1 + kind: DNSZone + namespace: default + name: drift-example From cec1d85ea7d375713adb02fcd9f744913095a558 Mon Sep 17 00:00:00 2001 From: Scot Wells Date: Sat, 25 Jul 2026 16:09:35 -0500 Subject: [PATCH 2/9] docs: Rewrite drift-detection doc to be concise and operator-focused Replace the engineering design/journey write-up with a product-focused doc centered on what the feature detects and how on-call responds: a plain-language explanation of the failure, a table of the three alerts with what each means and what to do, a short conceptual diagram, and a brief try-it/enable-in-prod section. Drops the internal build plan, phase/sequencing notes, code-path references, and open questions that belonged to the implementation rather than the product. Co-Authored-By: Claude Opus 4.8 --- .../controlplane-drift-detection.md | 194 ++++-------------- 1 file changed, 38 insertions(+), 156 deletions(-) diff --git a/docs/enhancements/controlplane-drift-detection.md b/docs/enhancements/controlplane-drift-detection.md index 9a688c7..af67806 100644 --- a/docs/enhancements/controlplane-drift-detection.md +++ b/docs/enhancements/controlplane-drift-detection.md @@ -1,177 +1,59 @@ -# Enhancement: Control-Plane Drift Detection via Resource Metrics +# DNS Control-Plane Drift Detection -**Status**: Proposed -**Author**: Engineering -**Created**: 2026-07-24 +Detects when a customer's DNS records fall out of sync between the control plane they manage (their project) and the control plane that actually serves DNS — and alerts on-call before it turns into a customer-visible outage. -## Summary +## Why this exists -Detect when the upstream (project) and downstream (root) control planes get out of sync — the failure class behind [engineering#346](https://github.com/datum-cloud/engineering/issues/346) — using the [`milo-os/resource-metrics`](https://github.com/datum-cloud/resource-metrics) controller to emit per-object state metrics from every control plane, plus Prometheus recording/alerting rules that diff the two sides. No new instrumentation in the DNS operator is required for the primary signal. +DNS records a customer creates in their project are replicated to a shared control plane that programs the live DNS servers. When that replication misbehaves, a record can be left behind on the serving side with nothing owning it anymore. A leftover like this keeps "reserving" a hostname, so the customer can no longer point that name anywhere — every new record is rejected as a duplicate. -## Motivation +That is exactly what happened in [engineering#346](https://github.com/datum-cloud/engineering/issues/346): a customer's `www` record was unresolvable for ~24 hours, and it was only found by hand. There was no signal for the underlying condition. This feature makes that condition an alert. -In #346 a `multicluster-runtime` sharding bug caused every replicator replica to reconcile events for clusters it did not own. Multiple replicas programmed DNS downstream, and on AI-Edge deletion a leftover internal record kept "reserving" `www.ab.dk`, so every later record for that name was rejected as a duplicate (PowerDNS 422). The customer domain was unresolvable for ~24 hours and it was found by hand. +## What it detects -We had no signal for the underlying condition: **a resource exists at one replication seam with no owner at the seam above it.** That condition is directly observable as a discrepancy between per-object metrics collected from each control plane. +Three alerts, each pointing at the specific record and customer project involved: -## Goals - -- Alert when a downstream `DNSRecordSet`/`DNSZone` has no surviving upstream owner (orphan — the #346 leftover). -- Alert when an upstream object is not replicated downstream (missing — stalled replication / shard-ownership gaps). -- Alert when an upstream object is stuck not-`Accepted`. -- Prove the whole path (emit → diff → alert) in e2e, building on the three-cluster harness from [PR #60](https://github.com/datum-cloud/dns-operator/pull/60). - -## Non-Goals - -- Downstream ↔ PowerDNS drift (PowerDNS is not a control plane; see Limitations). -- Detecting the shard-ownership root cause directly (runtime behavior, not resource state). The rules catch the *effect* (orphan), which is what pages. +| Alert | What it means | What to do | +|---|---|---| +| **DNSDownstreamOrphanRecordSet** | A record exists on the serving control plane with **no owner** in any customer project — a leftover that can block the customer from reusing that hostname (the #346 case). | Remove the orphaned record on the serving control plane. The alert labels name the project, namespace, and record. | +| **DNSDownstreamMissingRecordSet** | A record exists in a **customer project but was never replicated** to the serving side — the customer's change isn't taking effect. | Check replicator health for that project; the record isn't live until it replicates. | +| **DNSRecordSetNotAccepted** | A customer's record has been sitting **un-accepted** (e.g. a misconfigured zone). | Inspect the record's status conditions in the customer project. | ---- +Alerts only fire after the condition **persists** (`for: 10m`), so normal replication lag never pages anyone. -## Background: the replication topology +## How it works ``` -Project CP (upstream) Root CP (downstream) PowerDNS - DNSRecordSet / DNSZone ---> DNSRecordSet / DNSZone ---> RRsets - ^ replicator (mode: milo, per-project shards) ^ PowerDNS controller +Customer projects ──replicate──► Serving control plane ──► Live DNS + (upstream) (downstream) + │ │ + └──────── resource-metrics ─────────┘ emits one metric per DNS record, per side + │ + ▼ + Victoria Metrics + alerts (compare the two sides; a mismatch is drift) ``` -- The **replicator** (`internal/controller/dnsrecordset_replicator_controller.go`) uses `multicluster-runtime` with the Milo provider (`cmd/main.go` `initializeClusterDiscovery`, `discovery.mode: milo`) to watch every project control plane and copy DNS resources to the root control plane. -- Each downstream object is stamped with `meta.datumapis.com/upstream-{cluster-name,group,kind,name,namespace}` (`internal/downstreamclient/mappednamespace.go:117-121`). The downstream **namespace is remapped**, so the real upstream namespace lives only in the annotation. The cluster-name annotation value is prefixed `cluster-`. -- The **PowerDNS controller** (`internal/controller/dnsrecordset_powerdns_controller.go`, `internal/pdns/client.go`) programs RRsets and enforces name ownership; a conflicting owner is the 422 the customer hit. - -## Design - -### Collection: resource-metrics - -`resource-metrics` runs one controller centrally, discovers project control planes via the Milo provider (`discovery.mode: milo`), and — with `discovery.collectRootControlPlane: true` — also watches the root control plane. It evaluates CEL-defined gauge families per object and pushes OTLP → OTel collector → Victoria Metrics. Metric/label definitions live in a cluster-scoped `ResourceMetricsPolicy`. - -**Confirmed feasibility:** the root/downstream control plane is reachable via `collectRootControlPlane: true` (`resource-metrics internal/config/config.go`). The only open confirmation is that the DNS "downstream" CP *is* the Milo root CP; if it is a separate infra CP the Milo provider does not engage, discovery must be extended to include it. - -The policy (`config/observability/dns-metrics-policy.yaml`) defines two series: - -| Series | Emitted on | Join labels | -|---|---|---| -| `dns_recordset_upstream_info` | project CPs | `upstream_namespace`, `upstream_name` + Milo project label | -| `dns_recordset_downstream_info` | root CP | `upstream_cluster`, `upstream_namespace`, `upstream_name` (from annotations) | - -The downstream generator lifts the `upstream-*` annotations onto labels — this is the join key and the reason no operator code is needed. - -### Detection: recording + alerting rules - -`config/observability/dns-drift-prometheusrule.yaml`: - -- `dns:recordset_downstream_orphan` = `downstream unless on(join keys) upstream` → orphan (the #346 case). -- `dns:recordset_downstream_missing` = `upstream unless on(join keys) downstream` → not replicated. -- Alerts fire with `for: 10m` to ride out normal replication + OTLP push/staleness lag (avoids flapping during healthy eventual-consistency windows). - -**Label normalization (verified live):** the two sides label the source project differently, so the rules normalize both to a bare `proj` join key: -- Downstream `upstream_cluster` = `cluster-` + `replace(, "/", "_")`. The Milo provider keys a project cluster as `/alpha`, so the annotation is **`cluster-_alpha`** (note the underscore). -- Upstream is tagged by `resource-metrics` as `milo_project_name` = the bare name (`alpha`), and `milo_control_plane_type` = `project` vs `root`. - -The recording rules strip `cluster-_?` from the downstream label and use `milo_project_name` directly for the upstream, joining on `(proj, upstream_namespace, upstream_name)`. (An earlier assumption that both sides read `cluster-` was wrong — caught by live e2e, where the matched pair was falsely flagged as both orphan and missing until the normalization was fixed.) - -### Limitations / complementary work - -- **PowerDNS seam is invisible.** A leak purely downstream→PDNS (downstream object deleted, RRset leaked) won't show here. Keep a `dns_pdns_apply_errors_total{reason="conflict"}` counter in `internal/pdns` for the direct 422 symptom, and consider a PDNS RRset exporter that diffs against downstream objects. -- **Root cause (shard ownership) not directly observed.** Optionally add a `dns_replicator_cluster_reconcile_total{pod,cluster}` metric to alert when >1 pod reconciles one cluster — an early warning ahead of any drift. - -## E2E plan: Milo-based, build on PR #60 - -[PR #60](https://github.com/datum-cloud/dns-operator/pull/60) (`feat/dns-federation-test-env`) already stands up the topology and harness we need. It brings up **three** `datum-cloud/test-infra` kind clusters via a `Taskfile.yaml` (remote test-infra include, the same pattern `milo-os/activity` and `resource-metrics` use): +- [`milo-os/resource-metrics`](https://github.com/datum-cloud/resource-metrics) emits one metric per `DNSRecordSet`/`DNSZone` from every customer project and from the serving control plane — no new code in the DNS operator. +- Recording rules compare the two sides. A serving-side record with no matching project record is an **orphan**; a project record with no matching serving-side record is **missing**. +- Rules ship as a standard Prometheus `PrometheusRule` and evaluate in Victoria Metrics. -- **`dns-upstream`** — runs the **replicator** (`config/overlays/replicator`), pushes DNSZone/DNSRecordSet to control. -- **`dns-control`** — dns-operator agent + PowerDNS + RustFS; plays the **downstream** role for the replicator and the Lightningstream source for edge. -- **`dns-edge`** — PowerDNS only; receives via Lightningstream. +## What's included -The `dns-upstream → dns-control` hop **is the #346 seam.** #60 also already wires multi-cluster Chainsaw: `env:chainsaw-prepare-kubeconfigs` exports `kubeconfig-{upstream,control,edge}`, and `env:chainsaw` runs every suite under `test/e2e/`. So drift detection is almost purely additive on top of #60. +- `config/observability/` — the `dns-metrics` metrics policy and the `dns-controlplane-drift` alert/recording rules. +- `config/dependencies/` and `config/overlays/` — the environment used to validate it end-to-end. +- `test/e2e/controlplane-drift/` — automated tests for all three cases (healthy, orphan, missing). -### What #60 gives us for free +## Try it -- Task + remote `test-infra` harness, three-cluster bring-up/tear-down (`env:up` / `env:stack-up` / `env:down`). -- Cross-cluster addressing pattern (NodePort on the control-plane container) already used for RustFS and the replicator's downstream kubeconfig — reuse it to point OTLP at the collector. -- The pinned test-infra ref already ships `install-observability` — **Victoria Metrics + OTel Collector + the Prometheus-operator CRDs including `PrometheusRule`** (`prometheusrules.monitoring.coreos.com`). It's marked optional and #60 doesn't invoke it yet; we just call it. -- Multi-cluster Chainsaw suites addressing clusters by name, plus a `test/e2e/federation` suite to model the new one on. +Against a local Kubernetes (kind) setup: -### New work (additive to #60) - -1. **`config/dependencies/resource-metrics/`** — kustomize/Flux to deploy the `resource-metrics` controller in `discovery.mode: single` (default mode; watches its own cluster's API). Instantiate it on **both** `dns-upstream` and `dns-control`, each pushing OTLP to the OTel collector running on `dns-control` (cross-cluster via NodePort, same as RustFS). In single mode there is no auto project label, so set a static `upstream_cluster` resource attribute per instance so the join keys line up. -2. **Taskfile additions:** `env:observability-up` (call `test-infra:install-observability` on `dns-control`), `env:metrics-up` (deploy the two resource-metrics instances + `kubectl apply -k config/observability`). Fold both into `env:stack-up` behind a flag so the default federation flow stays lean. -3. **New Chainsaw suite `test/e2e/controlplane-drift/`** (added to `env:chainsaw`, addressing `upstream` + `control` by the already-exported kubeconfigs, querying VM's HTTP API): - - **Happy path:** create a DNSRecordSet on `dns-upstream` → replicator copies to `dns-control` → assert both `*_upstream_info` and `*_downstream_info` series exist in VM and `dns:recordset_downstream_orphan == 0`. - - **Orphan / #346 regression:** create the record, let it replicate, then delete the upstream object while preventing GC cascade (scale the replicator to 0, or drop the downstream owner/finalizer) → poll VM until `dns:recordset_downstream_orphan > 0` and assert `DNSDownstreamOrphanRecordSet` enters `firing`; restore and assert it clears. - - **Missing:** scale the replicator to 0, create an upstream object → assert `dns:recordset_downstream_missing > 0` / `DNSDownstreamMissingRecordSet` fires. -4. **CI:** extend #60's `e2e.yml` to run `env:observability-up` + `env:metrics-up` before the drift suite; keep it a separate job/flag so the base federation e2e stays fast. - -### Milo topology (production-accurate) - -We run the **real production path** so the e2e reproduces the #346 root cause, not just an injected orphan. `milo-apiserver` runs on the control cluster; the downstream is the Milo **core control plane**; a single `resource-metrics` in `mode: milo` + `collectRootControlPlane: true` collects both sides — exactly as production would. - -| Cluster (#60) | Role in Phase B | -|---|---| -| `dns-control` | Hosts `milo-apiserver` + `milo-controller-manager` (`--control-plane-scope=core`). Serves the **Milo core CP (downstream)** and **≥2 project CPs (upstream)** via aggregation. Also runs the PowerDNS agent (reading DNSRecordSets from the **core CP**), RustFS, the observability stack (OTel + VM), and the single `resource-metrics` controller. | -| `dns-upstream` | Hosts the **replicator** process only, `discovery.mode: milo`, **2 replicas, leader election off**, discovery + downstream both pointed at `milo-apiserver` on control. (The name is now a slight misnomer — upstream *data* lives in Milo project CPs, not this kind cluster.) | -| `dns-edge` | Unchanged from #60: PowerDNS + Lightningstream from control. | - -This rewires #60's control cluster in two ways: the **replicator's downstream target** and the **PowerDNS agent's read source** both move from the plain `dns-control` kind API to the Milo core CP (served by `milo-apiserver`). - -### Milo deployment — reuse resource-metrics' proven setup - -Lifted from `resource-metrics` (verified in its e2e): -- `config/dependencies/milo/` — Flux `OCIRepository` on `oci://ghcr.io/datum-cloud/milo-kustomize` (pin a tag) → `overlays/test-infra`, plus the separate `milo-infra-crds` Kustomization that installs the `ProjectControlPlane` CRD (the core controller crash-loops without it). Added to this repo as drafts. -- A `milo-kubeconfig` Secret targeting `https://milo-apiserver.milo-system.svc.cluster.local:6443` with the static `test-admin-token` from milo's test-infra overlay (`system:masters` in test only). -- `resource-metrics` server-config: `discovery.mode: milo`, `discoveryKubeconfigPath`/`projectKubeconfigPath` → the milo kubeconfig, `collectRootControlPlane: true`. -- `dns-operator` replicator server-config: `discovery.mode: milo`, same discovery/project kubeconfig, downstream client pointed at the Milo core CP. - -### Generator scoping — one policy, two control-plane roles - -The single `dns-metrics` policy is applied to all CPs, so **both** the upstream and downstream generators run on every control plane. They are separated by the `milo.project.name` label the Milo provider adds — `"root"` on the core CP, the project name on project CPs. The recording rules therefore filter: -- upstream view: `dns_recordset_upstream_info{milo_project_name!="root"}` -- downstream view: `dns_recordset_downstream_info{milo_project_name="root"}` - -Without this filter, the downstream generator's series on project CPs (empty `upstream_*` labels, annotations absent) and the upstream generator's series on the core CP would pollute the diff. **Verify** the exact promoted label name (`milo_project_name` vs `milo.project.name`) against a real series and adjust both the rules and the filter. - -### Chainsaw suite `test/e2e/controlplane-drift/` - -Added to `env:chainsaw`, addressing Milo project CPs (via aggregation-path kubeconfigs, the pattern `resource-metrics` uses) and querying VM's HTTP API: -- **Happy path:** create a DNSRecordSet on project CP `alpha` → replicator copies it to the core CP → assert both `*_upstream_info{milo_project_name="alpha"}` and `*_downstream_info{milo_project_name="root"}` exist and `dns:recordset_downstream_orphan == 0`. -- **Orphan / #346 regression (injected):** delete the upstream object while preventing GC cascade (scale replicator to 0, or drop the downstream owner/finalizer) → poll VM until `dns:recordset_downstream_orphan > 0` and assert `DNSDownstreamOrphanRecordSet` fires; restore and assert it clears. -- **Missing:** scale replicator to 0, create an upstream object → assert `dns:recordset_downstream_missing > 0`. -- **Genuine sharding repro (the reason for Phase B):** pin the **pre-fix** `multicluster-runtime` (before [datum-cloud/network-services-operator#320](https://github.com/datum-cloud/network-services-operator/pull/320) / [kubernetes-sigs/multicluster-runtime#173](https://github.com/kubernetes-sigs/multicluster-runtime/pull/173)), run **2 replicas across ≥2 project CPs**, exercise a create/delete cycle, and assert the drift alert fires from the real bug — every replica reconciling clusters it doesn't own — not an injected orphan. Flip to the fixed fork and assert it stays clean. - -### Taskfile additions (on #60) - -- `env:milo-up` — `kubectl apply -k config/dependencies/milo` on control, wait for the Flux Kustomizations, mint the `milo-kubeconfig`, create Org + ≥2 Projects (project CPs). -- `env:observability-up` — `test-infra:install-observability` on control (VM + OTel + PrometheusRule CRD). -- `env:metrics-up` — deploy the single `resource-metrics` (mode:milo, collectRootControlPlane) + `kubectl apply -k config/observability`. -- Rework `env:upstream-up` → deploy the replicator with the `mode: milo` overlay, 2 replicas, pointed at milo on control. -- Rework `env:control-up` → agent reads from the Milo core CP. -- Fold into `env:stack-up`; extend #60's `e2e.yml` to run the drift suite as a separate job. - -> Phase B is environment-heavy (Flux reconcile timing, milo token auth, cross-cluster addressing, 2-replica sharding). Expect live-cluster iteration to shake out the wiring; the config drafts here are the starting point, not turnkey. - -## Sequencing - -1. Land `config/observability/*` (policy + rules) — done as drafts in this change. -2. Add `config/dependencies/milo/`, the replicator `mode: milo` overlay + server-config, the `resource-metrics` deploy overlay, and the Taskfile `env:milo-up`/`env:observability-up`/`env:metrics-up` additions on top of PR #60. -3. Add the `test/e2e/controlplane-drift/` suite (happy / orphan / missing / genuine-sharding-repro) and wire it into `env:chainsaw` + CI. -4. Confirm the production downstream CP == Milo core CP; enable `collectRootControlPlane` in the prod `resource-metrics` config; validate series parity in staging for one retention window before wiring alerts to paging. -5. Complementary operator-side signals: PDNS conflict counter, replica-ownership metric. - -## Verified against datum-cloud/infra - -Reading the real staging/production deployment (`datum-cloud/infra`) resolved the assumptions that code alone couldn't: - -- **The Milo core control plane serves `DNSRecordSet`s.** `apps/dns-operator/control-plane/base/core-control-plane-resources.yaml` installs the DNS CRDs *into the core control plane* (via the `milo-configuration` cert, `system:control@services.miloapis.com`). So replicated records land on the core CP and `resource-metrics`' root collection can see them — the previously live-only caveat, now confirmed. -- **The replicator already runs `mode: milo` in prod** (`apps/dns-operator/control-plane/staging/config.yaml`) with discovery + project kubeconfigs, and `downstreamResourceManagement` **commented out** so it writes downstream via the pod's in-cluster SA — i.e. to the core control plane it runs on. Replicator (`control-plane/`) and agent (`downstream/`) both deploy to the same infra cluster (`datum-dns-system`). -- **`resource-metrics` is deployed but `collectRootControlPlane` is NOT set** (`apps/resource-metrics-system/{staging,production}/config.yaml` have only `mode: milo`). So the downstream/core-CP series this design needs are **not currently emitted** — enabling `collectRootControlPlane: true` is a required, net-new infra change, not just new rules. -- **Policies target the Milo core CP** — `resource-metrics` installs its CRDs into the core control plane (`apps/resource-metrics-system/base/milo-control-plane.yaml`), so the `dns-metrics` `ResourceMetricsPolicy` is applied there, not to a plain cluster. -- **Alert/recording rules stay a standard `PrometheusRule`.** The victoria-metrics-operator auto-converts `PrometheusRule` → `VMRule` (carrying labels through), so no VM-specific CRD is needed. What matters is the label `telemetry.miloapis.com/resource-metrics-aggregator`, which routes the rules to the dedicated `vmalert-datum-resource-metrics-aggregator` (the general vmalert selects the complement — that label `DoesNotExist`). `config/observability/dns-drift-rules.yaml` reflects this. +```sh +export TASK_X_REMOTE_TASKFILES=1 +task env:milo-all-up # bring up the full DNS platform + metrics + alerting +task env:chainsaw-milo # run the drift-detection tests (healthy / orphan / missing) +``` -Implication for the e2e topology: prod co-locates the replicator on the core/Milo cluster (no separate upstream cluster; "upstream" = Milo project CPs). The e2e should therefore run the replicator on `dns-control` alongside Milo (downstream via in-cluster SA), rather than on a separate `dns-upstream` kind cluster — simpler and more faithful than the #60-derived split. +The orphan test reproduces #346 end-to-end: replicate a record, break replication, delete the customer's record, and watch the orphan alert fire on the leftover — then clear once it's removed. -## Open questions +## Enabling in staging / production -- Enabling `collectRootControlPlane: true` in the infra `resource-metrics` config: confirm it doesn't balloon cardinality (it adds the whole core CP's objects) and that the root series carry `milo_control_plane_type="root"` as expected. -- Victoria Metrics staleness window vs. the `for: 10m` alert delay — tune together so orphans are neither missed nor flapped. -- Prod auth for the policy/agent paths uses cert-based `milo-configuration-kubeconfig` (not the e2e's static `test-admin-token`); keep the e2e's test-only creds out of any shared overlay. +The metrics and rules are additive, but the serving-side metrics require one platform change: set `discovery.collectRootControlPlane: true` in the `resource-metrics` config (it collects only customer projects today). Validate the new series in staging for one metrics-retention window before wiring the alerts to paging. From 99d9f69a226a478c8122cff4a2743c4139761c50 Mon Sep 17 00:00:00 2001 From: Scot Wells Date: Sat, 25 Jul 2026 16:27:00 -0500 Subject: [PATCH 3/9] refactor: Deploy resource-metrics from its published bundle, not vendored manifests MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Replace the vendored resource-metrics controller manifests (namespace, RBAC, Deployment, server-config, milo-kubeconfig Secret) with a Flux OCIRepository + Kustomization referencing the operator's published kustomize bundle (oci://ghcr.io/milo-os/resource-metrics-kustomize, overlays/test-infra) — the same pattern config/dependencies/milo already uses for milo. That overlay already provides the Deployment, RBAC, namespace, the token-based milo-kubeconfig Secret, and a mode:milo + collectRootControlPlane:true server-config, so we only override the controller image tag and the OTLP endpoint (our observability stack's collector lives in telemetry-system, not the overlay's default otel-collector-system). This drops the vendored manifests that would drift as the operator evolves and removes the local image build — the pinned bundle ships a multi-arch (amd64+arm64) image. The ResourceMetricsPolicy CRD + dns-metrics policy still install onto the Milo core CP unchanged. env:metrics-up now deploys via Flux and waits on the Kustomization instead of loading a local image. Verified live: the controller comes up on the published arm64 image, picks up the telemetry-system endpoint, and collects root + project control planes with no export errors. Once milo-os/resource-metrics#13 (multi-arch) and #14 (configurable OTLP endpoint) land in main, the pin bumps to a v0.0.0-main tag and the endpoint patch collapses to an OTEL_EXPORTER_OTLP_ENDPOINT env patch. Co-Authored-By: Claude Opus 4.8 --- Taskfile.yaml | 11 +-- .../dependencies/resource-metrics/README.md | 34 +++---- .../controller/deployment.yaml | 97 ------------------- .../controller/flux-install.yaml | 51 ++++++++++ .../controller/kustomization.yaml | 39 ++------ .../controller/milo-kubeconfig-secret.yaml | 40 -------- .../controller/namespace.yaml | 8 -- .../controller/ocirepository.yaml | 14 +++ .../resource-metrics/controller/rbac.yaml | 75 -------------- .../controller/server-config.yaml | 36 ------- 10 files changed, 93 insertions(+), 312 deletions(-) delete mode 100644 config/dependencies/resource-metrics/controller/deployment.yaml create mode 100644 config/dependencies/resource-metrics/controller/flux-install.yaml delete mode 100644 config/dependencies/resource-metrics/controller/milo-kubeconfig-secret.yaml delete mode 100644 config/dependencies/resource-metrics/controller/namespace.yaml create mode 100644 config/dependencies/resource-metrics/controller/ocirepository.yaml delete mode 100644 config/dependencies/resource-metrics/controller/rbac.yaml delete mode 100644 config/dependencies/resource-metrics/controller/server-config.yaml diff --git a/Taskfile.yaml b/Taskfile.yaml index f9b1afe..59e24ed 100644 --- a/Taskfile.yaml +++ b/Taskfile.yaml @@ -38,10 +38,6 @@ vars: # dns-control, via the envoy gateway NodePort on the control node's IP. MILO_UPSTREAM_KUBECONFIG: 'dev/milo.upstream.kubeconfig' MILO_GATEWAY_NODEPORT: '32648' - # resource-metrics controller image (loaded into the dns-control kind node). - # Published image is amd64-only (see milo-os/resource-metrics#13); on arm64 - # build locally: `docker build -t resource-metrics:arm64-dev `. - RM_IMG: 'resource-metrics:arm64-dev' # Dedicated kubeconfig for this environment (not the user's default # ~/.kube/config). go-task's `env:` blocks are implemented as process-wide # os.Setenv calls that leak between sibling task/cmd invocations (a known @@ -393,7 +389,6 @@ tasks: env:metrics-up: desc: "Deploy resource-metrics (mode:milo + collectRootControlPlane) + the dns-metrics policy + drift rules" cmds: - - kind load docker-image {{.RM_IMG}} --name {{.CONTROL_CLUSTER_NAME}} # ResourceMetricsPolicy CRD + the dns-metrics policy live on the Milo core CP - task: env:with-milo-admin vars: @@ -404,8 +399,12 @@ tasks: - task: env:with-milo-admin vars: CMD: kubectl apply -f config/observability/dns-metrics-policy.yaml - # Controller runs on the control kind cluster; drift rules load into VM + # Controller: reference resource-metrics' published overlays/test-infra + # bundle via Flux (OCIRepository + Kustomization) — no vendored manifests, + # no local image build (the pinned bundle ships a multi-arch image). - KUBECONFIG={{.ENV_KUBECONFIG}} kubectl --context kind-{{.CONTROL_CLUSTER_NAME}} apply -k config/dependencies/resource-metrics/controller + - KUBECONFIG={{.ENV_KUBECONFIG}} kubectl --context kind-{{.CONTROL_CLUSTER_NAME}} -n flux-system wait ocirepository/resource-metrics --for=condition=Ready --timeout=120s + - KUBECONFIG={{.ENV_KUBECONFIG}} kubectl --context kind-{{.CONTROL_CLUSTER_NAME}} -n flux-system wait kustomization/resource-metrics --for=condition=Ready --timeout=300s - KUBECONFIG={{.ENV_KUBECONFIG}} kubectl --context kind-{{.CONTROL_CLUSTER_NAME}} -n resource-metrics-system rollout status deploy/resource-metrics-controller-manager --timeout=150s - KUBECONFIG={{.ENV_KUBECONFIG}} kubectl --context kind-{{.CONTROL_CLUSTER_NAME}} -n telemetry-system apply -f config/observability/dns-drift-rules.yaml - echo "✅ resource-metrics + dns-metrics policy + drift rules deployed" diff --git a/config/dependencies/resource-metrics/README.md b/config/dependencies/resource-metrics/README.md index 0bbde2a..61ed759 100644 --- a/config/dependencies/resource-metrics/README.md +++ b/config/dependencies/resource-metrics/README.md @@ -36,31 +36,31 @@ targets**. They are applied separately and never combined into a single ### `controller/` — applied to the KIND (dns-control) context -The controller Deployment, its namespace, RBAC, the `milo-kubeconfig` Secret, -and the `mode: milo` server-config. Apply with the **kind kubeconfig/context** -for dns-control. The controller itself then talks to the Milo core CP through -the mounted `milo-kubeconfig` Secret. +Rather than vendoring the controller manifests, this **references the operator's +published kustomize bundle** (`oci://ghcr.io/milo-os/resource-metrics-kustomize`, +`overlays/test-infra`) via Flux — the same pattern `config/dependencies/milo` +uses. That overlay already ships the Deployment, RBAC, namespace, the +`milo-kubeconfig` Secret, and a `mode: milo` + `collectRootControlPlane: true` +server-config, so we only override two things. | File | Purpose | | --- | --- | -| `namespace.yaml` | `resource-metrics-system` namespace. | -| `rbac.yaml` | ServiceAccount + ClusterRole + ClusterRoleBinding (vendored from the operator's `controller_rbac` component). | -| `milo-kubeconfig-secret.yaml` | Kubeconfig Secret → in-cluster `milo-apiserver` + `test-admin-token`, `insecure-skip-tls-verify`. Modeled on the operator's `overlays/test-infra/milo-kubeconfig-secret.yaml`. **Test-only** credential. | -| `server-config.yaml` | `ResourceMetricsOperator` config: `discovery.mode: milo`, `discoveryKubeconfigPath`/`projectKubeconfigPath` → `/etc/milo/kubeconfig`, and `discovery.collectRootControlPlane: true` (so the root CP is collected as cluster `root`). Also the OTLP endpoint. Rendered into the `resource-metrics-service-config` ConfigMap. | -| `deployment.yaml` | Controller Deployment (base manager + test-infra patch folded in: `KUBECONFIG` env, milo-kubeconfig mount at `/etc/milo`, `--server-config=/etc/resource-metrics/server.yaml`, `imagePullPolicy: IfNotPresent`, no `--leader-elect`). | -| `kustomization.yaml` | Ties the above together; `images:` override for the controller image; `configMapGenerator` for the server-config. | +| `ocirepository.yaml` | Flux `OCIRepository` on `resource-metrics-kustomize`, pinned to a bundle tag that ships a multi-arch image. | +| `flux-install.yaml` | Flux `Kustomization` on `overlays/test-infra` with two overrides: `images:` (the controller image tag) and a patch pointing the OTLP endpoint at our collector (`telemetry-system`, not the overlay default `otel-collector-system`). | +| `kustomization.yaml` | Applies the two Flux resources into `flux-system`. | ```sh -# Uses the dns-control kind context. -kustomize build config/dependencies/resource-metrics/controller \ - | kubectl --context kind-dns-control apply -f - +kubectl --context kind-dns-control apply -k config/dependencies/resource-metrics/controller +kubectl --context kind-dns-control -n flux-system wait kustomization/resource-metrics --for=condition=Ready --timeout=300s ``` > [!NOTE] -> Override the controller image before applying if you are not using -> `ghcr.io/milo-os/resource-metrics:latest`, e.g. -> `kustomize edit set image ghcr.io/milo-os/resource-metrics=ghcr.io/milo-os/resource-metrics:` -> or load a `dev` image into kind and set `newTag: dev` in `kustomization.yaml`. +> The OTLP-endpoint override is a full-ConfigMap patch because the pinned bundle +> hardcodes the endpoint. Once [milo-os/resource-metrics#14](https://github.com/milo-os/resource-metrics/pull/14) +> (configurable endpoint) and [#13](https://github.com/milo-os/resource-metrics/pull/13) +> (multi-arch image) land in `main`, bump `ocirepository.yaml` to a `v0.0.0-main` +> tag and replace the patch with a Deployment env patch: +> `OTEL_EXPORTER_OTLP_ENDPOINT=otel-collector-collector.telemetry-system:4317`. ### `core-control-plane/` — applied with a MILO kubeconfig diff --git a/config/dependencies/resource-metrics/controller/deployment.yaml b/config/dependencies/resource-metrics/controller/deployment.yaml deleted file mode 100644 index 863ed23..0000000 --- a/config/dependencies/resource-metrics/controller/deployment.yaml +++ /dev/null @@ -1,97 +0,0 @@ -# resource-metrics controller Deployment for the DNS drift-detection e2e on the -# dns-control kind cluster. Vendored from milo-os/resource-metrics -# config/manager/manager.yaml with the config/overlays/test-infra -# deployment-patch folded in (this repo can't reference the operator's relative -# kustomize bases, so the merged result is inlined here). -# -# Key test-infra wiring: -# * KUBECONFIG env -> /etc/milo/kubeconfig so ctrl.GetConfigOrDie() (the local -# cluster handed to the multicluster manager) targets milo-apiserver, not the -# kind apiserver. The controller's real control plane (CRD, policies, leader -# lease) lives on milo. -# * --server-config=/etc/resource-metrics/server.yaml, backed by the -# resource-metrics-service-config ConfigMap (see kustomization.yaml). -# * milo-kubeconfig Secret mounted at /etc/milo for discovery + per-project -# clients (matches discovery{,project}KubeconfigPath in server-config.yaml). -# * --leader-elect omitted: single replica; the lease would otherwise need a -# namespace on milo that controller-runtime can't default to. -# * imagePullPolicy IfNotPresent so a kind-loaded image is used when present. -apiVersion: apps/v1 -kind: Deployment -metadata: - name: resource-metrics-controller-manager - namespace: resource-metrics-system - labels: - control-plane: controller-manager - app.kubernetes.io/name: resource-metrics - app.kubernetes.io/managed-by: kustomize -spec: - replicas: 1 - selector: - matchLabels: - control-plane: controller-manager - app.kubernetes.io/name: resource-metrics - template: - metadata: - annotations: - kubectl.kubernetes.io/default-container: manager - labels: - control-plane: controller-manager - app.kubernetes.io/name: resource-metrics - spec: - securityContext: - runAsNonRoot: true - seccompProfile: - type: RuntimeDefault - serviceAccountName: resource-metrics-controller-manager - terminationGracePeriodSeconds: 10 - containers: - - name: manager - command: - - /manager - args: - - --health-probe-bind-address=:8081 - - --server-config=/etc/resource-metrics/server.yaml - image: ghcr.io/milo-os/resource-metrics:latest - imagePullPolicy: IfNotPresent - env: - - name: KUBECONFIG - value: /etc/milo/kubeconfig - securityContext: - allowPrivilegeEscalation: false - capabilities: - drop: - - "ALL" - livenessProbe: - httpGet: - path: /healthz - port: 8081 - initialDelaySeconds: 15 - periodSeconds: 20 - readinessProbe: - httpGet: - path: /readyz - port: 8081 - initialDelaySeconds: 5 - periodSeconds: 10 - resources: - limits: - cpu: 1 - memory: 2Gi - requests: - cpu: 500m - memory: 512Mi - volumeMounts: - - name: service-config - mountPath: /etc/resource-metrics - readOnly: true - - name: milo-kubeconfig - mountPath: /etc/milo - readOnly: true - volumes: - - name: service-config - configMap: - name: resource-metrics-service-config - - name: milo-kubeconfig - secret: - secretName: milo-kubeconfig diff --git a/config/dependencies/resource-metrics/controller/flux-install.yaml b/config/dependencies/resource-metrics/controller/flux-install.yaml new file mode 100644 index 0000000..1193e17 --- /dev/null +++ b/config/dependencies/resource-metrics/controller/flux-install.yaml @@ -0,0 +1,51 @@ +apiVersion: kustomize.toolkit.fluxcd.io/v1 +kind: Kustomization +metadata: + name: resource-metrics + namespace: flux-system +spec: + interval: 10m + retryInterval: 1m + timeout: 5m + prune: true + wait: true + sourceRef: + kind: OCIRepository + name: resource-metrics + # resource-metrics' published overlays/test-infra already does what we need: + # discovery.mode=milo, collectRootControlPlane=true, the token-based + # milo-kubeconfig Secret, and the manager Deployment wiring. We only override + # the controller image (to the multi-arch tag) and the OTLP endpoint (our + # observability stack's collector lives in telemetry-system, not the overlay's + # default otel-collector-system). + path: overlays/test-infra + images: + - name: ghcr.io/milo-os/resource-metrics + newTag: v0.0.0-feat-multi-arch-arm64-image + patches: + # Point the operator at our OTel collector. Once milo-os/resource-metrics#14 + # lands, replace this whole ConfigMap patch with a Deployment env patch: + # env: [{name: OTEL_EXPORTER_OTLP_ENDPOINT, value: otel-collector-collector.telemetry-system:4317}] + - target: + kind: ConfigMap + name: resource-metrics-service-config + patch: | + apiVersion: v1 + kind: ConfigMap + metadata: + name: resource-metrics-service-config + data: + server.yaml: | + apiVersion: apiserver.config.miloapis.com/v1alpha1 + kind: ResourceMetricsOperator + discovery: + mode: milo + internalServiceDiscovery: false + discoveryKubeconfigPath: /etc/milo/kubeconfig + projectKubeconfigPath: /etc/milo/kubeconfig + collectRootControlPlane: true + otel: + endpoint: otel-collector-collector.telemetry-system:4317 + insecure: true + collectionInterval: 5s + defaultMetricPrefix: "" diff --git a/config/dependencies/resource-metrics/controller/kustomization.yaml b/config/dependencies/resource-metrics/controller/kustomization.yaml index a970901..1e66e3a 100644 --- a/config/dependencies/resource-metrics/controller/kustomization.yaml +++ b/config/dependencies/resource-metrics/controller/kustomization.yaml @@ -1,36 +1,9 @@ -# Deploys the resource-metrics controller into namespace resource-metrics-system -# on the dns-control kind cluster (the local/kind kubectl context). -# -# Apply with the KIND kubeconfig/context — NOT a milo kubeconfig. The controller -# then reaches the milo core control plane via the mounted milo-kubeconfig -# Secret. The resource-metrics CRD + dns-metrics ResourceMetricsPolicy live on -# the milo core CP and are installed separately (see ../core-control-plane). +# Deploys the resource-metrics controller by referencing its published kustomize +# bundle (overlays/test-infra) via Flux, instead of vendoring its manifests. +# Mirrors config/dependencies/milo. The controller runs on the control cluster +# and reads ResourceMetricsPolicy from the Milo core CP (see ../core-control-plane). apiVersion: kustomize.config.k8s.io/v1beta1 kind: Kustomization - -namespace: resource-metrics-system - resources: - - namespace.yaml - - rbac.yaml - - milo-kubeconfig-secret.yaml - - deployment.yaml - -# Pin/override the controller image. The upstream default tag is `latest`; -# override newTag here (or via `kustomize edit set image`) to the tag published -# for this e2e, or load a `dev` image into kind and set newTag: dev. -images: - - name: ghcr.io/milo-os/resource-metrics - newName: resource-metrics - newTag: arm64-dev - -# server-config.yaml -> mounted at /etc/resource-metrics/server.yaml and passed -# to --server-config. disableNameSuffixHash keeps the ConfigMap name stable so -# the Deployment volume reference resolves without a kustomize name reference. -configMapGenerator: - - name: resource-metrics-service-config - behavior: create - files: - - server.yaml=server-config.yaml - options: - disableNameSuffixHash: true + - ocirepository.yaml + - flux-install.yaml diff --git a/config/dependencies/resource-metrics/controller/milo-kubeconfig-secret.yaml b/config/dependencies/resource-metrics/controller/milo-kubeconfig-secret.yaml deleted file mode 100644 index 8ffb41e..0000000 --- a/config/dependencies/resource-metrics/controller/milo-kubeconfig-secret.yaml +++ /dev/null @@ -1,40 +0,0 @@ -# Kubeconfig the resource-metrics controller uses to reach the milo core -# control plane (the DOWNSTREAM / "root" CP) and to discover the project -# (UPSTREAM) control planes served by milo-apiserver. Modeled on -# milo-os/resource-metrics config/overlays/test-infra/milo-kubeconfig-secret.yaml. -# -# resource-metrics runs ON dns-control alongside milo-apiserver, so it uses the -# in-cluster Service endpoint directly (bypassing the Envoy gateway) — the pod -# serves a self-signed cert on :6443, so TLS verification is skipped. -# -# The token is the static `test-admin-token` wired into milo-apiserver's -# token-auth-file via secret milo-apiserver-auth-tokens (key tokens.csv) in -# namespace milo-system; it grants system:masters in the test environment. -# -# TEST-ONLY: production must use a least-privilege token or mTLS and source -# this Secret from a sealed/external secret store. -apiVersion: v1 -kind: Secret -metadata: - name: milo-kubeconfig - namespace: resource-metrics-system -type: Opaque -stringData: - kubeconfig: | - apiVersion: v1 - kind: Config - clusters: - - name: milo - cluster: - server: https://milo-apiserver.milo-system.svc.cluster.local:6443 - insecure-skip-tls-verify: true - users: - - name: milo-admin - user: - token: test-admin-token - contexts: - - name: milo - context: - cluster: milo - user: milo-admin - current-context: milo diff --git a/config/dependencies/resource-metrics/controller/namespace.yaml b/config/dependencies/resource-metrics/controller/namespace.yaml deleted file mode 100644 index 863bf48..0000000 --- a/config/dependencies/resource-metrics/controller/namespace.yaml +++ /dev/null @@ -1,8 +0,0 @@ -apiVersion: v1 -kind: Namespace -metadata: - name: resource-metrics-system - labels: - control-plane: controller-manager - app.kubernetes.io/name: resource-metrics - app.kubernetes.io/managed-by: kustomize diff --git a/config/dependencies/resource-metrics/controller/ocirepository.yaml b/config/dependencies/resource-metrics/controller/ocirepository.yaml new file mode 100644 index 0000000..6f1bc0e --- /dev/null +++ b/config/dependencies/resource-metrics/controller/ocirepository.yaml @@ -0,0 +1,14 @@ +apiVersion: source.toolkit.fluxcd.io/v1beta2 +kind: OCIRepository +metadata: + name: resource-metrics + namespace: flux-system +spec: + interval: 5m + url: oci://ghcr.io/milo-os/resource-metrics-kustomize + # Pinned to a branch bundle that ships a multi-arch (amd64+arm64) controller + # image (milo-os/resource-metrics#13). Bump to a v0.0.0-main tag once #13 and + # #14 (configurable OTLP endpoint) merge — at which point the endpoint patch + # below collapses to an OTEL_EXPORTER_OTLP_ENDPOINT env patch. + ref: + tag: v0.0.0-feat-multi-arch-arm64-image diff --git a/config/dependencies/resource-metrics/controller/rbac.yaml b/config/dependencies/resource-metrics/controller/rbac.yaml deleted file mode 100644 index 78fc04b..0000000 --- a/config/dependencies/resource-metrics/controller/rbac.yaml +++ /dev/null @@ -1,75 +0,0 @@ -# Cluster-scoped RBAC for the resource-metrics controller, vendored from -# milo-os/resource-metrics config/components/controller_rbac (service_account + -# role + role_binding). These grants apply on whatever API server the -# ServiceAccount authenticates against, but note: in this deployment the -# controller talks to the milo core control plane via the mounted -# milo-kubeconfig (system:masters test token), so the effective permissions -# there come from that token, not this ServiceAccount. This RBAC is retained -# for correctness/parity with the upstream bundle and for any in-kind API -# access (health/leader machinery). ---- -apiVersion: v1 -kind: ServiceAccount -metadata: - name: resource-metrics-controller-manager - namespace: resource-metrics-system - labels: - app.kubernetes.io/name: resource-metrics - app.kubernetes.io/managed-by: kustomize ---- -apiVersion: rbac.authorization.k8s.io/v1 -kind: ClusterRole -metadata: - name: resource-metrics-manager-role - labels: - app.kubernetes.io/name: resource-metrics - app.kubernetes.io/managed-by: kustomize -rules: -- apiGroups: - - authorization.k8s.io - resources: - - selfsubjectaccessreviews - verbs: - - create -- apiGroups: - - resourcemetrics.miloapis.com - resources: - - resourcemetricspolicies - verbs: - - create - - delete - - get - - list - - patch - - update - - watch -- apiGroups: - - resourcemetrics.miloapis.com - resources: - - resourcemetricspolicies/finalizers - verbs: - - update -- apiGroups: - - resourcemetrics.miloapis.com - resources: - - resourcemetricspolicies/status - verbs: - - get - - patch - - update ---- -apiVersion: rbac.authorization.k8s.io/v1 -kind: ClusterRoleBinding -metadata: - name: resource-metrics-manager-rolebinding - labels: - app.kubernetes.io/name: resource-metrics - app.kubernetes.io/managed-by: kustomize -roleRef: - apiGroup: rbac.authorization.k8s.io - kind: ClusterRole - name: resource-metrics-manager-role -subjects: -- kind: ServiceAccount - name: resource-metrics-controller-manager - namespace: resource-metrics-system diff --git a/config/dependencies/resource-metrics/controller/server-config.yaml b/config/dependencies/resource-metrics/controller/server-config.yaml deleted file mode 100644 index 13d32f0..0000000 --- a/config/dependencies/resource-metrics/controller/server-config.yaml +++ /dev/null @@ -1,36 +0,0 @@ -# Operator configuration consumed by the resource-metrics manager via the -# --server-config flag. Substituted into the `resource-metrics-service-config` -# ConfigMap by configMapGenerator (keyed server.yaml) in kustomization.yaml. -# -# apiVersion/kind and field names come from the operator's internal/config -# package (ResourceMetricsOperator, DiscoveryConfig, OtelConfig). -apiVersion: apiserver.config.miloapis.com/v1alpha1 -kind: ResourceMetricsOperator -discovery: - # milo mode: discover project (upstream) control planes through the milo - # multi-cluster provider, using the mounted milo kubeconfig for both the - # discovery client and per-project client construction. Without these paths - # the controller falls back to the in-cluster ServiceAccount (the kind - # apiserver), which does not serve resourcemanager.miloapis.com — so Project - # discovery would find nothing. - mode: milo - internalServiceDiscovery: false - discoveryKubeconfigPath: /etc/milo/kubeconfig - projectKubeconfigPath: /etc/milo/kubeconfig - # CRITICAL for drift detection: also collect the milo core/root control plane - # (the DOWNSTREAM, where the replicator writes the replicated copies) as - # cluster name "root". The upstream project CPs supply - # dns_recordset_upstream_info; the root CP supplies - # dns_recordset_downstream_info. Recording rules diff the two. - collectRootControlPlane: true -otel: - # OTLP gRPC endpoint of the OTel collector deployed by test-infra's - # `install-observability` task. That task applies an OpenTelemetryCollector - # CR named `otel-collector` in namespace `telemetry-system`; the OTel - # Operator renders a Service `-collector` (otel-collector-collector) - # and the CR's receivers.otlp.protocols.grpc listens on :4317. - # See README.md — confirm live if the collector name/namespace changes. - endpoint: otel-collector-collector.telemetry-system.svc.cluster.local:4317 - insecure: true - collectionInterval: 5s - defaultMetricPrefix: "" From 638d72b814c310feedeef39850d88b59576325ad Mon Sep 17 00:00:00 2001 From: Scot Wells Date: Sat, 25 Jul 2026 16:35:59 -0500 Subject: [PATCH 4/9] refactor: Install ResourceMetricsPolicy CRD from the upstream bundle too MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Stop vendoring the ResourceMetricsPolicy CRD. Replace it with a Flux Kustomization that installs the CRD onto the Milo core control plane straight from the operator's published bundle (path: crd) via kubeConfig.secretRef — mirroring infra apps/resource-metrics-system/base/milo-control-plane.yaml. The Flux object lives on the local cluster and targets the core CP through a milo-kubeconfig Secret in flux-system (in-cluster milo-apiserver + the test-only test-admin-token; production uses a cert-based kubeconfig). This was the one piece still vendored after moving the controller to the bundle; now the entire resource-metrics deployment (controller + CRD) comes from upstream, and only the dns-operator-owned dns-metrics policy is applied directly. env:metrics-up applies both Flux Kustomizations on the local cluster and waits on the CRD one before applying the policy. Verified live: the CRD Kustomization reconciles Ready ("Applied revision: @sha256…") and the CRD on the core CP is now Flux-managed (kustomize.toolkit.fluxcd.io/name=resource-metrics-crd). Co-Authored-By: Claude Opus 4.8 --- Taskfile.yaml | 32 +- .../dependencies/resource-metrics/README.md | 36 +-- .../core-control-plane/crd/flux-install.yaml | 23 ++ .../core-control-plane/crd/kustomization.yaml | 17 +- .../crd/milo-kubeconfig-secret.yaml | 31 ++ .../crd/resourcemetricspolicies-crd.yaml | 297 ------------------ 6 files changed, 88 insertions(+), 348 deletions(-) create mode 100644 config/dependencies/resource-metrics/core-control-plane/crd/flux-install.yaml create mode 100644 config/dependencies/resource-metrics/core-control-plane/crd/milo-kubeconfig-secret.yaml delete mode 100644 config/dependencies/resource-metrics/core-control-plane/crd/resourcemetricspolicies-crd.yaml diff --git a/Taskfile.yaml b/Taskfile.yaml index 59e24ed..fad340b 100644 --- a/Taskfile.yaml +++ b/Taskfile.yaml @@ -388,25 +388,27 @@ tasks: env:metrics-up: desc: "Deploy resource-metrics (mode:milo + collectRootControlPlane) + the dns-metrics policy + drift rules" + vars: + KCTL: KUBECONFIG={{.ENV_KUBECONFIG}} kubectl --context kind-{{.CONTROL_CLUSTER_NAME}} cmds: - # ResourceMetricsPolicy CRD + the dns-metrics policy live on the Milo core CP - - task: env:with-milo-admin - vars: - CMD: kubectl apply -k config/dependencies/resource-metrics/core-control-plane/crd - - task: env:with-milo-admin - vars: - CMD: kubectl wait --for=condition=Established crd/resourcemetricspolicies.resourcemetrics.miloapis.com --timeout=60s + # All Flux objects live on the LOCAL (dns-control) cluster. The controller + # Kustomization deploys onto it; the CRD Kustomization targets the Milo + # core CP via kubeConfig. Both reference the same OCIRepository, and both + # come from resource-metrics' published bundle (no vendored manifests, no + # local image build — the pinned bundle ships a multi-arch image). + - "{{.KCTL}} apply -k config/dependencies/resource-metrics/controller" + - "{{.KCTL}} apply -k config/dependencies/resource-metrics/core-control-plane/crd" + - "{{.KCTL}} -n flux-system wait ocirepository/resource-metrics --for=condition=Ready --timeout=120s" + # Installs the ResourceMetricsPolicy CRD onto the Milo core CP from the bundle. + - "{{.KCTL}} -n flux-system wait kustomization/resource-metrics-crd --for=condition=Ready --timeout=180s" + # The dns-metrics policy is dns-operator-owned (not in the bundle), so it's + # applied directly to the core CP after its CRD is Established. - task: env:with-milo-admin vars: CMD: kubectl apply -f config/observability/dns-metrics-policy.yaml - # Controller: reference resource-metrics' published overlays/test-infra - # bundle via Flux (OCIRepository + Kustomization) — no vendored manifests, - # no local image build (the pinned bundle ships a multi-arch image). - - KUBECONFIG={{.ENV_KUBECONFIG}} kubectl --context kind-{{.CONTROL_CLUSTER_NAME}} apply -k config/dependencies/resource-metrics/controller - - KUBECONFIG={{.ENV_KUBECONFIG}} kubectl --context kind-{{.CONTROL_CLUSTER_NAME}} -n flux-system wait ocirepository/resource-metrics --for=condition=Ready --timeout=120s - - KUBECONFIG={{.ENV_KUBECONFIG}} kubectl --context kind-{{.CONTROL_CLUSTER_NAME}} -n flux-system wait kustomization/resource-metrics --for=condition=Ready --timeout=300s - - KUBECONFIG={{.ENV_KUBECONFIG}} kubectl --context kind-{{.CONTROL_CLUSTER_NAME}} -n resource-metrics-system rollout status deploy/resource-metrics-controller-manager --timeout=150s - - KUBECONFIG={{.ENV_KUBECONFIG}} kubectl --context kind-{{.CONTROL_CLUSTER_NAME}} -n telemetry-system apply -f config/observability/dns-drift-rules.yaml + - "{{.KCTL}} -n flux-system wait kustomization/resource-metrics --for=condition=Ready --timeout=300s" + - "{{.KCTL}} -n resource-metrics-system rollout status deploy/resource-metrics-controller-manager --timeout=150s" + - "{{.KCTL}} -n telemetry-system apply -f config/observability/dns-drift-rules.yaml" - echo "✅ resource-metrics + dns-metrics policy + drift rules deployed" env:milo-all-up: diff --git a/config/dependencies/resource-metrics/README.md b/config/dependencies/resource-metrics/README.md index 61ed759..6286f27 100644 --- a/config/dependencies/resource-metrics/README.md +++ b/config/dependencies/resource-metrics/README.md @@ -62,38 +62,26 @@ kubectl --context kind-dns-control -n flux-system wait kustomization/resource-me > tag and replace the patch with a Deployment env patch: > `OTEL_EXPORTER_OTLP_ENDPOINT=otel-collector-collector.telemetry-system:4317`. -### `core-control-plane/` — applied with a MILO kubeconfig +### `core-control-plane/` — installed onto the Milo core control plane -These target the **Milo core control plane** (`milo-apiserver`), NOT the kind -apiserver. Apply them with a milo kubeconfig — the same in-cluster -endpoint + `test-admin-token` used by the controller, or an equivalent -admin kubeconfig. This mirrors milo-os/infra -`apps/resource-metrics-system/base/milo-control-plane.yaml`, which applies the -operator's `crd` path onto the Milo CP. +These land on the **Milo core control plane** (`milo-apiserver`), NOT the kind +apiserver, because that is where the controller reads its policy from. | Path | Purpose | | --- | --- | -| `core-control-plane/crd/` | The `ResourceMetricsPolicy` CRD (`resourcemetrics.miloapis.com`), vendored from the operator's `config/crd/bases`. Must be **Established first**. | -| `core-control-plane/policy/` | Applies the `dns-metrics` `ResourceMetricsPolicy`. It **references** `config/observability/dns-metrics-policy.yaml` (single source of truth) rather than duplicating it — so the build needs `--load-restrictor LoadRestrictionsNone`. | +| `core-control-plane/crd/` | Flux `Kustomization` (applied to the local cluster) that installs the `ResourceMetricsPolicy` CRD onto the core CP **from the published bundle** (`path: crd`) via `kubeConfig.secretRef`. No vendored CRD. Mirrors infra `apps/resource-metrics-system/base/milo-control-plane.yaml`. Includes the milo-kubeconfig Secret Flux targets the core CP with (test-only `test-admin-token`). | +| `core-control-plane/policy/` | Applies the `dns-metrics` `ResourceMetricsPolicy`. This one is dns-operator-owned (not in the bundle), so it **references** `config/observability/dns-metrics-policy.yaml` (single source of truth) and is applied directly to the core CP after the CRD is Established. | ```sh -# Point kubectl at the Milo core control plane. For example, extract the -# controller's kubeconfig from the Secret, or use any admin kubeconfig for -# milo-apiserver. Example using the same Secret the controller mounts: -kubectl --context kind-dns-control -n resource-metrics-system \ - get secret milo-kubeconfig -o jsonpath='{.data.kubeconfig}' \ - | base64 -d > /tmp/milo.kubeconfig - -# 1) Install the CRD and wait for it to be Established. -kustomize build config/dependencies/resource-metrics/core-control-plane/crd \ - | kubectl --kubeconfig /tmp/milo.kubeconfig apply -f - -kubectl --kubeconfig /tmp/milo.kubeconfig wait --for=condition=Established \ - crd/resourcemetricspolicies.resourcemetrics.miloapis.com --timeout=60s - -# 2) Apply the dns-metrics policy. +# 1) Install the CRD onto the core CP from the bundle (Flux objects go on the +# LOCAL cluster; the Kustomization targets the core CP via its kubeConfig). +kubectl --context kind-dns-control apply -k config/dependencies/resource-metrics/core-control-plane/crd +kubectl --context kind-dns-control -n flux-system wait kustomization/resource-metrics-crd --for=condition=Ready --timeout=180s + +# 2) Apply the dns-metrics policy to the core CP (point kubectl at milo-apiserver). kustomize build --load-restrictor LoadRestrictionsNone \ config/dependencies/resource-metrics/core-control-plane/policy \ - | kubectl --kubeconfig /tmp/milo.kubeconfig apply -f - + | kubectl --kubeconfig apply -f - ``` ## OTel endpoint assumption (confirm live) diff --git a/config/dependencies/resource-metrics/core-control-plane/crd/flux-install.yaml b/config/dependencies/resource-metrics/core-control-plane/crd/flux-install.yaml new file mode 100644 index 0000000..95f30a8 --- /dev/null +++ b/config/dependencies/resource-metrics/core-control-plane/crd/flux-install.yaml @@ -0,0 +1,23 @@ +apiVersion: kustomize.toolkit.fluxcd.io/v1 +kind: Kustomization +metadata: + name: resource-metrics-crd + namespace: flux-system +# Installs the ResourceMetricsPolicy CRD onto the Milo CORE control plane +# straight from the operator's published bundle (path crd) — no vendored copy. +# Mirrors infra apps/resource-metrics-system/base/milo-control-plane.yaml. +# Targets the core CP via kubeConfig (a different API server than the local +# cluster this Kustomization object lives on). +spec: + interval: 10m + retryInterval: 1m + timeout: 2m + prune: true + wait: true + sourceRef: + kind: OCIRepository + name: resource-metrics # shared with controller/ocirepository.yaml + path: crd + kubeConfig: + secretRef: + name: resource-metrics-milo-kubeconfig diff --git a/config/dependencies/resource-metrics/core-control-plane/crd/kustomization.yaml b/config/dependencies/resource-metrics/core-control-plane/crd/kustomization.yaml index 6b1821c..47a7783 100644 --- a/config/dependencies/resource-metrics/core-control-plane/crd/kustomization.yaml +++ b/config/dependencies/resource-metrics/core-control-plane/crd/kustomization.yaml @@ -1,15 +1,8 @@ -# Installs the resource-metrics CRD (ResourceMetricsPolicy, -# resourcemetrics.miloapis.com) INTO the Milo core control plane. -# -# TARGET: milo-apiserver (the core/downstream CP) — NOT the kind apiserver. -# Apply with a milo kubeconfig (see ../../README.md). Mirrors the intent of -# milo-os/infra apps/resource-metrics-system/base/milo-control-plane.yaml -# (which applies the operator's `crd` path onto the Milo CP via Flux). -# -# Must be Established before the dns-metrics ResourceMetricsPolicy in ../policy -# is applied. +# Installs the ResourceMetricsPolicy CRD onto the Milo core control plane from +# the published resource-metrics bundle via Flux (see flux-install.yaml). The +# milo-kubeconfig Secret it targets the core CP with is co-located here. apiVersion: kustomize.config.k8s.io/v1beta1 kind: Kustomization - resources: - - resourcemetricspolicies-crd.yaml + - milo-kubeconfig-secret.yaml + - flux-install.yaml diff --git a/config/dependencies/resource-metrics/core-control-plane/crd/milo-kubeconfig-secret.yaml b/config/dependencies/resource-metrics/core-control-plane/crd/milo-kubeconfig-secret.yaml new file mode 100644 index 0000000..50b284a --- /dev/null +++ b/config/dependencies/resource-metrics/core-control-plane/crd/milo-kubeconfig-secret.yaml @@ -0,0 +1,31 @@ +# Kubeconfig Flux uses to install resource-metrics CRDs onto the Milo core +# control plane (a different API server than the local kind cluster). Flux +# resolves kubeConfig.secretRef in the Kustomization's namespace (flux-system) +# and reads the kubeconfig from the `value` key. In-cluster milo-apiserver + +# the test-only static test-admin-token — mirrors infra's milo-configuration- +# kubeconfig, which uses a cert-based kubeconfig in production. +apiVersion: v1 +kind: Secret +metadata: + name: resource-metrics-milo-kubeconfig + namespace: flux-system +type: Opaque +stringData: + value: | + apiVersion: v1 + kind: Config + clusters: + - name: milo + cluster: + server: https://milo-apiserver.milo-system.svc.cluster.local:6443 + insecure-skip-tls-verify: true + users: + - name: milo-admin + user: + token: test-admin-token + contexts: + - name: milo + context: + cluster: milo + user: milo-admin + current-context: milo diff --git a/config/dependencies/resource-metrics/core-control-plane/crd/resourcemetricspolicies-crd.yaml b/config/dependencies/resource-metrics/core-control-plane/crd/resourcemetricspolicies-crd.yaml deleted file mode 100644 index 86e1d5d..0000000 --- a/config/dependencies/resource-metrics/core-control-plane/crd/resourcemetricspolicies-crd.yaml +++ /dev/null @@ -1,297 +0,0 @@ ---- -apiVersion: apiextensions.k8s.io/v1 -kind: CustomResourceDefinition -metadata: - annotations: - controller-gen.kubebuilder.io/version: v0.17.2 - discovery.miloapis.com/parent-contexts: Platform - name: resourcemetricspolicies.resourcemetrics.miloapis.com -spec: - group: resourcemetrics.miloapis.com - names: - kind: ResourceMetricsPolicy - listKind: ResourceMetricsPolicyList - plural: resourcemetricspolicies - singular: resourcemetricspolicy - scope: Cluster - versions: - - additionalPrinterColumns: - - jsonPath: .metadata.creationTimestamp - name: Age - type: date - name: v1alpha1 - schema: - openAPIV3Schema: - description: ResourceMetricsPolicy is the Schema for the resourcemetricspolicies - API. - properties: - apiVersion: - description: |- - APIVersion defines the versioned schema of this representation of an object. - Servers should convert recognized schemas to the latest internal value, and - may reject unrecognized values. - More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources - type: string - kind: - description: |- - Kind is a string value representing the REST resource this object represents. - Servers may infer this from the endpoint the client submits requests to. - Cannot be updated. - In CamelCase. - More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds - type: string - metadata: - type: object - spec: - description: ResourceMetricsPolicySpec defines the desired state of ResourceMetricsPolicy. - properties: - generators: - description: Generators defines the set of resource metric generators. - items: - description: GeneratorSpec defines a single resource metric generator. - properties: - families: - description: Families defines the metric families to emit for - each resource instance. - items: - description: MetricFamilySpec defines a Prometheus metric - family emitted per resource instance. - properties: - help: - description: Help is the help string for this metric family. - type: string - metrics: - description: |- - Metrics defines how to produce individual metric series from each resource. - Metrics in a family have no stable identity, so this list is treated as - atomic for server-side apply. - items: - description: MetricSpec defines a single metric series - within a family. - properties: - forEach: - description: |- - ForEach, when set, is a CEL expression that must evaluate to a list. - The metric is emitted once per element of that list. Within Value and - each label's Value expression, the variable "item" is bound to the - current list element (type dyn). When ForEach is absent, the metric - is emitted once per object (existing behaviour). - maxLength: 4096 - type: string - labels: - description: Labels defines the labels to attach - to this metric series. - items: - description: LabelSpec defines a single label - on a metric series. - properties: - name: - description: Name is the label name. Must - match the Prometheus label name syntax. - maxLength: 253 - pattern: ^[a-zA-Z_][a-zA-Z0-9_]*$ - type: string - value: - description: Value is a CEL expression evaluated - against the resource object. - type: string - required: - - name - - value - type: object - type: array - value: - description: |- - Value is a CEL expression evaluated against the resource object - that produces the metric value. Defaults to 1 if omitted. - type: string - type: object - minItems: 1 - type: array - x-kubernetes-list-type: atomic - name: - description: Name is the base metric name (e.g. "workload_info"). - maxLength: 253 - pattern: ^[a-zA-Z_:][a-zA-Z0-9_:]*$ - type: string - type: - default: gauge - description: Type is the Prometheus metric type. Only - "gauge" is supported in v1. - enum: - - gauge - type: string - required: - - name - type: object - type: array - x-kubernetes-list-map-keys: - - name - x-kubernetes-list-type: map - name: - description: Name is a unique name for this generator. - maxLength: 253 - pattern: ^[a-z][a-z0-9-]*$ - type: string - resource: - description: Resource identifies the Kubernetes API resource - to monitor. - properties: - group: - description: |- - Group is the API group of the resource (e.g. "compute.miloapis.com"). - Empty string targets core resources (configmaps, pods, namespaces, …). - type: string - resource: - description: Resource is the plural resource name (e.g. - "workloads"). - minLength: 1 - type: string - version: - description: Version is the API version (e.g. "v1alpha1"). - minLength: 1 - type: string - required: - - group - - resource - - version - type: object - required: - - name - - resource - type: object - minItems: 1 - type: array - x-kubernetes-list-map-keys: - - name - x-kubernetes-list-type: map - metricNamePrefix: - description: |- - MetricNamePrefix, when set, overrides the controller's - --default-metric-prefix flag for metrics emitted by this policy. - Must start with a letter, underscore, or colon and contain only - letters, digits, underscores, and colons. - maxLength: 32 - pattern: ^[a-zA-Z_:][a-zA-Z0-9_:]*$ - type: string - required: - - generators - type: object - status: - description: ResourceMetricsPolicyStatus defines the observed state of - ResourceMetricsPolicy. - properties: - activeGenerators: - description: |- - ActiveGenerators is the number of generators currently compiled and - actively emitting metrics for this policy. - format: int32 - type: integer - compilationFailures: - description: |- - CompilationFailures is the number of generators that failed to compile - (typically due to invalid CEL). - format: int32 - type: integer - conditions: - description: Conditions represent the latest observations of the resource's - state. - items: - description: Condition contains details for one aspect of the current - state of this API Resource. - properties: - lastTransitionTime: - description: |- - lastTransitionTime is the last time the condition transitioned from one status to another. - This should be when the underlying condition changed. If that is not known, then using the time when the API field changed is acceptable. - format: date-time - type: string - message: - description: |- - message is a human readable message indicating details about the transition. - This may be an empty string. - maxLength: 32768 - type: string - observedGeneration: - description: |- - observedGeneration represents the .metadata.generation that the condition was set based upon. - For instance, if .metadata.generation is currently 12, but the .status.conditions[x].observedGeneration is 9, the condition is out of date - with respect to the current state of the instance. - format: int64 - minimum: 0 - type: integer - reason: - description: |- - reason contains a programmatic identifier indicating the reason for the condition's last transition. - Producers of specific condition types may define expected values and meanings for this field, - and whether the values are considered a guaranteed API. - The value should be a CamelCase string. - This field may not be empty. - maxLength: 1024 - minLength: 1 - pattern: ^[A-Za-z]([A-Za-z0-9_,:]*[A-Za-z0-9_])?$ - type: string - status: - description: status of the condition, one of True, False, Unknown. - enum: - - "True" - - "False" - - Unknown - type: string - type: - description: type of condition in CamelCase or in foo.example.com/CamelCase. - maxLength: 316 - pattern: ^([a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*/)?(([A-Za-z0-9][-A-Za-z0-9_.]*)?[A-Za-z0-9])$ - type: string - required: - - lastTransitionTime - - message - - reason - - status - - type - type: object - type: array - x-kubernetes-list-map-keys: - - type - x-kubernetes-list-type: map - missingPermissions: - description: |- - MissingPermissions lists the GVRs for which the controller lacks the - RBAC permissions required to watch or list on at least one engaged - project control plane. - items: - description: GVRRef identifies a Kubernetes API resource by group, - version, and plural name. - properties: - group: - description: |- - Group is the API group of the resource. - Empty string targets core resources (configmaps, pods, namespaces, …). - type: string - resource: - description: Resource is the plural resource name. - minLength: 1 - type: string - version: - description: Version is the API version. - minLength: 1 - type: string - required: - - group - - resource - - version - type: object - type: array - x-kubernetes-list-type: atomic - observedGeneration: - description: |- - ObservedGeneration reflects the .metadata.generation the controller - has most recently acted upon. - format: int64 - type: integer - type: object - type: object - served: true - storage: true - subresources: - status: {} From c6d332f82405e14677f6d1483fc8af5bb4d4b7df Mon Sep 17 00:00:00 2001 From: Scot Wells Date: Sat, 25 Jul 2026 16:51:45 -0500 Subject: [PATCH 5/9] refactor: Toggle leader election via env var, not by overwriting args MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The replicator-milo overlay disabled leader election by restating the manager container's entire args list minus --leader-elect. Strategic-merge replaces scalar lists wholesale, so that overlay silently drops any flag the base later adds (it was already dropping the metrics/webhook flags other patches contribute). Default the leader-elect flag from a LEADER_ELECT env var (cmd/main.go), set it in the base manager env, and have the overlay patch just that env var — env lists merge by name, so the overlay now changes only what it means to and inherits the rest of the base args. The flag still works and still wins if passed explicitly. Same pattern the agent-powerdns-milo overlay already uses for KUBECONFIG. Verified: go vet passes and the rendered overlay keeps the full base args with LEADER_ELECT=false + the milo mount. Co-Authored-By: Claude Opus 4.8 --- cmd/main.go | 20 +++++++++++++++++-- config/manager/manager.yaml | 6 +++++- .../overlays/replicator-milo/patch-milo.yaml | 16 ++++++++------- 3 files changed, 32 insertions(+), 10 deletions(-) diff --git a/cmd/main.go b/cmd/main.go index eee383d..5a5ed08 100644 --- a/cmd/main.go +++ b/cmd/main.go @@ -9,6 +9,7 @@ import ( "flag" "fmt" "os" + "strconv" "time" // Import all Kubernetes client auth plugins (e.g. Azure, GCP, OIDC, etc.) @@ -62,6 +63,19 @@ func init() { // +kubebuilder:scaffold:scheme } +// envBool returns the boolean value of the named env var, or def when it is +// unset or unparseable. Used to default toggle flags from the environment so +// kustomize overlays can patch a single env var instead of overriding the +// container's whole args list. +func envBool(key string, def bool) bool { + if v, ok := os.LookupEnv(key); ok { + if b, err := strconv.ParseBool(v); err == nil { + return b + } + } + return def +} + // nolint:gocyclo func main() { var metricsAddr string @@ -82,9 +96,11 @@ func main() { flag.StringVar(&metricsAddr, "metrics-bind-address", "0", "The address the metrics endpoint binds to. "+ "Use :8443 for HTTPS or :8080 for HTTP, or leave as 0 to disable the metrics service.") flag.StringVar(&probeAddr, "health-probe-bind-address", ":8081", "The address the probe endpoint binds to.") - flag.BoolVar(&enableLeaderElection, "leader-elect", false, + flag.BoolVar(&enableLeaderElection, "leader-elect", envBool("LEADER_ELECT", false), "Enable leader election for controller manager. "+ - "Enabling this will ensure there is only one active controller manager.") + "Enabling this will ensure there is only one active controller manager. "+ + "Defaults from the LEADER_ELECT env var so overlays can toggle it "+ + "without overriding the whole args list.") flag.DurationVar(&leaderElectionLeaseDuration, "leader-elect-lease-duration", 10*time.Second, "The duration that non-leader candidates will wait to force acquire leadership.") flag.DurationVar(&leaderElectionRenewDeadline, "leader-elect-renew-deadline", 3*time.Second, diff --git a/config/manager/manager.yaml b/config/manager/manager.yaml index 58967d9..3c0bc8e 100644 --- a/config/manager/manager.yaml +++ b/config/manager/manager.yaml @@ -62,7 +62,6 @@ spec: - /manager args: - --role=replicator - - --leader-elect - --health-probe-bind-address=:8081 - --server-config=/config/server-config.yaml image: ghcr.io/datum-cloud/dns-operator:latest @@ -75,6 +74,11 @@ spec: valueFrom: fieldRef: fieldPath: metadata.name + # Leader election is set via env (defaulted in cmd/main.go) rather than a + # flag so overlays can toggle it by patching this one env var instead of + # overriding the whole args list. + - name: LEADER_ELECT + value: "true" ports: [] securityContext: readOnlyRootFilesystem: true diff --git a/config/overlays/replicator-milo/patch-milo.yaml b/config/overlays/replicator-milo/patch-milo.yaml index 8986f17..1d30560 100644 --- a/config/overlays/replicator-milo/patch-milo.yaml +++ b/config/overlays/replicator-milo/patch-milo.yaml @@ -1,6 +1,10 @@ -# Milo-mode replicator: 2 active replicas (leader election OFF so both replicas +# Milo-mode replicator: 2 replicas with leader election OFF so both replicas # reconcile — this is what exercises the multicluster per-cluster ownership that -# regressed in engineering#346), plus the milo-apiserver kubeconfig mount. +# regressed in engineering#346 — plus the milo-apiserver kubeconfig mount. +# +# Note we DON'T restate the container args here: env and volume/volumeMount +# lists merge by name under strategic merge, so we only patch what changes +# (LEADER_ELECT=false) and inherit the base args (--role, --server-config, …). apiVersion: apps/v1 kind: Deployment metadata: @@ -11,11 +15,9 @@ spec: spec: containers: - name: manager - # Replaces the base args wholesale — note NO --leader-elect. - args: - - --role=replicator - - --health-probe-bind-address=:8081 - - --server-config=/config/server-config.yaml + env: + - name: LEADER_ELECT + value: "false" volumeMounts: - name: milo-kubeconfig mountPath: /milo From 35f64f95ce13d913c9036dbb00eb82ec584d9665 Mon Sep 17 00:00:00 2001 From: Scot Wells Date: Sat, 25 Jul 2026 17:06:22 -0500 Subject: [PATCH 6/9] refactor: Use $(LEADER_ELECT) arg substitution instead of env-parsing in code MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Follow-up to c6d332f: revert the Go-side env parsing (envBool) and instead reference the env var from the flag via Kubernetes' native $(VAR) arg substitution — `--leader-elect=$(LEADER_ELECT)` with LEADER_ELECT set in the container env. This keeps the operator binary unchanged (standard --leader-elect flag) while still letting overlays toggle leader election by patching a single env var (env lists merge by name) rather than restating the whole args list. Verified live with a freshly built image: LEADER_ELECT=true acquires a leader-election Lease (the flag default is false and there is no bare --leader-elect, so it only turns on via the substituted env), LEADER_ELECT=false runs both replicas with no lease. Co-Authored-By: Claude Opus 4.8 --- cmd/main.go | 20 ++------------------ config/manager/manager.yaml | 9 ++++++--- 2 files changed, 8 insertions(+), 21 deletions(-) diff --git a/cmd/main.go b/cmd/main.go index 5a5ed08..eee383d 100644 --- a/cmd/main.go +++ b/cmd/main.go @@ -9,7 +9,6 @@ import ( "flag" "fmt" "os" - "strconv" "time" // Import all Kubernetes client auth plugins (e.g. Azure, GCP, OIDC, etc.) @@ -63,19 +62,6 @@ func init() { // +kubebuilder:scaffold:scheme } -// envBool returns the boolean value of the named env var, or def when it is -// unset or unparseable. Used to default toggle flags from the environment so -// kustomize overlays can patch a single env var instead of overriding the -// container's whole args list. -func envBool(key string, def bool) bool { - if v, ok := os.LookupEnv(key); ok { - if b, err := strconv.ParseBool(v); err == nil { - return b - } - } - return def -} - // nolint:gocyclo func main() { var metricsAddr string @@ -96,11 +82,9 @@ func main() { flag.StringVar(&metricsAddr, "metrics-bind-address", "0", "The address the metrics endpoint binds to. "+ "Use :8443 for HTTPS or :8080 for HTTP, or leave as 0 to disable the metrics service.") flag.StringVar(&probeAddr, "health-probe-bind-address", ":8081", "The address the probe endpoint binds to.") - flag.BoolVar(&enableLeaderElection, "leader-elect", envBool("LEADER_ELECT", false), + flag.BoolVar(&enableLeaderElection, "leader-elect", false, "Enable leader election for controller manager. "+ - "Enabling this will ensure there is only one active controller manager. "+ - "Defaults from the LEADER_ELECT env var so overlays can toggle it "+ - "without overriding the whole args list.") + "Enabling this will ensure there is only one active controller manager.") flag.DurationVar(&leaderElectionLeaseDuration, "leader-elect-lease-duration", 10*time.Second, "The duration that non-leader candidates will wait to force acquire leadership.") flag.DurationVar(&leaderElectionRenewDeadline, "leader-elect-renew-deadline", 3*time.Second, diff --git a/config/manager/manager.yaml b/config/manager/manager.yaml index 3c0bc8e..e1290b2 100644 --- a/config/manager/manager.yaml +++ b/config/manager/manager.yaml @@ -62,6 +62,10 @@ spec: - /manager args: - --role=replicator + # Value comes from the LEADER_ELECT env var via Kubernetes' + # $(VAR) substitution, so overlays toggle leader election by patching + # that one env var instead of overriding this whole args list. + - --leader-elect=$(LEADER_ELECT) - --health-probe-bind-address=:8081 - --server-config=/config/server-config.yaml image: ghcr.io/datum-cloud/dns-operator:latest @@ -74,9 +78,8 @@ spec: valueFrom: fieldRef: fieldPath: metadata.name - # Leader election is set via env (defaulted in cmd/main.go) rather than a - # flag so overlays can toggle it by patching this one env var instead of - # overriding the whole args list. + # Substituted into --leader-elect=$(LEADER_ELECT) above. Overlays patch + # this env var (env lists merge by name) rather than restating args. - name: LEADER_ELECT value: "true" ports: [] From befc97ba581d0953b90c46a23c76d9371f00baf6 Mon Sep 17 00:00:00 2001 From: Scot Wells Date: Sat, 25 Jul 2026 18:10:13 -0500 Subject: [PATCH 7/9] docs: Tighten dns-metrics policy and drift-rules comments MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Trim the API comments on the ResourceMetricsPolicy and PrometheusRule per the Google technical-writing style: present tense, active voice, one idea per sentence, and no restating what the names and expressions already say. Halves the comment volume (policy 29→15 lines, rules 49→25) with no change to any generator, label, CEL expression, or rule. Co-Authored-By: Claude Opus 4.8 --- config/observability/dns-drift-rules.yaml | 70 +++++++------------- config/observability/dns-metrics-policy.yaml | 38 ++++------- 2 files changed, 35 insertions(+), 73 deletions(-) diff --git a/config/observability/dns-drift-rules.yaml b/config/observability/dns-drift-rules.yaml index 49d8155..8e74721 100644 --- a/config/observability/dns-drift-rules.yaml +++ b/config/observability/dns-drift-rules.yaml @@ -1,34 +1,15 @@ -# Recording + alerting rules that detect drift between the upstream (project) -# and downstream (root) control planes, using the series emitted by the -# dns-metrics ResourceMetricsPolicy (see dns-metrics-policy.yaml). +# Recording + alerting rules that diff the dns-metrics series (dns-metrics-policy.yaml) +# to detect drift between upstream (project) and downstream (root) control planes. +# See docs/enhancements/controlplane-drift-detection.md. # -# Join key: (upstream_cluster, upstream_namespace, upstream_name). +# The policy runs on every control plane, so both generators emit everywhere. +# The rules partition on milo_control_plane_type ("root" vs "project") to keep +# each side's stray series out of the diff, and normalize both sides to a bare +# `proj` label to join on (proj, upstream_namespace, upstream_name). # -# A single dns-metrics policy is applied to every control plane, so BOTH the -# upstream and downstream generators emit on every CP. resource-metrics tags -# each series with the source CP type via milo.control_plane.type (promoted to -# the Prometheus label milo_control_plane_type): "root" on the core (downstream) -# CP, "project" on project (upstream) CPs. We partition on that label so the -# downstream generator's (empty-label) series on project CPs and the upstream -# generator's series on the core CP don't pollute the diff. -# -# Join normalization: the downstream series carries upstream_cluster="cluster- -# " (the replicator prefixes it; is the Milo cluster key, which for -# a cluster-scoped Project is the bare project name). The upstream series is -# tagged with milo_project_name = the project name (resource-metrics strips the -# leading "/" the Milo provider uses). We add the "cluster-" prefix to the -# upstream side so both align. Label names/values confirmed against -# resource-metrics v-current (milo.project.name / milo.control_plane.type, -# root value "root"); re-verify if resource-metrics changes its attribute keys. -# -# See docs/enhancements/controlplane-drift-detection.md for the full design. -# Standard Prometheus-operator PrometheusRule. datum-cloud/infra runs the -# victoria-metrics-operator, which converts PrometheusRule -> VMRule and carries -# labels through, so no VM-specific CRD is needed. The -# telemetry.miloapis.com/resource-metrics-aggregator label routes these rules to -# the dedicated `vmalert-datum-resource-metrics-aggregator` (which has the -# resource-metrics datasource); the general vmalert selects the complement (that -# label DoesNotExist). See infra .../victoria-metrics/base/vmalert.yaml. +# This is a standard PrometheusRule: victoria-metrics-operator converts it to a +# VMRule and preserves labels, so the resource-metrics-aggregator label below +# routes it to the vmalert that has the resource-metrics datasource. apiVersion: monitoring.coreos.com/v1 kind: PrometheusRule metadata: @@ -36,22 +17,18 @@ metadata: labels: managed-by: flux app.kubernetes.io/part-of: dns-operator - # Route to the resource-metrics aggregator vmalert (label presence is the - # selector; value is not significant). + # Selector for the resource-metrics aggregator vmalert; presence is what + # matters, not the value. telemetry.miloapis.com/resource-metrics-aggregator: "true" spec: groups: - name: dns.drift.recording interval: 30s rules: - # Both sides are normalized to a bare `proj` (project name) join key. - # Upstream: milo_project_name is already the bare name ("alpha"). - # Downstream: the replicator stamps upstream-cluster-name as - # "cluster-" + replace(, "/", "_"), and the milo - # provider keys a project cluster as "/alpha", so the annotation is - # "cluster-_alpha". Strip the "cluster-_?" prefix to recover "alpha". - # (Verified live: downstream upstream_cluster="cluster-_alpha" vs - # upstream milo_project_name="alpha".) + # Upstream project CPs, milo_project_name copied to the bare `proj` join + # key. Downstream's cluster label is normalized to match below: the + # annotation is "cluster-_" (the replicator maps the milo key + # "/alpha" to "_alpha"), verified live against milo_project_name="alpha". - record: dns:recordset_upstream:normalized expr: | label_replace( @@ -59,8 +36,8 @@ spec: "proj", "$1", "milo_project_name", "(.*)" ) - # Downstream view: core CP only (where replicated objects live), with - # the cluster annotation normalized to the bare project name. + # Downstream root CP records, cluster label stripped of "cluster-_" to + # recover the bare `proj`. - record: dns:recordset_downstream:scoped expr: | label_replace( @@ -68,16 +45,15 @@ spec: "proj", "$1", "upstream_cluster", "cluster-_?(.*)" ) - # ORPHANS: exists downstream, no surviving upstream owner. - # This is the failure mode from engineering#346 — a replicated record - # left "reserving" a name after its upstream owner was deleted. + # Orphan: a downstream record with no upstream owner — the engineering#346 + # leftover that keeps "reserving" a name after its owner is deleted. - record: dns:recordset_downstream_orphan expr: | dns:recordset_downstream:scoped unless on(proj, upstream_namespace, upstream_name) dns:recordset_upstream:normalized - # MISSING: exists upstream, never replicated / replication stalled. + # Missing: an upstream record with no downstream copy — replication stalled. - record: dns:recordset_downstream_missing expr: | dns:recordset_upstream:normalized @@ -87,8 +63,8 @@ spec: - name: dns.drift.alerts rules: - alert: DNSDownstreamOrphanRecordSet - # for: rides out normal replication + OTLP push/staleness lag so we - # only page on drift that actually persists. + # for: rides out normal replication/staleness lag, so only persistent + # drift pages. expr: dns:recordset_downstream_orphan > 0 for: 10m labels: diff --git a/config/observability/dns-metrics-policy.yaml b/config/observability/dns-metrics-policy.yaml index e812bdc..3c8cf00 100644 --- a/config/observability/dns-metrics-policy.yaml +++ b/config/observability/dns-metrics-policy.yaml @@ -1,22 +1,11 @@ -# ResourceMetricsPolicy consumed by the milo-os/resource-metrics controller. +# ResourceMetricsPolicy for control-plane DNS drift detection. resource-metrics +# emits one gauge per DNSRecordSet/DNSZone from each control plane; recording +# rules diff the two sides. See docs/enhancements/controlplane-drift-detection.md. # -# resource-metrics watches every project (upstream) control plane via the Milo -# multi-cluster provider, and — with discovery.collectRootControlPlane=true — -# the root/core (downstream) control plane as well. It emits one gauge series -# per matching object on whichever control plane the object lives on, pushing -# over OTLP to the platform OTel collector -> Victoria Metrics. -# -# The two generators below emit structurally different series on the two sides -# of the replication boundary so that recording rules can diff them: -# -# dns_recordset_upstream_info <- source of truth, on project control planes -# dns_recordset_downstream_info <- replicated copy, on the root control plane -# -# The downstream series lifts the replicator's `meta.datumapis.com/upstream-*` -# annotations onto labels so the two sides share a join key -# (upstream_cluster, upstream_namespace, upstream_name). -# -# See docs/enhancements/controlplane-drift-detection.md for the full design. +# Upstream generators run on project control planes (desired state). Downstream +# generators run on the root control plane (replicated copies) and lift the +# replicator's meta.datumapis.com/upstream-* annotations onto labels, so both +# sides share the join key (upstream_cluster, upstream_namespace, upstream_name). apiVersion: resourcemetrics.miloapis.com/v1alpha1 kind: ResourceMetricsPolicy metadata: @@ -41,9 +30,8 @@ spec: - name: upstream_name value: "object.metadata.name" - name: accepted - # Null-safe: a record with no status yet (no reconciler) has no - # status.conditions; guarding with has() avoids a CEL eval error - # that would make resource-metrics silently drop the whole series. + # has() guards a record that has no status yet: reading a + # missing status.conditions is a CEL error, which drops the series. value: "has(object.status) && has(object.status.conditions) && object.status.conditions.exists(c, c.type == 'Accepted' && c.status == 'True') ? 'true' : 'false'" - name: dnszone-upstream-info @@ -64,11 +52,9 @@ spec: value: "object.metadata.name" # ---- Downstream (root control plane): replicated / actual state ------- - # Join labels come from the annotations the replicator stamps in - # internal/downstreamclient/mappednamespace.go. NOTE: the namespace is - # remapped downstream, so metadata.namespace is NOT the upstream namespace; - # the real one is in the upstream-namespace annotation. The cluster-name - # annotation is prefixed "cluster-" — the recording rules normalize for it. + # Downstream namespaces are remapped, so the join labels come from the + # replicator's upstream-* annotations, not from metadata. The cluster-name + # annotation is prefixed "cluster-"; the recording rules normalize it. - name: dnsrecordset-downstream-info resource: group: dns.networking.miloapis.com From 447b67028c13145fa79189086332e20c22f40d90 Mon Sep 17 00:00:00 2001 From: Scot Wells Date: Sat, 25 Jul 2026 18:23:57 -0500 Subject: [PATCH 8/9] docs: Tighten and correct the drift-detection READMEs MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Apply the Google technical-writing style to the READMEs added for this feature (present tense, active voice, no restating the obvious, drop resolved "assumptions" sections) and fix content that went stale as the implementation settled. The resource-metrics README no longer describes the deleted controller/server-config.yaml or claims the image defaults to :latest (it references the published bundle now). The control-plane drift README documents the actual four-cluster chainsaw contract — including the replicator cluster on dns-upstream and the scale on deployment/dns-operator-controller-manager, not the earlier "infra hosts the replicator" / "scale --all" assumptions — and points at the kubeconfigs' real location. Cuts the two large READMEs roughly in half with no change to any command, path, or manifest. Co-Authored-By: Claude Opus 4.8 --- .../dependencies/resource-metrics/README.md | 119 ++++------------- config/overlays/agent-powerdns-milo/README.md | 46 +++---- test/e2e/controlplane-drift/README.md | 126 +++++------------- 3 files changed, 84 insertions(+), 207 deletions(-) diff --git a/config/dependencies/resource-metrics/README.md b/config/dependencies/resource-metrics/README.md index 6286f27..43a2ba2 100644 --- a/config/dependencies/resource-metrics/README.md +++ b/config/dependencies/resource-metrics/README.md @@ -1,121 +1,58 @@ # resource-metrics dependency (DNS control-plane drift detection) -Deploys the [milo-os/resource-metrics](https://github.com/milo-os/resource-metrics) -controller for the DNS control-plane drift-detection e2e, plus the pieces that -must live on the Milo core control plane. - -`resource-metrics` watches every project (UPSTREAM) control plane served by -`milo-apiserver` and — with `discovery.collectRootControlPlane: true` — the -Milo core/root (DOWNSTREAM) control plane as well. It evaluates the -`dns-metrics` `ResourceMetricsPolicy` and pushes one gauge series per matching -object over OTLP to the test-infra OTel collector, which forwards to Victoria -Metrics. Recording rules then diff the upstream (source of truth) and -downstream (replicated copy) series to detect drift. See +Deploys [milo-os/resource-metrics](https://github.com/milo-os/resource-metrics) +for the drift-detection e2e by referencing its published kustomize bundle via +Flux — no vendored manifests. resource-metrics emits one gauge per +DNSRecordSet/DNSZone from each control plane, and recording rules diff them. See `docs/enhancements/controlplane-drift-detection.md`. -## Topology (validated on the dns-control kind cluster) - -- **dns-control** hosts the Milo core control plane (the DOWNSTREAM / "root"), - the PowerDNS agent, RustFS, the observability stack, AND this - resource-metrics controller. -- Project control planes **alpha/beta** (served by `milo-apiserver`) are the - UPSTREAM. -- **dns-upstream** hosts the replicator; **dns-edge** hosts PowerDNS. - -Because resource-metrics runs on dns-control alongside `milo-apiserver`, it -reaches the core CP over the in-cluster Service -(`https://milo-apiserver.milo-system.svc.cluster.local:6443`, self-signed → -`insecure-skip-tls-verify`) using the static `test-admin-token` -(`system:masters`). - -## Two deployment slices, two different API servers - -This directory is split by **which control plane / kubeconfig each slice -targets**. They are applied separately and never combined into a single -`kubectl apply`. +The directory has two slices, applied separately because they target different +API servers. `env:metrics-up` in the Taskfile runs both. -### `controller/` — applied to the KIND (dns-control) context +## `controller/` — the controller, on the local (dns-control) cluster -Rather than vendoring the controller manifests, this **references the operator's -published kustomize bundle** (`oci://ghcr.io/milo-os/resource-metrics-kustomize`, -`overlays/test-infra`) via Flux — the same pattern `config/dependencies/milo` -uses. That overlay already ships the Deployment, RBAC, namespace, the -`milo-kubeconfig` Secret, and a `mode: milo` + `collectRootControlPlane: true` -server-config, so we only override two things. +Flux pulls the operator's `overlays/test-infra` bundle, which already ships the +Deployment, RBAC, namespace, the `milo-kubeconfig` Secret, and a `mode: milo` + +`collectRootControlPlane: true` server-config. This slice overrides two things. | File | Purpose | | --- | --- | -| `ocirepository.yaml` | Flux `OCIRepository` on `resource-metrics-kustomize`, pinned to a bundle tag that ships a multi-arch image. | -| `flux-install.yaml` | Flux `Kustomization` on `overlays/test-infra` with two overrides: `images:` (the controller image tag) and a patch pointing the OTLP endpoint at our collector (`telemetry-system`, not the overlay default `otel-collector-system`). | -| `kustomization.yaml` | Applies the two Flux resources into `flux-system`. | +| `ocirepository.yaml` | `OCIRepository` on `resource-metrics-kustomize`, pinned to a bundle tag with a multi-arch image. | +| `flux-install.yaml` | `Kustomization` on `overlays/test-infra`; overrides the image tag and patches the OTLP endpoint to our collector (`telemetry-system`, not the overlay default `otel-collector-system`). | +| `kustomization.yaml` | Applies both Flux resources into `flux-system`. | ```sh kubectl --context kind-dns-control apply -k config/dependencies/resource-metrics/controller kubectl --context kind-dns-control -n flux-system wait kustomization/resource-metrics --for=condition=Ready --timeout=300s ``` +The controller reaches the core CP over the in-cluster Service +(`milo-apiserver.milo-system.svc.cluster.local:6443`, self-signed → +`insecure-skip-tls-verify`) with the test-only `test-admin-token`. + > [!NOTE] -> The OTLP-endpoint override is a full-ConfigMap patch because the pinned bundle -> hardcodes the endpoint. Once [milo-os/resource-metrics#14](https://github.com/milo-os/resource-metrics/pull/14) -> (configurable endpoint) and [#13](https://github.com/milo-os/resource-metrics/pull/13) -> (multi-arch image) land in `main`, bump `ocirepository.yaml` to a `v0.0.0-main` -> tag and replace the patch with a Deployment env patch: +> The endpoint override is a full-ConfigMap patch because the pinned bundle +> hardcodes the endpoint. After resource-metrics +> [#14](https://github.com/milo-os/resource-metrics/pull/14) (configurable +> endpoint) and [#13](https://github.com/milo-os/resource-metrics/pull/13) +> (multi-arch image) merge, bump `ocirepository.yaml` to a `v0.0.0-main` tag and +> replace the patch with an env patch: > `OTEL_EXPORTER_OTLP_ENDPOINT=otel-collector-collector.telemetry-system:4317`. -### `core-control-plane/` — installed onto the Milo core control plane +## `core-control-plane/` — the CRD and policy, on the Milo core CP -These land on the **Milo core control plane** (`milo-apiserver`), NOT the kind -apiserver, because that is where the controller reads its policy from. +These target `milo-apiserver` — where the controller reads its policy — not the +kind apiserver. | Path | Purpose | | --- | --- | -| `core-control-plane/crd/` | Flux `Kustomization` (applied to the local cluster) that installs the `ResourceMetricsPolicy` CRD onto the core CP **from the published bundle** (`path: crd`) via `kubeConfig.secretRef`. No vendored CRD. Mirrors infra `apps/resource-metrics-system/base/milo-control-plane.yaml`. Includes the milo-kubeconfig Secret Flux targets the core CP with (test-only `test-admin-token`). | -| `core-control-plane/policy/` | Applies the `dns-metrics` `ResourceMetricsPolicy`. This one is dns-operator-owned (not in the bundle), so it **references** `config/observability/dns-metrics-policy.yaml` (single source of truth) and is applied directly to the core CP after the CRD is Established. | +| `crd/` | Flux `Kustomization` (on the local cluster) that installs the `ResourceMetricsPolicy` CRD onto the core CP from the bundle (`path: crd`) via `kubeConfig.secretRef`. Mirrors infra's `milo-control-plane.yaml`. Includes the test-only milo-kubeconfig Secret. | +| `policy/` | Applies the `dns-metrics` `ResourceMetricsPolicy`, which references `config/observability/dns-metrics-policy.yaml` (the single source), after the CRD is Established. | ```sh -# 1) Install the CRD onto the core CP from the bundle (Flux objects go on the -# LOCAL cluster; the Kustomization targets the core CP via its kubeConfig). kubectl --context kind-dns-control apply -k config/dependencies/resource-metrics/core-control-plane/crd kubectl --context kind-dns-control -n flux-system wait kustomization/resource-metrics-crd --for=condition=Ready --timeout=180s - -# 2) Apply the dns-metrics policy to the core CP (point kubectl at milo-apiserver). kustomize build --load-restrictor LoadRestrictionsNone \ config/dependencies/resource-metrics/core-control-plane/policy \ | kubectl --kubeconfig apply -f - ``` - -## OTel endpoint assumption (confirm live) - -`controller/server-config.yaml` sets: - -``` -otel.endpoint: otel-collector-collector.telemetry-system.svc.cluster.local:4317 -``` - -Rationale: test-infra's `install-observability` task applies an -`OpenTelemetryCollector` CR named `otel-collector` in namespace -`telemetry-system` -(`.test-infra/components/observability/otel-collector/opentelemetry-collector.yaml`). -The OpenTelemetry Operator renders a Service named `-collector` -(`otel-collector-collector`), and the CR's `receivers.otlp.protocols.grpc` -listens on `:4317`. The task even waits on -`daemonset/otel-collector-collector` in `telemetry-system`, confirming the -name. - -> [!NOTE] -> This differs from milo-os/infra, whose resource-metrics ships its own -> `metrics-collector` CR and points at -> `metrics-collector-collector.resource-metrics-system...:4317`. We reuse the -> shared test-infra collector in `telemetry-system` instead of deploying a -> second collector. If the collector CR name or namespace changes, update the -> endpoint. The collector CR is a **daemonset**; the OTel Operator still -> renders the `-collector` Service used above. - -## Assumptions needing live confirmation - -- **OTel endpoint** — as above; confirm `otel-collector-collector` exists in - `telemetry-system` and serves gRPC on 4317 after `install-observability`. -- **Controller image tag** — defaults to `:latest`; pin to whatever tag is - published/loaded for the e2e run. -- **`test-admin-token`** — must match milo-apiserver's `tokens.csv` - (`milo-apiserver-auth-tokens` in `milo-system`). diff --git a/config/overlays/agent-powerdns-milo/README.md b/config/overlays/agent-powerdns-milo/README.md index 611e8b7..f1309bd 100644 --- a/config/overlays/agent-powerdns-milo/README.md +++ b/config/overlays/agent-powerdns-milo/README.md @@ -1,39 +1,39 @@ # agent-powerdns-milo -PowerDNS agent overlay that points the DNS agent at the **Milo core control plane** (`milo-apiserver`, root scope) as its DNS read (and status-write) source, for the DNS drift-detection e2e. - -In this environment the DNS CRDs and `DNSRecordSet`s live on the Milo core control plane, **not** on the local kind API. The PowerDNS agent (`--role=downstream`) must read `DNSRecordSet`/`DNSZone` objects from the Milo core CP and program PowerDNS from them. +PowerDNS agent overlay that points the DNS agent at the **Milo core control +plane** (`milo-apiserver`, root scope) as its read and status-write source, for +the drift-detection e2e. Here the DNS CRDs and `DNSRecordSet`s live on the core +CP, not the local kind API, so the agent (`--role=downstream`) must read them +from there. ## What changed vs. `agent-powerdns-federated` -This overlay bases on `../agent-powerdns-federated` and leaves the PowerDNS, Lightningstream, and RustFS/S3 (`s3-credentials`) wiring completely intact. It changes only the agent's API target: +Bases on `../agent-powerdns-federated`, leaving the PowerDNS, Lightningstream, +and RustFS wiring intact. It only retargets the agent's API server: -1. **Replaces the agent server-config** (`configMapGenerator` `behavior: replace` on `agent-server-config`) with a milo-targeted `server-config.yaml`. -2. **Mounts a `milo-kubeconfig` Secret at `/milo`** on the `manager` container of the `pdns-auth` StatefulSet and sets `KUBECONFIG=/milo/kubeconfig` (`deployment-patch.yaml`). -3. **Ships the `milo-kubeconfig` Secret** (`milo-kubeconfig-secret.yaml`) pointing at the in-cluster `milo-apiserver` endpoint with the static `test-admin-token` and `insecure-skip-tls-verify`. +1. Replaces the `agent-server-config` ConfigMap with a milo-targeted `server-config.yaml`. +2. Mounts a `milo-kubeconfig` Secret at `/milo` on the `pdns-auth` StatefulSet's `manager` container and sets `KUBECONFIG=/milo/kubeconfig` (`deployment-patch.yaml`). +3. Ships that Secret (`milo-kubeconfig-secret.yaml`) — in-cluster `milo-apiserver` + the test-only `test-admin-token`. -`disableNameSuffixHash: true` is kept so the StatefulSet's existing `agent-server-config` volume reference stays valid after the replace. +`disableNameSuffixHash: true` is kept so the ConfigMap replace keeps the +StatefulSet's existing volume reference valid. -## Which field retargets the read source to the core CP +## How the retarget works > [!IMPORTANT] -> For `--role=downstream`, it is **not** a server-config field — it is the `KUBECONFIG` env var. - -`cmd/main.go`'s `case "downstream":` branch builds its manager and all three controllers (`DNSZone`, `DNSRecordSet`, `DNSRecordSetPowerDNS`) from `ctrl.GetConfigOrDie()` and `mgr.GetClient()`. It **never** consults `discovery.*` or `downstreamResourceManagement.kubeconfigPath` — those fields are read only by the `case "replicator":` branch (`serverConfig.DownstreamResourceManagement.RestConfig()` / `initializeClusterDiscovery`). So `discovery.mode: single` plus `downstreamResourceManagement.kubeconfigPath` cannot retarget a downstream agent's read source on their own. - -The retarget is done by `KUBECONFIG=/milo/kubeconfig` in `deployment-patch.yaml`. controller-runtime's `ctrl.GetConfig()` honors `KUBECONFIG` before falling back to the in-cluster config, and no `--kubeconfig` flag is registered on the binary — so both the reads (DNSRecordSet/DNSZone informers) and the writes (status updates via `mgr.GetClient()`) resolve to the mounted Milo core-CP kubeconfig instead of the in-cluster kind API. +> For `--role=downstream`, the read source is set by the `KUBECONFIG` env var, +> **not** a server-config field. -The milo-targeted values in `server-config.yaml` (`discovery` paths + `downstreamResourceManagement.kubeconfigPath` all set to `/milo/kubeconfig`) are kept for consistency and remain correct if this agent is ever run as `--role=replicator`; they are **inert** under `--role=downstream`. - -## Assumptions needing live confirmation - -- The Milo core CP is reachable at `https://milo-apiserver.milo-system.svc.cluster.local:6443` from the `dns-agent-system`/`dns-control` cluster and serves `dns.networking.miloapis.com/v1alpha1` `DNSRecordSet`/`DNSZone`/`DNSZoneClass` at its **root** endpoint (no aggregation path). Validated live per the task grounding; re-confirm if the endpoint or scope changes. -- `test-admin-token` (system:masters) is present in secret `milo-apiserver-auth-tokens` (`tokens.csv`) in ns `milo-system` and accepted by milo-apiserver's token-auth-file. -- The agent namespace is `dns-agent-system` (inherited from the federated overlay); the `milo-kubeconfig` Secret is created there. -- `KUBECONFIG` retargeting assumes the downstream binary registers no `--kubeconfig` flag (confirmed in `cmd/main.go`). If the code later adds one or stops using `ctrl.GetConfigOrDie()` in the downstream branch, this approach must be revisited (the minimal code alternative would be to have the downstream branch build its cluster from `serverConfig.DownstreamResourceManagement.RestConfig()`). +The downstream branch in `cmd/main.go` builds its manager from +`ctrl.GetConfigOrDie()`; it never reads `discovery.*` or +`downstreamResourceManagement.kubeconfigPath` (only the replicator branch does). +`ctrl.GetConfig()` honors `KUBECONFIG` before the in-cluster config, and the +binary registers no `--kubeconfig` flag, so both the reads and the status writes +resolve to the mounted core-CP kubeconfig. The milo values in `server-config.yaml` +are inert under `--role=downstream` but stay correct if run as `--role=replicator`. ## Validate -``` +```sh kustomize build --load-restrictor LoadRestrictionsNone config/overlays/agent-powerdns-milo ``` diff --git a/test/e2e/controlplane-drift/README.md b/test/e2e/controlplane-drift/README.md index 1a899ed..40f2cca 100644 --- a/test/e2e/controlplane-drift/README.md +++ b/test/e2e/controlplane-drift/README.md @@ -1,117 +1,57 @@ # control-plane drift e2e Chainsaw scenarios that prove upstream/downstream control-plane DNS drift -detection: the resource-metrics collectors emit per-object series from every -control plane, the `dns-controlplane-drift` recording/alerting rules diff the -two sides, and these tests assert the resulting series land in Victoria Metrics. +detection end to end: resource-metrics emits per-object series from each control +plane, the `dns-controlplane-drift` rules diff them, and these tests assert the +results in Victoria Metrics. See +`docs/enhancements/controlplane-drift-detection.md`. -See `docs/enhancements/controlplane-drift-detection.md` for the full design and -`config/observability/{dns-metrics-policy,dns-drift-rules}.yaml` for the policy -and rules under test. +Run with `task env:chainsaw-milo` against a running environment +(`task env:milo-all-up`). ## Scenarios | Dir | Proves | |---|---| -| `happy-path/` | Record created on project CP `alpha` replicates to the core CP; both `dns_recordset_upstream_info{milo_project_name="alpha"}` and `dns_recordset_downstream_info{milo_control_plane_type="root"}` exist in VM and `dns:recordset_downstream_orphan == 0`. | -| `orphan/` | engineering#346 regression. Replicate a record, scale the replicator to 0, delete the upstream object → downstream copy is orphaned → `dns:recordset_downstream_orphan > 0` and `DNSDownstreamOrphanRecordSet` becomes active. Restore the replicator → orphan clears. | +| `happy-path/` | A record on project CP `alpha` replicates to the core CP; both `*_upstream_info` and `*_downstream_info` land in VM and `dns:recordset_downstream_orphan == 0`. | +| `orphan/` | engineering#346: replicate a record, scale the replicator to 0, delete the upstream → the downstream copy is orphaned → `dns:recordset_downstream_orphan > 0` and `DNSDownstreamOrphanRecordSet` becomes active. Removing the leftover clears it. | | `missing/` | Scale the replicator to 0, create an upstream record → never replicated → `dns:recordset_downstream_missing > 0`. | -## Named clusters the Taskfile must provide +## Clusters -Each `chainsaw-test.yaml` declares its clusters inline (matching the -`federation/` and `zones-and-records/` suites) and expects these kubeconfig -files, one directory up (`test/e2e/`), exactly as the existing suites consume -`kubeconfig-{control,upstream,edge,downstream}`: +Each `chainsaw-test.yaml` names four clusters, with kubeconfigs in this directory +(referenced as `../kubeconfig-*`). `env:chainsaw-milo` generates them; they are +gitignored. -| Chainsaw cluster name | Kubeconfig file | Role / server URL | +| Name | Kubeconfig | Cluster | |---|---|---| -| `alpha` | `test/e2e/kubeconfig-alpha` | Project CP **alpha** (UPSTREAM). Aggregation path: `/apis/resourcemanager.miloapis.com/v1alpha1/projects/alpha/control-plane`. | -| `core` | `test/e2e/kubeconfig-core` | Milo **core** control plane (DOWNSTREAM), `` root. Replicated copies land here. | -| `infra` | `test/e2e/kubeconfig-infra` | The `dns-control` kind cluster that hosts Victoria Metrics + the OTel collector **and** the replicator `Deployment`. Used to run in-cluster `curl` against VM and to `kubectl scale` the replicator. | +| `alpha` | `kubeconfig-alpha` | Project CP alpha (upstream), milo aggregation path `.../projects/alpha/control-plane`. | +| `core` | `kubeconfig-core` | Milo core CP (downstream); replicated copies land here. | +| `infra` | `kubeconfig-infra` | dns-control kind cluster — hosts VM + OTel; runs the in-cluster `curl` VM queries. | +| `replicator` | `kubeconfig-replicator` | dns-upstream kind cluster — runs the replicator; the scenarios scale `deployment/dns-operator-controller-manager` in `dns-replicator-system` here. | -Notes for wiring: +`alpha` and `core` both reach milo-apiserver on dns-control with the +`test-admin-token`, differing only in the server URL. -- `fixtures/milo-projects.yaml` (already present, do not edit) creates the Org + - projects `alpha`/`beta` on the core CP; the Taskfile should apply it and mint - the `alpha` aggregation-path kubeconfig the same way `resource-metrics` does - (clone the milo kubeconfig, rewrite the `server:` URL to the project's - `.../projects/alpha/control-plane` path). -- `kubeconfig-beta` (project CP `beta`) is created by the fixture but is **not** - consumed by the current scenarios; provide it only if/when a multi-project - sharding scenario is added. -- `alpha`, `core`, and `infra` may all be served by the same `dns-control` kind - cluster (milo-apiserver serves the aggregation views); the three kubeconfigs - differ only in their `server:` URL and token. Admin token: `test-admin-token`. -- The replicator is scaled via `kubectl -n dns-replicator-system scale - deployment --all --replicas=0|1` — namespace `dns-replicator-system`, matched - by `--all` rather than a hard-coded deployment name. Confirm that namespace is - correct for the drift topology (the design doc co-locates the replicator on - the core cluster; adjust the `infra` kubeconfig / namespace if it differs). +## Victoria Metrics -## Victoria Metrics query endpoint - -Scenarios query VM over its HTTP API from a one-shot `curl` pod running inside -the `infra` cluster (the `kubectl run --rm ... curlimages/curl` pattern from the -resource-metrics suite). Default endpoint: +Scenarios query VM from a one-shot `curl` pod on the `infra` cluster. Default +endpoint (override with `VM_QUERY_URL`): ``` http://vmsingle-telemetry-system-vm.telemetry-system.svc.cluster.local:8428/api/v1/query ``` -Override per run by exporting `VM_QUERY_URL` (the scripts honour it). If the -observability stack uses a `vmselect`/cluster VM instead of `vmsingle`, point -`VM_QUERY_URL` at that `.../api/v1/query`. The endpoint must serve BOTH raw -series (`dns_recordset_*_info`) and the recording-rule / `ALERTS` series that -`vmalert` remote-writes back — i.e. `vmalert` must be configured against this -VM as its remote-write target (the `telemetry.miloapis.com/resource-metrics-aggregator` -`vmalert` in datum-cloud/infra). - -## Timing / `for:` considerations - -- **Recording rules are the primary gate.** `dns:recordset_downstream_orphan` - and `dns:recordset_downstream_missing` have no `for:` and are evaluated every - 30s, so the tests assert on those series (`> 0`) rather than on the alert's - full delay. This is fast and deterministic. -- **The alerts carry `for: 10m`.** After the recording rule fires, the alert - sits in `pending` for 10m before `firing`. Waiting the full period would blow - the e2e budget, so `orphan/` accepts `ALERTS{...alertstate="pending"|"firing"}` - (the alert is active on the correct series). To assert `firing` specifically, - deploy the drift rules with a shortened `for:` in a test-only overlay (e.g. - `for: 30s`); that is an infra/Taskfile change outside this suite. -- **Orphan detection lags by VM's staleness window.** The orphan only - materializes once the deleted upstream series ages out of VM's staleness - window (default ~5m) so the recording rule's `unless` no longer cancels the - downstream vector. `orphan/` therefore polls for up to ~9m (`timeout: 540s`). - Shortening VM's staleness window in the test-infra observability stack would - let this budget drop. -- **Pipeline latency.** watch → collect (OTel interval ~5s) → OTLP → remote-write - → VM ingest adds seconds; the happy-path VM polls allow a few minutes. -- **Run sequentially.** All scenarios share the `alpha`/`core`/`infra` clusters - and the `orphan`/`missing` scenarios scale the replicator globally, so run the - suite with `parallel: 1` (chainsaw's default when unset, or set it in the - Taskfile's chainsaw invocation). +VM must serve both the raw `dns_recordset_*` series and the vmalert +recording-rule / `ALERTS` series. -## Assumptions needing live confirmation +## Timing -- **Promoted label names/values** — `milo_project_name` (value `alpha`, and - `root` for the core CP) and `milo_control_plane_type` (`project` / `root`) are - taken from the resource-metrics OTLP attributes (`milo.project.name`, - `milo.control_plane.type`) after prometheus-remote-write dot→underscore - normalization. Confirm against a real series and adjust the queries + rules if - the aggregator promotes different keys. -- **Downstream join labels** — the downstream series lifts - `meta.datumapis.com/upstream-{cluster-name,namespace,name}` onto - `upstream_{cluster,namespace,name}`. The tests key on `upstream_name` - (= the upstream DNSRecordSet's `metadata.name`); verify the replicator stamps - the object name (not a derived value) into the `upstream-name` annotation. -- **Replicator namespace / deployment** — assumed `dns-replicator-system` on the - `infra` cluster. Confirm and adjust if the drift topology runs the replicator - elsewhere. -- **`vmalert` wiring** — assumes the drift `PrometheusRule`/`VMRule` is evaluated - and its recording results + `ALERTS` are queryable at `VM_QUERY_URL`. If the - observability stack doesn't remote-write vmalert results back to the same VM, - point the queries at the vmalert datasource instead. -- **Whether the upstream DNSRecordSet reaches `Accepted` on the project CP** is - not asserted (the condition-setting controller for that topology is - unconfirmed); the tests assert object existence + the VM series instead. +- Recording rules (`dns:recordset_downstream_{orphan,missing}`, no `for:`, 30s + interval) are the primary gate — the tests assert those `> 0`. +- The alerts carry `for: 10m`, so `orphan/` accepts + `alertstate="pending"|"firing"` rather than waiting for `firing`. +- The orphan only materializes after the deleted upstream series ages out of + VM's ~5m staleness window, so `orphan/` polls for up to ~9m. +- The suite runs sequentially (`.chainsaw.yaml` `parallel: 1`): the scenarios + share clusters and scale the single replicator. From ba4a37def0116a330d947010bdbaebb5a2969275 Mon Sep 17 00:00:00 2001 From: Scot Wells Date: Sat, 25 Jul 2026 18:28:33 -0500 Subject: [PATCH 9/9] docs: Make the resource-metrics dependency README less mechanism-heavy Rewrite it purpose-first: lead with what the controller does (emit per-object metrics so the rules can spot records that fell out of sync) and that env:metrics-up applies it, then explain the two-cluster split in plain language. Drop the Flux resource file-tables and the reproduced kubectl command blocks (those duplicated the Taskfile and were how the doc went stale), keeping only what an editor needs: where to change the image tag / OTel endpoint / bundle pin, and the upstream follow-up note. Co-Authored-By: Claude Opus 4.8 --- .../dependencies/resource-metrics/README.md | 75 +++++++------------ 1 file changed, 28 insertions(+), 47 deletions(-) diff --git a/config/dependencies/resource-metrics/README.md b/config/dependencies/resource-metrics/README.md index 43a2ba2..4d50e45 100644 --- a/config/dependencies/resource-metrics/README.md +++ b/config/dependencies/resource-metrics/README.md @@ -1,58 +1,39 @@ -# resource-metrics dependency (DNS control-plane drift detection) +# resource-metrics (drift-detection dependency) -Deploys [milo-os/resource-metrics](https://github.com/milo-os/resource-metrics) -for the drift-detection e2e by referencing its published kustomize bundle via -Flux — no vendored manifests. resource-metrics emits one gauge per -DNSRecordSet/DNSZone from each control plane, and recording rules diff them. See -`docs/enhancements/controlplane-drift-detection.md`. +Deploys the [resource-metrics](https://github.com/milo-os/resource-metrics) +controller that powers DNS control-plane drift detection: it emits a metric per +`DNSRecordSet`/`DNSZone` from every control plane so the alert rules can spot +records that have fallen out of sync between a customer's project and the +serving control plane. See `docs/enhancements/controlplane-drift-detection.md` +for the feature. -The directory has two slices, applied separately because they target different -API servers. `env:metrics-up` in the Taskfile runs both. +You don't normally apply this by hand — `task env:metrics-up` does it while +bringing up the e2e. The rest of this file is for changing or debugging it. -## `controller/` — the controller, on the local (dns-control) cluster +## What gets deployed, and where -Flux pulls the operator's `overlays/test-infra` bundle, which already ships the -Deployment, RBAC, namespace, the `milo-kubeconfig` Secret, and a `mode: milo` + -`collectRootControlPlane: true` server-config. This slice overrides two things. +resource-metrics comes from its own published release (a Flux-managed kustomize +bundle), not copied into this repo, so it tracks upstream. It's split in two +because the pieces live on different clusters: -| File | Purpose | -| --- | --- | -| `ocirepository.yaml` | `OCIRepository` on `resource-metrics-kustomize`, pinned to a bundle tag with a multi-arch image. | -| `flux-install.yaml` | `Kustomization` on `overlays/test-infra`; overrides the image tag and patches the OTLP endpoint to our collector (`telemetry-system`, not the overlay default `otel-collector-system`). | -| `kustomization.yaml` | Applies both Flux resources into `flux-system`. | +- **`controller/`** — the controller, on the local kind cluster. This is the + upstream `overlays/test-infra` bundle with our image tag and OTel endpoint + patched in. +- **`core-control-plane/`** — its CRD and the `dns-metrics` policy, on the Milo + core control plane (where the controller reads them). The CRD comes from the + same bundle; the policy is ours and points back to + `config/observability/dns-metrics-policy.yaml`. -```sh -kubectl --context kind-dns-control apply -k config/dependencies/resource-metrics/controller -kubectl --context kind-dns-control -n flux-system wait kustomization/resource-metrics --for=condition=Ready --timeout=300s -``` +## Changing it -The controller reaches the core CP over the in-cluster Service -(`milo-apiserver.milo-system.svc.cluster.local:6443`, self-signed → -`insecure-skip-tls-verify`) with the test-only `test-admin-token`. +- **Image tag / OTel endpoint** — `controller/flux-install.yaml`. +- **Bundle version** — pinned in `controller/ocirepository.yaml` (and reused by + `core-control-plane/crd/`). > [!NOTE] -> The endpoint override is a full-ConfigMap patch because the pinned bundle -> hardcodes the endpoint. After resource-metrics +> The OTel endpoint is overridden with a full-ConfigMap patch because the pinned +> bundle hardcodes it. After resource-metrics > [#14](https://github.com/milo-os/resource-metrics/pull/14) (configurable > endpoint) and [#13](https://github.com/milo-os/resource-metrics/pull/13) -> (multi-arch image) merge, bump `ocirepository.yaml` to a `v0.0.0-main` tag and -> replace the patch with an env patch: -> `OTEL_EXPORTER_OTLP_ENDPOINT=otel-collector-collector.telemetry-system:4317`. - -## `core-control-plane/` — the CRD and policy, on the Milo core CP - -These target `milo-apiserver` — where the controller reads its policy — not the -kind apiserver. - -| Path | Purpose | -| --- | --- | -| `crd/` | Flux `Kustomization` (on the local cluster) that installs the `ResourceMetricsPolicy` CRD onto the core CP from the bundle (`path: crd`) via `kubeConfig.secretRef`. Mirrors infra's `milo-control-plane.yaml`. Includes the test-only milo-kubeconfig Secret. | -| `policy/` | Applies the `dns-metrics` `ResourceMetricsPolicy`, which references `config/observability/dns-metrics-policy.yaml` (the single source), after the CRD is Established. | - -```sh -kubectl --context kind-dns-control apply -k config/dependencies/resource-metrics/core-control-plane/crd -kubectl --context kind-dns-control -n flux-system wait kustomization/resource-metrics-crd --for=condition=Ready --timeout=180s -kustomize build --load-restrictor LoadRestrictionsNone \ - config/dependencies/resource-metrics/core-control-plane/policy \ - | kubectl --kubeconfig apply -f - -``` +> (multi-arch image) merge, bump the pin to a `v0.0.0-main` tag and swap the +> patch for an `OTEL_EXPORTER_OTLP_ENDPOINT` env patch.