Management controllers re-reconcile idle workloads in a loop (no metric visibility)

[scotwells]

Problem

Compute's management controllers keep re-reconciling idle workloads in a self-perpetuating loop, and nothing alerts us when it happens. In staging, two idle workloads drive roughly 500 reconciles per minute across the workload, referenced-data, and workload-deployment-federator controllers — a single WorkloadDeployment reconciles up to 5 times a second — even though nothing about those workloads is changing.

That costs us in a few ways. Every reconcile writes back to the per-project control planes, so wasted load grows with every workload we add and eats directly into how many projects a management plane can carry. The churn also floods the logs (~48 healthy-looking lines/second), burying real events during incidents. And because nothing pages on it, a regression like this can run indefinitely until someone reads the logs by hand.

Root cause

The workload and referenced-data controllers fight over annotations on the same WorkloadDeployment. referenced-data merges an expected-referenced-data annotation onto the object, carefully preserving existing keys. The workload controller then overwrites that same metadata wholesale — it builds its desired WorkloadDeployment with no annotations and blindly assigns deployment.Annotations/deployment.Labels in its CreateOrUpdate — stripping the annotation the other controller just wrote.

Each controller undoes the other's write, and since the workload controller Owns the WorkloadDeployment while both others watch it, every write re-enqueues all three: an infinite ping-pong. It's metadata-only churn, so the object's generation never moves, which is why it hides in plain sight.

Visibility today

The metrics we need are already flowing. compute-manager exposes the standard controller-runtime metrics, and the platform's service-metrics vmagent already scrapes them into VictoriaMetrics — no scrape or ServiceMonitor change is required. The gap is that nothing alerts on the behavior, and the standard metrics are labeled by controller rather than by object, so the loop is only visible to someone who already knows to look. Its fingerprint is a sustained reconcile / workqueue_adds_total rate for the management controllers while workqueue_depth stays near zero.

Plan

Two PRs, sequenced so we prove the detection before we change behavior:

1. Alert + runbook first. A VMRule that fires on the reconcile-storm fingerprint, plus an on-call runbook. This ships to staging while the loop is still happening, so we can confirm it pages on the live incident before touching any code.
2. The fix. The workload controller merges controller-owned labels/annotations into the WorkloadDeployment instead of overwriting them, so expected-referenced-data survives and the ping-pong stops.

A namespace-scoped no-op reconcile counter would make this class of regression a first-class signal, but that's an optional follow-up.

Done when

• The alert fires against the current (still-looping) staging and then clears once the fix ships.
• With idle workloads, the reconcile rate for the three controllers drops to ~0 and the expected-referenced-data annotation stays put.
• The runbook documents how to spot this (e.g. workqueue_adds_total rate vs workqueue_depth).

We'll close this manually once that's confirmed in staging.