Convergence Reliability Work

[ecv]

Why this exists

This tracks our convergence-reliability work: making an HTTPProxy stand up quickly and reliably, and keeping one tenant or one control plane from dragging the whole fleet. Open issues are ranked by likely impact so we work the highest-leverage fixes first, not the freshest.

The lens comes from the recent per-stage investigation (https://github.com/datum-cloud/infra/issues/3622): the cost of standing up an HTTPProxy is queue serialization, requeue floors, and DNS-agent saturation — not compute. Ranking favors blast radius, user-facing breakage, and fixes we already have evidence for.

Fleet-wide / all-tenant blast radius

• OIDC SecurityPolicy with a missing clientSecret takes down the entire shared downstream gateway (EG v1.8.x); needs e2e reproduction #194 — one tenant's bad OIDC SecurityPolicy takes down the entire shared downstream gateway (all tenants, zero listeners). Live prod incident. Guard in flight + e2e upgrade-gate.
• connector/iroh-dns controllers fail to engage on 221 PCPs missing coordination.k8s.io discovery → Connector.Ready frozen indefinitely; 5s retry loop likely driving OOMKill #160 — 221 project control planes where connector/iroh-dns never engage, Ready frozen indefinitely + a 5s retry loop driving OOMKills. Feeds the shared-apiserver OOM at https://github.com/datum-cloud/infra/issues/3323.
• TrafficProtectionPolicy Enforce mode: 500 with truncated/empty body on all traffic #242 — TrafficProtectionPolicy Enforce → 500 with empty body on all traffic. WAF Enforce unusable. Blocked on an Envoy Gateway dependency.

User-facing feature broken

• Edge gateway HTTP-01 challenges stuck at ACME self-check (DNS/404/500/503) — NSO solver is live; control-plane vs member Challenge split #261 — edge HTTP-01 ACME challenges stuck, certificates don't issue, custom hostnames unusable.
• Stale hostname CNAME/ALIAS not reaped on hostname removal — breaks resolution (SERVFAIL) #283 — stale CNAME/ALIAS not reaped on hostname removal → SERVFAIL.
• Removing custom hostname from HTTPProxy leaves orphaned certificate resource, blocking TrafficProtectionPolicy acceptance #142 — orphaned certificate blocks TrafficProtectionPolicy acceptance after a hostname is removed.

Convergence latency (current theme, actionable now)

• Custom-hostname DNSRecordProgrammed stuck Pending >60s on fresh zone (breaches convergence SLO) #291 — DNS record programming >60s.
• Gateway *.datumproxy.net address resolves in 65–85s, exceeding the 60s convergence SLO #292 — gateway address resolving in 65–85s. Same root as Custom-hostname DNSRecordProgrammed stuck Pending >60s on fresh zone (breaches convergence SLO) #291: PowerDNS record-agent queue saturation (p95 ~79s queue wait, depth bursting to ~1555). One fix likely clears both — highest-leverage convergence win, evidence in hand. Fix lives in dns-operator + an infra overlay, not NSO.
• HTTPProxy reconcile backs off 3-4 min after 409 conflict burst at tunnel creation #166 — HTTPProxy reconcile backs off 3–4 min after a 409-conflict burst. Part of the ~14-min tunnel-creation story.
• Tunnel routing not live until next connector heartbeat fires after Programmed=True #167 — tunnel routing waits for a connector heartbeat instead of pushing addressing proactively (same story, second delay).

Observability / prevention

• Generalize reconcile-error-ratio alerts to all controllers #289 — reconcile-error-ratio alerts for all controllers. Fills the telemetry gap: NSO's project-CP controllers aren't scraped, so convergence regressions are invisible and fixes can't be confirmed from metrics. Cheap; unblocks measuring everything above.
• Catch silent edge misconfiguration before it reaches customers #247 — silent edge misconfiguration reaching customers.
• Edge serves stale TrafficProtectionPolicy config during upstream→edge propagation with no lag visibility #266 — edge serves stale TrafficProtectionPolicy config with no lag visibility.
• Domain verification backoff has no recovery path once the record is fixed #290 — domain verification has no recovery path once the record is fixed.
• bug (provisional): nil-hostname default listener silently dropped from downstream gateway while reported Programmed=True #235 — nil-hostname default listener silently dropped while reported Programmed.
• ResourceClaims not garbage-collected when HTTPProxy owner is deleted #182 — ResourceClaims not GC'd on HTTPProxy delete.

Lower / longer-horizon

Design-scale:

• AI Edge: eliminate slow/failed TLS cert provisioning at scale via Envoy Gateway extension server #216 — cert issuance at scale.
• WAF response-body buffering: proxy memory scales with concurrent inspected responses #253 — WAF response-body buffering memory.

Docs / e2e:

• docs: HTTPRoute attachment contract for Datum gateways (spec.hostnames rejected) #288 — HTTPRoute attachment contract docs.
• docs: document gateway certificate issuance model and downstream solver #269 — gateway certificate issuance model docs.
• Track version pins in the prod-fidelity e2e env (why + unpin criteria) #278 — track version pins in the prod-fidelity e2e env.
• HTTP usage metering: attribution is by project name — harden docs/e2e (#287) + investigate malformed subjects #286 — HTTP usage metering attribution docs/e2e.

Dashboards / status plumbing:

• Update HTTP Proxy dashboards to show when downstream proxy resources are unhealthy #104 — HTTP Proxy dashboards show downstream-unhealthy state.
• Monitor envoy gateway security policy status #96 — monitor envoy gateway security policy status.
• Propagate security policy status back from edge #97 — propagate security policy status back from edge.
• Automatically propagate secrets referenced in envoy gateway policies #95 — auto-propagate secrets referenced in envoy gateway policies.

GatewayClass hygiene:

• Add validating webhook to reject Gateway creation with unmanaged GatewayClass #152 — reject Gateway with unmanaged GatewayClass.
• Requeue gateway reconciliation on failed initial reconcile #153 — requeue gateway on failed initial reconcile.
• Investigate and remove unmanaged 'datum' GatewayClass from project control planes #154 — remove unmanaged 'datum' GatewayClass from project control planes.

Networking / deps:

• Investigate ways to make domain NS refresh faster than 24hrs #88 — make domain NS refresh faster than 24h.
• IPv6 private addressing strategy for project networking #163 — IPv6 private addressing strategy.
• Integrate IPAM and compute networking interface #164 — integrate IPAM and compute networking interface.
• Upgrade iroh to 1.0.2 across app, iroh-gateway, and infra #168 — upgrade iroh to 1.0.2.

Misc:

• Bug: changing origin endpoint from FQDN to IP puts the httpproxy is pending state #126 — FQDN↔IP origin flip leaves proxy pending (good first issue).
• Reject duplicate domain creation in webhook #86 — reject duplicate domain creation.
• Add github action to run the generation commands (make generate, manifests, etc) #90 — GitHub action to run generation commands.
• Add documentation for running the operator locally for development #19 — docs for running the operator locally.

Recommended next, best impact × actionable × rides current momentum

1. Custom-hostname DNSRecordProgrammed stuck Pending >60s on fresh zone (breaches convergence SLO) #291 + Gateway *.datumproxy.net address resolves in 65–85s, exceeding the 60s convergence SLO #292 — DNS-agent queue; metrics in hand, one fix, direct convergence win (dns-operator + infra overlay).
2. OIDC SecurityPolicy with a missing clientSecret takes down the entire shared downstream gateway (EG v1.8.x); needs e2e reproduction #194 — all-tenant outage; guard + e2e upgrade-gate.
3. Generalize reconcile-error-ratio alerts to all controllers #289 — telemetry, so the fixes above are verifiable.

Context

Convergence investigation and per-stage attribution: https://github.com/datum-cloud/infra/issues/3622.