Gateway DNS dual-writes CNAME and A/AAAA for v4./v6. — PowerDNS 422 hot-loop

[mattdjenkinson]

Summary

For gateway v4. / v6. hostnames, NSO ends up with two writers on the same FQDN: Gateway DNS creates CNAMEs to the apex, while another path (ExternalDNS off gateway addresses) creates A/AAAA. PowerDNS rejects the CNAME (422 Conflicts with pre-existing RRset), and dnsrecordset-powerdns retries forever.

Staging hit this on project datum-cloud (claude-test-connector-proxy, customer-api-l06tj1). Live resolution still works via the A/AAAA records; the damage is a stuck reconcile loop that trips ControllerReconcileErrorRatioCritical for dnsrecordset-powerdns (alert coverage expanded in #306 / NSO alerts applied 2026-07-22).

How we got here (staging)

1. Both CNAME and A/AAAA existed for the same v4. / v6. names (messy dual ownership; CNAME side already failing).
2. 2026-07-10 ~11:35 UTC — Domain RDAP refresh for prism.staging.env.datum.net briefly set status.nameservers to parent *.datumdomains.net (no overlap with DNSZone *.datum-staging.net). HasDNSAuthority failed, so garbageCollectDNSRecordSets deleted the four CNAMEs even though Gateway listeners/addresses had not changed.
3. 2026-07-11 ~14:54 UTC — next RDAP refresh restored the correct NS. Authority returned; NSO recreated the CNAMEs into the still-present A/AAAAs. Hot loop continues.

Actors for the Jul 10–11 delete/recreate were controllers only (control@networking.datumapis.com). No human edited the DNSRecordSets in that window.

Root cause

Two bugs that compound:

1. Dual-write. Gateway DNS (gateway_dns_controller.go) programs CNAMEs for claimed hostnames including v4. / v6. from status addresses. ExternalDNS (or the A/AAAA path feeding it) also programs those same names as address records. Conflict detection only flags same-hostname DNSRecordSets with a different managed-by, so same-FQDN A/AAAA from the other path do not block CNAME create.
2. GC on a single authority miss. ensureDNSRecordSets omits hostnames from desiredRecordSetNames when HasDNSAuthority is false, then GC deletes them. A transient Domain NS flap is enough to reap records the Gateway still wants.

dns-operator surfaces this as repeated 422s (related: datum-cloud/dns-operator#56 for cross-type conflict handling). Stopping the dual-write and softening GC is the NSO-side prevention.

Acceptance

• Exactly one writer owns v4. / v6. gateway hostnames (CNAME-to-apex or A/AAAA, not both). Prefer documenting which path is canonical and disabling the other.
• Conflict detection treats any conflicting RRset / DNSRecordSet at the same FQDN as a blocker (or clears it deliberately), not only differently-labeled peers.
• Transient Domain authority loss does not delete Gateway DNSRecordSets on a single failed check (sustained failure / backoff, or pause programming without reap).
• Unit or e2e coverage: recreate-after-authority-restore does not produce CNAME+A/AAAA at the same name; optional case that a Domain NS blip does not GC wanted CNAMEs.

Related

• Alert that pages this: NSO feat: generalize controller-health alerts to all controllers #306 (generic ControllerReconcileErrorRatioCritical)
• dns-operator#56 — PowerDNS 422 when writing over a pre-existing RRset (defense in depth, not the dual-write fix)
• dns-operator#59 — queue saturation (separate; this incident is error ratio from 422 retries, not wait latency)

[aflor024]

This flapped through seven notifications in prod today, ratios between 52% and 100%, and it's firing again as of 17:19 UTC. Acked rather than escalated, since resolution still works off the A/AAAA.

Today's burst looks like #318. It shipped in v0.24.8 at 10:08 UTC and rewrites every gateway-managed DNSRecordSet owner name from FQDN to relative form, which trips a delete bug in dns-operator: api and api.example.com. qualify to the same PowerDNS RRset, the update handler enqueues both spellings, and the retired one finds no owner and deletes what the live one just wrote. Recreating it then lands straight back on the A/AAAA that's still there — which is this issue. Full trace in a comment on #318.

Filed as datum-cloud/dns-operator#74, fixed in datum-cloud/dns-operator#75, merged 17:12 UTC. Two caveats on that:

It isn't in prod yet, and the reason is worse than a missing release. The dns-operator agent — the component that runs dnsrecordset-powerdns — had been frozen on v0.6.4 since 09:20 UTC because its Flux Kustomization was failing to apply on an env var collision, with nothing surfacing it. That's datum-cloud/infra#3716, fixed in datum-cloud/infra#3718. Prod's agent is now on v0.6.7. #75 still needs a release of its own — v0.6.7 was cut at 14:23, before it merged.

And it won't fully quiet the alert even then. #75 stops a recognised coexistence 422 from being retried as an error, but it only matches PowerDNS's single-error body shape — a conflict bundled into {"error": "Multiple errors found in RRset", "errors": [...]} still isn't classified and still hot-loops. More to the point, #75 only changes how conflicts are retried; it doesn't stop them being created, which is what this issue is about.

Worth someone checking whether any gateway hostname is currently NXDOMAIN while its DNSRecordSet still reads Programmed=True — that's what the delete bug leaves behind when the recreate loses the race, and there's no periodic requeue to recover it.

Edited: originally said prod was on v0.6.6 and needed a v0.6.7 cut. v0.6.7 already existed; the real blocker was the wedged agent Kustomization in infra#3716.

[aflor024]

Ran into a clean example of the dual-write in prod while checking something else:

ns-2f6dd006.../alt-cloud-website-proxy-3de98d4d   NotOwner   Another DNSRecordSet owns this record
ns-2f6dd006.../alt-cloud-website-proxy-e958b4f1   Conflict   A conflicting record already exists for this name

Two DNSRecordSets, same namespace, same base name. One holds ownership and fails against the pre-existing RRset, the other gets told NotOwner. Both stuck.

Six records show Conflict in prod right now, plus five more on NotOwner elsewhere that may or may not be the same shape.

dns-operator v0.6.8 shipped today so these no longer hot-loop — they sit at Programmed=False with a readable reason instead. Doesn't unstick them, still needs this fixed.