prod pdns backend returns 422 on PATCH /zones; new DNSZones never Accepted

[ecv]

Summary

Prod dns-operator cannot program a cohort of DNSZones. The manager container
of the dns-operator-agent-pdns-auth-0 StatefulSet (--role=downstream)
sends PowerDNS PATCH /api/v1/servers/localhost/zones/* payloads containing
duplicate records[] within a single RRset, and pdns-auth rejects the whole
RRset with HTTP 422. Affected zones never reach Accepted=true /
Programmed=true. Well-formed existing zones are unaffected.

Root cause is operator-side payload construction, not the pdns backend. pdns
is rejecting invalid input correctly.

Surfaced via the infra-repo scheduled Integration Chainsaw suite, whose
dns-setup test has failed on every run since ~2026-07-03 20:00 UTC. The test
creates a fresh e2e-dns-<ts>.datumchainsaw.art zone and waits 5m for
Accepted=true; it times out at the 5m mark because the zone's default NS
RRset carries duplicate records (see Mode 1 below), which fails on every
newly-created zone.

The 422 rejection reasons (captured live, 2026-07-05 17:10–17:11Z)

Source (corrects the original "pdns logs length only" note — the operator logs
the full body):

kubectl -n datum-dns-system logs dns-operator-agent-pdns-auth-0 -c manager \
  | grep "pdns apply failed"

Three distinct failure modes, one common cause (duplicate records[]):

Mode 1 — duplicate NS on the default RRset (the fleet-wide blocker, incl. Chainsaw):

{"error": "Multiple errors found in RRset", "errors": [
  "RRset estate-buddy.com. IN NS: duplicate record with content \"ns1.datumdns.net\"",
  "RRset estate-buddy.com. IN NS: duplicate record with content \"ns2.datumdns.net\"",
  "RRset estate-buddy.com. IN NS: duplicate record with content \"ns3.datumdns.net\"",
  "RRset estate-buddy.com. IN NS: duplicate record with content \"ns4.datumdns.net\""
]}

The default NS RRset the operator ensures on every new zone lists each
nameserver twice. Hits every newly-created zone, including the Chainsaw test.

Mode 2 — multi-valued / duplicate CNAME:

{"error": "Multiple errors found in RRset", "errors": [
  "RRset www.ibm.com. IN CNAME: only one such record allowed",
  "RRset www.ibm.com. IN CNAME: duplicate record with content \"outer-global-dual.ibmcom-tls12.edgekey.net\""
  // repeated ~7x
]}

Mode 3 — ALIAS conflicts with a pre-existing RRset:

status 422: {"error": "RRset www.ab.dk. IN ALIAS: Conflicts with pre-existing RRset"}

Also go.miloapis.com., www.alt-cloud.org., www.erupt.dev..

Why 2026-07-03 20:00, and why not the backend

• The pdns container in dns-operator-agent-pdns-auth-0 has restartCount: 0
  since 2026-06-30 — the backend binary/config did not change at the onset.
  (Image is powerdns/pdns-auth-51:latest, a floating tag — worth pinning, but
  not the trigger.)
• The onset lines up with a cohort of new zones (demo/import domains: ibm.com,
  ab.dk, erupt.dev, estate-buddy.com, in ns-<uuid> project namespaces)
  beginning to reconcile. Their record sets trip the duplicate/single-value/ALIAS
  rules. Pre-existing well-formed zones keep reconciling green.

Impact

• Any DNSZone whose emitted payload contains a duplicate record (esp. the
  default NS RRset) fails to program in prod.
• Blocks the infra Integration Chainsaw dns-setup test on every scheduled run.

Fix

Operator-side, in the RRset build path (internal/pdns):

1. Dedupe records[] before PATCH — collapse identical content within an
   RRset. Fixes Mode 1 (NS) and Mode 2 (CNAME), including the Chainsaw blocker.
2. Enforce single-value for CNAME/ALIAS before sending.
3. ALIAS coexistence — don't emit ALIAS at a name that already holds another
   RRset (or clear the conflicting set first).

Also (separate, diagnosability): surface the specific pdns error string into
the DNSRecordSet status condition instead of collapsing all 422s to the generic
FriendlyMessage — this incident was invisible via kubectl get dnsrecordset -o yaml because the detail only reached the logs (internal/pdns/errors.go).

Cross-ref: infra Integration Chainsaw Failure Tracker (datum-cloud/infra#2879).

[ecv]

Code-side analysis (dns-operator)

Walked the PATCH path in internal/pdns. Three findings that change the picture.

1. The 422 response body is already captured operator-side

The pdns webserver only logs the body length, but the operator reads the actual body and keeps it:

• applyRRSetPatch reads up to 64 KiB of the non-2xx response into pdnsAPIError{Status, Body} — internal/pdns/client.go:399-402.
• It is logged on every failing reconcile: internal/controller/dnsrecordset_powerdns_controller.go:137-139 → logger.Error(pdnsErr, "pdns apply failed"), whose error string is status 422: <body>.

So the rejection reason we said we "could not capture" is live in the manager pod logs, not the pdns pod. It rolled off before because we looked for the one-time creation-window log; the 422 actually recurs on every reconcile, so it is retrievable now:

kubectl -n datum-dns-system logs deploy/dns-operator-controller-manager \
  | grep "pdns apply failed"

2. Why the reason never reached the DNSRecordSet status

FriendlyMessage (internal/pdns/errors.go:39-58) matches a few known substrings (Invalid character, Not in zone, Conflicts with pre-existing RRset, …) and otherwise collapses all 422s to a generic "The DNS record was rejected as invalid. Check the record type and value." The specific pdns error field is dropped from the status condition — it only survives in logs. That is why kubectl get dnsrecordset -o yaml showed nothing actionable.

A follow-up PR will surface the specific pdns detail into the CR status so future incidents are diagnosable without pod-log access.

3. Not an operator regression — and broader than "new zones"

• Manager image v0.6.3 has been frozen since 2026-06-25; the 422s began ~2026-07-03 20:00 UTC. No operator change in that window → points at a backend change (pdns upgrade / config / stricter validation) around that time, matching the "what changed on the backend" ask.
• The evidence log shows PATCH 422 on existing zones too (www.ibm.com, miloapis.com, alt-cloud.org, …), with varying body lengths (77/79/81/395/1415) = per-zone distinct rejection text under a common trigger. So this is not limited to newly-created zones "never Accepted" — record-set reconciles are failing fleet-wide. The idempotent-skip claim doesn't hold; the operator is issuing PATCHes on established zones and they are being rejected.

Recommended next step

Pull the body from the manager logs (command above) to get the exact pdns error string — that determines whether the fix is backend-side (revert/adjust the ~07-03 change) or operator-side (adapt the payload to new validation).

[ecv]

The actual 422 rejection reasons

Pulled them live from prod (2026-07-05 17:10–17:11Z). Correction to my earlier comment: the pdns PATCH is not issued by deploy/dns-operator-controller-manager (that runs --role=replicator and never touches pdns — the grep there comes up empty). It's issued by the manager container of the dns-operator-agent-pdns-auth-0 StatefulSet (--role=downstream, PDNS_API_URL=http://localhost:8082). That is where pdns apply failed lives:

kubectl -n datum-dns-system logs dns-operator-agent-pdns-auth-0 -c manager \
  | grep "pdns apply failed"

The body we "couldn't capture" is fully present. It is not a single global rejection — pdns is rejecting specific malformed RRsets, in three distinct modes:

Mode 1 — duplicate NS records on the default RRset (this is the one blocking the Chainsaw test)

{"error": "Multiple errors found in RRset", "errors": [
  "RRset estate-buddy.com. IN NS: duplicate record with content \"ns1.datumdns.net\"",
  "RRset estate-buddy.com. IN NS: duplicate record with content \"ns2.datumdns.net\"",
  "RRset estate-buddy.com. IN NS: duplicate record with content \"ns3.datumdns.net\"",
  "RRset estate-buddy.com. IN NS: duplicate record with content \"ns4.datumdns.net\""
]}

The default NS RRset the operator ensures on every new zone carries each nameserver twice. pdns-auth rejects the whole RRset 422. This fires on every newly-created zone — including the Chainsaw e2e-dns-<ts>.datumchainsaw.art — which is why dns-setup times out at 5m. This is the generic, fleet-wide blocker.

Mode 2 — duplicate / multi-valued CNAME

{"error": "Multiple errors found in RRset", "errors": [
  "RRset www.ibm.com. IN CNAME: only one such record allowed",
  "RRset www.ibm.com. IN CNAME: duplicate record with content \"outer-global-dual.ibmcom-tls12.edgekey.net\"",
  ... (same pair repeated ~7x)
]}

CNAME is single-valued; the operator sends an RRset with the same CNAME content repeated many times.

Mode 3 — ALIAS conflicts with a pre-existing RRset

status 422: {"error": "RRset www.ab.dk. IN ALIAS: Conflicts with pre-existing RRset"}

Same for go.miloapis.com., www.alt-cloud.org., www.erupt.dev.. ALIAS cannot coexist with other data at the same name; the operator is writing an ALIAS where another RRset already exists.

What this changes about the diagnosis

• Common root cause across all three: the operator's PATCH payload contains duplicate records[] within a single RRset (and for single-valued types NS-dup / CNAME-dup / ALIAS-conflict, that's an immediate reject). This is operator-side payload construction, not a backend outage. pdns is behaving correctly — the payloads are invalid.
• Not a config/credentials change — the pdns container has restartCount: 0 since 2026-06-30, so the backend binary/config didn't change at the 07-03 20:00 onset. Note the image is powerdns/pdns-auth-51:latest (floating tag) — worth pinning regardless, but it's not what flipped here.
• Why 07-03 20:00: a cohort of new zones with these malformed record sets began reconciling around then (the demo/import domains — ibm.com, ab.dk, erupt.dev, estate-buddy.com — all in ns-<uuid> project namespaces). The pre-existing well-formed zones stay green because their idempotent reconciles produce valid payloads.

Fix

Operator-side, in the RRset build path (internal/pdns):

1. Dedupe records[] before PATCH — collapse identical content within an RRset. Fixes Mode 1 (NS) and Mode 2 (CNAME) directly, including the Chainsaw blocker.
2. Enforce single-value for CNAME/ALIAS — reject/collapse to one record before sending.
3. ALIAS coexistence — don't emit ALIAS at a name that already holds another RRset (or clear the conflicting set first).

The Mode-1 NS dedup is the highest-priority fix — it unblocks every new zone and the Integration Chainsaw suite. My separate follow-up (surface the pdns error string into the DNSRecordSet status condition instead of collapsing to the generic message) still stands — it would have made all of this visible via kubectl get dnsrecordset -o yaml without pod-log access.

_{Evidence: dns-operator-agent-pdns-auth-0 / manager container, prod core control plane, datum-dns-system, image v0.6.3.}

[ecv]

Fix status

Fix PR: #53 — dedupes RRset records by content at the payload boundary (buildRRSets / ReplaceRRSet) and at the source (normalizeStringSlice / normalizeDomainNameservers). Green on CI (lint + Chainsaw E2E). It resolves Mode 1 (duplicate NS) and Mode 2 (duplicate/multi-valued CNAME) — including the fleet-wide new-zone blocker and the infra Integration Chainsaw dns-setup failure.

A duplicate PR #54 (NS-only) was opened in parallel and closed as superseded by #53 (which is broader — #54 left ReplaceRRSet and normalizeDomainNameservers un-deduped).

Still open after #53 merges (keeping this issue open):

• Mode 3 — ALIAS Conflicts with pre-existing RRset: distinct semantic issue (ALIAS coexisting with another RRset at the same name), seen only on the demo/import zones, not the Chainsaw path.
• Surface the pdns error string into the DNSRecordSet status condition instead of collapsing all 422s to the generic FriendlyMessage (internal/pdns/errors.go) — this incident was invisible via kubectl get dnsrecordset -o yaml.