Skip to content

fix: Stop respelled owner names deleting live PowerDNS RRsets - #75

Merged
aflor024 merged 1 commit into
mainfrom
fix/pdns-conflict-requeue-and-stale-delete-guard
Jul 29, 2026
Merged

fix: Stop respelled owner names deleting live PowerDNS RRsets#75
aflor024 merged 1 commit into
mainfrom
fix/pdns-conflict-requeue-and-stale-delete-guard

Conversation

@aflor024

Copy link
Copy Markdown
Contributor

Summary

PowerDNS keys an RRset on a single absolute name, but the reconcile queue is keyed on the raw owner name from the spec. api, api.example.com. and @ at the apex all qualify to the same RRset, and the DNSRecordSet update handler enqueues both ObjectOld and ObjectNew — so rewriting an owner name produces two requests for one RRset. The request carrying the retired spelling finds no owning DNSRecordSet, falls into the cleanup branch, and deletes what the live request just wrote. The record vanishes from PowerDNS while the DNSRecordSet still reports Programmed=True, and with no periodic requeue on this reconciler it stays gone until the next watch event.

Both delete paths are now guarded: the RRset is only deleted once no other spelling of the owner name still claims it. The spelling being reconciled is excluded from that check, so an owner name that was genuinely removed still gets cleaned up.

This is reachable in production today — Gateway DNS moved to relative owner names shipped in network-services-operator v0.24.8 with enableDNSIntegration on in staging and production, so every gateway-managed record undergoes exactly this rewrite once.

Second change: a PowerDNS coexistence 422 no longer comes back as a reconcile error. It isn't transient — the record is well-formed but cannot share the name with data already there, so exponential retry can never clear it and only pins ControllerReconcileErrorRatioCritical high for this controller. It now requeues on a long interval instead. A slow poll rather than a terminal result, because the conflicting RRset may be removed outside Kubernetes with no watch event to wake us. Status still carries Programmed=False with reason Conflict, which is what #57 added.

QualifyOwner is exported so the controller can compare owner-name spellings for RRset identity, and NewAPIError so code outside the package can construct the error shapes IsConflict and FriendlyMessage already classify.

Note

This reduces the error ratio for conflicts and stops the data loss, but it does not stop the conflicts happening. The dual-write that creates them is network-services-operator#316.

Test plan

  • Retired owner-name spelling does not delete the live RRset, asserted in both queue orderings
  • Apex spellings alias correctly — @ and the absolute zone name are one RRset
  • A genuinely removed owner name still deletes its RRset (no over-blocking)
  • Coexistence 422 requeues without a reconcile error and reports Programmed=False/Conflict
  • All four new tests fail against main without this change
  • go test ./internal/controller/... ./internal/pdns/... (unit; the envtest suite needs local etcd/apiserver binaries)
  • golangci-lint run clean on v2.4.0, the version CI pins
  • Verify in staging that a gateway hostname surviving an owner-name rewrite keeps its RRset

Fixes #74
Related to #56
Related to #51

PowerDNS keys an RRset on one absolute name, but the reconcile queue is keyed
on the raw spec owner name. "api", "api.example.com." and "@" at the apex all
qualify to the same RRset, and the DNSRecordSet update handler enqueues both
ObjectOld and ObjectNew — so rewriting an owner name produces two requests for
one RRset. The request carrying the retired spelling finds no owning
DNSRecordSet, falls into the cleanup branch, and deletes what the live request
just wrote. The record disappears from PowerDNS while the DNSRecordSet still
reports Programmed=True, and the reconciler has no periodic requeue, so it stays
gone until the next watch event.

Guard both delete paths with aliasedOwnerExists: only delete once no other
spelling of the owner name still claims the RRset. The spelling being
reconciled is excluded, so a genuinely removed owner still cleans up.

This is reachable in production now — network-services-operator#318 moved
Gateway DNS to relative owner names, so every gateway-managed record undergoes
exactly this rewrite once.

Second, a coexistence 422 no longer returns as a reconcile error. It is not
transient: the record is well-formed but cannot share the name with existing
data, so exponential retry cannot clear it and only pins
ControllerReconcileErrorRatioCritical high for the controller. Requeue on a
long interval instead, since the conflicting RRset may be removed outside
Kubernetes with no watch event to wake us. Status still carries
Programmed=False with reason Conflict.

QualifyOwner is exported so the controller can compare owner-name spellings for
RRset identity, and NewAPIError so callers outside the package can build the
error shapes IsConflict and FriendlyMessage classify.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@cla-assistant

cla-assistant Bot commented Jul 29, 2026

Copy link
Copy Markdown

CLA assistant check
Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
You have signed the CLA already but the status is still pending? Let us recheck it.

@aflor024 aflor024 self-assigned this Jul 29, 2026
@aflor024
aflor024 requested review from a team and scotwells and removed request for a team July 29, 2026 15:28
@aflor024 aflor024 added the bug Something isn't working label Jul 29, 2026
@aflor024
aflor024 requested a review from a team July 29, 2026 15:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Renaming a DNSRecordSet owner name deletes the record that was just written

2 participants