fix: Stop respelled owner names deleting live PowerDNS RRsets - #75
Merged
Merged
Conversation
PowerDNS keys an RRset on one absolute name, but the reconcile queue is keyed on the raw spec owner name. "api", "api.example.com." and "@" at the apex all qualify to the same RRset, and the DNSRecordSet update handler enqueues both ObjectOld and ObjectNew — so rewriting an owner name produces two requests for one RRset. The request carrying the retired spelling finds no owning DNSRecordSet, falls into the cleanup branch, and deletes what the live request just wrote. The record disappears from PowerDNS while the DNSRecordSet still reports Programmed=True, and the reconciler has no periodic requeue, so it stays gone until the next watch event. Guard both delete paths with aliasedOwnerExists: only delete once no other spelling of the owner name still claims the RRset. The spelling being reconciled is excluded, so a genuinely removed owner still cleans up. This is reachable in production now — network-services-operator#318 moved Gateway DNS to relative owner names, so every gateway-managed record undergoes exactly this rewrite once. Second, a coexistence 422 no longer returns as a reconcile error. It is not transient: the record is well-formed but cannot share the name with existing data, so exponential retry cannot clear it and only pins ControllerReconcileErrorRatioCritical high for the controller. Requeue on a long interval instead, since the conflicting RRset may be removed outside Kubernetes with no watch event to wake us. Status still carries Programmed=False with reason Conflict. QualifyOwner is exported so the controller can compare owner-name spellings for RRset identity, and NewAPIError so callers outside the package can build the error shapes IsConflict and FriendlyMessage classify. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
|
aflor024
requested review from
a team and
scotwells
and removed request for
a team
July 29, 2026 15:28
ecv
approved these changes
Jul 29, 2026
This was referenced Jul 29, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
PowerDNS keys an RRset on a single absolute name, but the reconcile queue is keyed on the raw owner name from the spec.
api,api.example.com.and@at the apex all qualify to the same RRset, and theDNSRecordSetupdate handler enqueues bothObjectOldandObjectNew— so rewriting an owner name produces two requests for one RRset. The request carrying the retired spelling finds no owningDNSRecordSet, falls into the cleanup branch, and deletes what the live request just wrote. The record vanishes from PowerDNS while theDNSRecordSetstill reportsProgrammed=True, and with no periodic requeue on this reconciler it stays gone until the next watch event.Both delete paths are now guarded: the RRset is only deleted once no other spelling of the owner name still claims it. The spelling being reconciled is excluded from that check, so an owner name that was genuinely removed still gets cleaned up.
This is reachable in production today — Gateway DNS moved to relative owner names shipped in network-services-operator v0.24.8 with
enableDNSIntegrationon in staging and production, so every gateway-managed record undergoes exactly this rewrite once.Second change: a PowerDNS coexistence 422 no longer comes back as a reconcile error. It isn't transient — the record is well-formed but cannot share the name with data already there, so exponential retry can never clear it and only pins
ControllerReconcileErrorRatioCriticalhigh for this controller. It now requeues on a long interval instead. A slow poll rather than a terminal result, because the conflicting RRset may be removed outside Kubernetes with no watch event to wake us. Status still carriesProgrammed=Falsewith reasonConflict, which is what #57 added.QualifyOwneris exported so the controller can compare owner-name spellings for RRset identity, andNewAPIErrorso code outside the package can construct the error shapesIsConflictandFriendlyMessagealready classify.Note
This reduces the error ratio for conflicts and stops the data loss, but it does not stop the conflicts happening. The dual-write that creates them is network-services-operator#316.
Test plan
@and the absolute zone name are one RRsetProgrammed=False/Conflictmainwithout this changego test ./internal/controller/... ./internal/pdns/...(unit; the envtest suite needs local etcd/apiserver binaries)golangci-lint runclean on v2.4.0, the version CI pinsFixes #74
Related to #56
Related to #51