docs: add downstream-orphan troubleshooting doc - #61
Open
scotwells wants to merge 7 commits into
Open
Conversation
scotwells
force-pushed
the
feat/repro-replicator-downstream-orphan
branch
from
July 24, 2026 19:14
fb1afdc to
5890bc2
Compare
scotwells
changed the base branch from
feat/dns-federation-test-env
to
docs/architecture-reorg
July 24, 2026 19:14
Support-facing doc for the DNSRecordSet downstream-orphan failure mode behind engineering#346: symptoms, why it happens, and concrete steps to investigate and resolve it. Links out to the topology and replication docs for background on the upstream/downstream design.
scotwells
force-pushed
the
feat/repro-replicator-downstream-orphan
branch
from
July 24, 2026 19:19
5890bc2 to
410bfa4
Compare
scotwells
changed the title
Convert the symptom description and the investigation/resolution callouts to GitHub's blockquote-based alert syntax instead of bolded lead-ins.
This doc is public-facing; the wiki should link to it, not the other way around. Replace the wiki cross-link with a one-sentence inline explanation of upstream/downstream, and keep the links to this repo's own architecture docs.
Leftover from the wiki-link removal; nothing for it to point at anymore.
The intro paragraph and 'What this means' both defined the terms almost verbatim; keep the definition in the intro only.
'Skipped' read as if the cleanup were optional; it's a bug, not a deliberate omission.
- Use "downstream copy" consistently (one spot said "downstream
shadow").
- Replace ambiguous "this"/"it" references with concrete nouns.
- Switch passive constructions ("gets rejected", "is missed", "is
never told") to active voice with a clear subject.
- Cut filler words (really, actually, just, quietly, currently,
itself).
- Tighten run-on sentences into shorter, single-idea sentences.
ecv
approved these changes
Jul 26, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds a troubleshooting doc for a "conflicting record" error that's hard to diagnose without it: a customer can't create or edit a DNS record because PowerDNS reports a naming conflict, even though nothing in their project references that name anymore. The doc walks through recognizing the symptom, confirming it's this specific failure mode, and resolving it with concrete
kubectlcommands — no need to already know the replicator's internals to act on it.This same failure caused a real production incident (engineering#346): a customer's
www.ab.dkrecord was stuck for hours because its old downstream copy was never cleaned up after the customer deleted it. This doc turns that incident into a repeatable runbook.Root cause
A DNS record exists both upstream (what the customer manages) and downstream (what actually gets programmed into PowerDNS). Deleting the upstream record is supposed to clean up its downstream copy too. That cleanup depends on the replicator catching the deletion while it happens; if a create and delete race closely enough, the replicator can miss it; a stale cached read tells it the downstream copy doesn't exist yet, so it finishes up and moves on without ever deleting the copy it just created. The result looks and behaves like a healthy record, so it's easy to miss — until a customer tries to reuse that name and gets rejected as a duplicate, indefinitely.
What's included
docs/troubleshooting/dnsrecordset-downstream-orphan.md— symptoms, why it happens, and concrete steps (withkubectlcommands) to investigate and resolve it. Links out to Topology and Replication from docs: Reorganize README and architecture documentation #64 for background on the upstream/downstream design.Stacked on #64 so those links resolve against real files.
Test plan
kubectlcommands in the investigation/resolution steps are accurate against a real upstream/downstream pair