From 410bfa4dc708353f58bb1d90486516495e7789e0 Mon Sep 17 00:00:00 2001 From: Scot Wells Date: Fri, 24 Jul 2026 14:18:56 -0500 Subject: [PATCH 1/7] docs: add downstream-orphan troubleshooting doc Support-facing doc for the DNSRecordSet downstream-orphan failure mode behind engineering#346: symptoms, why it happens, and concrete steps to investigate and resolve it. Links out to the topology and replication docs for background on the upstream/downstream design. --- .../dnsrecordset-downstream-orphan.md | 147 ++++++++++++++++++ 1 file changed, 147 insertions(+) create mode 100644 docs/troubleshooting/dnsrecordset-downstream-orphan.md diff --git a/docs/troubleshooting/dnsrecordset-downstream-orphan.md b/docs/troubleshooting/dnsrecordset-downstream-orphan.md new file mode 100644 index 0000000..c431b0b --- /dev/null +++ b/docs/troubleshooting/dnsrecordset-downstream-orphan.md @@ -0,0 +1,147 @@ +# A DNS record can't be created because of a "conflicting record" error + +**Symptom:** A customer can't create or edit a DNS record. The record's +status shows it failed to program, with a message like: + +``` +PDNSError: A conflicting record already exists for this name. Remove the +existing record and try again. +``` + +...but nothing visible in the customer's project actually references that +name anymore. + +Background on how DNS records are represented across control planes (and +how to access each one) is covered in the wiki's +[Multi-Tenancy: Upstream & Downstream Control Planes](https://wiki.datum.net/infrastructure/dns/multi-tenancy) +doc — read that first if the upstream/downstream terminology below is +unfamiliar. For how this operator itself is designed, see this repo's +architecture docs: +- [Topology](../architecture/topology.md) — the replicator and + downstream-agent roles, the control planes involved, and discovery modes. +- [Replication](../architecture/replication.md) — the shadow-object model, + namespace mapping, and status synthesis this doc relies on below. + +## What this means + +Every DNS record has an upstream copy (what the customer sees) and a +downstream copy (what actually gets written to the DNS provider). Deleting +the upstream record is supposed to clean up the downstream copy too. When +that cleanup is skipped, the downstream copy keeps quietly reasserting +itself as the owner of that name — with no upstream object left for the +customer, or for us, to point at from their side. Any new record request +for that same name then gets rejected as a duplicate, indefinitely, because +nothing in the system still considers itself responsible for removing the +old one. + +## Why this happens + +The cleanup of a downstream copy currently depends on catching the upstream +record's deletion *while it's happening*. If that moment is missed — for +example, because of a service restart or a timing gap right around when the +record was deleted — the downstream copy is never told its upstream is +gone, and nothing later re-checks for that condition on its own. It's not +a stuck or crashed object; it looks and behaves perfectly healthy, which is +part of what makes it easy to miss. + +This caused a [production incident](https://github.com/datum-cloud/engineering/issues/346) +where a customer's `www.ab.dk` couldn't be repointed to a new destination +for about 24 hours, because a downstream record from a previously-deleted +AI Edge was never cleaned up. + +## How to investigate + +These steps assume `kubectl` contexts named `upstream` (the customer's +project control plane) and `downstream` (the shared DNS infrastructure +cluster) — substitute whatever your environment actually calls them. + +1. **Confirm the upstream record really doesn't exist.** List + `DNSRecordSet`s in the customer's project namespace for the name in + question: + + ```bash + kubectl --context upstream -n get dnsrecordset \ + -o custom-columns=NAME:.metadata.name,TYPE:.spec.recordType + ``` + + If nothing there references the stuck name, but the zone still reports + a conflict for it, that's the signal to keep going. + +2. **Find the downstream namespace for this project.** The replicator maps + each upstream namespace to a downstream namespace named `ns-`, + where `` is the upstream namespace's `metadata.uid` (see + [Replication](../architecture/replication.md) for the mapping + strategy). Look it up directly instead of guessing: + + ```bash + uid=$(kubectl --context upstream get namespace \ + -o jsonpath='{.metadata.uid}') + echo "ns-$uid" + ``` + +3. **Look for a leftover downstream copy** in that namespace on the shared + DNS infrastructure cluster: + + ```bash + kubectl --context downstream -n "ns-$uid" get dnsrecordset -o yaml + ``` + + You're looking for a record that claims the same name the customer is + stuck on, is otherwise healthy (`status.conditions[Programmed]=True`), + but whose upstream reference no longer resolves to anything. + +4. **Confirm it's actually orphaned.** Every downstream shadow carries the + upstream namespace it came from in an annotation. Check it, then verify + that namespace (or the record in it) is really gone, not just + temporarily unreachable: + + ```bash + kubectl --context downstream -n "ns-$uid" get dnsrecordset \ + -o jsonpath='{.metadata.annotations.meta\.datumapis\.com/upstream-namespace}' + + # then, using that value: + kubectl --context upstream -n get dnsrecordset + # expect: Error from server (NotFound) + ``` + + If that upstream `get` returns `NotFound` while the downstream copy + above is healthy and has no `deletionTimestamp`, this is a confirmed + orphan, not a timing artifact — a genuinely deleted object leaves no + `DeletionTimestamp` behind for anything to react to. + +5. **Optional:** confirm what the DNS provider itself currently has on + record for that name, to double check it matches what you found + downstream before touching anything. See + [the PowerDNS backend doc](../architecture/backends/powerdns.md) for how + records are queried directly against the backend. + +## How to resolve + +1. Delete the orphaned downstream record — this should trigger the normal + cleanup path (removing the record from the DNS provider), not leave it + out of sync: + + ```bash + kubectl --context downstream -n "ns-$uid" delete dnsrecordset + ``` + + Confirm nothing else references the object first (check + `metadata.ownerReferences` for an anchor `ConfigMap` — see + [Replication](../architecture/replication.md) — before deleting). + +2. Confirm the customer's originally-stuck record programs successfully + shortly afterward: + + ```bash + kubectl --context upstream -n get dnsrecordset \ + -o jsonpath='{.status.conditions}' + ``` + +## Longer-term fix + +This is a gap, not a one-off fluke: the system only knows how to react to +*watching* an upstream deletion happen, not to *discovering* after the +fact that an upstream record is permanently gone. Until that's fixed, +expect this to recur under similar conditions (deletions racing with a +restart or reconnect), and treat any similar "conflicting record" report +with no matching upstream object as a candidate for this same root cause. From d1b09748e2c99f80a6f36c4112a43ab937408af1 Mon Sep 17 00:00:00 2001 From: Scot Wells Date: Fri, 24 Jul 2026 14:29:14 -0500 Subject: [PATCH 2/7] docs: use GitHub callout syntax in the troubleshooting doc Convert the symptom description and the investigation/resolution callouts to GitHub's blockquote-based alert syntax instead of bolded lead-ins. --- .../dnsrecordset-downstream-orphan.md | 44 ++++++++++--------- 1 file changed, 24 insertions(+), 20 deletions(-) diff --git a/docs/troubleshooting/dnsrecordset-downstream-orphan.md b/docs/troubleshooting/dnsrecordset-downstream-orphan.md index c431b0b..8c79ec8 100644 --- a/docs/troubleshooting/dnsrecordset-downstream-orphan.md +++ b/docs/troubleshooting/dnsrecordset-downstream-orphan.md @@ -1,15 +1,16 @@ # A DNS record can't be created because of a "conflicting record" error -**Symptom:** A customer can't create or edit a DNS record. The record's -status shows it failed to program, with a message like: - -``` -PDNSError: A conflicting record already exists for this name. Remove the -existing record and try again. -``` - -...but nothing visible in the customer's project actually references that -name anymore. +> [!NOTE] +> **Symptom:** A customer can't create or edit a DNS record. The record's +> status shows it failed to program, with a message like: +> +> ``` +> PDNSError: A conflicting record already exists for this name. Remove the +> existing record and try again. +> ``` +> +> ...but nothing visible in the customer's project actually references that +> name anymore. Background on how DNS records are represented across control planes (and how to access each one) is covered in the wiki's @@ -51,9 +52,10 @@ AI Edge was never cleaned up. ## How to investigate -These steps assume `kubectl` contexts named `upstream` (the customer's -project control plane) and `downstream` (the shared DNS infrastructure -cluster) — substitute whatever your environment actually calls them. +> [!NOTE] +> These steps assume `kubectl` contexts named `upstream` (the customer's +> project control plane) and `downstream` (the shared DNS infrastructure +> cluster) — substitute whatever your environment actually calls them. 1. **Confirm the upstream record really doesn't exist.** List `DNSRecordSet`s in the customer's project namespace for the name in @@ -104,10 +106,11 @@ cluster) — substitute whatever your environment actually calls them. # expect: Error from server (NotFound) ``` - If that upstream `get` returns `NotFound` while the downstream copy - above is healthy and has no `deletionTimestamp`, this is a confirmed - orphan, not a timing artifact — a genuinely deleted object leaves no - `DeletionTimestamp` behind for anything to react to. + > [!IMPORTANT] + > If that upstream `get` returns `NotFound` while the downstream copy + > above is healthy and has no `deletionTimestamp`, this is a confirmed + > orphan, not a timing artifact — a genuinely deleted object leaves no + > `DeletionTimestamp` behind for anything to react to. 5. **Optional:** confirm what the DNS provider itself currently has on record for that name, to double check it matches what you found @@ -125,9 +128,10 @@ cluster) — substitute whatever your environment actually calls them. kubectl --context downstream -n "ns-$uid" delete dnsrecordset ``` - Confirm nothing else references the object first (check - `metadata.ownerReferences` for an anchor `ConfigMap` — see - [Replication](../architecture/replication.md) — before deleting). + > [!WARNING] + > Confirm nothing else references the object first (check + > `metadata.ownerReferences` for an anchor `ConfigMap` — see + > [Replication](../architecture/replication.md)) before deleting. 2. Confirm the customer's originally-stuck record programs successfully shortly afterward: From 2d737dd01b8e8ed92f97fcfd1b0d2a1fb6b004cf Mon Sep 17 00:00:00 2001 From: Scot Wells Date: Fri, 24 Jul 2026 14:30:27 -0500 Subject: [PATCH 3/7] docs: drop the internal wiki link, inline the terminology instead This doc is public-facing; the wiki should link to it, not the other way around. Replace the wiki cross-link with a one-sentence inline explanation of upstream/downstream, and keep the links to this repo's own architecture docs. --- docs/troubleshooting/dnsrecordset-downstream-orphan.md | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/docs/troubleshooting/dnsrecordset-downstream-orphan.md b/docs/troubleshooting/dnsrecordset-downstream-orphan.md index 8c79ec8..0955eec 100644 --- a/docs/troubleshooting/dnsrecordset-downstream-orphan.md +++ b/docs/troubleshooting/dnsrecordset-downstream-orphan.md @@ -12,11 +12,11 @@ > ...but nothing visible in the customer's project actually references that > name anymore. -Background on how DNS records are represented across control planes (and -how to access each one) is covered in the wiki's -[Multi-Tenancy: Upstream & Downstream Control Planes](https://wiki.datum.net/infrastructure/dns/multi-tenancy) -doc — read that first if the upstream/downstream terminology below is -unfamiliar. For how this operator itself is designed, see this repo's +Every DNS record exists in two places: an **upstream** copy in the +customer's own project control plane, and a **downstream** copy on the +shared DNS infrastructure cluster that actually gets programmed into the +DNS provider. Read that first if the upstream/downstream terminology below +is unfamiliar. For how this operator itself is designed, see this repo's architecture docs: - [Topology](../architecture/topology.md) — the replicator and downstream-agent roles, the control planes involved, and discovery modes. From 487e2688c2813a11708fbfb761b4df0e70943b3b Mon Sep 17 00:00:00 2001 From: Scot Wells Date: Fri, 24 Jul 2026 14:33:15 -0500 Subject: [PATCH 4/7] docs: fix dangling 'read that first' reference Leftover from the wiki-link removal; nothing for it to point at anymore. --- docs/troubleshooting/dnsrecordset-downstream-orphan.md | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-) diff --git a/docs/troubleshooting/dnsrecordset-downstream-orphan.md b/docs/troubleshooting/dnsrecordset-downstream-orphan.md index 0955eec..fdfe496 100644 --- a/docs/troubleshooting/dnsrecordset-downstream-orphan.md +++ b/docs/troubleshooting/dnsrecordset-downstream-orphan.md @@ -15,9 +15,8 @@ Every DNS record exists in two places: an **upstream** copy in the customer's own project control plane, and a **downstream** copy on the shared DNS infrastructure cluster that actually gets programmed into the -DNS provider. Read that first if the upstream/downstream terminology below -is unfamiliar. For how this operator itself is designed, see this repo's -architecture docs: +DNS provider. Keep that distinction in mind for the terminology below. For +how this operator itself is designed, see this repo's architecture docs: - [Topology](../architecture/topology.md) — the replicator and downstream-agent roles, the control planes involved, and discovery modes. - [Replication](../architecture/replication.md) — the shadow-object model, From a17cf4ef271cee87df2b7bf97220c842389ab6c7 Mon Sep 17 00:00:00 2001 From: Scot Wells Date: Fri, 24 Jul 2026 14:34:46 -0500 Subject: [PATCH 5/7] docs: remove duplicate upstream/downstream definition The intro paragraph and 'What this means' both defined the terms almost verbatim; keep the definition in the intro only. --- docs/troubleshooting/dnsrecordset-downstream-orphan.md | 6 ++---- 1 file changed, 2 insertions(+), 4 deletions(-) diff --git a/docs/troubleshooting/dnsrecordset-downstream-orphan.md b/docs/troubleshooting/dnsrecordset-downstream-orphan.md index fdfe496..22c4e4d 100644 --- a/docs/troubleshooting/dnsrecordset-downstream-orphan.md +++ b/docs/troubleshooting/dnsrecordset-downstream-orphan.md @@ -24,10 +24,8 @@ how this operator itself is designed, see this repo's architecture docs: ## What this means -Every DNS record has an upstream copy (what the customer sees) and a -downstream copy (what actually gets written to the DNS provider). Deleting -the upstream record is supposed to clean up the downstream copy too. When -that cleanup is skipped, the downstream copy keeps quietly reasserting +Deleting the upstream record is supposed to clean up the downstream copy +too. When that cleanup is skipped, the downstream copy keeps quietly reasserting itself as the owner of that name — with no upstream object left for the customer, or for us, to point at from their side. Any new record request for that same name then gets rejected as a duplicate, indefinitely, because From ecb293b0186c35e287c13180cb359553e3786036 Mon Sep 17 00:00:00 2001 From: Scot Wells Date: Fri, 24 Jul 2026 14:36:46 -0500 Subject: [PATCH 6/7] docs: clarify the cleanup failure isn't intentional 'Skipped' read as if the cleanup were optional; it's a bug, not a deliberate omission. --- .../dnsrecordset-downstream-orphan.md | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/docs/troubleshooting/dnsrecordset-downstream-orphan.md b/docs/troubleshooting/dnsrecordset-downstream-orphan.md index 22c4e4d..89a2d73 100644 --- a/docs/troubleshooting/dnsrecordset-downstream-orphan.md +++ b/docs/troubleshooting/dnsrecordset-downstream-orphan.md @@ -25,12 +25,12 @@ how this operator itself is designed, see this repo's architecture docs: ## What this means Deleting the upstream record is supposed to clean up the downstream copy -too. When that cleanup is skipped, the downstream copy keeps quietly reasserting -itself as the owner of that name — with no upstream object left for the -customer, or for us, to point at from their side. Any new record request -for that same name then gets rejected as a duplicate, indefinitely, because -nothing in the system still considers itself responsible for removing the -old one. +too. When that cleanup fails to happen, the downstream copy keeps quietly +reasserting itself as the owner of that name — with no upstream object left +for the customer, or for us, to point at from their side. Any new record +request for that same name then gets rejected as a duplicate, indefinitely, +because nothing in the system still considers itself responsible for +removing the old one. ## Why this happens From d6fdcb68766ec6736684db1e6e5c49d8cb15d750 Mon Sep 17 00:00:00 2001 From: Scot Wells Date: Fri, 24 Jul 2026 14:38:50 -0500 Subject: [PATCH 7/7] docs: apply Google technical writing guidelines - Use "downstream copy" consistently (one spot said "downstream shadow"). - Replace ambiguous "this"/"it" references with concrete nouns. - Switch passive constructions ("gets rejected", "is missed", "is never told") to active voice with a clear subject. - Cut filler words (really, actually, just, quietly, currently, itself). - Tighten run-on sentences into shorter, single-idea sentences. --- .../dnsrecordset-downstream-orphan.md | 102 +++++++++--------- 1 file changed, 50 insertions(+), 52 deletions(-) diff --git a/docs/troubleshooting/dnsrecordset-downstream-orphan.md b/docs/troubleshooting/dnsrecordset-downstream-orphan.md index 89a2d73..7bfbe04 100644 --- a/docs/troubleshooting/dnsrecordset-downstream-orphan.md +++ b/docs/troubleshooting/dnsrecordset-downstream-orphan.md @@ -9,14 +9,13 @@ > existing record and try again. > ``` > -> ...but nothing visible in the customer's project actually references that -> name anymore. +> But nothing in the customer's project references that name anymore. Every DNS record exists in two places: an **upstream** copy in the customer's own project control plane, and a **downstream** copy on the -shared DNS infrastructure cluster that actually gets programmed into the -DNS provider. Keep that distinction in mind for the terminology below. For -how this operator itself is designed, see this repo's architecture docs: +shared DNS infrastructure cluster that gets programmed into the DNS +provider. This doc uses upstream and downstream in that sense throughout. +For details on how this operator is designed, see: - [Topology](../architecture/topology.md) — the replicator and downstream-agent roles, the control planes involved, and discovery modes. - [Replication](../architecture/replication.md) — the shadow-object model, @@ -24,25 +23,25 @@ how this operator itself is designed, see this repo's architecture docs: ## What this means -Deleting the upstream record is supposed to clean up the downstream copy -too. When that cleanup fails to happen, the downstream copy keeps quietly -reasserting itself as the owner of that name — with no upstream object left -for the customer, or for us, to point at from their side. Any new record -request for that same name then gets rejected as a duplicate, indefinitely, -because nothing in the system still considers itself responsible for -removing the old one. +Deleting the upstream record should also clean up the downstream copy. When +that cleanup fails, the downstream copy keeps acting as the owner of that +name, and no upstream object remains for the customer, or for support, to +point to. PowerDNS then rejects any new record request for that name as a +duplicate, indefinitely, because nothing in the system is responsible for +removing the old downstream copy. ## Why this happens -The cleanup of a downstream copy currently depends on catching the upstream -record's deletion *while it's happening*. If that moment is missed — for -example, because of a service restart or a timing gap right around when the -record was deleted — the downstream copy is never told its upstream is -gone, and nothing later re-checks for that condition on its own. It's not -a stuck or crashed object; it looks and behaves perfectly healthy, which is -part of what makes it easy to miss. +The replicator only cleans up a downstream copy by catching the upstream +record's deletion *while it happens*. If the replicator misses that moment +— for example, during a service restart, or a timing gap right around the +deletion — the downstream copy never learns its upstream is gone. Nothing +later re-checks for that condition. The downstream copy isn't stuck or +crashed; it looks and behaves like a healthy record, which makes it easy to +miss. -This caused a [production incident](https://github.com/datum-cloud/engineering/issues/346) +This failure mode caused a +[production incident](https://github.com/datum-cloud/engineering/issues/346) where a customer's `www.ab.dk` couldn't be repointed to a new destination for about 24 hours, because a downstream record from a previously-deleted AI Edge was never cleaned up. @@ -52,25 +51,24 @@ AI Edge was never cleaned up. > [!NOTE] > These steps assume `kubectl` contexts named `upstream` (the customer's > project control plane) and `downstream` (the shared DNS infrastructure -> cluster) — substitute whatever your environment actually calls them. +> cluster). Substitute whatever your environment calls them. -1. **Confirm the upstream record really doesn't exist.** List - `DNSRecordSet`s in the customer's project namespace for the name in - question: +1. **Confirm the upstream record doesn't exist.** List `DNSRecordSet`s in + the customer's project namespace for the name in question: ```bash kubectl --context upstream -n get dnsrecordset \ -o custom-columns=NAME:.metadata.name,TYPE:.spec.recordType ``` - If nothing there references the stuck name, but the zone still reports - a conflict for it, that's the signal to keep going. + If nothing references the name upstream, but the zone still reports a + conflict for it, continue to the next step. 2. **Find the downstream namespace for this project.** The replicator maps each upstream namespace to a downstream namespace named `ns-`, where `` is the upstream namespace's `metadata.uid` (see [Replication](../architecture/replication.md) for the mapping - strategy). Look it up directly instead of guessing: + strategy). Look up the value directly instead of guessing it: ```bash uid=$(kubectl --context upstream get namespace \ @@ -85,14 +83,14 @@ AI Edge was never cleaned up. kubectl --context downstream -n "ns-$uid" get dnsrecordset -o yaml ``` - You're looking for a record that claims the same name the customer is - stuck on, is otherwise healthy (`status.conditions[Programmed]=True`), - but whose upstream reference no longer resolves to anything. + Look for a record with the same name the customer is stuck on, showing + `status.conditions[Programmed]=True`, whose upstream reference no + longer resolves to anything. -4. **Confirm it's actually orphaned.** Every downstream shadow carries the - upstream namespace it came from in an annotation. Check it, then verify - that namespace (or the record in it) is really gone, not just - temporarily unreachable: +4. **Confirm the record is orphaned.** Every downstream copy carries the + upstream namespace it came from in an annotation. Check that + annotation, then verify the namespace (or the record in it) is gone, + not temporarily unreachable: ```bash kubectl --context downstream -n "ns-$uid" get dnsrecordset \ @@ -105,30 +103,29 @@ AI Edge was never cleaned up. > [!IMPORTANT] > If that upstream `get` returns `NotFound` while the downstream copy - > above is healthy and has no `deletionTimestamp`, this is a confirmed - > orphan, not a timing artifact — a genuinely deleted object leaves no + > above is healthy and has no `deletionTimestamp`, you've confirmed an + > orphan, not a timing artifact. A deleted object leaves no > `DeletionTimestamp` behind for anything to react to. -5. **Optional:** confirm what the DNS provider itself currently has on - record for that name, to double check it matches what you found - downstream before touching anything. See - [the PowerDNS backend doc](../architecture/backends/powerdns.md) for how - records are queried directly against the backend. +5. **Optional: confirm what the DNS provider has on record** for that + name, and check it matches what you found downstream, before making + changes. See [the PowerDNS backend doc](../architecture/backends/powerdns.md) + for how to query records directly against the backend. ## How to resolve -1. Delete the orphaned downstream record — this should trigger the normal - cleanup path (removing the record from the DNS provider), not leave it - out of sync: +1. Delete the orphaned downstream record. Deleting it triggers the normal + cleanup path, removing the record from the DNS provider, instead of + leaving the provider out of sync: ```bash kubectl --context downstream -n "ns-$uid" delete dnsrecordset ``` > [!WARNING] - > Confirm nothing else references the object first (check + > Before deleting, confirm nothing else references the object (check > `metadata.ownerReferences` for an anchor `ConfigMap` — see - > [Replication](../architecture/replication.md)) before deleting. + > [Replication](../architecture/replication.md)). 2. Confirm the customer's originally-stuck record programs successfully shortly afterward: @@ -140,9 +137,10 @@ AI Edge was never cleaned up. ## Longer-term fix -This is a gap, not a one-off fluke: the system only knows how to react to -*watching* an upstream deletion happen, not to *discovering* after the -fact that an upstream record is permanently gone. Until that's fixed, -expect this to recur under similar conditions (deletions racing with a -restart or reconnect), and treat any similar "conflicting record" report -with no matching upstream object as a candidate for this same root cause. +This failure mode is a gap in the replicator, not a one-off fluke. The +replicator only knows how to react to *watching* an upstream deletion +happen. It has no way to *discover*, after the fact, that an upstream +record is permanently gone. Until that gap is fixed, expect this failure to +recur under similar conditions, such as a deletion racing a restart or +reconnect. Treat any "conflicting record" report with no matching upstream +object as a candidate for this same root cause.