Summary
When an HTTPProxy claims a custom hostname under a freshly-created DNSZone, the per-hostname DNSRecordProgrammed condition sits at status=False, reason=Pending ("DNS record is pending") for more than 60 seconds before flipping to True (RecordCreated). That exceeds the reasonable-expectation threshold for managed-DNS/record programming (60s) defined in datum-cloud/infra#3692.
The hostname is claimed quickly (Available=True), but the DNS record it depends on stays Pending well past what a customer would reasonably wait for the hostname to start working.
Evidence
Measured across 5 back-to-back end-to-end runs on staging (each creates a fresh per-run zone, an HTTPProxy with one custom hostname, and polls DNSRecordProgrammed):
- 3 of 5 runs never reached
DNSRecordProgrammed=True within 60s (12 × 5s polls) — all 12 polls returned False / Pending, e.g.:
attempt 12/12: DNSRecordProgrammed=False reason=Pending, retrying...
timeout: <host> never reached DNSRecordProgrammed=True (RecordCreated/RecordUpdated)
DNSRecordProgrammed=False (Pending: DNS record is pending)
- The other 2 runs programmed within the window. So healthy convergence straddles ~60s — the p50 is fine, the tail is not.
Why it matters
DNSRecordProgrammed is the signal that the customer's hostname will actually resolve. A record stuck Pending for >60s is a slow-but-eventual success: the hostname eventually works, every gate stays green, but the customer experiences a hostname that takes minutes to come alive.
Where to look
The condition is owned here (NSO reflects downstream record readiness as Pending), but the underlying latency may originate in the downstream DNS record programming / PowerDNS propagation (dns-operator) rather than NSO's own reconcile. Worth confirming whether NSO is slow to create/requeue the DNSRecordSet, or whether it is faithfully waiting on a downstream program that is itself slow — and splitting the fix accordingly.
Threshold reference
enhancements#828 — "Managed DNS zone and record sets program: 60s". Per that enhancement's contract, a convergence slower than the threshold is a defect to fix, not headroom to grant.
Summary
When an
HTTPProxyclaims a custom hostname under a freshly-createdDNSZone, the per-hostnameDNSRecordProgrammedcondition sits atstatus=False, reason=Pending("DNS record is pending") for more than 60 seconds before flipping toTrue (RecordCreated). That exceeds the reasonable-expectation threshold for managed-DNS/record programming (60s) defined in datum-cloud/infra#3692.The hostname is claimed quickly (
Available=True), but the DNS record it depends on stays Pending well past what a customer would reasonably wait for the hostname to start working.Evidence
Measured across 5 back-to-back end-to-end runs on staging (each creates a fresh per-run zone, an
HTTPProxywith one custom hostname, and pollsDNSRecordProgrammed):DNSRecordProgrammed=Truewithin 60s (12 × 5s polls) — all 12 polls returnedFalse / Pending, e.g.:Why it matters
DNSRecordProgrammedis the signal that the customer's hostname will actually resolve. A record stuck Pending for >60s is a slow-but-eventual success: the hostname eventually works, every gate stays green, but the customer experiences a hostname that takes minutes to come alive.Where to look
The condition is owned here (NSO reflects downstream record readiness as
Pending), but the underlying latency may originate in the downstream DNS record programming / PowerDNS propagation (dns-operator) rather than NSO's own reconcile. Worth confirming whether NSO is slow to create/requeue theDNSRecordSet, or whether it is faithfully waiting on a downstream program that is itself slow — and splitting the fix accordingly.Threshold reference
enhancements#828 — "Managed DNS zone and record sets program: 60s". Per that enhancement's contract, a convergence slower than the threshold is a defect to fix, not headroom to grant.