A production outage on 2026-07-27 blocked every DNS record create/update across all projects for ~1h: the DNSRecordSet mutating webhook was registered against a released image that contained no webhook server, so admission failed closed (datum-cloud/infra#3688, #67).
Nothing in this repo asserts that the manager actually serves the webhook. internal/webhook/dnsrecordset_mutating_test.go covers the handler's mutation logic against a constructed request, but the handler is only reachable if cmd/main.go wires it into a running webhook server — and that wiring lives inside the role switch, so it can be present for one role and absent for another (or absent entirely from a given build) with no test failing.
That is exactly the gap the outage fell through: the code was on main, the released v0.6.4 had no internal/webhook/ at all, and nothing signalled the difference.
Proposal
Add an envtest-backed test that boots the manager for each role that is supposed to serve admission (replicator, all) and asserts:
- the webhook server is listening on the configured port
POST /mutate-dns-networking-miloapis-com-v1alpha1-dnsrecordset returns a valid AdmissionReview (not a connection error, not a 404)- a
DNSRecordSet create through the envtest API server with the MutatingWebhookConfiguration applied comes back carrying the display-name / display-value annotations
The third assertion is the valuable one: it exercises registration, TLS, path, and handler as one unit, which is the shape the outage broke.
Roles that intentionally do not serve admission (downstream) should be asserted as such, so the split stays deliberate rather than incidental.
Notes
This is the component-owned half of the fix. The other half — shipping the MutatingWebhookConfiguration alongside the image so config and code cannot version-skew — is filed separately.
A production outage on 2026-07-27 blocked every DNS record create/update across all projects for ~1h: the
DNSRecordSetmutating webhook was registered against a released image that contained no webhook server, so admission failed closed (datum-cloud/infra#3688, #67).Nothing in this repo asserts that the manager actually serves the webhook.
internal/webhook/dnsrecordset_mutating_test.gocovers the handler's mutation logic against a constructed request, but the handler is only reachable ifcmd/main.gowires it into a running webhook server — and that wiring lives inside theroleswitch, so it can be present for one role and absent for another (or absent entirely from a given build) with no test failing.That is exactly the gap the outage fell through: the code was on
main, the releasedv0.6.4had nointernal/webhook/at all, and nothing signalled the difference.Proposal
Add an envtest-backed test that boots the manager for each role that is supposed to serve admission (
replicator,all) and asserts:POST /mutate-dns-networking-miloapis-com-v1alpha1-dnsrecordsetreturns a validAdmissionReview(not a connection error, not a 404)DNSRecordSetcreate through the envtest API server with theMutatingWebhookConfigurationapplied comes back carrying thedisplay-name/display-valueannotationsThe third assertion is the valuable one: it exercises registration, TLS, path, and handler as one unit, which is the shape the outage broke.
Roles that intentionally do not serve admission (
downstream) should be asserted as such, so the split stays deliberate rather than incidental.Notes
This is the component-owned half of the fix. The other half — shipping the
MutatingWebhookConfigurationalongside the image so config and code cannot version-skew — is filed separately.