Skip to content

fix: Harden install-components against cert-manager/kyverno races - #36

Open
scotwells wants to merge 1 commit into
mainfrom
fix/install-components-webhook-races
Open

fix: Harden install-components against cert-manager/kyverno races#36
scotwells wants to merge 1 commit into
mainfrom
fix/install-components-webhook-races

Conversation

@scotwells

Copy link
Copy Markdown
Contributor

Summary

A cold task install-components fails intermittently in two spots (hit repeatedly bringing up a fresh cluster for the dns-operator drift e2e):

  1. cert-manager webhook race. install-cert-manager waits only for the cert-manager HelmRelease to be Ready, which does not mean the validating webhook is serving. Since install-components runs install-cert-manager and install-envoy-gateway-operator as concurrent deps, the gateway resources — which include a cert-manager Certificate — often apply before the webhook is up and fail:
    failed calling webhook "webhook.cert-manager.io": ... connect: connection refused
    
  2. Kyverno transient fetch. install-kyverno builds a manifest from a remote source and occasionally dies on a cold run:
    accumulating resources from '.../kyverno/releases/download/...': net/http: TLS handshake timeout
    

Changes

  • install-cert-manager now waits for the cert-manager-webhook Deployment and its Service endpoints before reporting ready, so downstream Certificate/Issuer applies don't hit a cold webhook.
  • install-kyverno retries its build+apply (transient remote-manifest fetch).
  • install-envoy-gateway-operator retries the gateway-resources apply (belt-and-suspenders against the webhook race / cold-start caBundle injection).

All three mirror the retry pattern already used for the OTel operator webhook in install-observability.

Test plan

  • Taskfile.yml parses (task --list-all) and is valid YAML.
  • Repeated cold task cluster-up / install-components runs no longer flake on the cert-manager webhook or the kyverno fetch.

🤖 Generated with Claude Code

A cold install-components fails intermittently in two spots. install-cert-manager waits only for the cert-manager HelmRelease to be Ready, which does not mean its validating webhook is serving yet; because install-components runs cert-manager and install-envoy-gateway-operator as concurrent deps, the gateway resources (which include a cert-manager Certificate) frequently apply before the webhook is up and fail with "connection refused" against cert-manager-webhook. Separately, install-kyverno builds a manifest from a remote source and occasionally dies on a transient TLS handshake timeout on a cold run. Wait for the cert-manager webhook Deployment + its Service endpoints before declaring cert-manager ready, and retry both the kyverno apply and the gateway-resources apply, mirroring the existing retry pattern already used for the OTel operator webhook in install-observability.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant