fix: Harden install-components against cert-manager/kyverno races - #36
Open
scotwells wants to merge 1 commit into
Open
fix: Harden install-components against cert-manager/kyverno races#36scotwells wants to merge 1 commit into
scotwells wants to merge 1 commit into
Conversation
A cold install-components fails intermittently in two spots. install-cert-manager waits only for the cert-manager HelmRelease to be Ready, which does not mean its validating webhook is serving yet; because install-components runs cert-manager and install-envoy-gateway-operator as concurrent deps, the gateway resources (which include a cert-manager Certificate) frequently apply before the webhook is up and fail with "connection refused" against cert-manager-webhook. Separately, install-kyverno builds a manifest from a remote source and occasionally dies on a transient TLS handshake timeout on a cold run. Wait for the cert-manager webhook Deployment + its Service endpoints before declaring cert-manager ready, and retry both the kyverno apply and the gateway-resources apply, mirroring the existing retry pattern already used for the OTel operator webhook in install-observability. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
A cold
task install-componentsfails intermittently in two spots (hit repeatedly bringing up a fresh cluster for the dns-operator drift e2e):install-cert-managerwaits only for the cert-manager HelmRelease to beReady, which does not mean the validating webhook is serving. Sinceinstall-componentsrunsinstall-cert-managerandinstall-envoy-gateway-operatoras concurrent deps, the gateway resources — which include a cert-managerCertificate— often apply before the webhook is up and fail:install-kyvernobuilds a manifest from a remote source and occasionally dies on a cold run:Changes
install-cert-managernow waits for thecert-manager-webhookDeployment and its Service endpoints before reporting ready, so downstream Certificate/Issuer applies don't hit a cold webhook.install-kyvernoretries its build+apply (transient remote-manifest fetch).install-envoy-gateway-operatorretries thegateway-resourcesapply (belt-and-suspenders against the webhook race / cold-start caBundle injection).All three mirror the retry pattern already used for the OTel operator webhook in
install-observability.Test plan
Taskfile.ymlparses (task --list-all) and is valid YAML.task cluster-up/install-componentsruns no longer flake on the cert-manager webhook or the kyverno fetch.🤖 Generated with Claude Code