Advanced VM install: cert-manager DNS-01, drop Caddy/lego#1336
Draft
RAVEENSR wants to merge 7 commits into
Draft
Advanced VM install: cert-manager DNS-01, drop Caddy/lego#1336RAVEENSR wants to merge 7 commits into
RAVEENSR wants to merge 7 commits into
Conversation
Contributor
|
Important Review skippedDraft detected. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Pro Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
The advanced installer used a host Caddy container plus dockerized lego, with a five-mode TLS matrix (letsencrypt | letsencrypt-dns | byoc | selfsigned | upstream). Replace all of it with a single publicly-trusted path: cert-manager's ACME DNS-01 challenge issues a wildcard certificate in-cluster and kgateway terminates TLS on :443. Works on a public or a private VM, since DNS-01 validates via a DNS TXT record (egress only, no inbound). - Add lib-certmanager.sh: renders the DNS-01 provider Secret, the ACME ClusterIssuer, the wildcard Certificate, and the single :443 gateway. Four providers (cloudflare | route53 | clouddns | azuredns) via one code path; each emits the matching cert-manager dns01 solver block. - Collapse the config to DNS-01 only (DOMAIN_BASE/ACME_EMAIL/DNS_PROVIDER + that provider's credentials); drop TLS_MODE and the byoc validate. - Consolidate the three plane gateways onto one :443 gateway with a cert-manager wildcard cert (validated: cross-namespace routes attach via allowedRoutes from All). Fixed-service routes are repointed via ocIngress overrides; deployed-agent routes via the ClusterDataPlane external ingress (now parameterized in install.sh); the rest are attached post-install. - Expose host :443 in the k3d config (advanced only; the simple path stays loopback + Caddy). - Delete lib-tls.sh (lego issuance, self-signed CA, systemd renewal) — cert-manager auto-renews in-cluster. Rework the test suites. Related to wso2#1329
The advanced tab described the removed Caddy + lego + five-TLS-mode flow with separate Public/Private network sub-tabs. Replace it with the single DNS-01 path: prerequisites, config keys, the four DNS-provider credential blocks (cloudflare/route53/clouddns/azuredns), a how-TLS-works diagram (cert-manager issues a wildcard cert, kgateway terminates :443), the one-command bootstrap install, and teardown. The public/private split collapses because DNS-01 issuance is egress-only and identical for both. Related to wso2#1329
main wso2#1340 consolidated observability into agent-manager-observer and renamed the observability chart's top-level key tracesObserver -> amObserver. Update the advanced installer's consolidated-gateway override to amObserver.ocIngress.gatewayName/gatewayNamespace so the observer route still attaches to the single :443 gateway. Related to wso2#1329
RAVEENSR
force-pushed
the
vm-advanced-cert-manager
branch
from
July 17, 2026 12:12
9587a38 to
f0e252c
Compare
The advanced installer used to consolidate every route onto one gateway in the control plane, reparenting the data-plane and observability routes onto it cross-namespace. That broke the dynamic tiers: OpenChoreo births each deployed agent's route on the data-plane gateway-default, and nothing moved it to the :443 gateway, so agents deployed after install were unreachable until their route was manually attached. Invert the model. The consolidated :443 gateway now terminates TLS and forwards by host to each plane's own gateway-default via three static front-proxy HTTPRoutes (gateway->gateway), authorized cross-namespace by ReferenceGrants. Every plane keeps its native routes, so plane separation is preserved, and the *.agents and *.thunder wildcards cover the dynamic tiers permanently — agents and per-env Thunder are reachable with no reparenting. This drops the ocIngress/external-gateway overrides and the post-install attach step entirely. Validated on a fresh VM install (DNS-01 staging): all endpoints reachable on :443, routes born native on their plane gateways, and the *.agents chain live.
The OTEL export endpoint injected into deployed agents pointed at the public https gateway (`https://<gateway-host>/otel`). In-cluster that only works by hairpinning out through the VM's public IP; a private VM has no such path, and the data-plane->control-plane :443 hop is blocked by network policy, so trace export failed there. Export is an in-cluster hop, so target the data-plane gateway over http :19080 instead, and resolve the gateway host in-cluster to the data-plane gateway service via a CoreDNS rewrite. External agents still resolve the host to the public IP via DNS and keep the public endpoint; only in-cluster resolution is redirected. Applies to both VM flavors (the override + CoreDNS render are shared). The eval/monitor LLM-judge and deployed-agent LLM proxy reach the gateway the same way and have the same latent issue; that fix lives in agent-manager-service (buildProxyURL's unimplemented internal branch) and is tracked separately.
buildProxyURL took an isInternal flag but ignored it, always returning the gateway's public vhost. In-cluster consumers that pass isInternal (the monitor/eval LLM-judge and deployed-agent LLM proxy) therefore addressed the gateway by its public https vhost. On a VM that only works by hairpinning out through the public IP; on a private VM there is no such path and the data-plane->control-plane :443 hop is blocked, so the monitor LLM-judge timed out. Implement the isInternal branch. On an on-prem deployment (IsOnPremDeployment, the default) rewrite the vhost to plain http on the kgateway port (DefaultGatewayPort), keeping the host so the gateway's host-based routing still matches — it resolves in-cluster to the data-plane gateway via the installer's CoreDNS rewrite. Cloud deployments (IsOnPremDeployment=false) keep the public vhost, which is reachable in-cluster there, so their behavior is unchanged. Companion to routing agent OTEL export at the internal data-plane gateway; both keep in-cluster gateway traffic off the public endpoint.
The goheader linter requires the repository's exact header (2026, and a double space after "implied."); the new test file used a slightly different one, failing the Lint check.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Purpose
Continues #1329. The advanced VM installer used a host Caddy container plus dockerized lego, with a five-mode TLS matrix (
letsencrypt | letsencrypt-dns | byoc | selfsigned | upstream). This replaces all of it with a single publicly-trusted path: cert-manager's ACME DNS-01 challenge issues a wildcard certificate in-cluster and kgateway terminates TLS on:443— no Caddy, no lego. Because DNS-01 validates via a DNS TXT record (egress only), the same flow works on a public or a private/firewalled VM.The simple path (sslip.io + Caddy + TLS-ALPN-01) is intentionally unchanged; this PR only touches the advanced path.
Related to #1329. Builds on #1331 (simple path + one-command delivery, merged). Depends on #1335 (hook-RBAC fix) for a clean fresh install.
Goals
Approach
deployments/vm/lib-certmanager.sh(new): renders the DNS-01 provider Secret, the ACMEClusterIssuer, the wildcardCertificate, and the consolidated:443Gateway. Four providers via one code path; each emits the matching cert-managerdns01solver block.DOMAIN_BASE/ACME_EMAIL/DNS_PROVIDER+ that provider's credentials.TLS_MODEand the byoc cert validation are removed.:443gateway: the three plane gateways are consolidated onto one HTTPS gateway with the cert-manager wildcard cert (allowedRoutes: from All). Fixed-service routes are repointed viaocIngressgateway overrides; deployed-agent routes via theClusterDataPlaneexternal ingress (now parameterized ininstall.sh); remaining routes are attached post-install.deployments/vm/lib-tls.shdeleted (lego issuance, self-signed CA, systemd renewal) — cert-manager auto-renews in-cluster.:443(the simple path stays loopback + Caddy).User stories
As an operator with my own domain, I can install Agent Manager on a public or private VM and get publicly-trusted TLS with no Caddy/lego and no per-host certificate steps — cert-manager issues and renews a wildcard cert via my DNS provider.
Release note
The advanced VM installer now issues publicly-trusted certificates via cert-manager's ACME DNS-01 challenge (Cloudflare, Route 53, Google Cloud DNS, or Azure DNS) and terminates TLS at the in-cluster gateway, replacing the Caddy + lego front door. It works on public and private VMs alike.
Documentation
Updated in this PR: the Advanced tab of
documentation/docs/getting-started/on-a-vm.mdx.Training
N/A — no training-content impact.
Certification
N/A — no impact on certification exams.
Marketing
N/A — install-experience improvement.
Automation tests
Security checks
Samples
N/A.
Related PRs
Migrations (if applicable)
N/A — VM installer path; existing installs are unaffected.
Test environment
k3d (Kubernetes v1.32) + OpenChoreo on Ubuntu; cert-manager (jetstack); Helm 3/4. Shell verified on macOS (bash 3.2 compatibility retained) and Linux.
Learning
Confirmed cert-manager cannot replicate Caddy's on-demand per-hostname issuance, so the DNS-01 wildcard (which requires a controlled zone) is what makes dropping Caddy viable — hence the advanced path requires an owned domain while the simple path keeps Caddy + sslip.io.