fix(observability): make a cold install-observability succeed from zero

[scotwells]

Problem

task install-observability on a fresh cluster fails partway through and needs manual recovery to finish. Independent causes, each of which aborts the install on its own.

Fixes

1. OTel Collector CR install was racy — now ordered and retried. The OpenTelemetryCollector CR was listed in components/observability/kustomization.yaml, so the task's first kustomize build | kubectl apply tried to create it before its CRD/webhook existed (no matches for kind). The task does also apply it in a dedicated step — but that step only waited for the operator HelmRelease Ready + CRD Established, and neither guarantees the operator's admission webhook is serving with an injected caBundle (both are async after the Helm install), so it still failed with an x509 / no endpoints webhook error on a cold cluster. Fix: drop the CR from the bundle (so it is installed exactly once, in order), and in the dedicated step wait for the operator Deployment to be Available and retry the apply until the webhook is ready.

2. cert-manager cainjector OOMKilled. cainjector caches every CRD/APIService/webhook to inject CA bundles, so its memory scales with cluster object count. At 64Mi it OOMKills (exit 137) once the stack's CRDs/webhooks land — and until it's healthy it never injects the OTel operator's webhook caBundle (cause #1's x509). Raised to 256Mi.

3. Loki HelmRelease never went Ready. The Loki chart enables memcached chunk/results caches by default, each requesting ~8-9Gi, which stay Pending (Insufficient memory) on a single-node cluster and hold the Loki HelmRelease not-Ready — so wait helmrelease/loki fails even though everything else is up. Disabled both caches (unneeded for the SingleBinary + filesystem dev setup).

How the collector is installed now

Still installed as part of install-observability — just once, in order. It was being applied twice: prematurely in the bulk kustomize build | kubectl apply at the top of the task (before its CRD/webhook existed → the cold-start failure), and again in a dedicated step. This drops the premature bundle copy and keeps the dedicated step, which now (a) waits for the CRD Established, (b) waits for the operator Deployment Available, and (c) retries kubectl apply of the CR until the admission webhook accepts it — then waits on the collector DaemonSet rollout. components/observability has no other (e.g. Flux/GitOps) consumer, so this is the single install path.

Result

task install-observability completes clean from a fresh cluster with no manual intervention.

Verification

Applied all of this on a live cold install: cainjector stable at 256Mi, OTel Collector DaemonSet Ready, and end-to-end telemetry confirmed — assistant metrics scraped into Victoria Metrics (up{job="assistant"}=1) and traces landing in Tempo.