QAO-358: gate calypso.live proxy on container /health readiness

[lucatume]

Summary

Makes dserve wait until a started container's /health endpoint returns 200 OK before proxying user requests to it. While the container is running-but-not-healthy, HTML navigations see the existing auto-refresh loading page; non-HTML requests see 503 Retry-After: 2.

Addresses QAO-358: users clicking the PR comment link in Automattic/wp-calypso PRs were experiencing timeouts and wrong-site redirects (being bounced to jetpack.me etc.) because the link was served before the wp-calypso Node server had finished booting. QAO-281 measured this as a 4–16s window. The /health endpoint itself was added in Automattic/wp-calypso#109316, but that only fixed E2E test infrastructure — this PR closes the loop for the browser-facing PR-link UX.

What changed

• Health gate. New src/health.ts exposes probeContainerHealth (single probe) and pollUntilHealthy (500ms loop, 30s ceiling). A per-container healthyContainers: Set<string> in APIState tracks probe results.
• Proxy gating. Extracted decideRouteAction into src/route-actions.ts (pure function, unit-tested). The * handler and /status handler both gate on isContainerReadyToServe instead of isContainerRunning. /status now distinguishes Ready (probed healthy) from Starting (running but not yet probed).
• Fresh-start path. startContainer's success branch kicks off a probe via ensureHealthProbeFor.
• Restart-recovery path. refreshContainers (runs every 5s) calls ensureHealthProbesForRunningContainers, which probes any pre-existing dserve container not yet marked healthy — dedupe via a new probingContainers set. This is the path that catches containers surviving a dserve process restart; without it they would be stuck on the loading page forever.
• Kill switch. Env var DSERVE_HEALTH_GATE_ENABLED=false restores pre-QAO-358 behavior (proxy to any running container). Requires a dserve restart to take effect.
• Fail-open. After the 30s ceiling expires with no successful probe, the container is marked healthy anyway and a WARN log line is emitted. This restores today's behavior if /health itself is broken — dserve should never wedge a working container just because its health endpoint regressed.
• Observability. Five new StatsD counters: dserve.health.probe.{started,healthy,fail_open,aborted,error}. Alarm suggestion: rate(dserve.health.probe.fail_open[5m]) / rate(dserve.health.probe.started[5m]) > 0.1 catches silent /health regressions across the fleet.

Design notes

• Health state is keyed by Docker container Id, not commit hash, because multiple envs share a commit hash but have distinct containers.
• Non-HTML requests during the unhealthy window get 503 Retry-After: 2 rather than the loading HTML, so subresource fetches fail cleanly instead of half-rendering.
• refreshContainers calls ensureHealthProbesForRunningContainers() fire-and-forget (does NOT await it). Awaiting would stretch the 5s refresh cadence to 30s under slow probes. A regression test (test/refresh-fire-and-forget.test.ts) pins this invariant.
• ensureHealthProbeFor returns Promise<void> for test determinism. Production callers ignore the returned promise; tests await it to drain the chain without setImmediate bookkeeping. Chain ends with a terminal .catch(() => {}) as belt-and-suspenders against future drift.
• ensureHealthProbesForRunningContainers uses Promise.allSettled (not Promise.all) to avoid fail-fast orphaning sibling probes.
• startContainer's post-success lookup now passes env to getRunningContainerForHash — arguably a latent bug fix (previously could return the wrong env's container when multiple envs shared a commit).

Out of scope

• The workflow that posts the PR comment in Automattic/wp-calypso. The Linear ticket proposes making the workflow wait for /health or post-then-update. That's infeasible here because dserve builds are lazy — the container doesn't exist until someone clicks the link. The fix delivered is equivalent: the link can be posted immediately, and dserve won't proxy until the container is actually ready.
• src/image-runner.ts (the /api/images/* runner): separate code path, different UX expectations.

Test plan

Automated (56 tests, 11 suites)

• yarn test — 56 tests passing, including dedicated suites for probe helpers, state selectors, route decisions, restart recovery, and the fire-and-forget invariant.
• yarn run build-ts clean.
• Metric-name audit: grep -rn "health.probe" src/ shows exactly the 5 expected counters.

Local e2e (all scenarios passed)

Tested locally using mock Docker containers and a non-default port (38400), following the README's "e2e test locally" pattern. Port 3000 was temporarily patched to read process.env.DSERVE_PORT || 3000 for the session.

Mock container setup: Built a Python-based Docker image tagged with dserve's expected prefix (dserve-wpcalypso:<hex-sha>). The mock server returns 503 from /health for the first 15 seconds after process start, then 200 — simulating a real Calypso container's startup delay. All other paths return 200 <html>mock app</html>. Three identical images were built with distinct hex SHA tags for isolation between scenarios. Containers were triggered via ?hash=<sha> against calypso.localhost:38400.

Scenario A — Starting → Ready transition: ✅
After triggering the container, /status returned Starting at the ~8s mark (no "Container reported healthy" in logs yet). After ~18s total, /status returned Ready and the log showed "msg":"Container reported healthy" — matching the mock's 15s /health delay.

Scenario B — Non-HTML requests get 503 during unhealthy window: ✅
During the unhealthy window, curl -H 'Accept: */*' returned HTTP 503, Retry-After: 2, body Container not ready. After the container became healthy, the same curl returned HTTP 200 with mock HTML.

Scenario C — Restart recovery: ✅
Killed dserve while containers from A and B were still running, then restarted. Within ~10s (one refresh cycle), logs showed "msg":"Container reported healthy" for both existing containers, and /status returned Ready — without any user request triggering startContainer. Confirms ensureHealthProbesForRunningContainers re-discovers and re-probes containers surviving a dserve restart.

Scenario D — Kill-switch DSERVE_HEALTH_GATE_ENABLED=false: ✅
Started dserve with DSERVE_HEALTH_GATE_ENABLED=false. Triggered a fresh container. curl -H 'Accept: */*' returned HTTP 200 with mock HTML immediately, even though /health would still return 503 at that time. /status returned Ready. The background health probe still ran (one "Container reported healthy" entry appeared later), which is expected — the kill switch disables the gating policy in isContainerReadyToServe, not the probe itself.

Post-merge monitoring (production = calypso.live, auto-deployed from master every ~15 minutes)

• Tail https://calypso.live/log for the first 30 minutes after deploy. Watch for Container /health did not return 200 before ceiling; failing open warnings — a handful is normal (slow boots), a flood means something is wrong.
• Spot-check a fresh PR's Calypso Live link: the link should now be reliable — no redirect to jetpack.me, no timeout.

Rollback

Two options, pick based on urgency:

1. Instant (seconds): set DSERVE_HEALTH_GATE_ENABLED=false in dserve's environment and restart the process. Proxy behavior returns to pre-QAO-358 immediately. No code change needed.
2. Clean (~15 min): git revert the merge commit and merge — auto-deploys within the next 15-minute poll.

No DB migrations, no data format changes, no cross-service coordination required.

🤖 Generated with Claude Code

[lucatume]

@mreishus while having unit tests is great, I think it could help having a proper end-to-end testing framework in place.
bats-core seems to fit the bill for a pure bash testing framework, but the tests I've ran could as well run using Playwright.
What do you think?