test: gate OIDC missing-secret gateway isolation - #312
Conversation
Add the end-to-end upgrade gate for issue #194: one tenant's OIDC SecurityPolicy that references a missing clientSecret must not black out the shared datum-downstream-gateway for every other tenant. The poison is Envoy Gateway behaviour (envoyproxy/gateway#6123, open): on a missing clientSecret, EG v1.8.x emits a config-less oauth2 listener filter, Envoy rejects it, and the atomic listener snapshot drops the whole listener set. An already-running proxy hides this by keeping its last-good config; only a fresh pod pulling the xDS snapshot cold comes up serving nothing. A steady-state test therefore passes on a broken build. Key changes: - Add a reusable proxy-restart step (_steps/restart-downstream-proxy.yaml) that deletes the shared proxy pod and waits for a fresh Ready pod, so the fresh-xDS-pull outage becomes observable. - Add the oidc-missing-secret-isolation scenario with two arms. Arm 1 asserts the #304 guard holds the policy off the shared gateway (Accepted=False, reason=PendingSecret; no downstream projection) and the neighbour keeps serving. Arm 2 hand-delivers the poison directly onto the shared gateway, restarts the proxy, and asserts the neighbour still serves (benign 200 / attack 403), the bad route fails on its own (per-route 5xx), the restarted proxy carries active listener filter chains, and EG logs no "config must be present" rejection. This is the EG-upgrade gate: green on the pinned v1.7.4 and with the guard, red only if EG is bumped to v1.8.x while the upstream bug is unfixed. The scenario header and README say so, so "green" is never misread as "the upstream bug is fixed".
|
@ecv should I see a failing test here? |
The federated edge suite runs an explicit scenario allowlist, so a new scenario folder is inert until it is named there. The OIDC missing-secret isolation test was never executed by CI — the green Federated E2E check carried no signal for it, which is the failure mode the test exists to prevent. Add the scenario to DEFAULT_SCENARIOS so the two-cluster run picks it up, and list its folder in the suite README alongside the other guarantees.
|
Oops! Yeah, there totally should be. There is now. At least, the test should run green, and only fail if envoyproxy/gateway#6123 hits us. |
|
The scenario was never actually running. First real run came back red, and it's a genuine find rather than the upgrade gate tripping: Arm 1 times out waiting for Filed as #322. Arm 2 never gets to run, so the EG v1.7.4 upgrade gate is still unverified — moved this to draft until #322 lands. |
2 participants
Summary
One tenant's broken login policy can black out the shared gateway for every tenant on it. A tenant OIDC
SecurityPolicypointing at a missing secret gets pushed to the shared proxy; the running proxy masks the damage by holding its last-good config, but the next restart reloads from scratch and comes up serving nothing — taking every neighbour down.This adds the e2e test that locks in the containment: one tenant's mistake stays on its own route, even across a proxy restart. The load-bearing piece is a reusable proxy-restart step — without a forced restart the test would pass on a broken build.
Arm 1 — bad policy never reaches the shared proxy. OIDC policy whose secret is never created is held back (
Accepted=False, reason=PendingSecret), nothing projected downstream, neighbour keeps serving.Arm 2 — from-scratch reload stays healthy. Bad config forced directly onto the shared gateway, proxy restarted; afterwards the neighbour still serves, the bad route fails alone, listeners come up intact.
It also guards upgrades: passes on pinned Envoy Gateway v1.7.4 plus the #304 hold, goes red only if Envoy Gateway is bumped back into the still-open upstream bug. The scenario and README say so, so a green run is never misread as "upstream fixed".
Note
Root cause is an open upstream Envoy Gateway bug (envoyproxy/gateway#6123). This test does not fix it — it stops a silent re-upgrade into it and proves the #304 hold works.
Test plan
e2e-edge suite against the real shared data plane; not runnable in envtest or locally.
oidc-missing-secret-isolationpasses on pinned Envoy Gateway v1.7.4Related to #194