TrafficProtectionPolicy Enforce mode: 500 with truncated/empty body on all traffic

[kevwilliams]

Summary

TrafficProtectionPolicy (TPP) in Enforce mode causes a 100% reproducible 500 with a truncated response body, on a plain HTTPProxy with no unusual traffic (small static page, GET /, no request body). Disabling the TPP fixes it immediately.

Environment

• Project: default-ec6p8s
• Proxy: kev1n-proxy (HTTPProxy -> Gateway/HTTPRoute -> edge hostname 52f1481fa40341799b03b2bd02432592.datumproxy.net)
• Backend: https://kev1n-org.vercel.app (static Vercel-hosted site, 2408-byte HTML response)
• TrafficProtectionPolicy spec:

  apiVersion: networking.datumapis.com/v1alpha
  kind: TrafficProtectionPolicy
  spec:
    mode: Enforce
    ruleSets:
    - type: OWASPCoreRuleSet
      owaspCoreRuleSet:
        paranoiaLevels: {detection: 1, blocking: 2}
        scoreThresholds: {inbound: 5, outbound: 4}
    targetRefs:
    - group: gateway.networking.k8s.io
      kind: Gateway
      name: kev1n-proxy

• TPP status: Accepted=True (policy is correctly attached, not a translation/acceptance failure)

Repro

$ curl -v https://52f1481fa40341799b03b2bd02432592.datumproxy.net/
< HTTP/1.1 500 Internal Server Error
< content-length: 13001
< content-type: text/html; charset=UTF-8
<
* TLS alert, close notify
* end of response with 13001 bytes missing
curl: (18) end of response with 13001 bytes missing

• 100% reproducible across 5+ consecutive requests.
• Direct-to-origin (curl https://kev1n-org.vercel.app/) returns a clean 200, 2408-byte body.
• Content-Length: 13001 does not match the origin's real body size (2408 bytes) — it's suspected to be the size of the NSO extension server's branded 5xx error page (errorPage.bodyPath: /etc/datum/error-pages/error-5xx.html in the Coraza config), but zero bytes of any body are actually delivered before the connection closes.
• Disabling the TPP (deleting/detaching it from the Gateway) immediately restores normal 200 responses with the full 2408-byte body.

Isolation performed

To rule out a filter-chain interaction, I removed the HTTPProxy's own ResponseHeaderModifier filter (which adds an x-kevin-test response header) while leaving the TPP in Enforce mode:

• Before: response included x-kevin-test: true header, then 500 / Content-Length: 13001 / 0 bytes delivered.
• After removing the filter: x-kevin-test header confirmed gone (change propagated), but identical failure signature persisted — same 500, same Content-Length: 13001, same 0 bytes delivered.

This rules out interaction with other Gateway API filters (ResponseHeaderModifier) as the cause. The failure is isolated to the TPP/Coraza WAF path itself — likely a bug in how the extension server (or the underlying coraza-envoy-go-filter, currently pinned at v1.3.0-alpha4 — an alpha release) computes/sends the branded error-page Content-Length header without actually writing the corresponding body to the response, leaving the client with a truncated/incomplete response.

Given the response body here is small (2408 bytes, well under any sane SecResponseBodyLimit) and the request has no body at all (GET /), this does not appear to be a body-size-limit rejection (which was previously fixed for the response-body case in 888f22f0f, "fix: 500 response body over limit issue") — it looks like a distinct defect in the error-page/local-reply body-write path.

Suggested places to look

• apps/network-services-operator/downstream/base/extension-server-config.yaml — gateway.errorPage config (enabled: true, bodyPath: /etc/datum/error-pages/error-5xx.html).
• The extension server's / coraza-envoy-go-filter's local-reply / error-page body-write logic — headers (including a correct Content-Length) appear to be sent, but the body write step seems to be skipped or the stream closed prematurely.
• Coraza engine internal error/panic handling during the response phase that falls back to the branded error page.

Impact

Any customer with a TrafficProtectionPolicy in Enforce mode attached to a Gateway/HTTPRoute may see 100% failure (500, truncated body) on all traffic through that route, unrelated to actual WAF rule matches — this is not a false-positive block, it's every request failing. The only current workaround is to disable/detach the TPP.

[ecv]

Likely root cause: edge is running a coraza-waf build that predates the response-body-limit fix

Traced this through NSO + infra. Strong hypothesis (not yet confirmed against the filter binary source):

The symptom is the "response body over limit → Reject/abort" signature — Content-Length for the branded 5xx page is emitted, then the connection is torn down with zero body bytes (end of response with N bytes missing). That is exactly the failure mode 888f22f0f ("fix: 500 response body over limit issue") was meant to eliminate.

That fix is config-only, and it depends on the WAF binary honoring it:

• The fix adds a SecLang directive, SecResponseBodyLimitAction ProcessPartial. It is deployed to edge — present in the base and prod configs (apps/network-services-operator/downstream/base/extension-server-config.yaml:11, control-plane/production/config.yaml:27).
• But the edge data plane runs coraza-waf:v1.3.0-alpha4, hardcoded, with no $imagepolicy marker (apps/network-services-operator/downstream/edge/patches/downstream-gateway.yaml:18). Edge's image-updater only targets iroh-gateway (downstream/edge/image-updater-patch.yaml), so coraza-waf on edge never auto-advances.
• Staging runs alpha8, auto-bumped by Flux (downstream/staging/patches/downstream-gateway.yaml:59).

So the config workaround is in place, but the alpha4 binary on edge appears not to honor it (the body-flush / partial-processing behavior likely landed in a later alpha).

NSO code is not the culprit. The branded body is unconditionally inline in the xDS local_reply_config (network-services-operator/internal/extensionserver/mutate/localreply.go:145-152), and NSO never sets Content-Length — Envoy derives it. The header-set / body-missing split is happening inside the coraza-envoy-go-filter binary at response time.

Suggested fix

1. Bump edge coraza-waf from v1.3.0-alpha4 → v1.3.0-alpha8 (match staging) and confirm the fix on that build.
2. Wire a Flux image-updater for coraza-waf on edge so it stops drifting behind staging (currently only iroh-gateway is auto-updated on edge).

Repro caveats

• Do not validate in staging. Staging's downstream gateway is a different stack (TEG v1.7.1 / Envoy contrib-v1.36.2 + its own local-reply-5xx EnvoyPatchPolicy) and coraza alpha8 — the edge alpha4 bug won't reproduce there. See infra#2135.
• Do not bump Envoy Gateway to test. Prod EG is pinned v1.7.4 on purpose (NSO#194 — a bad OIDC SecurityPolicy poisons the whole shared-listener snapshot on v1.8.x). Bumping EG risks a full-gateway outage.
• Green Chainsaw WAF tests are not proof: the proxy-waf tests only assert API acceptance and skip: true when the gateway isn't fully programmed (infra#2937), so they stay green under this 500.

Next step: confirm which alpha introduced the response-body flush fix in the coraza-envoy-go-filter repo, then bump edge.

[kevwilliams]

I dont think this is related to the response body fix although bumping the version shouldnt hurt.

[ecv]

Filed test-coverage gap this surfaced: the WAF Chainsaw tests assert status code only (curl -o /dev/null), never body integrity — so the truncated/empty-body signature here would pass a 403 check. datum-cloud/infra#3321

[ecv]

Correction + refined hypothesis: ABI is exonerated; the real prior fix was the EG v1.7.x upgrade

My earlier "Envoy version delta (edge 1.37.1 vs staging 1.36.2) is the prime suspect" had the direction backwards — retracting it. Full ABI picture below.

The Coraza .so is ABI-locked to Envoy contrib-v1.37.1

Both alpha4 and alpha8 go.mod pin github.com/envoyproxy/envoy v1.37.1 (the contrib/golang/common/go/api filter SDK). Since alpha8 == alpha4 + a Dockerfile-only commit, go.mod is byte-identical → same ABI target. The filter .so requires host Envoy contrib-v1.37.1.

Edge has been ABI-matched since 2026-04-06

when	edge Envoy	coraza filter (ABI target)	match
before 2026-04-06	contrib-v1.36.2	alpha4 (1.37.1)	✗ mismatch
16aa7c18c 2026-04-06 (EG v1.7.x)	contrib-v1.37.1	alpha4 (1.37.1)	✓ matched
3efcad844 2026-07-09 (alpha8 bump)	contrib-v1.37.1	alpha8 (1.37.1)	✓ matched (no-op)

The realignment ("we think we fixed this before") was the EG v1.7.x upgrade on 2026-04-06 that moved edge Envoy to 1.37.1 — not the alpha8 bump (which changed nothing; code + ABI identical to alpha4).

#242 was filed 2026-07-09, ~3 months after edge became ABI-matched. So the Envoy/ABI dimension is exonerated — edge host (1.37.1) has matched the filter ABI the entire time the 500s have been observed.

Aside: staging contrib-v1.36.2 + alpha8 is now the mismatched pairing, but per infra#2135 staging is the TEG stack (different Envoy image, local-reply-5xx EnvoyPatchPolicy branding, V2 listener naming) — not a clean control for anything.

Root cause — two stacked defects (see the consolidated comment below for full detail)

Update: my first version of this section pinned the 500 on validateResponseBody → ProcessResponseBody() returning a Go error. A source audit of coraza v3.4.0 shows ProcessResponseBody() essentially cannot return an error on benign traffic (the only benign error path needs an invalid ctl:responseBodyProcessor, which stock CRS never sets). Downgrading that mechanism. Also note a normal CRS block is 403, not 500 — so the 500 we see is not an ordinary WAF block.

The real picture is two independent defects:

1. Delivery (confirmed): any response/encode-phase SendLocalReply with an empty body hits Envoy #39775 — headers (incl. Content-Length) are committed, then the body never flushes and the stream resets. This is the "CL set, 0 body bytes" signature. Tracked for a filter-side fix in infra#3324.
2. Trigger (narrowing): what makes the filter emit a 500 on every Enforce response. Candidates, ranked: a Go-filter panic during response-phase disruptive handling; an interruption whose status is 500/0 (drop → status 0, or a rule/SecDefaultAction status:500). Ruled out: body-limit Reject (body is tiny) and the ProcessResponseBody Go-error path (v3.4.0 can't hit it on benign traffic). One edge log line disambiguates — see the consolidated comment.

NSO's local_reply_config is the victim, not the cause; leave it as-is.

[ecv]

Live, deterministic repro (upstream-agnostic — kills the size/limit theory)

@kevwilliams pointed a second proxy at httpbin.org and it 500s the same way with TPP Enforce enabled — so it is not backend-specific. Captured the wire signature:

$ curl -sS -D- -o /dev/null https://f2d100f9d26b4ca39541741acc134361.datumproxy.net/
HTTP/1.1 500 Internal Server Error
alt-svc: h3=":443"; ma=86400
content-length: 13001
content-type: text/html; charset=UTF-8
date: Thu, 09 Jul 2026 21:46:18 GMT

curl: (18) transfer closed with 13001 bytes remaining to read

3/3 identical. size_download = 0 bytes every time.

What it proves

• content-length: 13001 = our branded 5xx HTML page. So NSO's local_reply_config fired, formed the branded reply, and Envoy stamped the correct content-length for it.
• Zero of those 13001 bytes reach the wire — headers go out, then the stream is torn down before the substituted body flushes (transfer closed with 13001 bytes remaining).
• Backend is httpbin.org (tiny JSON) → not a response-body-size / SecResponseBodyLimit / ProcessPartial issue. The 13001 is our own error page, not upstream content.
• Reproduces on any proxy with TPP in Enforce mode; Observe/Disabled are fine (Enforce = the only mode where SecRuleEngine On, per the NSO translation — Enforce vs Observe differ by exactly that one token).

Mechanism this points to

A response-encode-path body-flush failure in the Coraza go filter when SecRuleEngine On lets it attempt a response-phase local reply: the filter takes its encode-path local-reply branch (SendLocalReply(...) + zero-fills the response buffer + return api.LocalReply); local_reply_config substitutes the 13001-byte branded body and sets content-length, but the body never streams and Envoy (contrib-v1.37.1) resets the stream. It is not a clean CRS rule block (that would be a branded 403/500 with a body).

Independent of both the WAF alpha tag and the Envoy version (edge is ABI-matched, see prior comment).

One-liner to re-check after any fix

curl -sS -D- -o /dev/null https://f2d100f9d26b4ca39541741acc134361.datumproxy.net/
# expect: HTTP 200 with a full body, not "HTTP 500 ... transfer closed with N bytes remaining"

[ecv]

Consolidated root cause: two stacked defects

Investigated across the filter (datum-labs/coraza-envoy-go-filter@60d8b1a), coraza v3.4.0, NSO's xDS emission, and the Envoy layer. The 500 is two independent bugs, not one.

Layer A — delivery defect (confirmed) → the "truncated/empty body"

Any response/encode-phase SendLocalReply with an empty body triggers envoyproxy/envoy#39775: the upstream response headers are already committed, so Envoy emits a Content-Length and then tears the stream down without writing a body. Closed not planned — unfixed in 1.37.x. This is the exact wire signature (content-length: 13001, 0 body bytes, transfer closed).

• The filter emits an empty body and relies on NSO's local_reply_config to fill it (filter.go validateResponseBody and handleInterruption). On the decode path that's fine (headers not yet sent, Envoy recomputes CL per fixed #12690). On the encode path it's too late → body lost.
• NSO's local_reply_config (mutate/localreply.go) is the victim, not the cause — leave it as-is.
• Filter-side fix tracked in infra#3324 (emit the branded body inline with explicit CL, or reset the stream).

Bonus, same file: EncodeData returns api.Continue on non-final chunks (flushing the held 200 headers + early chunks) but only computes the verdict on the final endStream chunk → multi-chunk responses truncate mid-body rather than getting a clean error.

Layer B — the 500 trigger (narrowed; one log line closes it)

Key correction to my earlier hypothesis: coraza v3.4.0 ProcessResponseBody() cannot return a Go error on benign traffic (the only benign error path needs an invalid ctl:responseBodyProcessor, which stock CRS never sets). And a normal CRS block is 403, not 500. So the 500 is not an ordinary WAF block and not the ProcessResponseBody-error path.

NSO emission is clean (thanks @-analysis): Enforce vs Observe differ by exactly one token — SecRuleEngine On vs DetectionOnly. SecResponseBodyAccess On comes from @recommended-conf in all modes, so the response encode path runs in Observe too; Enforce is only the mode where a phase-3/4 match can disrupt.

Ranked triggers for "500 + empty body on every benign response, Enforce-only":

1. Go-filter panic during response-phase disruptive handling → Envoy converts to 500 empty body. Enforce-only (disruptive eval gated to RuleEngineOn), fits "every response / tiny body" without needing a rule to match all traffic. Strong fit, unverified.
2. Interruption with status 500/0 — drop action (Status 0 → invalid → 500) or a rule / SecDefaultAction status:500. A block, but surfaces as 500 not 403.
3. ❌ Body-limit Reject (hard-codes 500) — body is tiny (httpbin), and it's not mode-gated. Ruled out.
4. ❌ ProcessResponseBody Go error — needs invalid processor; can't hit on benign traffic in v3.4.0.

Decisive check (needs edge Loki access):

{app_kubernetes_io_name="envoy", gateway_envoyproxy_io_owning_gatewayclass="datum-downstream-gateway"}
  |~ "Transaction interrupted|failed to process response body|panic|response validation failed"

• Transaction interrupted ... action=deny/drop status=... → block (mech 1/2); read the action/status.
• Go stack trace / panic → mech 1.
• failed to process response body → error path (mech 4).

Gotcha: don't filter on coraza_outcome="blocked" — the response-error branch never calls recordOutcome, so it defaults to allowed. Key on http_status_code=500 + the log strings.

Bottom line

• Fixing Layer A (infra#3324) makes every response-phase 500 deliver a proper branded body instead of a truncated/empty one — worth doing regardless of Layer B.
• Fixing Layer B requires the one log line above to name the trigger (panic vs status-0 drop vs explicit status:500).
• Neither is the WAF alpha tag (alpha8≡alpha4) nor the Envoy version (edge ABI-matched since April).

Not-this-incident, but flagged

• Latent NSO bug: dormant EPP-emission path trafficprotectionpolicy_controller.go:1232 aliases the shared base-directives slice — would corrupt directives across policies if eppEmissionEnabled is ever turned on. (Live path is safe: index.go uses make+copy.)

[ecv]

Worth noting this rhymes with engineering#93 (TPP Enforce → 500 through the proxy while origin returns 200). Same symptom family, but different bug — that incident was large responses tripping SecResponseBodyLimit (>65KB, response_payload_too_large), fixed with ProcessPartial. This one hits tiny responses on all traffic and isn't size-related. So: not a regression of it, just a cousin. That fix is still in place.

[scotwells]

Should be able to validate this in a local environment and debug there. There's a full functioning e2e environment now that includes WAF

[ecv]

Local repro on a full kind WAF stack — found a distinct NSO-side error-page defect

Stood up the full data plane locally (two kind clusters via colima, extension-server + extensionManager Envoy Gateway + Coraza coraza-waf:v1.3.0-alpha8, a demo-gw Gateway/HTTPRoute + TrafficProtectionPolicy in Enforce) to validate this. TPP Enforce reproduces 100% traffic failure on all requests, as reported. But the mechanism I can pin down here is upstream of the filter's response-phase behavior — it's in NSO's own local_reply_config, which the consolidated analysis had marked as "the victim, leave as-is." At least the embedded default error page is malformed and takes the whole listener down.

What Envoy actually does

With TPP Enforce attached, the merged Envoy rejects the listener outright:

[warning][config] delta config for type.googleapis.com/envoy.config.listener.v3.Listener
  rejected: Error adding/updating listener(s) tcp-80: Incorrect configuration: <!DOCTYPE html>
  ...
  . Couldn't find valid command at position 330

Because the downstream EG runs failOpen: false, a rejected listener blocks the xDS update fleet-wide — Envoy ends up with no tcp-80 listener at all (only the admin/stats listeners in /listeners), so every request gets curl: (52) Empty reply from server. The backend itself is healthy (direct hit returns 200). Delete the TPP and — modulo EG re-translation — the listener comes back.

Root cause

internal/extensionserver/mutate/localreply.go injects the branded page as an Envoy SubstitutionFormatString / TextFormatSource:

BodyFormatOverride: &corev3.SubstitutionFormatString{
    ContentType: cfg.ContentType,
    Format: &corev3.SubstitutionFormatString_TextFormatSource{
        TextFormatSource: &corev3.DataSource{
            Specifier: &corev3.DataSource_InlineString{InlineString: cfg.BodyHTML},
        },
    },
},

Envoy parses that string as a format string, so every literal % in the HTML is treated as a command operator. %RESPONSE_CODE% is intended and valid, but any bare % is not. The embedded default page internal/extensionserver/assets/error-5xx-default.html has an unescaped % at byte 330 — it's the % in html, body { height: 100%; margin: 0; } — hence "Couldn't find valid command at position 330". The page has 12 single % and zero %% escapes.

The infra-mounted page (apps/envoy-error-pages/.../error-5xx.html, 13047 B) does escape them (width: 100%%, etc.), which is why prod hasn't been taken out by this specific path. So the defect bites any Enforce gateway that falls back to the embedded page (no envoy-error-pages ConfigMap mounted).

Fix

Escape literal % → %% in error-5xx-default.html (and anything else fed into a SubstitutionFormatString), or, if substitution isn't wanted for the body, stop routing arbitrary HTML through TextFormatSource. A validation/round-trip check on the assembled local_reply_config at extension-server startup would have caught this before it ever reached Envoy.

Relationship to the existing analysis

• This is distinct from infra#3324 (filter emits an empty body → Envoy drops it on the encode path → 500 with Content-Length set, no body). That's a delivery defect after a valid listener exists. This one is a config-validity defect in NSO's local_reply_config itself — the listener never programs.
• It qualifies the earlier "local_reply_config is the victim, not the cause — leave it as-is": the embedded default page is not innocent.

Caveats (what I verified vs. didn't)

• Reproduced on Envoy contrib-v1.35.0 (the tag in config/dev/downstream_resources/downstream-gateway.yaml); edge runs contrib-v1.37.1. The wire symptom differs by version: here the listener is rejected → "empty reply"; the reported edge signature is 500 + content-length: 13001 + 0 body + transfer closed. Both are consistent with the branded error page never producing deliverable bytes — a newer Envoy likely accepts the listener but fails per-response rendering, where 1.35 rejects it up front. Same unescaped-% root is a strong candidate for both.
• After mounting the escaped 13047 B prod page and restarting the extension-server (which logged using error-page body override ... bytes=13047), the pushed listener config still carried the embedded page (height: 100%, not 100%%) and stayed rejected. That's most likely EG caching the pre-override translation (re-applying an unchanged TPP doesn't retrigger the xDS hook), but it's worth confirming the loaded override actually reaches InjectLocalReplyConfig and isn't shadowed by the embedded default.

Repro manifests + the exact local-stack setup (colima/kind, arm64 caveat: the coraza .so is amd64-only and is dlopen'd in-process, so Envoy has to run emulated) are captured; happy to push them if useful for a regression test.

[ecv]

Filed the NSO-side cause/fix as sub-issue #243 (unescaped % in the embedded error page → SubstitutionFormatString listener rejection). Chainsaw regression test + fix incoming there.