[Enhancement]: Mid-stream OpenAI socket resets can break Codex account affinity

[luvs01]

Client or integration

Codex App

Area

Proxy and routing

Summary

A native OpenAI Responses stream can return HTTP 200 and partial SSE data, then fail while OpenCodex reads the upstream body with a socket reset such as ECONNRESET.

Codex correctly treats the resulting stream disconnected before completion: Transport error: network error: error decoding response body as retryable and replays the logical request. OpenCodex should not replay a partially delivered stream because that could duplicate tool calls or other side effects.

The routing side effect is incorrect: the passthrough inspection path converts a mid-stream body-read failure into synthetic status 502 and records it through recordCodexUpstreamOutcome. Repeated client retries or concurrent streams can then reach upstreamFailoverThreshold, soft-avoid a healthy credential, clear thread affinity, and promote another account even though no account-specific HTTP rejection occurred.

Expected behavior:

• Keep the synthetic mid-stream 502 in request diagnostics.
• Do not apply that synthetic transport terminal to account failure streak, cooldown, affinity, or active-account selection.
• Release only a quota-probe lease owned by the interrupted request.
• Continue recording actual HTTP 429/402/5xx and semantic SSE response.failed outcomes.
• Do not replay the partially delivered request inside OpenCodex; leave bounded retry to Codex.
• Preserve the selected account for the Codex retry unless an actual account-specific rejection occurs.
• Record enough correlation-safe metadata to distinguish upstream body-read failure, downstream pipe failure, client cancellation, and terminal delivery.

Relevant source:

• src/server/relay.ts: passthrough inspection maps body-read failure to a synthetic 502 terminal.
• src/server/responses/core.ts: native passthrough terminal recording forwards that synthetic status to account routing.
• src/codex/routing.ts: repeated transient outcomes can clear affinity and promote another account.
• structure/04_transports-and-sidecars.md: mid-stream SSE failures are intentionally not replayed by the proxy.

Required regression coverage:

• HTTP 200 SSE emits a partial delta, then its body throws ECONNRESET three times for the same thread.
• Each request remains observable as a synthetic mid-stream failure, but account health, affinity, and active selection remain unchanged.
• A held scoped or account-wide probe lease is released exactly once.
• An actual SSE response.failed with status 502 still records an account failure.
• A client cancellation remains distinct and does not become an upstream failure.
• No extra upstream request is sent by OpenCodex after partial delivery.

Related: #914 handles DNS/reachability failures before response headers; this issue handles failures after HTTP 200 and partial body delivery. #913 covers compact failover only after an actual upstream 429/402. The same client-visible error is tracked upstream in openai/codex#29087.

Reproduction

1. Configure OpenAI Codex-login pool routing and bind a thread to account A.
2. Return HTTP 200 with text/event-stream from the OpenAI upstream.
3. Deliver at least one valid SSE delta, then make the upstream body reader throw an ECONNRESET-shaped error before response.completed.
4. Let Codex retry the request several times or reproduce the same failure concurrently.

Current: OpenCodex records synthetic 502 account failures. At the configured threshold it can clear A's affinity or promote another account without an actual HTTP rejection.

Expected: Codex can retry, but OpenCodex keeps account routing state unchanged and never replays the partial stream itself.

Version

2.10.0; the relevant passthrough terminal and routing behavior remained on dev at c72acb3e368d2613c41d6d2b15a771895843f03f when checked on 2026-08-03.

Operating system

Windows 11 Pro 10.0.26200. The account-outcome classification issue is platform-independent; the observed resets occurred in the Windows native passthrough path.

Provider and model

OpenAI (Codex login / forward account pool); native Codex Responses streaming.

Logs or error output

stream disconnected before completion: Transport error: network error: error decoding response body
TypeError: The socket connection was closed unexpectedly
code: ECONNRESET

Screenshots and supporting files

None. Exact request IDs, thread IDs, account identifiers, quota values, and timestamps are intentionally omitted.

Redacted configuration

{
  "providers": {
    "openai": {
      "adapter": "openai-responses",
      "authMode": "forward",
      "codexAccountMode": "pool"
    }
  },
  "upstreamFailoverThreshold": 3
}

Checks

• I searched existing issues and documentation.
• I removed secrets, tokens, account details, request credentials, and personal data.

[github-actions]

_{Automated translation bookkeeping — detected language: unknown.}

[lidge-jun]

Triaged, and the answer is more interesting than a yes or no. Two corrections and a reframe.

OpenCodex does not replay a partially delivered stream. Retry helpers operate before a Response is returned (src/lib/upstream-retry.ts:203-238), and the failed-tail relay documents and implements "not a resend" (src/server/relay.ts:98-137). Any logical replay you observed is the client's own subsequent request. That half of the report is already satisfied.

The routing side effect is real. consumeForInspection() maps an upstream body-read exception to a synthetic failed, 502 labeled transportPhase="mid_stream", terminalSource="synthetic" (src/server/relay.ts:988), and codexForwardTerminalOutcomeRecorder() (src/server/responses/core.ts:427-469) records it as an account failure. Enough of them soft-avoid the account and clear affinity.

But it was designed that way, deliberately. devlog/_fin/260722_issue_bug_sweep/030_patch_s_sticky_502.md introduced this exact behavior, and its test matrix lists upstream read rejection (200 후 socket reset) with the expected outcome transient 실패 기록, affinity 해제. The comment is still in the source. So this is not a bug that slipped in — it is a policy someone chose, and reversing it is a policy decision.

I spent four audit rounds trying to make it a local fix and failed each time. terminalSource: "synthetic" records who manufactured the terminal event, not who caused the failure — and the eager path's catch (src/server/relay-eager.ts:256) additionally wraps inspection and rewriting, so the same label can mean an upstream body failure or a local exception in our own relay.

The real question is whether post-200 transport reliability should affect account routing at all. Arguments both ways: every pool account shares the host, so rotating cannot repair a host-side drop — but account health tracks transient reliability generally, which is why ordinary 5xx responses count.

Recorded with #914 in devlog/_plan/260803_transport_attribution/000_plan.md, since they are the same question before and after HTTP 200. Leaving open until that policy is decided rather than shipping a guard that suppresses real signal.

[luvs01]

A new live reproduction on installed opencodex 2.10.0 confirms the routing side effect described here.

Observed on 2026-08-03 UTC, with account identifiers and request contents omitted:

• One ongoing conversation produced a burst of terminal HTTP 502 records between 12:34:31 and 12:36:24. Multiple records had a non-null firstOutputMs, so at least part of the burst occurred after output had started.
• Codex's structured log then recorded stream disconnected before completion: Transport error: network error: error decoding response body and an ordinary sampling retry against http://127.0.0.1:10100/v1/responses.
• During the same window, the OpenCodex usage log changed from the base openai provider label to a different privacy-safe pool label. That pool account returned 429 twice, followed by additional router-level 429 responses. Codex turns then terminated with exceeded retry limit, last status: 429 Too Many Requests.
• Codex continued to address only the localhost proxy and had no account-selection event. The provider/account transition exists only in OpenCodex records, so the rotation was proxy-side.
• Runtime configuration was fill-first with autoSwitchThreshold: 100. Ongoing affinity should not rotate for ordinary fill-first selection, which leaves the transient-failure path as the matching trigger. The installed routing code defaults upstreamFailoverThreshold to 3 and records synthetic mid-stream 502 as an account transient failure that can clear affinity and promote an alternate.
• After the incident, quota refresh showed the main account below its weekly limit while both added pool accounts were at 100%. This does not prove the cache knew those values at selection time, but it demonstrates the user-visible failure mode: a network/body-decode incident displaced an ongoing thread onto an account that returned 429, and client retries ended in a retry-limit failure.

This evidence does not establish that every post-200 transport failure should be account-neutral. It does establish that the current synthetic-502 policy can break ongoing affinity and turn a recoverable transport failure into account rotation plus 429 exhaustion. A regression test should cover both relay modes and assert that client cancellation remains neutral, while policy work separates host/transport health from credential/account health.

[lidge-jun]

Re-labeled bug → enhancement, keeping proxy, streaming, tools. The behavior you describe is real and reproduces; the reclassification is about what kind of change would resolve it, not about whether it happens.

The mid-stream reset clearing account affinity was introduced deliberately. devlog/_fin/260722_issue_bug_sweep/030_patch_s_sticky_502.md records the intended outcome as transient 실패 기록, affinity 해제 — a mid-stream socket reset is recorded as a transient failure precisely so account health stops treating that account as sticky. Escalated transient health then requires two successes to recover.

So the current behavior is the policy working as written, and changing it is a policy decision about how aggressively affinity should survive a transport failure. That is an enhancement request against a recorded design, not a defect against an intended one.

Two things worth knowing for whoever picks this up:

The synthetic/mid-stream label is not a shortcut to the answer. It records who manufactured the terminal event, not who caused the failure, and the eager path's catch wraps our own inspection and rewriting too — so keying the new behavior off it would produce a guard that looks right and misattributes.

#922 is adjacent but not this. It makes shared network failures account-neutral, which is a different question from whether a confirmed mid-stream reset should drop affinity.

Keeping it open with the label corrected so it sorts with the other accepted behavior changes rather than sitting in the bug queue.