[Bug]: DNS and network reachability failures incorrectly rotate Codex pool accounts

[luvs01]

Client or integration

Codex App

Area

Authentication and account pool

Summary

Pre-connection DNS and local network failures are currently recorded as failures of the selected Codex account. Regular Responses and native compact map every thrown non-timeout transport error to connect_error; after upstreamFailoverThreshold failures, OpenCodex soft-avoids that account, clears thread affinity, and changes the effective account.

This is incorrect for source-proven reachability failures such as ENOTFOUND, EAI_AGAIN, ENETUNREACH, ENETDOWN, EHOSTUNREACH, ECONNREFUSED, and equivalent bounded Bun error codes. Credential replacement cannot repair a machine-wide DNS or network outage, and concurrent agents can cross the default threshold almost immediately.

Expected behavior:

• Return the existing client-visible 502, but classify proven pre-connection DNS/network reachability failures as account-neutral.
• Do not increment account failure streak, set soft-avoid/cooldown, clear affinity, or change runtime/persisted active account.
• Release any owned quota-probe lease without treating the failure as account evidence.
• Inspect stable codes through a bounded cause chain; do not trust an unanchored message substring.
• Preserve current handling for ECONNRESET, EPIPE, established-socket closure, TimeoutError, HTTP 5xx, actual 429/402, 401/403, and client cancellation.

Relevant source:

• src/server/responses/core.ts: native passthrough transport failure recording.
• src/server/responses/compact.ts: compact transport failure recording.
• src/codex/routing.ts: transient streak, soft-avoid, affinity clear, and active failover.
• src/lib/upstream-retry.ts: existing narrow transport classification.
• src/codex/auth-context.ts: probe-lease release.

Required tests: table-driven bounded cause-chain classification; negative message-only cases; routing state unchanged after three neutral failures; regular and compact end-to-end ENOTFOUND cases; concurrent neutral failures; and proof that an owned probe lease is released. Existing reset/socket/timeout/HTTP failover regressions must remain.

Related: #186/#194 (established-connection and transient recovery, which this must not undo), #714 (retry/amplification rather than account-state mutation), and #504/#506 (thresholded account failover).

Reproduction

1. Configure at least two Codex pool accounts and bind a thread to account A.
2. Keep the default upstreamFailoverThreshold: 3.
3. Make the provider hostname unresolvable, or inject a pre-response error with stable code ENOTFOUND.
4. Send three concurrent or sequential requests.
5. Restore network reachability and inspect the next routing decision.

Current: A is soft-avoided, affinity can be cleared, and another account can become effective.

Expected: requests may return 502 during the outage, but A's health, affinity, and active-account state remain unchanged.

Version

2.10.0; the relevant transport branches remained on dev at 075c2f34c060d068ed56985e76924a9f23aca39d when checked on 2026-08-03.

Operating system

Windows 11 Pro 10.0.26200; equivalent Node/Bun shapes should be covered on every supported platform.

Provider and model

OpenAI (Codex login / forward account pool); native Codex models.

Logs or error output

Provider unreachable: getaddrinfo ENOTFOUND <provider-host>

Screenshots and supporting files

None. Hostnames, account identifiers, and request identifiers are removed.

Redacted configuration

{
  "providers": {
    "openai": {
      "adapter": "openai-responses",
      "authMode": "forward",
      "codexAccountMode": "pool"
    }
  },
  "accountPoolStrategy": "fill-first",
  "upstreamFailoverThreshold": 3,
  "activeCodexAccountId": "__main__"
}

Checks

• I searched existing issues and documentation.
• I removed secrets, tokens, account details, request credentials, and personal data.

[github-actions]

_{Automated translation bookkeeping — detected language: unknown.}

[lidge-jun]

Triaged against current dev. The defect is real and your reasoning about the mechanism is correct — but three separate fixes for it were designed and rejected at review, and recording why matters more than a fourth attempt.

The path you describe is exactly right: src/server/responses/core.ts:1640-1656 maps every non-timeout fetch rejection to connect_error and records it against the selected account, src/server/responses/compact.ts:275-301 does the same, routing treats connect_error as transient (src/codex/routing.ts:297-307), and at upstreamFailoverThreshold the account is soft-avoided with thread affinity cleared (:1417-1465). A DNS failure on your machine rotates you off a working account, and the rotation cannot help.

Why the obvious fix does not work. The first design matched the Node error codes you named. Under Bun 1.3.14 they never arrive:

fetch("https://no-such-host.invalid/x") -> code:"ConnectionRefused", errno:0, no cause
fetch("http://127.0.0.1:1/x")           -> code:"ConnectionRefused", errno:0, no cause
dns.lookup("no-such-host.invalid")      -> code:"ENOTFOUND"

A unit test injecting code:"ENOTFOUND" would have passed while the branch never fired in production. Worse, Bun gives DNS failure and connection refusal the same label, so no code list separates them. Repeated calls to the same host even alternate ConnectionRefused / FailedToOpenSocket as Bun evicts its DNS cache.

Why resolving the hostname does not work either. The second design probed with dns.lookup() on a rejection: lookup fails means neutral, lookup succeeds means the account is at fault. But a hostname that resolves can still refuse TCP behind a firewall, VPN, or fake-IP proxy — a configuration this repository explicitly supports (src/lib/destination-policy.ts:175). And the probe observes the resolver now, not during the failed fetch.

Why "no headers arrived" does not work. The third design skipped health recording for every rejection, reasoning that a rejected fetch means no server evaluated the credential. Two counterexamples reached the real wrapper: Bun follows redirects by default, so a server can receive the authenticated request, return 307 to a dead host, and reject after headers arrived; and a server that reads Authorization then closes yields ECONNRESET with the credential already seen.

What that adds up to. Every design tried to answer "was this the credential's fault?" from evidence that is insufficient in principle. The likely correct direction is different: connection-level failures belong to a (provider, host) pair, not a credential — every pool account shares the host, which is why rotation cannot repair a network fault. A distinct health key with its own threshold makes the attribution question moot for the common case.

That analysis, all three rejected designs with their probe output, and one further trap (a 5xx observed during a retry vanishes from the outer catch, so any design keying on "the promise rejected" inherits a hole) are recorded in devlog/_plan/260803_transport_attribution/000_plan.md on dev.

Staying open. Nothing in the current sweep fixes it, and claiming otherwise would be exactly the overreach the review rounds prevented.

[Yuxin-Qiao]

Implemented the host-health direction recorded in devlog/_plan/260803_transport_attribution/000_plan.md — PR #966 (targeting dev).

What changed:

• Proven pre-connection reachability failures (DNS failure / TCP refusal, Bun codes ConnectionRefused / FailedToOpenSocket and the Node equivalents, matched through a bounded cause chain, never message text) are recorded in a new (provider, host) ledger with its own threshold instead of the account streak. Account health, soft-avoid, thread affinity, and active account stay untouched; owned quota-probe leases are released.
• Pool sends (/v1/responses, alternate-account send, /v1/responses/compact) now use redirect: "manual" with an explicit 3xx policy (relayed, never health evidence), removing the redirect-to-dead-host counterexample. Read-then-close (ECONNRESET) stays account-attributed with a regression test, so the 2026-07-22 decision is preserved.
• Same classification on the search/live/images/web-search/vision sidecar outcome sites.

Probe evidence on Bun 1.3.14 (same runtime as the plan probe): ConnectionRefused/FailedToOpenSocket for both DNS and TCP failures (errno 0, no cause, labels alternate); ECONNRESET for read-then-close. Validation: typecheck, privacy scan, focused e2e (regular + compact, concurrent, real dead-port rejection, read-then-close), full suite 7933 pass with only the 10 pre-existing provider management validation failures that reproduce on clean dev.

[Ingwannu]

Thanks — PR #966 is linked and its host-health direction matches the accepted design, but the current head 80cd0cee remains a draft and is not ready to claim as the fix.

The open review blockers are material to the attribution boundary: redirect handling is still incomplete across search/sidecar and custom keyed image paths, reset evidence can be lost when a retry later fails as a reachability error, and the redirect tests/documentation do not yet pin the full observable contract. Those gaps can still misclassify a failure or break a working same-origin provider redirect.

Please resolve the active review threads on #966 and rerun the focused transport tests plus the full cross-platform gates. Keeping #914 open until the PR is review-ready and the corrected head lands on dev.