feat(proxy): opt-in same-target 429 wait-and-retry before key failover (#487)

devlog/_plan/260802_429_same_target_retry/000_research.md

@@ -0,0 +1,39 @@
+# 000 — same-target 429 retry: why the client cannot do it
+
+## 요약
+
+Codex turns die instantly when an upstream provider (e.g. BLSC with a single API key)
+returns HTTP 429. The proxy forwards the error with `Retry-After` (#514), but Codex
+never acts on it.
+
+## Evidence
+
+- **Upstream [openai/codex#30471](https://github.com/openai/codex/issues/30471) (open):**
+  `codex-rs/codex-api/src/provider.rs` has a retry policy with an explicit `retry_429` flag,
+  and the endpoint configs set it to `false`; HTTP 429 is only mapped to `UsageLimitReached`
+  when the body matches the Codex usage-limit shape, otherwise it falls through to a generic
+  transport/API error and surfaces as the misleading "exceeded retry limit" message. The
+  suggested fix is to reword the error, deliberately preserving `retry_429=false` — there is
+  no user-facing knob to enable client-side 429 retry.
+- **opencodex issue #487 (auto-closed as `not_planned` for missing detail):** the author
+  requested exactly this feature and noted "The Codex client itself does not retry 429
+  (upstream: openai/codex#30471) — it only retries 5xx — so a proxy-side retry would fill a
+  real gap."
+- **opencodex PR #514 (merged):** attaches a client `Retry-After` (upstream header → message
+  hint → default 2) across Responses, chat completions, Claude messages, and passthrough
+  paths. Its own test plan lists the Codex recovery check as *unverified*. Claude Code honors
+  `Retry-After` and absorbs 429s (#507); Codex does not.
+
+## Current proxy behavior (v2.8.0 / dev)
+
+- `src/server/responses/core.ts` recovery loop: the only 429 retry is multi-key failover
+  (`hasKeyPoolFailover` requires ≥2 keys in `apiKeyPool`). Single-key pools no-op, and the
+  429 falls through to `rate_limit_error` with `Retry-After`.
+- `src/server/chat-completions.ts` and the routed Claude path reuse `handleResponses`, so one
+  insertion point covers all three inbound surfaces.
+
+## Conclusion
+
+Client-side 429 retry does not exist and is not planned. The fix must live in the proxy:
+an opt-in wait-and-retry that replays the identical pre-stream request on the same key before
+any failover.

devlog/_plan/260802_429_same_target_retry/010_design.md

@@ -0,0 +1,117 @@
+# 010 — design: `retryOn429` same-target wait-and-retry
+
+의존: `000_research.md`
+
+## Goal
+
+Provider-level opt-in knob: on HTTP 429, wait (upstream `Retry-After` or a fixed interval)
+and replay the identical request on the same key, up to `attempts` extra times, before the
+existing multi-key failover runs. Default off → zero behavior change for existing setups.
+
+## Surface
+
+```jsonc
+// ~/.opencodex/config.json → providers.<name>
+"retryOn429": {
+  "enabled": true,            // object presence also enables; false disables
+  "attempts": 3,              // extra replays after the first 429 (1..20)
+  "intervalMs": 5000,         // fixed wait when no usable Retry-After
+  "maxIntervalMs": 60000,     // cap for any single wait
+  "respectRetryAfter": true   // prefer the upstream Retry-After when parseable
+}
+```
+
+## Implementation
+
+- `src/types.ts`: `RateLimitRetryPolicy` interface + `OcxProviderConfig.retryOn429`.
+- `src/config.ts`: zod schema entry (zod's default strip inside the object — an unknown key is
+  dropped, never a config-rejecting error; the outer provider schema stays passthrough).
+  Load-time degradation: one hand-edited invalid optional field (e.g. `attempts: 0`) is dropped
+  with a warning instead of tripping the whole schema and hiding all providers behind a default
+  config; misnamed keys (e.g. `attempt`) are warned about too; the management write boundary
+  still rejects invalid policies.
+- `src/providers/key-failover.ts`: `rateLimitRetryPolicyFor` (normalize/default) and
+  `rateLimitRetryDelayMs` (Retry-After seconds/HTTP-date → capped, else `intervalMs`),
+  reusing the existing `parseRetryAfterMs` cooldown parser; the fixed fallback is capped at
+  `maxIntervalMs` too, so a single wait never exceeds the cap. Fail closed: only `authMode:
+  "key"` (or the documented omitted default for custom API-key providers) may use replays —
+  OAuth/forward are never replayed on the same token, local runtimes have no remote key to
+  preserve, and unknown values are rejected. `providerConfigSeed` preserves the registry auth
+  kind (including `"local"`) so the gate survives the seed round-trip.
+- `src/usage/log.ts`: new `AttemptRecoveryKind` member `"rate-limit-429"`.
+- `src/server/responses/core.ts`: in the pre-stream recovery loop, BEFORE the multi-key
+  failover `while`, wait then `rebuildAndRefetch("rate-limit-429")`. Abort during the wait
+  cancels the client request (the unread 429 body is released first). The retry budget lives
+  OUTSIDE the recovery loop, so a 413/401 replay that comes back 429 cannot re-arm a fresh
+  budget — bounded to `attempts` per request. After attempts are exhausted the existing
+  failover and error mapping run unchanged. The same wait-and-replay applies to the other
+  key-auth surfaces that bypass that loop:
+  - Responses passthrough wire (`openai-responses` key-auth gateways, e.g. the built-in
+    DeepSeek preset) — pre-relay, before the forward-pool logic;
+  - image/video bridge and web-search sidecar loops (`src/images/loop.ts`,
+    `src/web-search/loop.ts`) — before their `on429` key rotation;
+  - Anthropic terminal-guard continuations — before key/account failover.
+  Bridge retries apply to HTTP adapters only: a custom transport that enters the
+  `adapter.runTurn` branch (`src/images/loop.ts`) returns before the HTTP 429 retry loop and
+  therefore does not receive the wait-and-replay policy.
+  Every surface releases (and awaits the cancellation of) the unread 429 body BEFORE the
+  backoff, records the `rate-limit-429` recovery kind on replay sends, and (bridges) clears the
+  old response-header deadline before the wait and starts a fresh one afterward, re-checking
+  client cancellation before telemetry and replay.
+  Covers Responses, chat completions, and routed Claude messages (they all enter
+  `handleResponses`).
+
+## Safety
+
+- Pre-stream only: a 429 arrives before any bytes are relayed, so replaying the string-body
+  request is lossless (same invariant as the transient-5xx layer in `lib/upstream-retry.ts`).
+- Ordering: same-key retries run before failover, so "primary-first" users keep their key on
+  rate-limit blips; failover still works after retries exhaust.
+- Retry-wait bound: the SLEEP component is at most `attempts × maxIntervalMs` (default
+  3 × 60 s = 180 s) when honoring upstream `Retry-After`; `attempts × intervalMs`
+  (default 15 s) when `respectRetryAfter=false` or no header is present. Total request
+  latency is higher: every attempt also consumes its own connect/response time (bounded by
+  `connectTimeoutMs`), so the documented bound covers deliberate waits only.
+- Identical replay: rebuilds are deterministic for the same parsed request (same serialized
+  body and auth headers); the passthrough/continuation/e2e tests assert byte-identical bodies
+  and identical auth headers across replays, not just send counts.
+- Abort during the wait: the sleep is abort-aware — when the server observes the client
+  disconnect (Bun propagates this asynchronously, observed 1–10 s), the wait is interrupted,
+  the unread 429 body is released, and the request is cancelled with 499 before any replay.
+  Because the propagation is async, a replay can still precede the cancel if the interval
+  elapses first; that is bounded by the same `attempts` budget. Terminal continuations sleep
+  on the upstream signal, so a body-cancel (SSE already streaming) aborts the wait too.
+- Concurrency: each request honors its own policy independently — no process-wide cooldown is
+  shared between concurrent requests (unlike the Kiro 429 pattern). Upstream volume per
+  request: same-key replays add at most `attempts` sends, then multi-key failover adds up to
+  `poolKeys − 1` more (or Anthropic account rotations), so the combined bound is
+  `attempts + poolKeys` sends (pool size = configured `apiKeyPool` length, fixed per request) —
+  a storm multiplies by that factor per request, not by `attempts + 1`.
+- Header deadlines: the image/video and web-search bridge loops restart their response-header
+  deadline after each deliberate wait, so backoffs never consume the connect budget and a
+  rate-limit wait is never misattributed as a 504 header timeout. The old deadline is cleared
+  BEFORE the sleep and client cancellation is re-checked after it, so 499 always wins over a
+  stale-deadline edge.
+- Expired `Retry-After`: a valid HTTP-date already in the past retries immediately (same as
+  numeric `Retry-After: 0`) instead of falling back to the fixed interval.
+- Recovery observability: every retry surface records the `rate-limit-429` recovery kind
+  (normal loop, passthrough wire, image/video bridge, web-search sidecar, terminal
+  continuations), so usage logs explain the extra sends.
+- Final 429 still carries `Retry-After` for clients that honor it (Claude Code).
+
+## Tests
+
+- `tests/rate-limit-retry.test.ts` — policy normalization (incl. OAuth/forward gating) +
+  delay computation (seconds, HTTP-date, `0`, malformed, cap; deterministic) + abort during
+  the wait: a directly-invoked `handleResponses` with a controlled abort signal returns 499,
+  cancels the unread 429 body, and performs no further upstream sends (deterministic — no
+  real-socket disconnect timing involved).
+- `tests/usage-log.test.ts` — the `rate-limit-429` recovery kind survives persisted usage logs.
+- `tests/server-rate-limit-retry-e2e.test.ts` — single-key replay to success, immediate
+  passthrough without the knob, exhausted attempts surface 429, and retry-before-failover
+  ordering with a 2-key pool, plus key-auth `openai-responses` passthrough replaying 429 on
+  the same key.
+- `tests/terminal-guard-server.test.ts` — an Anthropic terminal-guard continuation that 429s
+  is replayed on the same key before the error surfaces (3 upstream sends).
+- `tests/images/loop.test.ts` + `tests/web-search.test.ts` — the bridge loops replay 429 on
+  the same key before `on429` rotation runs (same-key sends counted, rotations zero).

docs-site/src/content/docs/guides/providers.md

@@ -39,6 +39,12 @@ labels local presets separately; those normally omit both `authMode` and `apiKey
 | `forward` | Relays **your incoming Codex auth headers** verbatim to the provider — no key stored. This is the ChatGPT-login passthrough. | OpenAI (`openai-responses` adapter). |
 | `oauth` | Resolves a stored OAuth access token (auto-refreshed before expiry) and uses it as the bearer key. | xAI, Anthropic, Kimi, Kiro, Google Antigravity, Cursor, GitHub Copilot. |
 
+The [`retryOn429`](/reference/configuration/) same-key 429 replay applies only to API-key
+providers (`authMode: "key"`). OAuth, forward, and local presets are excluded — their
+credentials must never be replayed on the same token, and local runtimes have no remote key to
+preserve. It is opt-in: when the option is absent the feature is off; object presence enables
+it unless `enabled: false`.
+
 ## 1. ChatGPT login (forward / passthrough)
 
 The `openai` provider needs **no API key**. Direct forwards credentials from your existing

docs-site/src/content/docs/ja/guides/providers.md

@@ -9,7 +9,7 @@ description: opencodex が LLM プロバイダーを認証し通信するすべ
 ## OpenAI アカウントモード
 
 | プロバイダー ID | 用途 | 認証情報/アカウットルール |
- --- | --- | --- |
+| --- | --- | --- |
 | `openai` | Codex ログイン | Pool(デフォルト)はメイン + 追加アカウントを選び、Direct は現在の caller/メインログインのみを使います。 |
 | `openai-apikey` | OpenAI API | 設定された API キー/キープールのみを使い、Codex アカウントは読みません。 |
 
@@ -30,11 +30,16 @@ max input 922,000 で `*-pro` virtual ID は公開状態を維持し、wire で
 ローカルプリセットを別に分類します。ローカルプリセットでは通常 `authMode` と `apiKey` を両方使いません。
 
 | `authMode` | 認証方式 | 用途 |
- --- | --- | --- |
+| --- | --- | --- |
 | `key` | API キーを送信します(`Authorization: Bearer …`、またはアダプターにより `x-api-key` / `api-key`)。キーはリテラルまたは `${ENV_VAR}` 参照です。 | 大半のプロバイダー。 |
 | `forward` | **受け取った Codex 認証ヘッダーを**プロバイダーにそのまま中継します — キーを保存しません。ChatGPT ログインのパススルーです。 | OpenAI(`openai-responses` アダプター)。 |
 | `oauth` | 保存された OAuth アクセストークンを読み込み bearer キーとして使い、期限切れ前に自動更新します。 | xAI、Anthropic、Kimi、Kiro、Google Antigravity、Cursor。 |
 
+[`retryOn429`](/ja/reference/configuration/)（同一キーでの 429 リトライ）は API キー プロバイダー
+（`authMode: "key"`）のみに適用されます。OAuth・forward・ローカル プリセットは除外されます —
+同じトークンを再送すべきではなく、ローカルランタイムには保存すべきリモートキーがありません。
+オプトインです: オプションが無ければ無効、オブジェクトがあれば `enabled: false` でない限り有効です。
+
 ## 1. ChatGPT ログイン(forward / パススルー)
 
 デフォルトプロバイダーは**API キー不要**です。既存の `codex login` の認証情報を OpenAI Responses バックエンドに
@@ -138,7 +143,7 @@ Cline IDE/CLI のみで API からは使えません。`minimax/minimax-m2.5` 
 無料試用モデルとして文書化されています。
 
 | プロバイダー | ベース URL |
- --- | --- |
+| --- | --- |
 | **OpenAI (API キー)** | `https://api.openai.com/v1` |
 | **Anthropic (API キー)** | `https://api.anthropic.com` |
 | **OpenRouter** | `https://openrouter.ai/api/v1` |
@@ -238,7 +243,7 @@ OAuth、API キープールを確認・切り替えできます。完全なコ
 Sol/Terra/Luna をフォールバックリストに入れています。
 
 | Codex 経路 | 事前登録されたモデル ID | Codex に表示されるコンテキスト |
- --- | --- | --- |
+| --- | --- | --- |
 | Codex ログイン(Pool または Direct) | `gpt-5.6-*` | 372,000 |
 | OpenAI (API キー) | `openai-apikey/gpt-5.6-*` と `*-pro` | 1,050,000 (max input 922,000) |
 | OpenRouter | `openrouter/openai/gpt-5.6-sol`、`openrouter/openai/gpt-5.6-terra`、`openrouter/openai/gpt-5.6-luna` | 1,050,000 |
@@ -260,6 +265,15 @@ Amazon Bedrock ネイティブ API のような、これらの実装のいずれ
 **サブスクリプショントークン**(通常の API キーではない)で認証します。**Cloudflare AI
 Gateway** は URL にアカウント + ゲートウェイ ID を埋める必要があります。
 
+Copilot は混在 wire カタログを提供します。GPT-5 系モデル（`gpt-5.3-codex`、`gpt-5.4`、
+`gpt-5.4-mini`、`gpt-5.5`、`gpt-5.6-luna`、`gpt-5.6-sol`、`gpt-5.6-terra`）はエージェント
+通信の `/chat/completions` を拒否するため、opencodex はこれらのモデルを組み込みデフォルトで
+Responses API 経由にルーティングし、他の Copilot モデルはすべて chat completions のままです。
+優先順位は次のとおりです: ハード wire ピン → 明示的な
+[`modelAdapters`](/ja/reference/configuration/providers/) エントリ → レジストリのデフォルト →
+プロバイダー全体の adapter。組み込みデフォルトのないモデル（例: `gpt-5.4-nano`）を Responses
+に移すには、`"modelAdapters": { "gpt-5.4-nano": "openai-responses" }` を設定してください。
+
 Cursor は別の実験的アダプターとして追跡します。`adapter: "cursor"` は `ocx init` とダッシュボード Add
 Provider ピッカーに実験的 local config 項目として表示され、Cursor の静的フォールバックモデルカタログ
 メタデータを保存します。Cursor アクセストークンを設定すると opencodex は Cursor ライブ HTTP/2 トランスポートを
@@ -293,7 +307,7 @@ Ollama の `:size` タグに寛容なので `gpt-oss` は `gpt-oss:120b` と `gp
 opencodex をローカルの OpenAI 互換サーバーに向けてください — 通常は空キーで使います:
 
 | プロバイダー | ベース URL |
- --- | --- |
+| --- | --- |
 | Ollama (local) | `http://localhost:11434/v1` |
 | vLLM | `http://localhost:8000/v1` |
 | LM Studio | `http://localhost:1234/v1` |

docs-site/src/content/docs/ja/reference/adapters.md

@@ -46,6 +46,11 @@ interface ProviderAdapter {
 **対象:** OpenAI **Responses API**。**`passthrough: true`** — 元のリクエスト本文をそのまま渡し、レスポンスを **変換せずに** ストリーミングします。
 **認証:** `forward`（呼び出し元ヘッダー中継）または `key`。
 
+`key` 認証では、[`retryOn429`](/ja/reference/configuration/) もここに適用されます: プリストリームの
+429 は、翻訳された `openai-chat` / Anthropic リクエスト経路と同様に、他の処理やフェイルオーバーに
+先立って、同じキーで同一リクエストを待機して再送します。カスタム `runTurn` トランスポートは
+HTTP リトライ ループの対象外です。
+
 - `forward` URL → `{baseUrl}/responses`。`key` provider はデフォルトで従来の `{baseUrl}/v1/responses` 構築を使います。
 - `key` provider は検証済みの相対 `responsesPath` を設定できます。adapter は `baseUrl` 末尾の `/` を 1 つ除き、`{trimmedBaseUrl}{responsesPath}` に送信します。Ark Agent Plan では `baseUrl: "https://ark.cn-beijing.volces.com/api/plan/v3"` と `responsesPath: "/responses"` を使います。
 - `forward` モードでは安全なヘッダー許可リスト（`FORWARD_HEADERS`）だけを中継します。authorization、ChatGPT account id、OpenAI beta/originator/session ヘッダーが対象です。この ChatGPT ログイン経路は [サイドカー](/ja/guides/sidecars/) にも使われます。