diff --git a/devlog/_plan/260802_429_same_target_retry/000_research.md b/devlog/_plan/260802_429_same_target_retry/000_research.md new file mode 100644 index 000000000..2e5dd6673 --- /dev/null +++ b/devlog/_plan/260802_429_same_target_retry/000_research.md @@ -0,0 +1,39 @@ +# 000 — same-target 429 retry: why the client cannot do it + +## 요약 + +Codex turns die instantly when an upstream provider (e.g. BLSC with a single API key) +returns HTTP 429. The proxy forwards the error with `Retry-After` (#514), but Codex +never acts on it. + +## Evidence + +- **Upstream [openai/codex#30471](https://github.com/openai/codex/issues/30471) (open):** + `codex-rs/codex-api/src/provider.rs` has a retry policy with an explicit `retry_429` flag, + and the endpoint configs set it to `false`; HTTP 429 is only mapped to `UsageLimitReached` + when the body matches the Codex usage-limit shape, otherwise it falls through to a generic + transport/API error and surfaces as the misleading "exceeded retry limit" message. The + suggested fix is to reword the error, deliberately preserving `retry_429=false` — there is + no user-facing knob to enable client-side 429 retry. +- **opencodex issue #487 (auto-closed as `not_planned` for missing detail):** the author + requested exactly this feature and noted "The Codex client itself does not retry 429 + (upstream: openai/codex#30471) — it only retries 5xx — so a proxy-side retry would fill a + real gap." +- **opencodex PR #514 (merged):** attaches a client `Retry-After` (upstream header → message + hint → default 2) across Responses, chat completions, Claude messages, and passthrough + paths. Its own test plan lists the Codex recovery check as *unverified*. Claude Code honors + `Retry-After` and absorbs 429s (#507); Codex does not. + +## Current proxy behavior (v2.8.0 / dev) + +- `src/server/responses/core.ts` recovery loop: the only 429 retry is multi-key failover + (`hasKeyPoolFailover` requires ≥2 keys in `apiKeyPool`). Single-key pools no-op, and the + 429 falls through to `rate_limit_error` with `Retry-After`. +- `src/server/chat-completions.ts` and the routed Claude path reuse `handleResponses`, so one + insertion point covers all three inbound surfaces. + +## Conclusion + +Client-side 429 retry does not exist and is not planned. The fix must live in the proxy: +an opt-in wait-and-retry that replays the identical pre-stream request on the same key before +any failover. diff --git a/devlog/_plan/260802_429_same_target_retry/010_design.md b/devlog/_plan/260802_429_same_target_retry/010_design.md new file mode 100644 index 000000000..6b390c2b9 --- /dev/null +++ b/devlog/_plan/260802_429_same_target_retry/010_design.md @@ -0,0 +1,117 @@ +# 010 — design: `retryOn429` same-target wait-and-retry + +의존: `000_research.md` + +## Goal + +Provider-level opt-in knob: on HTTP 429, wait (upstream `Retry-After` or a fixed interval) +and replay the identical request on the same key, up to `attempts` extra times, before the +existing multi-key failover runs. Default off → zero behavior change for existing setups. + +## Surface + +```jsonc +// ~/.opencodex/config.json → providers. +"retryOn429": { + "enabled": true, // object presence also enables; false disables + "attempts": 3, // extra replays after the first 429 (1..20) + "intervalMs": 5000, // fixed wait when no usable Retry-After + "maxIntervalMs": 60000, // cap for any single wait + "respectRetryAfter": true // prefer the upstream Retry-After when parseable +} +``` + +## Implementation + +- `src/types.ts`: `RateLimitRetryPolicy` interface + `OcxProviderConfig.retryOn429`. +- `src/config.ts`: zod schema entry (zod's default strip inside the object — an unknown key is + dropped, never a config-rejecting error; the outer provider schema stays passthrough). + Load-time degradation: one hand-edited invalid optional field (e.g. `attempts: 0`) is dropped + with a warning instead of tripping the whole schema and hiding all providers behind a default + config; misnamed keys (e.g. `attempt`) are warned about too; the management write boundary + still rejects invalid policies. +- `src/providers/key-failover.ts`: `rateLimitRetryPolicyFor` (normalize/default) and + `rateLimitRetryDelayMs` (Retry-After seconds/HTTP-date → capped, else `intervalMs`), + reusing the existing `parseRetryAfterMs` cooldown parser; the fixed fallback is capped at + `maxIntervalMs` too, so a single wait never exceeds the cap. Fail closed: only `authMode: + "key"` (or the documented omitted default for custom API-key providers) may use replays — + OAuth/forward are never replayed on the same token, local runtimes have no remote key to + preserve, and unknown values are rejected. `providerConfigSeed` preserves the registry auth + kind (including `"local"`) so the gate survives the seed round-trip. +- `src/usage/log.ts`: new `AttemptRecoveryKind` member `"rate-limit-429"`. +- `src/server/responses/core.ts`: in the pre-stream recovery loop, BEFORE the multi-key + failover `while`, wait then `rebuildAndRefetch("rate-limit-429")`. Abort during the wait + cancels the client request (the unread 429 body is released first). The retry budget lives + OUTSIDE the recovery loop, so a 413/401 replay that comes back 429 cannot re-arm a fresh + budget — bounded to `attempts` per request. After attempts are exhausted the existing + failover and error mapping run unchanged. The same wait-and-replay applies to the other + key-auth surfaces that bypass that loop: + - Responses passthrough wire (`openai-responses` key-auth gateways, e.g. the built-in + DeepSeek preset) — pre-relay, before the forward-pool logic; + - image/video bridge and web-search sidecar loops (`src/images/loop.ts`, + `src/web-search/loop.ts`) — before their `on429` key rotation; + - Anthropic terminal-guard continuations — before key/account failover. + Bridge retries apply to HTTP adapters only: a custom transport that enters the + `adapter.runTurn` branch (`src/images/loop.ts`) returns before the HTTP 429 retry loop and + therefore does not receive the wait-and-replay policy. + Every surface releases (and awaits the cancellation of) the unread 429 body BEFORE the + backoff, records the `rate-limit-429` recovery kind on replay sends, and (bridges) clears the + old response-header deadline before the wait and starts a fresh one afterward, re-checking + client cancellation before telemetry and replay. + Covers Responses, chat completions, and routed Claude messages (they all enter + `handleResponses`). + +## Safety + +- Pre-stream only: a 429 arrives before any bytes are relayed, so replaying the string-body + request is lossless (same invariant as the transient-5xx layer in `lib/upstream-retry.ts`). +- Ordering: same-key retries run before failover, so "primary-first" users keep their key on + rate-limit blips; failover still works after retries exhaust. +- Retry-wait bound: the SLEEP component is at most `attempts × maxIntervalMs` (default + 3 × 60 s = 180 s) when honoring upstream `Retry-After`; `attempts × intervalMs` + (default 15 s) when `respectRetryAfter=false` or no header is present. Total request + latency is higher: every attempt also consumes its own connect/response time (bounded by + `connectTimeoutMs`), so the documented bound covers deliberate waits only. +- Identical replay: rebuilds are deterministic for the same parsed request (same serialized + body and auth headers); the passthrough/continuation/e2e tests assert byte-identical bodies + and identical auth headers across replays, not just send counts. +- Abort during the wait: the sleep is abort-aware — when the server observes the client + disconnect (Bun propagates this asynchronously, observed 1–10 s), the wait is interrupted, + the unread 429 body is released, and the request is cancelled with 499 before any replay. + Because the propagation is async, a replay can still precede the cancel if the interval + elapses first; that is bounded by the same `attempts` budget. Terminal continuations sleep + on the upstream signal, so a body-cancel (SSE already streaming) aborts the wait too. +- Concurrency: each request honors its own policy independently — no process-wide cooldown is + shared between concurrent requests (unlike the Kiro 429 pattern). Upstream volume per + request: same-key replays add at most `attempts` sends, then multi-key failover adds up to + `poolKeys − 1` more (or Anthropic account rotations), so the combined bound is + `attempts + poolKeys` sends (pool size = configured `apiKeyPool` length, fixed per request) — + a storm multiplies by that factor per request, not by `attempts + 1`. +- Header deadlines: the image/video and web-search bridge loops restart their response-header + deadline after each deliberate wait, so backoffs never consume the connect budget and a + rate-limit wait is never misattributed as a 504 header timeout. The old deadline is cleared + BEFORE the sleep and client cancellation is re-checked after it, so 499 always wins over a + stale-deadline edge. +- Expired `Retry-After`: a valid HTTP-date already in the past retries immediately (same as + numeric `Retry-After: 0`) instead of falling back to the fixed interval. +- Recovery observability: every retry surface records the `rate-limit-429` recovery kind + (normal loop, passthrough wire, image/video bridge, web-search sidecar, terminal + continuations), so usage logs explain the extra sends. +- Final 429 still carries `Retry-After` for clients that honor it (Claude Code). + +## Tests + +- `tests/rate-limit-retry.test.ts` — policy normalization (incl. OAuth/forward gating) + + delay computation (seconds, HTTP-date, `0`, malformed, cap; deterministic) + abort during + the wait: a directly-invoked `handleResponses` with a controlled abort signal returns 499, + cancels the unread 429 body, and performs no further upstream sends (deterministic — no + real-socket disconnect timing involved). +- `tests/usage-log.test.ts` — the `rate-limit-429` recovery kind survives persisted usage logs. +- `tests/server-rate-limit-retry-e2e.test.ts` — single-key replay to success, immediate + passthrough without the knob, exhausted attempts surface 429, and retry-before-failover + ordering with a 2-key pool, plus key-auth `openai-responses` passthrough replaying 429 on + the same key. +- `tests/terminal-guard-server.test.ts` — an Anthropic terminal-guard continuation that 429s + is replayed on the same key before the error surfaces (3 upstream sends). +- `tests/images/loop.test.ts` + `tests/web-search.test.ts` — the bridge loops replay 429 on + the same key before `on429` rotation runs (same-key sends counted, rotations zero). diff --git a/docs-site/src/content/docs/guides/providers.md b/docs-site/src/content/docs/guides/providers.md index 5fce1c79c..b19adf39b 100644 --- a/docs-site/src/content/docs/guides/providers.md +++ b/docs-site/src/content/docs/guides/providers.md @@ -39,6 +39,12 @@ labels local presets separately; those normally omit both `authMode` and `apiKey | `forward` | Relays **your incoming Codex auth headers** verbatim to the provider — no key stored. This is the ChatGPT-login passthrough. | OpenAI (`openai-responses` adapter). | | `oauth` | Resolves a stored OAuth access token (auto-refreshed before expiry) and uses it as the bearer key. | xAI, Anthropic, Kimi, Kiro, Google Antigravity, Cursor, GitHub Copilot. | +The [`retryOn429`](/reference/configuration/) same-key 429 replay applies only to API-key +providers (`authMode: "key"`). OAuth, forward, and local presets are excluded — their +credentials must never be replayed on the same token, and local runtimes have no remote key to +preserve. It is opt-in: when the option is absent the feature is off; object presence enables +it unless `enabled: false`. + ## 1. ChatGPT login (forward / passthrough) The `openai` provider needs **no API key**. Direct forwards credentials from your existing diff --git a/docs-site/src/content/docs/ja/guides/providers.md b/docs-site/src/content/docs/ja/guides/providers.md index d55a8a843..0fa811e4a 100644 --- a/docs-site/src/content/docs/ja/guides/providers.md +++ b/docs-site/src/content/docs/ja/guides/providers.md @@ -9,7 +9,7 @@ description: opencodex が LLM プロバイダーを認証し通信するすべ ## OpenAI アカウントモード | プロバイダー ID | 用途 | 認証情報/アカウットルール | - --- | --- | --- | +| --- | --- | --- | | `openai` | Codex ログイン | Pool(デフォルト)はメイン + 追加アカウントを選び、Direct は現在の caller/メインログインのみを使います。 | | `openai-apikey` | OpenAI API | 設定された API キー/キープールのみを使い、Codex アカウントは読みません。 | @@ -30,11 +30,16 @@ max input 922,000 で `*-pro` virtual ID は公開状態を維持し、wire で ローカルプリセットを別に分類します。ローカルプリセットでは通常 `authMode` と `apiKey` を両方使いません。 | `authMode` | 認証方式 | 用途 | - --- | --- | --- | +| --- | --- | --- | | `key` | API キーを送信します(`Authorization: Bearer …`、またはアダプターにより `x-api-key` / `api-key`)。キーはリテラルまたは `${ENV_VAR}` 参照です。 | 大半のプロバイダー。 | | `forward` | **受け取った Codex 認証ヘッダーを**プロバイダーにそのまま中継します — キーを保存しません。ChatGPT ログインのパススルーです。 | OpenAI(`openai-responses` アダプター)。 | | `oauth` | 保存された OAuth アクセストークンを読み込み bearer キーとして使い、期限切れ前に自動更新します。 | xAI、Anthropic、Kimi、Kiro、Google Antigravity、Cursor。 | +[`retryOn429`](/ja/reference/configuration/)（同一キーでの 429 リトライ）は API キー プロバイダー +（`authMode: "key"`）のみに適用されます。OAuth・forward・ローカル プリセットは除外されます — +同じトークンを再送すべきではなく、ローカルランタイムには保存すべきリモートキーがありません。 +オプトインです: オプションが無ければ無効、オブジェクトがあれば `enabled: false` でない限り有効です。 + ## 1. ChatGPT ログイン(forward / パススルー) デフォルトプロバイダーは**API キー不要**です。既存の `codex login` の認証情報を OpenAI Responses バックエンドに @@ -138,7 +143,7 @@ Cline IDE/CLI のみで API からは使えません。`minimax/minimax-m2.5` 無料試用モデルとして文書化されています。 | プロバイダー | ベース URL | - --- | --- | +| --- | --- | | **OpenAI (API キー)** | `https://api.openai.com/v1` | | **Anthropic (API キー)** | `https://api.anthropic.com` | | **OpenRouter** | `https://openrouter.ai/api/v1` | @@ -238,7 +243,7 @@ OAuth、API キープールを確認・切り替えできます。完全なコ Sol/Terra/Luna をフォールバックリストに入れています。 | Codex 経路 | 事前登録されたモデル ID | Codex に表示されるコンテキスト | - --- | --- | --- | +| --- | --- | --- | | Codex ログイン(Pool または Direct) | `gpt-5.6-*` | 372,000 | | OpenAI (API キー) | `openai-apikey/gpt-5.6-*` と `*-pro` | 1,050,000 (max input 922,000) | | OpenRouter | `openrouter/openai/gpt-5.6-sol`、`openrouter/openai/gpt-5.6-terra`、`openrouter/openai/gpt-5.6-luna` | 1,050,000 | @@ -260,6 +265,15 @@ Amazon Bedrock ネイティブ API のような、これらの実装のいずれ **サブスクリプショントークン**(通常の API キーではない)で認証します。**Cloudflare AI Gateway** は URL にアカウント + ゲートウェイ ID を埋める必要があります。 +Copilot は混在 wire カタログを提供します。GPT-5 系モデル（`gpt-5.3-codex`、`gpt-5.4`、 +`gpt-5.4-mini`、`gpt-5.5`、`gpt-5.6-luna`、`gpt-5.6-sol`、`gpt-5.6-terra`）はエージェント +通信の `/chat/completions` を拒否するため、opencodex はこれらのモデルを組み込みデフォルトで +Responses API 経由にルーティングし、他の Copilot モデルはすべて chat completions のままです。 +優先順位は次のとおりです: ハード wire ピン → 明示的な +[`modelAdapters`](/ja/reference/configuration/providers/) エントリ → レジストリのデフォルト → +プロバイダー全体の adapter。組み込みデフォルトのないモデル（例: `gpt-5.4-nano`）を Responses +に移すには、`"modelAdapters": { "gpt-5.4-nano": "openai-responses" }` を設定してください。 + Cursor は別の実験的アダプターとして追跡します。`adapter: "cursor"` は `ocx init` とダッシュボード Add Provider ピッカーに実験的 local config 項目として表示され、Cursor の静的フォールバックモデルカタログ メタデータを保存します。Cursor アクセストークンを設定すると opencodex は Cursor ライブ HTTP/2 トランスポートを @@ -293,7 +307,7 @@ Ollama の `:size` タグに寛容なので `gpt-oss` は `gpt-oss:120b` と `gp opencodex をローカルの OpenAI 互換サーバーに向けてください — 通常は空キーで使います: | プロバイダー | ベース URL | - --- | --- | +| --- | --- | | Ollama (local) | `http://localhost:11434/v1` | | vLLM | `http://localhost:8000/v1` | | LM Studio | `http://localhost:1234/v1` | diff --git a/docs-site/src/content/docs/ja/reference/adapters.md b/docs-site/src/content/docs/ja/reference/adapters.md index 2f5cd1e50..0bd13fbc5 100644 --- a/docs-site/src/content/docs/ja/reference/adapters.md +++ b/docs-site/src/content/docs/ja/reference/adapters.md @@ -46,6 +46,11 @@ interface ProviderAdapter { **対象:** OpenAI **Responses API**。**`passthrough: true`** — 元のリクエスト本文をそのまま渡し、レスポンスを **変換せずに** ストリーミングします。 **認証:** `forward`（呼び出し元ヘッダー中継）または `key`。 +`key` 認証では、[`retryOn429`](/ja/reference/configuration/) もここに適用されます: プリストリームの +429 は、翻訳された `openai-chat` / Anthropic リクエスト経路と同様に、他の処理やフェイルオーバーに +先立って、同じキーで同一リクエストを待機して再送します。カスタム `runTurn` トランスポートは +HTTP リトライ ループの対象外です。 + - `forward` URL → `{baseUrl}/responses`。`key` provider はデフォルトで従来の `{baseUrl}/v1/responses` 構築を使います。 - `key` provider は検証済みの相対 `responsesPath` を設定できます。adapter は `baseUrl` 末尾の `/` を 1 つ除き、`{trimmedBaseUrl}{responsesPath}` に送信します。Ark Agent Plan では `baseUrl: "https://ark.cn-beijing.volces.com/api/plan/v3"` と `responsesPath: "/responses"` を使います。 - `forward` モードでは安全なヘッダー許可リスト（`FORWARD_HEADERS`）だけを中継します。authorization、ChatGPT account id、OpenAI beta/originator/session ヘッダーが対象です。この ChatGPT ログイン経路は [サイドカー](/ja/guides/sidecars/) にも使われます。 diff --git a/docs-site/src/content/docs/ja/reference/configuration/providers.md b/docs-site/src/content/docs/ja/reference/configuration/providers.md index 39a451397..0285abddb 100644 --- a/docs-site/src/content/docs/ja/reference/configuration/providers.md +++ b/docs-site/src/content/docs/ja/reference/configuration/providers.md @@ -85,6 +85,7 @@ namespace 付き combo alias はその namespace prefix に selector を再利 | `noPenaltyModels?` | `string[]` |存在/周波数ペナルティを拒否するモデル。 | | `parallelToolCalls?` | `boolean` |並列ツール呼び出しを切り替えます。 OpenAI Chat はデフォルトでオンになっています。非チャット アダプターは明示的な `true` でのみアドバタイズします。 | | `responsesItemIdRepair?` | `{ message?: string[]; reasoning?: string[]; repairMissingTerminalIds?: boolean }` |正確なプレースホルダー ID および欠落している端末 ID に対するダウンストリーム SSE 修復はデフォルトで無効になっています。関数呼び出し ID は決して書き換えられません。 | +| `retryOn429?` | `{ enabled?: boolean; attempts?: number; intervalMs?: number; maxIntervalMs?: number; respectRetryAfter?: boolean }` | API-key プロバイダーのみ(`authMode: "key"`)。オプトインの同一ターゲット 429 リトライ: `retryOn429` が無ければ無効で、オブジェクトがあれば `enabled: false` でない限り有効になります。429 時に待機(上流の `Retry-After` または固定間隔)してから、キー フェイルオーバーの前に同一キーで同一リクエストを再送します — メインのテキストターン回復ループ、Responses passthrough、画像/動画ブリッジ、web-search サイドカー、ターミナル継続要求をすべてカバーします。再送の対象はプリストリームの HTTP 429 応答のみで、カスタム `runTurn` トランスポートは HTTP リトライループの対象外です。`attempts` は最初の 429 以降の同一キー再送回数(合計送信数 = `attempts` + 1)で、メインの回復ループ・ターミナルガード継続・ブリッジ再試行で共有されるリクエスト単位の予算です。`attempts` を使い切っても同一キーでの再送が止まるだけで、通常のキー フェイルオーバーまたは最終エラー処理が利用可能なターゲットに応じて続きます — キー認証の passthrough ワイヤにはフェイルオーバーがないため、使い切った 429 はそのまま返ります。Codex 自体は 429 をリトライしないため、単一キーのプロバイダーでは唯一の防御です。デフォルト: `enabled: true`、`attempts: 3`、`intervalMs: 5000`、`maxIntervalMs: 60000`(1回の待機は `maxIntervalMs` で上限、その上限は 600000)、`respectRetryAfter: true`。 | | `autoToolChoiceOnlyModels?` | `string[]` | `tool_choice` が `auto` または `none` のみを受け入れるモデル。強制的な選択は格下げされます。 | | `preserveReasoningContentModels?` | `string[]` |チャット履歴に以前のアシスタント `reasoning_content` が必要なモデル。 | | `thinkingToggleModels?` | `string[]` |エフォート ラダーではなく `thinking.enabled` を使用してモデルをチャットします。 | diff --git a/docs-site/src/content/docs/ko/guides/providers.md b/docs-site/src/content/docs/ko/guides/providers.md index ce1609aba..032766ab5 100644 --- a/docs-site/src/content/docs/ko/guides/providers.md +++ b/docs-site/src/content/docs/ko/guides/providers.md @@ -35,6 +35,11 @@ shipped v1 config는 marker 2의 단일 옵션 행으로 자동 이관됩니다. | `forward` | **수신된 Codex 인증 헤더를** 프로바이더에 그대로 중계합니다 — 키를 저장하지 않습니다. ChatGPT 로그인 패스스루입니다. | OpenAI (`openai-responses` 어댑터). | | `oauth` | 저장된 OAuth 액세스 토큰을 불러와 bearer 키로 사용하며, 만료 전에 자동 갱신합니다. | xAI, Anthropic, Kimi, Kiro, Google Antigravity, Cursor. | +[`retryOn429`](/ko/reference/configuration/)（동일 키 429 재시도）는 API 키 프로바이더 +（`authMode: "key"`）에만 적용됩니다. OAuth·forward·로컬 프리셋은 제외됩니다 — 같은 토큰을 +재전송해서는 안 되며, 로컬 런타임에는 보존할 원격 키가 없습니다. 옵트인입니다: 옵션이 없으면 +꺼져 있고, 객체가 있으면 `enabled: false`가 아닌 한 활성화됩니다. + ## 1. ChatGPT 로그인 (forward / 패스스루) 기본 프로바이더는 **API 키가 필요 없습니다**. 기존 `codex login`의 자격 증명을 OpenAI Responses 백엔드로 @@ -261,6 +266,15 @@ Amazon Bedrock 네이티브 API처럼 이 구현 중 어느 것과도 맞지 않 **구독 토큰**(일반 API 키가 아님)으로 인증합니다. **Cloudflare AI Gateway**는 URL에 계정 + 게이트웨이 id를 채워야 합니다. +Copilot은 혼합 wire 카탈로그를 제공합니다. GPT-5 계열 모델(`gpt-5.3-codex`, `gpt-5.4`, +`gpt-5.4-mini`, `gpt-5.5`, `gpt-5.6-luna`, `gpt-5.6-sol`, `gpt-5.6-terra`)은 에이전트 +트래픽에 대해 `/chat/completions`를 거부하므로 opencodex는 이 모델들을 내장 기본값으로 +Responses API를 통해 라우팅하고, 다른 Copilot 모델은 모두 chat completions를 유지합니다. +우선순위는 하드 wire 핀 → 명시적 [`modelAdapters`](/ko/reference/configuration/providers/) +항목 → 레지스트리 기본값 → 프로바이더 전체 adapter 순입니다. 내장 기본값이 없는 모델(예: +`gpt-5.4-nano`)을 Responses로 전환하려면 `"modelAdapters": { "gpt-5.4-nano": "openai-responses" }`를 +설정하세요. + Cursor는 별도의 실험적 어댑터로 추적합니다. `adapter: "cursor"`는 `ocx init`과 dashboard Add Provider picker에 실험적 local config 항목으로 표시되며, Cursor의 static fallback model catalog metadata를 저장합니다. Cursor access token이 설정되면 opencodex는 Cursor live HTTP/2 transport를 diff --git a/docs-site/src/content/docs/ko/reference/adapters.md b/docs-site/src/content/docs/ko/reference/adapters.md index 57fb881b9..bddf944b7 100644 --- a/docs-site/src/content/docs/ko/reference/adapters.md +++ b/docs-site/src/content/docs/ko/reference/adapters.md @@ -53,6 +53,11 @@ interface ProviderAdapter { **변환하지 않은 채** 스트리밍합니다. **인증:** `forward`(호출자 헤더 중계) 또는 `key`. +`key` 인증에서는 [`retryOn429`](/ko/reference/configuration/)도 여기에 적용됩니다: 사전 스트림 +429는 번역된 `openai-chat`/Anthropic 요청 경로와 동일하게 다른 처리나 페일오버보다 먼저 +같은 키로 동일 요청을 대기 후 재전송합니다. 커스텀 `runTurn` 전송은 HTTP 재시도 루프에 +포함되지 않습니다. + - `forward` URL → `{baseUrl}/responses`. `key` provider는 기본적으로 기존 `{baseUrl}/v1/responses` 구성을 사용합니다. - `key` provider는 검증된 상대 `responsesPath`를 설정할 수 있습니다. adapter는 `baseUrl` 끝의 `/` 하나를 제거하고 `{trimmedBaseUrl}{responsesPath}`로 전송합니다. Ark Agent Plan은 `baseUrl: "https://ark.cn-beijing.volces.com/api/plan/v3"`와 `responsesPath: "/responses"`를 사용합니다. - `forward` 모드에서는 안전한 헤더 허용 목록(`FORWARD_HEADERS`)만 중계합니다. authorization, diff --git a/docs-site/src/content/docs/ko/reference/configuration/providers.md b/docs-site/src/content/docs/ko/reference/configuration/providers.md index f3d08deef..38a499c6c 100644 --- a/docs-site/src/content/docs/ko/reference/configuration/providers.md +++ b/docs-site/src/content/docs/ko/reference/configuration/providers.md @@ -33,7 +33,7 @@ provider 및 예약된 `openai` / `combo` 충돌은 대소문자를 구분하지 combo alias는 selector를 namespace prefix로 재사용할 수 없습니다. 설정된 pool id와 다른 selector target도 selector로 재사용할 수 없습니다. raw account id와 email은 비공개로 유지하고 selector를 공개 이름으로 사용하세요. 명시적 선택 동작과 우선순위는 -[라우팅 설정](/reference/configuration/routing/)을 참고하십시오. +[라우팅 설정](/ko/reference/configuration/routing/)을 참고하십시오. ## 예약된 OpenAI 공급자 @@ -85,6 +85,7 @@ target도 selector로 재사용할 수 없습니다. raw account id와 email은 | `noPenaltyModels?` | `string[]` | presence/frequency penalty를 허용하지 않는 모델입니다. | | `parallelToolCalls?` | `boolean` | 병렬 도구 호출을 켜거나 끕니다. OpenAI Chat은 기본으로 켜져 있고, 비-chat 어댑터는 명시적으로 `true`일 때만 이를 노출합니다. | | `responsesItemIdRepair?` | `{ message?: string[]; reasoning?: string[]; repairMissingTerminalIds?: boolean }` | 기본값이 꺼진 downstream SSE 복구입니다. 정확한 자리표시자 id와 누락된 종료 id를 복구합니다. function-call id는 다시 쓰지 않습니다. | +| `retryOn429?` | `{ enabled?: boolean; attempts?: number; intervalMs?: number; maxIntervalMs?: number; respectRetryAfter?: boolean }` | API-key 프로바이더 전용(`authMode: "key"`). 동일 대상 429 재시도: `retryOn429`가 없으면 기능이 꺼져 있고, 객체가 있으면 `enabled: false`가 아닌 한 활성화됩니다. 429 시 대기(업스트림 `Retry-After` 또는 고정 간격) 후 키 장애 조치 전에 동일 키로 동일 요청을 재전송합니다 — 일반 텍스트 턴 복구 루프, Responses passthrough, 이미지/비디오 브리지, web-search 사이드카, 터미널 연속 요청을 모두 포함합니다. 재전송 대상은 프리스트림 HTTP 429 응답뿐이며, 커스텀 `runTurn` 전송은 HTTP 재시도 루프에서 제외됩니다. `attempts`는 첫 429 이후의 동일 키 재전송 횟수(총 전송 = `attempts` + 1)이며, 메인 복구 루프·터미널 가드 연속 요청·브리지 재시도가 공유하는 요청 단위 예산입니다. `attempts`를 모두 소진해도 동일 키 재전송만 중단되며, 이후에는 일반 키 장애 조치 또는 최종 오류 처리가 사용 가능한 대상에 따라 진행됩니다 — 키 인증 passthrough 와이어에는 장애 조치가 없으므로 소진된 429가 그대로 반환됩니다. Codex 자체는 429를 재시도하지 않으므로 단일 키 프로바이더의 유일한 방어선입니다. 기본값: `enabled: true`, `attempts: 3`, `intervalMs: 5000`, `maxIntervalMs: 60000`(단일 대기는 `maxIntervalMs`로 상한, 그 자체는 600000으로 상한), `respectRetryAfter: true`. | | `autoToolChoiceOnlyModels?` | `string[]` | `tool_choice`가 `auto` 또는 `none`만 받는 모델입니다. 강제 선택은 낮은 수준으로 바뀝니다. | | `preserveReasoningContentModels?` | `string[]` | chat 기록에서 이전 assistant `reasoning_content`가 필요한 모델입니다. | | `thinkingToggleModels?` | `string[]` | effort 계층 대신 `thinking.enabled`를 쓰는 chat 모델입니다. | diff --git a/docs-site/src/content/docs/reference/adapters.md b/docs-site/src/content/docs/reference/adapters.md index 011d837b9..970557956 100644 --- a/docs-site/src/content/docs/reference/adapters.md +++ b/docs-site/src/content/docs/reference/adapters.md @@ -53,6 +53,11 @@ provider — xAI, Kimi, DeepSeek, GLM, Groq, OpenRouter, Ollama (local & cloud), streams the response back **untranslated**. **Auth:** `forward` (relay the caller's headers) or `key`. +For `key` auth, [`retryOn429`](/reference/configuration/) applies here too: a pre-stream 429 +waits and replays the identical request on the same key before any other handling, exactly like +the translated `openai-chat` / Anthropic request path. Custom `runTurn` transports are not part +of the HTTP retry loop. + - `forward` URL → `{baseUrl}/responses`. A `key` provider defaults to the legacy `{baseUrl}/v1/responses` construction. - A `key` provider may set a validated relative `responsesPath`; the adapter removes one trailing slash from `baseUrl` and sends `{trimmedBaseUrl}{responsesPath}`. For Ark Agent Plan, use `baseUrl: "https://ark.cn-beijing.volces.com/api/plan/v3"` with `responsesPath: "/responses"`. - In `forward` mode only a safe header allowlist is relayed (`FORWARD_HEADERS`): authorization, diff --git a/docs-site/src/content/docs/reference/configuration/providers.md b/docs-site/src/content/docs/reference/configuration/providers.md index a04a142ef..7e27d048b 100644 --- a/docs-site/src/content/docs/reference/configuration/providers.md +++ b/docs-site/src/content/docs/reference/configuration/providers.md @@ -92,6 +92,7 @@ differing backup and rewrites known legacy namespaced selected ids to bare ids. | `noPenaltyModels?` | `string[]` | Models that reject presence/frequency penalties. | | `parallelToolCalls?` | `boolean` | Toggle parallel tool calls. OpenAI Chat defaults on; non-chat adapters advertise only on explicit `true`. | | `responsesItemIdRepair?` | `{ message?: string[]; reasoning?: string[]; repairMissingTerminalIds?: boolean }` | Disabled-by-default downstream SSE repair for exact placeholder ids and missing terminal ids. Function-call ids are never rewritten. | +| `retryOn429?` | `{ enabled?: boolean; attempts?: number; intervalMs?: number; maxIntervalMs?: number; respectRetryAfter?: boolean }` | API-key providers only (`authMode: "key"`). Opt-in same-target 429 retry: when `retryOn429` is absent the feature is off; object presence enables it unless `enabled: false`. On 429 the proxy waits (upstream `Retry-After` or the fixed interval) and replays the identical request on the same key before any key failover — across the main text-turn recovery loop, the Responses passthrough wire, the image/video bridge, the web-search sidecar, and terminal continuations. Only pre-stream HTTP 429 responses are eligible for replay; custom `runTurn` transports are outside the HTTP retry loop. `attempts` counts same-key replays after the first 429 (total sends = `attempts` + 1) and is one request-wide budget shared by the main recovery loop, the terminal-guard continuation, and bridge retries. Exhausting `attempts` only stops further same-key replays: normal key failover or final-error handling then applies per the available targets — on the key-auth passthrough wire there is no failover, so the exhausted 429 surfaces as-is. Codex itself never retries 429, so this is the only defense for single-key providers. Defaults: `enabled: true`, `attempts: 3`, `intervalMs: 5000`, `maxIntervalMs: 60000` (any single wait is capped at `maxIntervalMs`, itself capped at 600000), `respectRetryAfter: true`. | | `autoToolChoiceOnlyModels?` | `string[]` | Models whose `tool_choice` accepts only `auto` or `none`; forced choices are downgraded. | | `preserveReasoningContentModels?` | `string[]` | Models requiring prior assistant `reasoning_content` in chat history. | | `thinkingToggleModels?` | `string[]` | Chat models using `thinking.enabled` rather than an effort ladder. | diff --git a/docs-site/src/content/docs/ru/guides/providers.md b/docs-site/src/content/docs/ru/guides/providers.md index 2ef2dfdb1..1be9218c0 100644 --- a/docs-site/src/content/docs/ru/guides/providers.md +++ b/docs-site/src/content/docs/ru/guides/providers.md @@ -41,6 +41,12 @@ description: Все способы, которыми opencodex аутентиф | `forward` | Передаёт провайдеру **входящие заголовки аутентификации Codex** без изменений — ключ не хранится. Это сквозной режим (passthrough) входа через ChatGPT. | OpenAI (адаптер `openai-responses`). | | `oauth` | Берёт сохранённый OAuth-токен доступа (автоматически обновляется до истечения срока) и использует его как bearer-ключ. | xAI, Anthropic, Kimi, Kiro, Google Antigravity, Cursor, GitHub Copilot. | +Повтор при 429 на том же ключе ([`retryOn429`](/ru/reference/configuration/)) применим только к +провайдерам с API-ключом (`authMode: "key"`). Пресеты OAuth, forward и local исключены — их +учётные данные нельзя повторно отправлять по тому же токену, а у локальных сред выполнения нет +удалённого ключа. Это opt-in: при отсутствии опции функция выключена; наличие объекта включает +её, если только `enabled: false`. + ## 1. Вход через ChatGPT (forward / passthrough) Провайдеру `openai` **не нужен API-ключ**. Direct пересылает учётные данные вашего существующего @@ -272,6 +278,15 @@ Assist), `azure` / `azure-openai`, `kiro` и `cursor`. Проприетарны **GitLab Duo** остаётся шлюзом с ключом/токеном подписки на своей OpenAI-совместимой конечной точке. **Cloudflare AI Gateway** требует подставить в URL id аккаунта и шлюза. +Copilot предоставляет каталог со смешанными проводами: его семейство GPT-5 (`gpt-5.3-codex`, +`gpt-5.4`, `gpt-5.4-mini`, `gpt-5.5`, `gpt-5.6-luna`, `gpt-5.6-sol`, `gpt-5.6-terra`) +отклоняет `/chat/completions` для агентного трафика, поэтому opencodex по умолчанию +маршрутизирует эти модели через Responses API, а все остальные модели Copilot остаются на +chat completions. Приоритет: жёсткий wire-пин → явная запись +[`modelAdapters`](/ru/reference/configuration/providers/) → дефолт реестра → adapter всего +провайдера. Чтобы перевести модель без встроенного дефолта (например, `gpt-5.4-nano`) на +Responses, задайте `"modelAdapters": { "gpt-5.4-nano": "openai-responses" }`. + Cursor отслеживается отдельно как экспериментальный адаптер. `adapter: "cursor"` появляется в `ocx init` и в селекторе Add Provider дашборда как экспериментальная запись локальной конфигурации с метаданными статического резервного каталога моделей Cursor. Когда настроен токен доступа Cursor, diff --git a/docs-site/src/content/docs/ru/reference/adapters.md b/docs-site/src/content/docs/ru/reference/adapters.md index 49ba48c89..d84c86693 100644 --- a/docs-site/src/content/docs/ru/reference/adapters.md +++ b/docs-site/src/content/docs/ru/reference/adapters.md @@ -57,6 +57,11 @@ interface ProviderAdapter { запроса и стримит ответ обратно **без преобразования**. **Аутентификация:** `forward` (ретрансляция заголовков вызывающей стороны) или `key`. +При `key`-аутентификации [`retryOn429`](/ru/reference/configuration/) действует и здесь: 429 до +начала потока ждёт и, до любой другой обработки или фейловера, повторяет идентичный запрос на +том же ключе, как и в переводимом пути `openai-chat`/Anthropic. Пользовательские транспорты +`runTurn` в цикл HTTP-повторов не входят. + - URL для `forward` → `{baseUrl}/responses`. Провайдер с `key` по умолчанию сохраняет прежнее построение `{baseUrl}/v1/responses`. - Провайдер с `key` может задать проверенный относительный `responsesPath`: адаптер удаляет один завершающий `/` из `baseUrl` и отправляет запрос на `{trimmedBaseUrl}{responsesPath}`. Для Ark Agent Plan используйте `baseUrl: "https://ark.cn-beijing.volces.com/api/plan/v3"` и `responsesPath: "/responses"`. - В режиме `forward` ретранслируется только безопасный allowlist заголовков (`FORWARD_HEADERS`): diff --git a/docs-site/src/content/docs/ru/reference/configuration/providers.md b/docs-site/src/content/docs/ru/reference/configuration/providers.md index 18d884525..4003682d2 100644 --- a/docs-site/src/content/docs/ru/reference/configuration/providers.md +++ b/docs-site/src/content/docs/ru/reference/configuration/providers.md @@ -95,6 +95,7 @@ cross-route credential fallback не существует. Строки API GPT- | `noPenaltyModels?` | `string[]` | Модели, отвергающие penalty presence/frequency. | | `parallelToolCalls?` | `boolean` | Переключатель parallel tool call'ов. Для OpenAI Chat по умолчанию включено; не-chat adapter'ы рекламируют это только при явном `true`. | | `responsesItemIdRepair?` | `{ message?: string[]; reasoning?: string[]; repairMissingTerminalIds?: boolean }` | По умолчанию выключенная downstream SSE-repair для exact placeholder-id и отсутствующих terminal-id. Function-call id никогда не переписываются. | +| `retryOn429?` | `{ enabled?: boolean; attempts?: number; intervalMs?: number; maxIntervalMs?: number; respectRetryAfter?: boolean }` | Только для провайдеров с API-ключом (`authMode: "key"`). Опциональный повтор при 429 на том же таргете: если `retryOn429` отсутствует, функция выключена; наличие объекта включает её, если только `enabled: false`. При 429: ожидание (`Retry-After` апстрима или фиксированный интервал) и повтор идентичного запроса на том же ключе до любого фейловера ключей — покрывает основной цикл восстановления текстовых ходов, passthrough-канал Responses, мост изображений/видео, sidecar web-search и терминальные продолжения. Повтор допустим только для HTTP 429, полученных до начала потока; пользовательские транспорты `runTurn` не входят в цикл HTTP-повторов. `attempts` — это число повторов на том же ключе после первого 429 (всего отправок = `attempts` + 1) и единый бюджет на запрос, общий для основного цикла восстановления, терминального продолжения и повторов моста. Исчерпание `attempts` лишь останавливает дальнейшие повторы на том же ключе; далее применяется обычный фейловер ключей или финальная обработка ошибки в зависимости от доступных таргетов — на passthrough-канале с ключевой аутентификацией фейловера нет, поэтому исчерпанный 429 возвращается как есть. Codex сам никогда не повторяет 429, поэтому это единственная защита для провайдеров с одним ключом. По умолчанию: `enabled: true`, `attempts: 3`, `intervalMs: 5000`, `maxIntervalMs: 60000` (любое ожидание ограничено `maxIntervalMs`, который сам ограничен 600000), `respectRetryAfter: true`. | | `autoToolChoiceOnlyModels?` | `string[]` | Модели, у которых `tool_choice` принимает только `auto` или `none`; forced choice понижается. | | `preserveReasoningContentModels?` | `string[]` | Модели, которым нужен предыдущий assistant `reasoning_content` в chat history. | | `thinkingToggleModels?` | `string[]` | Chat-модели, использующие `thinking.enabled` вместо effort-ladder. | diff --git a/docs-site/src/content/docs/zh-cn/guides/providers.md b/docs-site/src/content/docs/zh-cn/guides/providers.md index 2680857dc..6ce56c9bc 100644 --- a/docs-site/src/content/docs/zh-cn/guides/providers.md +++ b/docs-site/src/content/docs/zh-cn/guides/providers.md @@ -34,6 +34,10 @@ shipped v1 配置自动迁移到 marker 2 的单一选项行。原配置只保 | `forward` | 将**你传入的 Codex 认证请求头**原样转发给提供商——不存储任何密钥。这就是 ChatGPT 登录的透传方式。 | OpenAI（`openai-responses` adapter）。 | | `oauth` | 读取已存储的 OAuth 访问令牌（过期前自动刷新），并将其用作 bearer 密钥。 | xAI、Anthropic、Kimi、Kiro、Google Antigravity、Cursor。 | +[`retryOn429`](/zh-cn/reference/configuration/)（同 key 的 429 重试）仅适用于 API-key 提供商 +（`authMode: "key"`）。OAuth、forward 与本地预设均被排除——同一 token 绝不可重放，本地运行时 +也没有需要保留的远程 key。仅在配置后启用，默认关闭；配置了对象即启用，除非 `enabled: false`。 + ## 1. ChatGPT 登录（forward / 透传） 默认提供商**不需要 API 密钥**。它将你现有 `codex login` 的凭据直接转发到 OpenAI Responses 后端： @@ -242,6 +246,14 @@ GPT-5.6 Sol/Terra/Luna 会预置在提供商的回退列表中，因此即使实 使用 Bearer **订阅令牌**（而非普通 API 密钥）进行认证。 **Cloudflare AI Gateway** 需要将 account 和 gateway id 填入 URL。 +Copilot 提供混合 wire 目录：其 GPT-5 系列模型（`gpt-5.3-codex`、`gpt-5.4`、 +`gpt-5.4-mini`、`gpt-5.5`、`gpt-5.6-luna`、`gpt-5.6-sol`、`gpt-5.6-terra`）会拒绝面向 +agent 流量的 `/chat/completions`，因此 opencodex 默认将这些模型路由到 Responses API，而其他 +Copilot 模型仍走 chat completions。优先级为：硬 wire 固定 → 显式 +[`modelAdapters`](/zh-cn/reference/configuration/providers/) 条目 → 注册表默认值 → 提供商级 +adapter。若要将没有内置默认值的模型（例如 `gpt-5.4-nano`）接入 Responses，请设置 +`"modelAdapters": { "gpt-5.4-nano": "openai-responses" }`。 + Cursor 作为单独的实验性 adapter 进行跟踪。`adapter: "cursor"` 会作为实验性本地配置出现在 `ocx init` 和 dashboard Add Provider picker 中，并保存 Cursor 的静态回退模型目录 metadata。配置 Cursor access token 后，opencodex 会使用 Cursor live HTTP/2 transport。内置回退列表包含上下文为 diff --git a/docs-site/src/content/docs/zh-cn/reference/adapters.md b/docs-site/src/content/docs/zh-cn/reference/adapters.md index f1e8e1a71..032df8806 100644 --- a/docs-site/src/content/docs/zh-cn/reference/adapters.md +++ b/docs-site/src/content/docs/zh-cn/reference/adapters.md @@ -50,6 +50,10 @@ interface ProviderAdapter { **不经转换**地流式传回。 **认证：** `forward`（转发调用方 header）或 `key`。 +使用 `key` 认证时，[`retryOn429`](/zh-cn/reference/configuration/) 同样适用：流开始前的 429 +会等待并先于其他处理或故障转移，在相同 key 上重放完全相同请求，与翻译后的 +`openai-chat`/Anthropic 请求路径一致。自定义 `runTurn` 传输不在 HTTP 重试循环之内。 + - `forward` URL → `{baseUrl}/responses`。`key` provider 默认保留原有的 `{baseUrl}/v1/responses` 构造。 - `key` provider 可设置经过验证的相对 `responsesPath`；adapter 会移除 `baseUrl` 末尾的一个 `/`，并向 `{trimmedBaseUrl}{responsesPath}` 发送请求。Ark Agent Plan 使用 `baseUrl: "https://ark.cn-beijing.volces.com/api/plan/v3"` 和 `responsesPath: "/responses"`。 - `forward` 模式只会转发安全的 header allowlist（`FORWARD_HEADERS`）：authorization、ChatGPT diff --git a/docs-site/src/content/docs/zh-cn/reference/configuration/providers.md b/docs-site/src/content/docs/zh-cn/reference/configuration/providers.md index 3702c0046..4d672e90e 100644 --- a/docs-site/src/content/docs/zh-cn/reference/configuration/providers.md +++ b/docs-site/src/content/docs/zh-cn/reference/configuration/providers.md @@ -32,7 +32,7 @@ pool account id（不能是内部 `__main__`），或用 `"@main"` 表示 Codex 保留的 `openai` / `combo` 冲突时不区分大小写；带 namespace 的 combo alias 不能把 selector 复用为 其 namespace prefix，已配置的 pool id 和其他 selector target 也不能复用为 selector。raw account id 与 email 应保持私密，selector 才是公开名称。明确选择的行为和优先级见 -[路由配置](/reference/configuration/routing/)。 +[路由配置](/zh-cn/reference/configuration/routing/)。 ## 保留的 OpenAI 提供者 @@ -84,6 +84,7 @@ pool account id（不能是内部 `__main__`），或用 `"@main"` 表示 Codex | `noPenaltyModels?` | `string[]` | 会拒绝 presence/frequency penalty 的模型。 | | `parallelToolCalls?` | `boolean` | 切换并行工具调用。OpenAI Chat 默认开启；非 chat 适配器只有显式 `true` 时才会声明支持。 | | `responsesItemIdRepair?` | `{ message?: string[]; reasoning?: string[]; repairMissingTerminalIds?: boolean }` | 默认关闭的下游 SSE 修复，用于精确占位 id 和缺失的终止 id。function-call id 永远不会被重写。 | +| `retryOn429?` | `{ enabled?: boolean; attempts?: number; intervalMs?: number; maxIntervalMs?: number; respectRetryAfter?: boolean }` | 仅限 API-key 提供商（`authMode: "key"`）。可选的同目标 429 重试：未配置 `retryOn429` 时功能关闭；对象存在即启用，除非 `enabled: false`。收到 429 时等待（上游 `Retry-After` 或固定间隔）后在相同 key 上重放完全相同请求，再进入任何 key 故障转移——覆盖主文本恢复循环、Responses passthrough、图像/视频桥、web-search 侧车与终结续接。重放仅适用于流开始前的 HTTP 429 响应；自定义 `runTurn` 传输不在 HTTP 重试循环范围内。`attempts` 是首个 429 之后的同 key 重放次数（总发送次数 = `attempts` + 1），是主恢复循环、终结守卫续接与桥接重试共享的按请求统一预算；`attempts` 耗尽只会停止进一步的同 key 重放：随后按可用目标进行正常的 key 故障转移或最终错误处理——key 认证的 passthrough 线路上没有故障转移，因此耗尽的 429 会原样透出。Codex 自身从不重试 429，因此这是单 key 提供商唯一的防线。默认值：`enabled: true`、`attempts: 3`、`intervalMs: 5000`、`maxIntervalMs: 60000`（单次等待以 `maxIntervalMs` 为上限，其本身上限 600000）、`respectRetryAfter: true`。 | | `autoToolChoiceOnlyModels?` | `string[]` | `tool_choice` 只接受 `auto` 或 `none` 的模型；强制选择会被降级。 | | `preserveReasoningContentModels?` | `string[]` | 需要在聊天历史中保留先前 assistant `reasoning_content` 的模型。 | | `thinkingToggleModels?` | `string[]` | 使用 `thinking.enabled` 而不是 effort 阶梯的 chat 模型。 | diff --git a/gui/src/i18n/de.ts b/gui/src/i18n/de.ts index 9fa38f909..ae49552c2 100644 --- a/gui/src/i18n/de.ts +++ b/gui/src/i18n/de.ts @@ -1,6 +1,9 @@ // German — generated from en.ts. Must match TKey set (compile-checked). import type { TKey } from "./en"; +/** + * German i18n catalog, generated from en.ts. Must match the `TKey` set (compile-checked). + */ export const de: Record = { "nav.dashboard": "Übersicht", "nav.startup": "Startsicherheit", @@ -549,6 +552,14 @@ export const de: Record = { "logs.detail.attempt.reason": "Ergebnis / Grund", "logs.detail.attempt.completed": "Abgeschlossen", "logs.detail.attempt.e2eNote": "Tok/s auf oberster Ebene ist Ende-zu-Ende; jeder Versuch nutzt seine eigene Dauer.", + "logs.detail.attempt.recovery.transient5xx": "Vorübergehender 5xx-Fehler", + "logs.detail.attempt.recovery.connectionReset": "Verbindung zurückgesetzt", + "logs.detail.attempt.recovery.oauth401": "OAuth-Neuanmeldung", + "logs.detail.attempt.recovery.key429": "Schlüssel ratenbegrenzt (429)", + "logs.detail.attempt.recovery.rateLimit429": "Ratenbegrenzt (429)", + "logs.detail.attempt.recovery.anthropicOauth429": "Anthropic OAuth ratenbegrenzt (429)", + "logs.detail.attempt.recovery.image413": "Bildnutzlast zu groß (413)", + "logs.detail.attempt.recovery.unknown": "Unbekannter Wiederherstellungsgrund", "logs.detail.reason.usage_missing": "Nutzung wurde nicht gemeldet.", "logs.detail.reason.usage_unsupported": "Dieser Anbieter meldet keine Nutzung.", "logs.detail.reason.output_missing": "Es wurden keine positiven Ausgabe-Tokens gemeldet.", diff --git a/gui/src/i18n/en.ts b/gui/src/i18n/en.ts index bca1c399e..edde53e1f 100644 --- a/gui/src/i18n/en.ts +++ b/gui/src/i18n/en.ts @@ -1,5 +1,10 @@ // English — source of truth. Its keys define the TKey type; ko/zh/ja must match (compile-checked). // Strings with {cmd} render a chip via ; {var} are plain interpolations. +/** + * English i18n catalog — source of truth. Its keys define the compile-checked `TKey` set; + * other locales must match (compile-checked). `{cmd}` renders a chip via ; + * `{var}` are plain interpolations. + */ export const en = { // sidebar / nav / common "nav.dashboard": "Dashboard", @@ -574,6 +579,14 @@ export const en = { "logs.detail.attempt.reason": "Result / reason", "logs.detail.attempt.completed": "Completed", "logs.detail.attempt.e2eNote": "Top-level tok/s is end-to-end; each attempt uses its own duration.", + "logs.detail.attempt.recovery.transient5xx": "Transient 5xx", + "logs.detail.attempt.recovery.connectionReset": "Connection reset", + "logs.detail.attempt.recovery.oauth401": "OAuth re-authentication", + "logs.detail.attempt.recovery.key429": "Key rate-limited (429)", + "logs.detail.attempt.recovery.rateLimit429": "Rate-limited (429)", + "logs.detail.attempt.recovery.anthropicOauth429": "Anthropic OAuth rate-limited (429)", + "logs.detail.attempt.recovery.image413": "Image payload too large (413)", + "logs.detail.attempt.recovery.unknown": "Unknown recovery reason", "logs.detail.reason.usage_missing": "Usage was not reported.", "logs.detail.reason.usage_unsupported": "This provider does not report usage.", "logs.detail.reason.output_missing": "No positive output token count was reported.", diff --git a/gui/src/i18n/ja.ts b/gui/src/i18n/ja.ts index 9c84de37d..bf7c1266e 100644 --- a/gui/src/i18n/ja.ts +++ b/gui/src/i18n/ja.ts @@ -1,5 +1,8 @@ import type { TKey } from "./en"; +/** + * Japanese i18n catalog; must match the `TKey` set (compile-checked). + */ export const ja: Record = { // sidebar / nav / common "nav.dashboard": "ダッシュボード", @@ -534,6 +537,14 @@ export const ja: Record = { "logs.detail.attempt.reason": "結果 / 理由", "logs.detail.attempt.completed": "完了", "logs.detail.attempt.e2eNote": "トップレベルの tok/s はエンドツーエンドです; 各試行は自身の所要時間を使います。", + "logs.detail.attempt.recovery.transient5xx": "一時的な5xxエラー", + "logs.detail.attempt.recovery.connectionReset": "接続がリセットされました", + "logs.detail.attempt.recovery.oauth401": "OAuth 再認証", + "logs.detail.attempt.recovery.key429": "キーがレート制限 (429)", + "logs.detail.attempt.recovery.rateLimit429": "レート制限 (429)", + "logs.detail.attempt.recovery.anthropicOauth429": "Anthropic OAuth レート制限 (429)", + "logs.detail.attempt.recovery.image413": "画像ペイロードが大きすぎます (413)", + "logs.detail.attempt.recovery.unknown": "不明なリカバリ理由", "logs.detail.reason.usage_missing": "使用量が報告されませんでした。", "logs.detail.reason.usage_unsupported": "このプロバイダーは使用量を報告しません。", "logs.detail.reason.output_missing": "正の出力トークン数が報告されませんでした。", diff --git a/gui/src/i18n/ko.ts b/gui/src/i18n/ko.ts index c43cb067d..ca0e2a241 100644 --- a/gui/src/i18n/ko.ts +++ b/gui/src/i18n/ko.ts @@ -1,5 +1,8 @@ import type { TKey } from "./en"; +/** + * Korean i18n catalog; must match the `TKey` set (compile-checked). + */ export const ko: Record = { // sidebar / nav / common "nav.dashboard": "대시보드", @@ -568,6 +571,14 @@ export const ko: Record = { "logs.detail.attempt.reason": "결과 / 사유", "logs.detail.attempt.completed": "완료", "logs.detail.attempt.e2eNote": "상위 tok/s는 전체 요청 기준이며 각 시도는 자체 소요 시간을 사용합니다.", + "logs.detail.attempt.recovery.transient5xx": "일시적 5xx 오류", + "logs.detail.attempt.recovery.connectionReset": "연결이 재설정됨", + "logs.detail.attempt.recovery.oauth401": "OAuth 재인증", + "logs.detail.attempt.recovery.key429": "키 요청 한도 초과 (429)", + "logs.detail.attempt.recovery.rateLimit429": "요청 한도 초과 (429)", + "logs.detail.attempt.recovery.anthropicOauth429": "Anthropic OAuth 요청 한도 초과 (429)", + "logs.detail.attempt.recovery.image413": "이미지 페이로드가 너무 큼 (413)", + "logs.detail.attempt.recovery.unknown": "알 수 없는 복구 사유", "logs.detail.reason.usage_missing": "usage가 보고되지 않았습니다.", "logs.detail.reason.usage_unsupported": "이 프로바이더는 usage 보고를 지원하지 않습니다.", "logs.detail.reason.output_missing": "양수 출력 토큰 수가 보고되지 않았습니다.", diff --git a/gui/src/i18n/ru.ts b/gui/src/i18n/ru.ts index bbf9d590f..266ac5276 100644 --- a/gui/src/i18n/ru.ts +++ b/gui/src/i18n/ru.ts @@ -1,5 +1,8 @@ import type { TKey } from "./en"; +/** + * Russian i18n catalog; must match the `TKey` set (compile-checked). + */ export const ru: Record = { // sidebar / nav / common "nav.dashboard": "Дашборд", @@ -566,6 +569,14 @@ export const ru: Record = { "logs.detail.attempt.reason": "Результат / причина", "logs.detail.attempt.completed": "Завершено", "logs.detail.attempt.e2eNote": "Общий tok/s — сквозной показатель; для каждой попытки используется её собственная длительность.", + "logs.detail.attempt.recovery.transient5xx": "Временная ошибка 5xx", + "logs.detail.attempt.recovery.connectionReset": "Соединение сброшено", + "logs.detail.attempt.recovery.oauth401": "Повторная авторизация OAuth", + "logs.detail.attempt.recovery.key429": "Ключ ограничен (429)", + "logs.detail.attempt.recovery.rateLimit429": "Ограничение частоты запросов (429)", + "logs.detail.attempt.recovery.anthropicOauth429": "Anthropic OAuth ограничен (429)", + "logs.detail.attempt.recovery.image413": "Слишком большой размер изображения (413)", + "logs.detail.attempt.recovery.unknown": "Неизвестная причина восстановления", "logs.detail.reason.usage_missing": "Данные об использовании не были сообщены.", "logs.detail.reason.usage_unsupported": "Этот провайдер не сообщает данные об использовании.", "logs.detail.reason.output_missing": "Положительное число выходных токенов не было сообщено.", diff --git a/gui/src/i18n/zh.ts b/gui/src/i18n/zh.ts index 20e4c23e4..cb0b6446c 100644 --- a/gui/src/i18n/zh.ts +++ b/gui/src/i18n/zh.ts @@ -1,5 +1,8 @@ import type { TKey } from "./en"; +/** + * Chinese i18n catalog; must match the `TKey` set (compile-checked). + */ export const zh: Record = { // sidebar / nav / common "nav.dashboard": "仪表盘", @@ -561,6 +564,14 @@ export const zh: Record = { "logs.detail.attempt.reason": "结果 / 原因", "logs.detail.attempt.completed": "已完成", "logs.detail.attempt.e2eNote": "顶层 tok/s 为端到端值；每次尝试使用各自耗时。", + "logs.detail.attempt.recovery.transient5xx": "临时 5xx 错误", + "logs.detail.attempt.recovery.connectionReset": "连接已重置", + "logs.detail.attempt.recovery.oauth401": "OAuth 重新认证", + "logs.detail.attempt.recovery.key429": "密钥被限流 (429)", + "logs.detail.attempt.recovery.rateLimit429": "被限流 (429)", + "logs.detail.attempt.recovery.anthropicOauth429": "Anthropic OAuth 被限流 (429)", + "logs.detail.attempt.recovery.image413": "图片载荷过大 (413)", + "logs.detail.attempt.recovery.unknown": "未知的恢复原因", "logs.detail.reason.usage_missing": "未上报 usage。", "logs.detail.reason.usage_unsupported": "该提供方不支持上报 usage。", "logs.detail.reason.output_missing": "未上报正数输出 token。", diff --git a/gui/src/pages/Logs.tsx b/gui/src/pages/Logs.tsx index 69b1316fd..ebe177532 100644 --- a/gui/src/pages/Logs.tsx +++ b/gui/src/pages/Logs.tsx @@ -76,11 +76,17 @@ interface LogDisplayMetrics { cost: CostResult; } +/** + * Recovery kinds recorded on a log attempt; rendered as localized labels in the logs + * detail dialog instead of raw wire values. + */ type AttemptRecoveryKind = | "transient-5xx" | "connection-reset" | "oauth-401" | "key-429" + | "rate-limit-429" + | "anthropic-oauth-429" | "image-413"; interface LogAttempt { @@ -286,6 +292,20 @@ const ESTIMATE_REASON_KEYS = { expected_price_overlay: "logs.detail.estimate.expected_price_overlay", } as const satisfies Record; +/** + * i18n keys for every {@link AttemptRecoveryKind}, so the logs detail dialog renders a + * localized label instead of the raw wire value (e.g. `rate-limit-429`). + */ +const RECOVERY_KIND_KEYS = { + "transient-5xx": "logs.detail.attempt.recovery.transient5xx", + "connection-reset": "logs.detail.attempt.recovery.connectionReset", + "oauth-401": "logs.detail.attempt.recovery.oauth401", + "key-429": "logs.detail.attempt.recovery.key429", + "rate-limit-429": "logs.detail.attempt.recovery.rateLimit429", + "anthropic-oauth-429": "logs.detail.attempt.recovery.anthropicOauth429", + "image-413": "logs.detail.attempt.recovery.image413", +} as const satisfies Record; + function metricReasonKey(reason: MetricUnavailableReason) { return METRIC_REASON_KEYS[reason]; } @@ -294,6 +314,15 @@ function estimateReasonKey(reason: CostEstimateReason) { return ESTIMATE_REASON_KEYS[reason]; } +/** + * Map one attempt recovery kind to its i18n key for the logs detail dialog. + */ +function recoveryKindKey(kind: AttemptRecoveryKind) { + // A stale/malformed cached row can carry a kind outside the known set; fall back to a + // localized label instead of handing `t()` an undefined key. + return RECOVERY_KIND_KEYS[kind] ?? "logs.detail.attempt.recovery.unknown"; +} + function verificationKey(status: MatchedPriceInfo["status"]): "logs.detail.verification.verified" | "logs.detail.verification.derived" { return status === "verified" ? "logs.detail.verification.verified" : "logs.detail.verification.derived"; } @@ -960,7 +989,9 @@ function LogDetailDialog({ const attemptReasoningWire = reasoningWireLabel(attempt); const matched = attemptCost?.kind === "value" ? attemptCost.estimate.price : undefined; const reason = attempt.errorCode - ?? (attempt.recoveryKinds.length ? attempt.recoveryKinds.join(", ") : undefined) + ?? (attempt.recoveryKinds.length + ? attempt.recoveryKinds.map(kind => t(recoveryKindKey(kind))).join(", ") + : undefined) ?? (attemptCost?.kind === "unavailable" ? t(metricReasonKey(attemptCost.reason)) : t("logs.detail.attempt.completed")); return ( diff --git a/src/config.ts b/src/config.ts index fe523323a..2a11d0d36 100644 --- a/src/config.ts +++ b/src/config.ts @@ -23,6 +23,7 @@ import { import { recordOwnedConfigPath } from "./lib/config-ownership"; import { assertNotRealHomeUnderTest } from "./lib/test-home-guard"; import { providerDestinationConfigError } from "./lib/destination-policy"; +import { redactSecretString } from "./lib/redact"; import { openRouterRoutingConfigError } from "./providers/openrouter-routing"; import { isWirePinnedModel, @@ -550,6 +551,27 @@ export function reconcileConfigWarningMemos(generation: number): number { return removed; } +/** + * Bounds for the opt-in same-target 429 wait-and-retry policy. Single source of truth + * shared by the config schema, the load-time sanitizer, and the management write + * boundary. Strict, so an unknown key is rejected at every validation boundary instead + * of being silently ignored (the load-time sanitizer still degrades unknown keys with a + * warning before schema validation, so hand-edited configs keep loading). + */ +const retryOn429PolicySchema = z.object({ + enabled: z.boolean().optional(), + attempts: z.number().int().min(1).max(20).optional(), + intervalMs: z.number().int().min(100).max(600_000).optional(), + // The effective cap for a single wait is MAX_COOLDOWN_MS (10 min) in key-failover.ts; + // larger configured values would be dead config. + maxIntervalMs: z.number().int().min(100).max(600_000).optional(), + respectRetryAfter: z.boolean().optional(), +}).strict(); + +/** + * Zod schema for one provider entry: known fields are validated strictly while unknown + * fields pass through (preserved for runtime extensions). + */ const providerConfigSchema = z.object({ adapter: z.string().min(1), baseUrl: z.string().min(1), @@ -562,6 +584,7 @@ const providerConfigSchema = z.object({ supportsServiceTier: z.boolean().optional(), preserveResponsesReasoningContent: z.boolean().optional(), allowPrivateNetwork: z.boolean().optional(), + retryOn429: retryOn429PolicySchema.optional(), codexAccountMode: z.enum(["pool", "direct"]).optional(), responsesItemIdRepair: z.object({ message: z.array(z.string().min(1)).optional(), @@ -1294,6 +1317,96 @@ function warnDegradedStreamMode(rawParsed: unknown, validated: OcxConfig): void } } +/** + * Load-time degradation for `retryOn429` (loadConfig only): one hand-edited invalid optional + * field (e.g. `attempts: 0` or a string) must not trip the whole provider schema and hide every + * provider/key behind a default config. Invalid fields are dropped with a warning; the management + * write boundary still rejects invalid policies explicitly. + */ +function sanitizeRetryOn429ForLoad(parsed: unknown): void { + if (!parsed || typeof parsed !== "object") return; + const root = parsed as Record; + const providers = root.providers; + if (!providers || typeof providers !== "object" || Array.isArray(providers)) return; + for (const [name, provider] of Object.entries(providers as Record)) { + // This sanitizer runs BEFORE schema validation, so the provider name is untrusted: redact + // secret-shaped names and JSON-escape control characters before it reaches any warning. + const safeProviderName = JSON.stringify(redactSecretString(name)); + if (!provider || typeof provider !== "object" || Array.isArray(provider)) continue; + const p = provider as Record; + const policy = p.retryOn429; + if (policy === undefined) continue; + if (!policy || typeof policy !== "object" || Array.isArray(policy)) { + delete p.retryOn429; + // Never serialize the value: an accidental `retryOn429: "sk-..."` would leak the secret. + console.warn(`⚠️ config.json providers.${safeProviderName}.retryOn429 (${typeof policy}) is invalid — ignoring the policy`); + continue; + } + const policyRecord = policy as Record; + // An explicitly present but invalid master switch must not silently default to ENABLED: + // drop the whole policy so a hand-edit that tried to disable retries stays disabled. + if ("enabled" in policyRecord && typeof policyRecord.enabled !== "boolean") { + delete p.retryOn429; + console.warn(`⚠️ config.json providers.${safeProviderName}.retryOn429.enabled (${typeof policyRecord.enabled}) is invalid — ignoring the whole policy`); + continue; + } + // Field checks derive from the shared policy schema so the bounds cannot drift + // between the load-time sanitizer, the config schema, and the write boundary. + const policyShape = retryOn429PolicySchema.shape; + const hadPolicyEntries = Object.keys(policyRecord).length > 0; + const cleaned: Record = {}; + for (const [key, fieldSchema] of Object.entries(policyShape)) { + const value = policyRecord[key]; + if (value === undefined) continue; + if (fieldSchema.safeParse(value).success) cleaned[key] = value; + // Log only the received type, never the value (provider config can hold secrets). + else console.warn(`⚠️ config.json providers.${safeProviderName}.retryOn429.${key} (${typeof value}) is invalid — ignoring the field`); + } + const knownKeys = new Set(Object.keys(policyShape)); + for (const key of Object.keys(policyRecord)) { + if (!knownKeys.has(key)) { + // Redact the field NAME before logging: a malformed hand-edit can place a secret in a + // property name (`retryOn429: { "sk-...": true }`). Ordinary typos (e.g. `attempt`) + // stay readable, secret-shaped names become [REDACTED]. JSON-escape afterwards so a + // control-character property name (newline/ANSI) can never forge a log line. + console.warn(`⚠️ config.json providers.${safeProviderName}.retryOn429.${JSON.stringify(redactSecretString(key))} is not a recognized field — ignoring it`); + } + } + if (hadPolicyEntries && Object.keys(cleaned).length === 0) { + // Every supplied field was invalid: drop the whole policy. Persisting `{}` here would + // opt IN to retries with defaults, which is the opposite of what a malformed + // disable-oriented edit (`retryOn429: { enabled: "false" }`, `attempts: 0`) asked for. + delete p.retryOn429; + console.warn(`⚠️ config.json providers.${safeProviderName}.retryOn429 has no valid fields left — removing the policy (an empty policy would enable retries with defaults)`); + } else { + // Preserve an intentionally empty `retryOn429: {}` (presence = opt-in with defaults). + p.retryOn429 = cleaned; + } + } +} + +/** + * Management write-boundary validation for `retryOn429` (fail closed). Unlike the + * lenient load-time sanitizer, invalid values and unknown keys are rejected outright so + * a POST/PATCH cannot persist a policy the proxy would then silently degrade. Reuses the + * shared policy schema. Never echoes values, and secret-shaped unknown field names are + * redacted (a malformed write can place a secret in a property name). + */ +export function retryOn429PolicyConfigError(policy: unknown): string | null { + if (policy === undefined) return null; + const result = retryOn429PolicySchema.safeParse(policy); + if (result.success) return null; + const first = result.error.issues[0]; + if (!first) return "retryOn429 is invalid"; + if (first.code === "unrecognized_keys") { + const names = first.keys.map(key => JSON.stringify(redactSecretString(key))).join(", "); + return `retryOn429 has unrecognized field${first.keys.length > 1 ? "s" : ""}: ${names}`; + } + if (first.path.length === 0) return `retryOn429 is invalid (${first.message})`; + const field = String(first.path[first.path.length - 1]); + return `retryOn429.${field} is invalid (${first.message})`; +} + /** * Companion to {@link warnDegradedStreamMode} for a blank persisted `hostname`. The bind * falls back to loopback, which is the safe direction but not what the file asked for — @@ -1505,6 +1618,7 @@ export function loadConfig(): OcxConfig { try { const raw = readFileSync(configPath, "utf-8").replace(/^\uFEFF/, ""); const parsed = JSON.parse(raw); + sanitizeRetryOn429ForLoad(parsed); const result = configSchema.safeParse(parsed); if (result.success) { const config = normalizeApiKeyIds(result.data as OcxConfig); @@ -1672,6 +1786,10 @@ export function validateConfigCandidate(value: unknown): { ok: true; config: Ocx function configDiagnosticsFromRaw(raw: string): ConfigDiagnostics { try { const parsed = JSON.parse(raw.replace(/^\uFEFF/, "")); + // Same degradation as loadConfig: a hand-edited invalid retryOn429 must not trip the + // schema and send the caller a default-config fallback (the config command could then + // persist that fallback over the user's providers/keys). + sanitizeRetryOn429ForLoad(parsed); const result = configSchema.safeParse(parsed); if (result.success) { return validFileConfigDiagnostics(normalizeApiKeyIds(result.data as OcxConfig), parsed); diff --git a/src/images/loop.ts b/src/images/loop.ts index 535f295df..2caa34996 100644 --- a/src/images/loop.ts +++ b/src/images/loop.ts @@ -14,12 +14,14 @@ import type { AdapterRequest, IncomingMeta, ProviderAdapter } from "../adapters/ import { existsSync } from "node:fs"; import { pathToFileURL } from "node:url"; import { createAdapterEventQueue } from "../adapters/run-turn-queue"; -import type { AdapterEvent, OcxMessage, OcxParsedRequest, OcxProviderContinuationState, OcxRequestOptions, OcxThinkingContent, OcxUsage } from "../types"; +import type { AdapterEvent, OcxMessage, OcxParsedRequest, OcxProviderContinuationState, OcxRequestOptions, OcxThinkingContent, OcxUsage, RateLimitRetryPolicy } from "../types"; import { namespacedToolName } from "../types"; +import type { AttemptRecoveryKind } from "../usage/log"; import { bridgeToResponsesSSE } from "../bridge"; import { clearableDeadline, idleDeadline } from "../lib/abort"; import { readBoundedResponseBody } from "../lib/bounded-body"; -import { fetchWithResetRetry } from "../lib/upstream-retry"; +import { fetchWithResetRetry, prepareSameTarget429Wait } from "../lib/upstream-retry"; +import { rateLimitRetryDelayMs } from "../providers/key-failover"; import { isTranslatorBudgetExceededError, TRANSLATOR_MAX_TURN_BYTES, @@ -202,6 +204,10 @@ class LoopError extends Error { } } +/** + * Dependencies for one image-bridge iteration: parsed request, active adapter, incoming + * metadata, and the optional image/video bridge plans. + */ export interface ImageBridgeDeps { parsed: OcxParsedRequest; adapter: ProviderAdapter; @@ -212,8 +218,8 @@ export interface ImageBridgeDeps { videoTimeoutMs?: number; /** Headers forwarded from the original request (e.g. Codex auth). Cloned per iteration. */ forwardHeaders?: Headers; - /** Called before each routed-model dispatch in the bridge loop, for attempt telemetry. */ - onAttemptSend?: () => void; + /** Called before each routed-model dispatch in the bridge loop, for attempt telemetry. Same-target 429 replays pass the `rate-limit-429` recovery kind. */ + onAttemptSend?: (recovery?: AttemptRecoveryKind) => void; /** Called after each upstream request is built (parity with web-search / normal path). */ onRequestBuilt?: (request: AdapterRequest) => void; abortSignal?: AbortSignal; @@ -233,6 +239,8 @@ export interface ImageBridgeDeps { * rotated key, or null when the pool is exhausted. */ on429?: (retryAfterHeader: string | null) => ProviderAdapter | null; + /** Opt-in same-target 429 policy (key-auth providers). When present, 429 replays on the SAME key before on429 rotation. */ + retryOn429Policy?: Required | null; /** Called when the bridged Responses stream completes (parity with runTurn / routed paths). */ onCompletedResponse?: (response: Record, providerState?: OcxProviderContinuationState) => void; /** WebSocket Responses path only — leave response id empty for protocol compatibility. */ @@ -331,9 +339,20 @@ export async function runWithImageBridge(deps: ImageBridgeDeps): Promise; + // Same-target 429 budget is per REQUEST, not per model iteration: later image rounds inherit + // what earlier rounds left of `attempts`, so a bounded multi-round turn can never exceed the + // configured replay count in total (a per-round reset would multiply it by maxRounds). + const rateLimitRetryPolicy = deps.retryOn429Policy ?? null; + let rateLimitRetries = 0; + // Acquire one iteration's final response headers. The first call is drained eagerly so an initial // connect/header/HTTP failure stays a non-2xx JSON response — except for runTurn adapters, which // have no HTTP status surface and must not block SSE headers behind queue.collect(). + /** + * Fetch one image-bridge iteration's final response headers, applying the response-header + * deadline and the same-target 429 retry policy (with awaited body release and deadline + * restart) before the `on429` key rotation. + */ const prepareIterationEvents = async function* (forceFinal: boolean): AsyncGenerator { const iterParsed: OcxParsedRequest = { ...parsed, @@ -422,27 +441,51 @@ export async function runWithImageBridge(deps: ImageBridgeDeps): Promise => { - const request = await requestAdapter.buildRequest(iterParsed, { - headers: deps.forwardHeaders ? new Headers(deps.forwardHeaders) : new Headers(), - abortSignal: headerDeadline.signal, - translatorBudget, - }); - try { deps.onRequestBuilt?.(request); } catch { /* diagnostics are best-effort */ } - deps.onAttemptSend?.(); + /** + * Build and fetch one image-bridge iteration on the given adapter, under the iteration + * header deadline. The caller owns same-target 429 replays and key rotation around it. + * The outbound request is cached per adapter so a same-target replay reuses the EXACT + * URL, serialized body, and headers (builder runs once per target sequence). + */ + let cachedRequest: AdapterRequest | undefined; + let cachedAdapter: ProviderAdapter | undefined; + /** + * Build and fetch one image-bridge iteration on the given adapter, under the iteration + * header deadline. The caller owns same-target 429 replays and key rotation around it. + */ + const fetchOnce = async (requestAdapter: ProviderAdapter, recovery?: AttemptRecoveryKind): Promise => { + let request: AdapterRequest; + if (cachedRequest !== undefined && cachedAdapter === requestAdapter) { + request = cachedRequest; + } else { + request = await requestAdapter.buildRequest(iterParsed, { + headers: deps.forwardHeaders ? new Headers(deps.forwardHeaders) : new Headers(), + abortSignal: headerDeadline.signal, + translatorBudget, + }); + try { deps.onRequestBuilt?.(request); } catch { /* diagnostics are best-effort */ } + cachedRequest = request; + cachedAdapter = requestAdapter; + } let response: Response; try { - response = requestAdapter.fetchResponse - ? await requestAdapter.fetchResponse(request, { + if (requestAdapter.fetchResponse) { + deps.onAttemptSend?.(recovery); + response = await requestAdapter.fetchResponse(request, { abortSignal: headerDeadline.signal, timeoutMs: connectTimeoutMs, returnRawErrors: true, stream: true, - }) - : await fetchWithResetRetry( - () => { + }); + } else { + response = await fetchWithResetRetry( + (retryRecovery) => { + // Record every helper-driven send (the callback runs for the first attempt and + // each connection-reset replay); preserve the caller's recovery kind + // (rate-limit-429 / key-429) when the retry layer supplies none. + deps.onAttemptSend?.(retryRecovery ?? recovery); const h = new Headers(request.headers); if (!h.has("accept-encoding")) h.set("accept-encoding", "identity"); return fetchImpl(request.url, { @@ -453,7 +496,8 @@ export async function runWithImageBridge(deps: ImageBridgeDeps): Promise {}); } catch { /* already closed */ } adapter = rotated; yield { type: "heartbeat" }; - prepared = await fetchOnce(adapter); + prepared = await fetchOnce(adapter, "key-429"); } // Final headers have arrived. Clear only the deadline timer before ANY body read. diff --git a/src/lib/upstream-retry.ts b/src/lib/upstream-retry.ts index f767f045e..82023549c 100644 --- a/src/lib/upstream-retry.ts +++ b/src/lib/upstream-retry.ts @@ -71,6 +71,107 @@ export async function sleepWithAbort(ms: number, signal?: AbortSignal): Promise< }); } +/** + * Best-effort, bounded cancellation of a response body before a retry backoff. + * + * The 429 paths release the unread body before waiting so sockets do not accumulate under a + * rate-limit storm, but a never-settling `cancel()` promise must not be able to block the + * abort-aware backoff (client cancel, `maxIntervalMs`, or the cumulative header deadline). + * Cancellation is started and its rejection observed; the await is bounded by `timeoutMs` + * and the abort signal. This mirrors the rotation-path guarantee (release is initiated, not + * awaited forever) while preserving the resource-release intent of the same-target paths. + */ +export async function releaseResponseBodyBestEffort( + body: ReadableStream | null, + signal: AbortSignal | undefined, + timeoutMs = 1_000, +): Promise { + if (!body) return; + if (signal?.aborted) { + void body.cancel().catch(() => {}); + return; + } + const cancel = body.cancel().catch(() => {}); + if (!signal) { + await Promise.race([cancel, new Promise(resolve => setTimeout(resolve, timeoutMs))]); + return; + } + await new Promise(resolve => { + let timer: ReturnType; + /** + * Abort hook: clear the bounded-body release timer and settle the promise so a + * never-settling cancel() can never block the abort-aware backoff. + */ + const onAbort = () => { + clearTimeout(timer); + resolve(); + }; + timer = setTimeout(() => { + signal.removeEventListener("abort", onAbort); + resolve(); + }, timeoutMs); + signal.addEventListener("abort", onAbort, { once: true }); + void cancel.then(() => { + clearTimeout(timer); + signal.removeEventListener("abort", onAbort); + resolve(); + }); + }); +} + +/** + * Abort-aware sleep that yields an adapter `heartbeat` at least every `heartbeatIntervalMs`. + * The Responses bridge treats a returned iterator event as upstream liveness and aborts turns + * that stay silent past the stall budget (default 300s), while a retryOn429 wait may legally + * reach 600s — so deliberate waits must keep the watchdog fed or a long backoff is killed + * mid-turn. The final chunk always yields once, which doubles as the post-wait liveness beat. + */ +export async function* sleepWithHeartbeats( + ms: number, + signal?: AbortSignal, + heartbeatIntervalMs = 10_000, +): AsyncGenerator<{ type: "heartbeat" }> { + if (ms <= 0) return; + // Guard against a non-positive interval: a zero/negative step would spin the loop forever + // while sleepWithAbort early-returns without ever observing the abort signal. NaN must be + // normalized too: Math.max(1, NaN) is NaN, which would abort the wait after one beat. + const stepMs = Number.isNaN(heartbeatIntervalMs) ? 1 : Math.max(1, heartbeatIntervalMs); + let remaining = ms; + while (remaining > 0) { + const chunk = Math.min(remaining, stepMs); + await sleepWithAbort(chunk, signal); + remaining -= chunk; + yield { type: "heartbeat" }; + } +} + +export interface SameTarget429WaitOptions { + body: ReadableStream | null; + signal?: AbortSignal; + delayMs: number; + /** + * When set, the wait yields adapter heartbeats so bridge stall watchdogs stay fed. + * Omit for pre-stream recovery paths that have no stall watchdog. + */ + heartbeatIntervalMs?: number; +} + +/** + * Shared pre-replay prep for opt-in same-target 429 waits: + * release the unread 429 body, then sleep (optionally with heartbeats). + * Callers still own attempt budgeting, abort re-checks, and the replay itself. + */ +export async function* prepareSameTarget429Wait( + options: SameTarget429WaitOptions, +): AsyncGenerator<{ type: "heartbeat" }> { + await releaseResponseBodyBestEffort(options.body, options.signal); + if (options.heartbeatIntervalMs === undefined) { + await sleepWithAbort(options.delayMs, options.signal); + return; + } + yield* sleepWithHeartbeats(options.delayMs, options.signal, options.heartbeatIntervalMs); +} + export function isConnectionResetError(err: unknown): boolean { if (!(err instanceof Error)) return false; // Aborts and timeouts are caller decisions / honest failures — never retryable. diff --git a/src/providers/derive.ts b/src/providers/derive.ts index ba4fc6244..1cfbbd5dc 100644 --- a/src/providers/derive.ts +++ b/src/providers/derive.ts @@ -104,13 +104,20 @@ function cloneNestedRecord(input: Record>): Recor return Object.fromEntries(Object.entries(input).map(([key, value]) => [key, { ...value }])); } +/** + * Build the provider config a registry entry contributes when a preset is materialized. + * The registry auth kind is preserved verbatim (including `"local"`) so fail-closed gates + * keep distinguishing local runtimes from API-key providers after the seed round-trip. + */ export function providerConfigSeed(entry: ProviderRegistryEntry): OcxProviderConfig { return { adapter: entry.adapter, baseUrl: entry.baseUrl, ...(entry.apiKeyTransport !== undefined ? { apiKeyTransport: entry.apiKeyTransport } : {}), ...(entry.responsesPath ? { responsesPath: entry.responsesPath } : {}), - authMode: entry.authKind === "local" ? undefined : entry.authKind, + // Preserve the registry auth kind verbatim (including "local") so fail-closed gates that + // distinguish local runtimes from API-key providers keep working after the seed round-trip. + authMode: entry.authKind, ...(entry.codexAccountMode ? { codexAccountMode: entry.codexAccountMode } : {}), ...(entry.keyOptional !== undefined ? { keyOptional: entry.keyOptional } : {}), ...(entry.freeTier !== undefined ? { freeTier: entry.freeTier } : {}), diff --git a/src/providers/key-failover.ts b/src/providers/key-failover.ts index 32a308514..7a2d2d330 100644 --- a/src/providers/key-failover.ts +++ b/src/providers/key-failover.ts @@ -9,7 +9,7 @@ * Modelled after src/codex/routing.ts cooldown logic but scoped to plain API-key pools. */ import { saveConfigPreservingClaudeCode } from "../config"; -import type { OcxConfig, OcxProviderConfig } from "../types"; +import type { OcxConfig, OcxProviderConfig, RateLimitRetryPolicy } from "../types"; import { resolveProviderTransport, type OcxProviderTransport } from "./xai-transport"; import { sweepExpiredOnWrite } from "../lib/state-store-sweeper"; @@ -22,6 +22,18 @@ interface KeyCooldown { const DEFAULT_COOLDOWN_MS = 60_000; const MAX_COOLDOWN_MS = 10 * 60_000; // cap at 10 min for api-key rotation +/** + * Default same-target 429 retry policy used when a provider opts in via a bare + * `retryOn429: {}` (presence = opt-in with these defaults). + */ +const DEFAULT_RATE_LIMIT_RETRY = { + enabled: true, + attempts: 3, + intervalMs: 5_000, + maxIntervalMs: 60_000, + respectRetryAfter: true, +} as const satisfies Required; + /** Map<`${providerName}\0${keyId}`, KeyCooldown> */ const keyCooldowns = new Map(); @@ -29,21 +41,32 @@ function cooldownKey(providerName: string, keyId: string): string { return `${providerName}\0${keyId}`; } +/** + * Parse an upstream `Retry-After` header: numeric seconds (including `0`) or an HTTP-date. + * Returns a bounded delay in ms (1..MAX_COOLDOWN_MS), or undefined when the value is + * malformed. An HTTP-date already in the past yields an immediate (1 ms) retry. + */ function parseRetryAfterMs(value: string | null | undefined, now = Date.now()): number | undefined { const text = value?.trim(); if (!text) return undefined; if (/^\d+(?:\.\d+)?$/.test(text)) { const seconds = Number(text); - if (Number.isFinite(seconds) && seconds > 0) { + if (Number.isFinite(seconds) && seconds >= 0) { return Math.min(Math.max(Math.ceil(seconds * 1000), 1), MAX_COOLDOWN_MS); } } const timestamp = Date.parse(text); if (!Number.isFinite(timestamp)) return undefined; const delay = timestamp - now; - return delay > 0 ? Math.min(delay, MAX_COOLDOWN_MS) : undefined; + // A valid HTTP-date whose retry time has already passed is an immediate retry, exactly like + // numeric `Retry-After: 0` — never a malformed-header fallback to the fixed interval. + return Math.min(Math.max(delay, 1), MAX_COOLDOWN_MS); } +/** + * True while the given key is inside its 429 cooldown window (lazily evicting the entry once the + * window expires). Used to skip keys that the upstream just rate-limited during failover. + */ function isKeyInCooldown(providerName: string, keyId: string, now = Date.now()): boolean { const entry = keyCooldowns.get(cooldownKey(providerName, keyId)); if (!entry) return false; @@ -65,6 +88,51 @@ export function hasKeyPoolFailover(provider: OcxProviderConfig): boolean { return (provider.apiKeyPool?.length ?? 0) >= 2; } +/** + * Normalize a provider's `retryOn429` policy, or return null when the knob is absent, + * explicitly disabled, or the provider is not key-auth (OAuth/forward credentials must not be + * replayed on the same token, forward passthrough never reaches the recovery loop anyway, and + * local runtimes have no remote key to preserve). The returned policy is fully defaulted so + * callers never re-check fields. + */ +export function rateLimitRetryPolicyFor( + provider: Pick, +): Required | null { + const policy = provider.retryOn429; + if (!policy || policy.enabled === false) return null; + // Fail closed: only explicit key auth or the documented omitted-default (undefined == key for + // custom API-key providers) may use same-key replays. OAuth/forward are never replayed on the + // same token, local runtimes have no remote key to preserve, and unknown/custom values are + // rejected rather than guessed at. + if (provider.authMode !== undefined && provider.authMode !== "key") return null; + return { + enabled: policy.enabled ?? DEFAULT_RATE_LIMIT_RETRY.enabled, + attempts: policy.attempts ?? DEFAULT_RATE_LIMIT_RETRY.attempts, + intervalMs: policy.intervalMs ?? DEFAULT_RATE_LIMIT_RETRY.intervalMs, + maxIntervalMs: policy.maxIntervalMs ?? DEFAULT_RATE_LIMIT_RETRY.maxIntervalMs, + respectRetryAfter: policy.respectRetryAfter ?? DEFAULT_RATE_LIMIT_RETRY.respectRetryAfter, + }; +} + +/** + * Wait before the next same-target replay: upstream Retry-After (seconds or HTTP-date) when + * `respectRetryAfter` is on and the header parses, capped at `maxIntervalMs`; otherwise the + * fixed `intervalMs`, also capped at `maxIntervalMs` (a single wait never exceeds the cap). + * Malformed headers fall back to the fixed interval. + */ +export function rateLimitRetryDelayMs( + policy: Required, + retryAfterHeader: string | null | undefined, + now = Date.now(), +): number { + const raw = retryAfterHeader?.trim(); + if (policy.respectRetryAfter && raw) { + const parsed = parseRetryAfterMs(raw, now); + if (parsed !== undefined) return Math.min(parsed, policy.maxIntervalMs); + } + return Math.min(policy.intervalMs, policy.maxIntervalMs); +} + /** * Record a 429 for the current key and attempt to switch to the next available one. * diff --git a/src/providers/xai-transport.ts b/src/providers/xai-transport.ts index 2c6060592..1e56506ec 100644 --- a/src/providers/xai-transport.ts +++ b/src/providers/xai-transport.ts @@ -53,7 +53,7 @@ function withoutUserOverridden( function withGeneratedRequestId( init: RequestInit | undefined, - configuredRequestId: string | undefined, + pinnedRequestId: string, stableHeaders: Readonly>, ): RequestInit { const headers = new Headers(init?.headers); @@ -63,7 +63,7 @@ function withGeneratedRequestId( if (!headers.has(XAI_GROK_COMPATIBILITY.headers.requestId)) { headers.set( XAI_GROK_COMPATIBILITY.headers.requestId, - configuredRequestId ?? randomUUID(), + pinnedRequestId, ); } return { ...init, headers }; @@ -83,7 +83,9 @@ export function deriveXaiConvId(promptCacheKey: string): string { /** * Resolve xAI's runtime transport without mutating persisted config. Conversation/session - * affinity is stable for this resolved transport; request identity is generated per fetch. + * affinity is stable for this resolved transport; request identity is pinned per resolved + * transport (= per logical request until key rotation), so same-target replays and transient + * retries carry the same id while a rotated key gets a fresh one. * Agent, deployment, model-override, turn, mode, and user identity headers are intentionally * omitted because opencodex has no truthful values for the official fields. */ @@ -128,9 +130,14 @@ export function resolveProviderTransport( provider.headers, XAI_GROK_COMPATIBILITY.headers.requestId, ); + // Pin the request id per resolved transport (= per logical request until key rotation): + // same-target 429 replays must carry the SAME x-grok-req-id as the original dispatch, and + // transient retries reuse one id so the upstream can dedupe them. A rotated key resolves a + // fresh transport, which gets its own id. + const requestId = configuredRequestId ?? randomUUID(); const baseFetch = provider.fetch ?? globalThis.fetch; const attemptFetch = ((input, init) => - baseFetch(input, withGeneratedRequestId(init, configuredRequestId, stableHeaders))) as typeof globalThis.fetch; + baseFetch(input, withGeneratedRequestId(init, requestId, stableHeaders))) as typeof globalThis.fetch; return { ...provider, diff --git a/src/server/auth-cors.ts b/src/server/auth-cors.ts index 79b02c329..2acea2848 100644 --- a/src/server/auth-cors.ts +++ b/src/server/auth-cors.ts @@ -11,8 +11,10 @@ import { providerBaseUrlConfigError, providerHeadersConfigError, reasoningSummaryDeliveryRecordConfigError, + retryOn429PolicyConfigError, } from "../config"; import { providerDestinationConfigError } from "../lib/destination-policy"; +import { redactSecretString } from "../lib/redact"; import { getProviderRegistryEntry, providerCodexAccountMode, providerMatchesRegistryTransport, registryEntryForProviderDestination } from "../providers/registry"; import { providerConfigSeed } from "../providers/derive"; import type { OcxConfig, OcxProviderConfig } from "../types"; @@ -389,6 +391,11 @@ function sameCanonicalProviderSeed(actual: Record, expected: Oc return actualKeys.every(key => JSON.stringify(actual[key]) === JSON.stringify((expected as unknown as Record)[key])); } +/** + * Validate a provider object arriving at the management write boundary. Returns an error + * string, or null when the provider may be persisted. Caller-controlled names/fields are + * redacted and JSON-escaped so secrets never reach the response. + */ export function providerManagementConfigError(name: unknown, provider: unknown): string | null { if (typeof name !== "string" || !provider || typeof provider !== "object" || Array.isArray(provider)) { return "provider must be a plain object"; @@ -420,6 +427,12 @@ export function providerManagementConfigError(name: unknown, provider: unknown): if (destinationError) return `provider ${name} ${destinationError}`; const headersError = providerHeadersConfigError(typed.headers); if (headersError) return `provider ${name} ${headersError}`; + const retryOn429Error = retryOn429PolicyConfigError(raw.retryOn429); + if (retryOn429Error) { + // The provider name is caller-controlled and can be token-shaped; redact and JSON-escape + // it before it reaches the management API response. + return `provider ${JSON.stringify(redactSecretString(name))} ${retryOn429Error}`; + } const apiKeyTransportError = apiKeyTransportConfigError(typed); if (apiKeyTransportError) return `provider ${name} ${apiKeyTransportError}`; const maxInputError = positiveIntegerRecordConfigError(raw.modelMaxInputTokens, "modelMaxInputTokens"); diff --git a/src/server/responses/core.ts b/src/server/responses/core.ts index 2271b04fd..8b7940013 100644 --- a/src/server/responses/core.ts +++ b/src/server/responses/core.ts @@ -85,7 +85,12 @@ import { recordCodexUpstreamOutcome, type CodexUpstreamOutcome, } from "../../codex/routing"; -import { fetchWithResetRetry, fetchWithTransientRetry, applyUpstreamRecoveryInit } from "../../lib/upstream-retry"; +import { + applyUpstreamRecoveryInit, + fetchWithResetRetry, + fetchWithTransientRetry, + prepareSameTarget429Wait, +} from "../../lib/upstream-retry"; import { ForwardAdmissionCredentialError, validateForwardAdmissionCredential } from "../auth-cors"; import { createTranslatorBudget, isTranslatorBudgetExceededError, type TranslatorBudget } from "../../lib/translator-budget"; import { listOpenAiForwardSidecarCandidates, resolveFirstUsableOpenAiSidecar, type ResolvedOpenAiForwardSidecar } from "../../providers/openai-sidecar"; @@ -96,7 +101,13 @@ import { isUsageDebugEnabled } from "../../usage/debug"; import { readJsonRequestBody, DecompressedBodyTooLargeError, UnsupportedContentEncodingError } from "../request-decompress"; import { resolveAdapter, resolveWireProtocolOverride } from "../adapter-resolve"; import { providerModelWebsocketUpstreamStreaming, type InboundWire } from "../../providers/registry"; -import { hasKeyPoolFailover, rotateProviderTransportOn429 } from "../../providers/key-failover"; +import type { AdapterRequest } from "../../adapters/base"; +import { + hasKeyPoolFailover, + rateLimitRetryDelayMs, + rateLimitRetryPolicyFor, + rotateProviderTransportOn429, +} from "../../providers/key-failover"; import { shouldAttemptImageTierRetry } from "../image-retry"; import { resolveProviderTransport } from "../../providers/xai-transport"; import type { WsData } from "../ws-bridge"; @@ -563,6 +574,10 @@ export interface HandleResponsesOptions { +/** + * Build the 499 JSON error the proxy returns when the client disconnects before the + * response completes (`client_cancelled`). + */ export function clientCancelledResponse(): Response { return formatErrorResponse(499, "client_cancelled", "Client cancelled request"); } @@ -1207,6 +1222,10 @@ export function applyServiceTierGate( options.serviceTier = undefined; } +/** + * Route one `/v1/responses` request through the adapter pipeline: recovery loop, passthrough + * wire, image/web-search bridges, and the terminal-guard continuation. + */ export async function handleResponses( req: Request, config: OcxConfig, @@ -1224,6 +1243,10 @@ export async function handleResponses( } } +/** + * Inner implementation of `handleResponses`; owns the pre-stream recovery loop and the + * per-request same-target 429 retry budgets. + */ async function handleResponsesInner( req: Request, config: OcxConfig, @@ -1708,6 +1731,59 @@ async function handleResponsesInner( request.releaseBodyObservation?.(); } + // Same-target 429 wait-and-retry (opt-in `retryOn429`) for key-auth providers on the + // passthrough wire. This branch returns before the recovery loop below, so Responses-shaped + // key-auth gateways (e.g. the built-in DeepSeek preset) would otherwise surface 429 + // immediately with no same-key replay. Pre-stream only — nothing has been relayed yet, so + // the replay is lossless (same invariant as the recovery loop). Forward/OAuth providers + // keep their pool logic below (rateLimitRetryPolicyFor returns null for them). + const rateLimitPolicy = rateLimitRetryPolicyFor(route.provider); + let rateLimitRetries = 0; + while ( + upstreamResponse.status === 429 + && rateLimitPolicy !== null + && rateLimitRetries < rateLimitPolicy.attempts + ) { + rateLimitRetries += 1; + // Release unread body + deliberate wait via the shared same-target helper. + const retryAfterHeader = upstreamResponse.headers.get("retry-after"); + try { + for await (const _ of prepareSameTarget429Wait({ + body: upstreamResponse.body, + signal: options.abortSignal, + delayMs: rateLimitRetryDelayMs(rateLimitPolicy, retryAfterHeader, Date.now()), + })) { + // pre-stream: no stall watchdog to feed + } + } catch { + upstream.abort(); + return clientCancelledResponse(); + } + // Client cancellation wins over any stale timer edge: re-check before dispatching the + // replay so the wire never starts work for a request the client already abandoned. + if (options.abortSignal?.aborted || upstream.signal.aborted) { + upstream.abort(); + return clientCancelledResponse(); + } + try { + upstreamResponse = await fetchWithTransientRetry( + recovery => { + // The first send of every replay is itself a rate-limit retry; inner transient-5xx + // recoveries keep their own label (recovery is provided for those). + noteAttemptSend(logCtx.activeAttempt, passthroughEstimate, recovery ?? "rate-limit-429"); + return fetchWithHeaderTimeout(request.url, applyUpstreamRecoveryInit({ + method: request.method, + headers: request.headers, + body: request.body, + }, recovery), upstream.signal, connectMs, parsed.stream, providerFetch(route.provider)); + }, + { abortSignal: upstream.signal, label: safeHostLabel(request.url) }, + ); + } catch (err) { + return transportFailureResponse(err); + } + } + if (usesCodexForwardPoolAuth(authCtx, route.provider) && !authCtx.fixedAccount) { let poolRetryOutcome: number | undefined; if (await shouldRetryCodexPoolAccountModel400( @@ -2115,7 +2191,8 @@ async function handleResponsesInner( ...(imgPlan ? { plan: imgPlan } : {}), ...(vidPlan ? { videoPlan: vidPlan } : {}), forwardHeaders: selectedForwardHeaders, - onAttemptSend: () => noteAttemptSend(logCtx.activeAttempt, logCtx.usageLogInputTokens), + onAttemptSend: (recovery?: AttemptRecoveryKind) => + noteAttemptSend(logCtx.activeAttempt, logCtx.usageLogInputTokens, recovery), abortSignal: options.abortSignal, maxRounds: imgPlan && vidPlan ? clampImageMaxRounds(Math.min(config.images?.maxRounds ?? 3, config.images?.videoMaxRounds ?? 2)) @@ -2154,6 +2231,7 @@ async function handleResponsesInner( config.cacheRetention, ); }, + retryOn429Policy: rateLimitRetryPolicyFor(route.provider), ...(options.onFirstOutput ? { onFirstOutput: options.onFirstOutput } : {}), ...(options.forceEmptyResponseId ? { forceEmptyResponseId: true } : {}), onCompletedResponse: (response, providerState) => @@ -2182,7 +2260,6 @@ async function handleResponsesInner( // through web-search instead of being swallowed. runTurn adapters never enter this branch. if (canRunWebSearch && wsPlan) { parsed.context.tools = [...(parsed.context.tools ?? []), buildWebSearchTool()]; - noteAttemptSend(logCtx.activeAttempt, logCtx.usageLogInputTokens); const wsResponse = await runWithWebSearch({ parsed, adapter, incomingMeta: { headers: selectedForwardHeaders, abortSignal: options.abortSignal, translatorBudget }, @@ -2197,6 +2274,8 @@ async function handleResponsesInner( abortSignal: options.abortSignal, ...(options.onFirstOutput ? { onFirstOutput: options.onFirstOutput } : {}), onRequestBuilt: request => recordAdapterReasoning(logCtx, request), + onAttemptSend: (recovery?: AttemptRecoveryKind) => + noteAttemptSend(logCtx.activeAttempt, logCtx.usageLogInputTokens, recovery), onUsage: usage => { logCtx.usageFromBridge = true; if (usage) { @@ -2222,6 +2301,7 @@ async function handleResponsesInner( config.cacheRetention, ); }, + retryOn429Policy: rateLimitRetryPolicyFor(route.provider), }); // Register the sidecar stream as an active turn so drainAndShutdown waits for (or aborts) // in-flight web-search turns instead of skipping them during graceful shutdown. @@ -2360,19 +2440,58 @@ async function handleResponsesInner( const upstream = new AbortController(); const cleanupUpstreamAbort = linkAbortSignal(upstream, options.abortSignal); const connectMs = config.connectTimeoutMs ?? 200_000; + // Bridge stall budget (seconds of silence before upstream_stall_timeout); the retry backoff + // heartbeat interval is derived from it so the watchdog is always fed during deliberate waits. + const stallTimeoutMs = typeof config.stallTimeoutSec === "number" && Number.isFinite(config.stallTimeoutSec) && config.stallTimeoutSec > 0 + ? Math.floor(config.stallTimeoutSec * 1000) + : 300_000; let activeAdapter = adapter; - const request = await activeAdapter.buildRequest(parsed, { headers: selectedForwardHeaders, translatorBudget }); - recordAdapterReasoning(logCtx, request); - const inputTokenEstimate = typeof request.usageLog?.inputTokens === "number" - ? request.usageLog.inputTokens - : undefined; - if (inputTokenEstimate !== undefined) logCtx.usageLogInputTokens = inputTokenEstimate; + // One immutable, body-safe outbound request per same-target sequence (URL, serialized body, + // auth headers, generated compat headers). Same-target 429 replays reuse it verbatim; the + // builder runs again only after a key/account/adapter rotation, an oauth refresh, or an + // image-tier bias change (transportToken bump). `body` is always a serialized string, so + // reuse is safe, and releaseBodyObservation is idempotent per build. + let initialRequest: AdapterRequest | undefined; + let inputTokenEstimate: number | undefined; + try { + initialRequest = await activeAdapter.buildRequest(parsed, { headers: selectedForwardHeaders, translatorBudget }); + recordAdapterReasoning(logCtx, initialRequest); + inputTokenEstimate = typeof initialRequest.usageLog?.inputTokens === "number" + ? initialRequest.usageLog.inputTokens + : undefined; + if (inputTokenEstimate !== undefined) logCtx.usageLogInputTokens = inputTokenEstimate; + } catch (err) { + // A throwing buildRequest never returned a request; if a post-build step threw, release + // the serialized-body observation (idempotent) so the translator budget is not leaked. + // The build runs after linkAbortSignal, so a failure must also tear the link down and + // abort the upstream controller instead of escaping handleResponses unmapped. + initialRequest?.releaseBodyObservation?.(); + cleanupUpstreamAbort(); + upstream.abort(); + if (options.abortSignal?.aborted) return clientCancelledResponse(); + const msg = err instanceof Error ? err.message : String(err); + return formatErrorResponse(400, "invalid_request_error", redactSecretString(msg)); + } + // The catch path above always returns, so the request is definitely assigned here. + // Capture it in a const so the fetch callbacks read a narrowed, immutable value + // (TypeScript drops narrowing for a `let` captured by a nested function). + const builtInitialRequest = initialRequest; + let sameTargetRequest: AdapterRequest | undefined = builtInitialRequest; + let sameTargetParsed: OcxParsedRequest | undefined = parsed; + let sameTargetToken = 0; + let transportToken = 0; + /** + * Invalidate the same-target request cache. Every credential/adapter/parsed mutation MUST + * go through here: the cache keys on `parsed` REFERENCE identity, so an in-place mutation + * is invisible to it and a missed bump would replay a request built with a stale key. + */ + const invalidateSameTargetRequest = (): void => { transportToken += 1; }; let upstreamResponse: Response; try { if (activeAdapter.fetchResponse) { noteAttemptSend(logCtx.activeAttempt, inputTokenEstimate); - upstreamResponse = await activeAdapter.fetchResponse(request, { + upstreamResponse = await activeAdapter.fetchResponse(builtInitialRequest, { abortSignal: upstream.signal, timeoutMs: connectMs, stream: parsed.stream, @@ -2381,13 +2500,13 @@ async function handleResponsesInner( upstreamResponse = await fetchWithResetRetry( recovery => { noteAttemptSend(logCtx.activeAttempt, inputTokenEstimate, recovery); - return fetchWithHeaderTimeout(request.url, applyUpstreamRecoveryInit({ - method: request.method, - headers: request.headers, - body: request.body, + return fetchWithHeaderTimeout(builtInitialRequest.url, applyUpstreamRecoveryInit({ + method: builtInitialRequest.method, + headers: builtInitialRequest.headers, + body: builtInitialRequest.body, }, recovery), upstream.signal, connectMs, parsed.stream, providerFetch(route.provider)); }, - { abortSignal: upstream.signal, label: safeHostLabel(request.url) }, + { abortSignal: upstream.signal, label: safeHostLabel(builtInitialRequest.url) }, ); } } catch (err) { @@ -2397,27 +2516,60 @@ async function handleResponsesInner( const msg = describeUpstreamConnectFailure(err, connectMs); return formatErrorResponse(502, "upstream_error", msg); } finally { - request.releaseBodyObservation?.(); + builtInitialRequest.releaseBodyObservation?.(); } + // Same-target 429 retry budget is per REQUEST: it lives OUTSIDE the recovery loop (so a 413/401 + // replay that comes back 429 cannot silently re-arm a fresh budget) and is SHARED with the + // terminal-guard continuation below, so the main loop + one continuation can never exceed + // `attempts` same-key replays in total (bounded per request). + const rateLimitPolicy = rateLimitRetryPolicyFor(route.provider); + let rateLimitRetries = 0; + // Shared with the terminal-guard continuation below: an image-tier reduction that let the + // main request clear a 413 must not be forgotten on the very next continuation build. + let imageTierBias = 0; if (!upstreamResponse.ok) { // Recovery loop: multi-key 429 failover + at most ONE anthropic 413 tightened retry // (devlog/260714_image_normalization_pipeline/030). One mutable activeAdapter serves // both paths so a 429→413 sequence never rebuilds against a stale pre-rotation // adapter, and imageTierBias — once armed — rides EVERY subsequent rebuild so a // 413→429 rotation cannot silently undo the tightening. - let imageTierBias = 0; let imageRetryAttempted = false; let oauth401ReplayAttempted = false; + /** + * Rebuild the request from the current parsed input (and any image-tier bias) and refetch + * it once, tagging the attempt with the given recovery kind. Rebuilds are deterministic + * for the same parsed request, so same-target replays stay byte-identical. + */ const rebuildAndRefetch = async ( recovery: AttemptRecoveryKind, ): Promise => { - const retryRequest = await activeAdapter.buildRequest(parsed, { - headers: selectedForwardHeaders, - translatorBudget, - ...(imageTierBias > 0 ? { imageTierBias } : {}), - }); - recordAdapterReasoning(logCtx, retryRequest); + let retryRequest: AdapterRequest; + if (sameTargetRequest !== undefined && sameTargetParsed === parsed && sameTargetToken === transportToken) { + // Same target (key/adapter/parsed/tier unchanged): replay the exact cached request. + retryRequest = sameTargetRequest; + } else { + try { + retryRequest = await activeAdapter.buildRequest(parsed, { + headers: selectedForwardHeaders, + translatorBudget, + ...(imageTierBias > 0 ? { imageTierBias } : {}), + }); + recordAdapterReasoning(logCtx, retryRequest); + } catch (err) { + // A rotated/rebuilt adapter build failure is a request-shaping error, not an + // upstream connect failure: tear the abort link down and map it as 400 (no 413 + // translator-budget mapping here — that stays with parseRequest/buildToolBridgeMaps). + cleanupUpstreamAbort(); + upstream.abort(); + if (options.abortSignal?.aborted) return { failed: clientCancelledResponse() }; + const msg = err instanceof Error ? err.message : String(err); + return { failed: formatErrorResponse(400, "invalid_request_error", redactSecretString(msg)) }; + } + sameTargetRequest = retryRequest; + sameTargetParsed = parsed; + sameTargetToken = transportToken; + } const retryEstimate = typeof retryRequest.usageLog?.inputTokens === "number" ? retryRequest.usageLog.inputTokens : undefined; @@ -2472,6 +2624,7 @@ async function handleResponsesInner( route.providerName === "github-copilot" ? getOAuthCredentialApiBaseUrl(route.providerName) : undefined, ); route.provider = refreshedProvider; + invalidateSameTargetRequest(); activeAdapter = resolveAdapter( resolveWireProtocolOverride(route.providerName, route.modelId, refreshedProvider, inboundWire), config.cacheRetention, @@ -2482,6 +2635,45 @@ async function handleResponsesInner( continue recovery; } + // Same-target 429 wait-and-retry (opt-in `retryOn429`, issue #487). Codex never retries + // 429 itself (it retries 5xx only), and single-key pools cannot use the failover below, + // so wait (Retry-After or the fixed interval) and replay the IDENTICAL request on the + // same key first. Pre-stream only: a 429 arrives before any bytes are relayed, so the + // replay is lossless. Runs before key failover so "primary-first" setups keep the same + // key on rate-limit blips; only after the attempts are exhausted does failover run. + while ( + upstreamResponse.status === 429 + && rateLimitPolicy !== null + && rateLimitRetries < rateLimitPolicy.attempts + ) { + rateLimitRetries += 1; + // Release unread body + deliberate wait via the shared same-target helper. + const retryAfterHeader = upstreamResponse.headers.get("retry-after"); + try { + for await (const _ of prepareSameTarget429Wait({ + body: upstreamResponse.body, + signal: options.abortSignal, + delayMs: rateLimitRetryDelayMs(rateLimitPolicy, retryAfterHeader, Date.now()), + })) { + // pre-stream: no stall watchdog to feed + } + } catch { + cleanupUpstreamAbort(); + upstream.abort(); + return clientCancelledResponse(); + } + // Client cancellation wins over any stale timer edge: re-check before dispatching the + // replay so an adapter never starts work for a request the client already abandoned. + if (options.abortSignal?.aborted || upstream.signal.aborted) { + cleanupUpstreamAbort(); + upstream.abort(); + return clientCancelledResponse(); + } + const result = await rebuildAndRefetch("rate-limit-429"); + if ("failed" in result) return result.failed; + upstreamResponse = result; + } + // Multi-key 429 failover: rotate to the next pool key (cooldown-aware) and retry the // SAME request once per remaining key. OAuth/forward providers and single-key pools // return null immediately, so this stays a no-op for them (src/providers/key-failover.ts). @@ -2497,6 +2689,7 @@ async function handleResponsesInner( // until runtime cleanup (one per rotated key under a rate-limit storm). try { void upstreamResponse.body?.cancel().catch(() => {}); } catch { /* already consumed/closed */ } route.provider = rotated; + invalidateSameTargetRequest(); activeAdapter = resolveAdapter( resolveWireProtocolOverride(route.providerName, route.modelId, route.provider, inboundWire), config.cacheRetention, @@ -2527,6 +2720,7 @@ async function handleResponsesInner( anthropicPoolAccountId = nextAccountId; anthropicPoolFailovers += 1; route.provider = { ...route.provider, apiKey: accessToken }; + invalidateSameTargetRequest(); promoteAnthropicActiveAccount(nextAccountId); logCtx.provider = formatAnthropicProviderForLog("anthropic", nextAccountId, config); activeAdapter = resolveAdapter( @@ -2551,6 +2745,7 @@ async function handleResponsesInner( })) { imageRetryAttempted = true; imageTierBias = 1; + invalidateSameTargetRequest(); try { void upstreamResponse.body?.cancel().catch(() => {}); } catch { /* already consumed/closed */ } const result = await rebuildAndRefetch("image-413"); if ("failed" in result) return result.failed; @@ -2595,64 +2790,149 @@ async function handleResponsesInner( cancelBodyOnAbort(upstreamResponse.body, upstream.signal); - // Claude can return a clean end_turn after announcing an edit without emitting any tool call. - // Keep the normal request/recovery path above intact, and use this bounded callback only for the - // one internal continuation pass. A continuation failure becomes an in-stream adapter error so - // the client never sees a second hidden HTTP response or an unbounded retry loop. + // Anthropic-only: one bounded internal continuation re-ask for clean end_turn turns that + // announced an edit without emitting a tool call. const terminalGuardEnabled = activeAdapter.name === "anthropic" && !options.comboAttempt && !routedCompaction; + /** + * One bounded internal re-ask for Anthropic end_turn-without-tool-call turns. Replays the + * continuation on a 429 with the same-key retry budget (hoisted per request), then falls + * back to key/account failover; a failure becomes an in-stream adapter error so the client + * never sees a second hidden HTTP response or an unbounded retry loop. + */ const fetchTerminalGuardContinuation = async function* (nextParsed: OcxParsedRequest): AsyncGenerator { - let imageTierBias = 0; let response: Response | undefined; + // One-shot recovery label for the next top-of-loop continuation send after a failover rotation. + let nextContinuationRecoveryKind: AttemptRecoveryKind | undefined; + /** + * Build and fetch one terminal-guard continuation. `recoveryKind` tags same-target and + * failover sends (`rate-limit-429`, `key-429`, `anthropic-oauth-429`, `image-413`); the + * adapter rebuild is deterministic for the same parsed request (tests assert byte-identical + * replays). + */ + const fetchContinuation = async (recoveryKind?: AttemptRecoveryKind): Promise => { + let continuationRequest: AdapterRequest | undefined; + if (sameTargetRequest !== undefined && sameTargetParsed === nextParsed && sameTargetToken === transportToken) { + // Same target (key/adapter/parsed/tier unchanged): replay the exact cached request. + continuationRequest = sameTargetRequest; + } else { + try { + continuationRequest = await activeAdapter.buildRequest(nextParsed, { + headers: selectedForwardHeaders, + translatorBudget, + ...(imageTierBias > 0 ? { imageTierBias } : {}), + }); + recordAdapterReasoning(logCtx, continuationRequest); + } catch (err) { + // The main body is already streaming, so there is no HTTP error surface: release + // any partial body observation and surface the failure as an in-stream error via + // the outer catch (no upstream.abort() — that would kill the live body stream). + continuationRequest?.releaseBodyObservation?.(); + throw err; + } + sameTargetRequest = continuationRequest; + sameTargetParsed = nextParsed; + sameTargetToken = transportToken; + } + // Both branches assign the request (the build catch rethrows), so capture it in a + // const for the fetch callback and finally below — a `let` read inside a nested + // function keeps its undefined half, which would break the byte-identical replay. + const builtContinuationRequest = continuationRequest; + const continuationEstimate = typeof builtContinuationRequest.usageLog?.inputTokens === "number" + ? builtContinuationRequest.usageLog.inputTokens + : undefined; + if (continuationEstimate !== undefined) logCtx.usageLogInputTokens = continuationEstimate; + // Optional recovery label for same-target / failover continuation sends. + const replayKind: AttemptRecoveryKind | undefined = recoveryKind; + try { + if (activeAdapter.fetchResponse) { + noteAttemptSend(logCtx.activeAttempt, continuationEstimate, replayKind); + return await activeAdapter.fetchResponse(builtContinuationRequest, { + abortSignal: upstream.signal, + timeoutMs: connectMs, + stream: nextParsed.stream, + }); + } + return await fetchWithResetRetry( + recovery => { + noteAttemptSend(logCtx.activeAttempt, continuationEstimate, recovery ?? replayKind); + return fetchWithHeaderTimeout( + builtContinuationRequest.url, + applyUpstreamRecoveryInit({ + method: builtContinuationRequest.method, + headers: builtContinuationRequest.headers, + body: builtContinuationRequest.body, + }, recovery), + upstream.signal, + connectMs, + nextParsed.stream, + providerFetch(route.provider), + ); + }, + { abortSignal: upstream.signal, label: safeHostLabel(builtContinuationRequest.url) }, + ); + } finally { + builtContinuationRequest.releaseBodyObservation?.(); + } + }; while (true) { try { - const continuationRequest = await activeAdapter.buildRequest(nextParsed, { - headers: selectedForwardHeaders, - translatorBudget, - ...(imageTierBias > 0 ? { imageTierBias } : {}), - }); - recordAdapterReasoning(logCtx, continuationRequest); - const continuationEstimate = typeof continuationRequest.usageLog?.inputTokens === "number" - ? continuationRequest.usageLog.inputTokens - : undefined; - if (continuationEstimate !== undefined) logCtx.usageLogInputTokens = continuationEstimate; + const recoveryKind = nextContinuationRecoveryKind; + nextContinuationRecoveryKind = undefined; + response = await fetchContinuation(recoveryKind); + } catch (error) { + if (options.abortSignal?.aborted || upstream.signal.aborted) { + yield { type: "error", message: "client closed request during terminal continuation", status: 499 }; + } else { + yield { type: "error", message: `Provider continuation failed: ${redactSecretString(error instanceof Error ? error.message : String(error))}` }; + } + return; + } + + // Same-target 429 wait-and-retry (opt-in `retryOn429`) before key/account failover: + // a primary-key rate-limit blip replays on the SAME key, matching the main recovery + // loop; only after the attempts are exhausted does the continuation fail over. + while ( + response.status === 429 + && rateLimitPolicy !== null + && rateLimitRetries < rateLimitPolicy.attempts + ) { + rateLimitRetries += 1; + // Release unread body + heartbeat-fed wait via the shared same-target helper. + const retryAfterHeader = response.headers.get("retry-after"); try { - if (activeAdapter.fetchResponse) { - noteAttemptSend(logCtx.activeAttempt, continuationEstimate); - response = await activeAdapter.fetchResponse(continuationRequest, { - abortSignal: upstream.signal, - timeoutMs: connectMs, - stream: nextParsed.stream, - }); + yield* prepareSameTarget429Wait({ + body: response.body, + // Listen on the upstream signal: once the SSE body is being streamed, a client + // cancel aborts `upstream` through the bridge, and upstream is also linked from + // options.abortSignal — so this covers both cancellation paths. + signal: upstream.signal, + delayMs: rateLimitRetryDelayMs(rateLimitPolicy, retryAfterHeader, Date.now()), + heartbeatIntervalMs: Math.min(10_000, Math.max(250, stallTimeoutMs / 2)), + }); + } catch { + if (options.abortSignal?.aborted || upstream.signal.aborted) { + yield { type: "error", message: "client closed request during terminal continuation", status: 499 }; } else { - response = await fetchWithResetRetry( - recovery => { - noteAttemptSend(logCtx.activeAttempt, continuationEstimate, recovery); - return fetchWithHeaderTimeout( - continuationRequest.url, - applyUpstreamRecoveryInit({ - method: continuationRequest.method, - headers: continuationRequest.headers, - body: continuationRequest.body, - }, recovery), - upstream.signal, - connectMs, - nextParsed.stream, - providerFetch(route.provider), - ); - }, - { abortSignal: upstream.signal, label: safeHostLabel(continuationRequest.url) }, - ); + yield { type: "error", message: "Provider continuation failed: retry wait interrupted" }; } - } finally { - continuationRequest.releaseBodyObservation?.(); + return; } - } catch (error) { - if (options.abortSignal?.aborted) { + // Client cancellation wins over any stale timer edge: re-check before dispatching the + // replay so the continuation never starts work for a request the client abandoned. + if (options.abortSignal?.aborted || upstream.signal.aborted) { yield { type: "error", message: "client closed request during terminal continuation", status: 499 }; - } else { - yield { type: "error", message: `Provider continuation failed: ${error instanceof Error ? error.message : String(error)}` }; + return; + } + try { + response = await fetchContinuation("rate-limit-429"); + } catch (error) { + if (options.abortSignal?.aborted || upstream.signal.aborted) { + yield { type: "error", message: "client closed request during terminal continuation", status: 499 }; + } else { + yield { type: "error", message: `Provider continuation failed: ${redactSecretString(error instanceof Error ? error.message : String(error))}` }; + } + return; } - return; } if (response.status === 429 && hasKeyPoolFailover(route.provider)) { @@ -2665,10 +2945,12 @@ async function handleResponsesInner( if (rotated) { try { void response.body?.cancel().catch(() => {}); } catch { /* already closed */ } route.provider = rotated; + invalidateSameTargetRequest(); activeAdapter = resolveAdapter( resolveWireProtocolOverride(route.providerName, route.modelId, route.provider, inboundWire), config.cacheRetention, ); + nextContinuationRecoveryKind = "key-429"; continue; } } @@ -2691,6 +2973,7 @@ async function handleResponsesInner( anthropicPoolAccountId = nextAccountId; anthropicPoolFailovers += 1; route.provider = { ...route.provider, apiKey: accessToken }; + invalidateSameTargetRequest(); promoteAnthropicActiveAccount(nextAccountId); logCtx.provider = formatAnthropicProviderForLog("anthropic", nextAccountId, config); activeAdapter = resolveAdapter( @@ -2698,6 +2981,7 @@ async function handleResponsesInner( config.cacheRetention, ); sealRequestAttemptIdentity(logCtx.activeAttempt, logCtx.provider, activeAdapter.name); + nextContinuationRecoveryKind = "anthropic-oauth-429"; continue; } catch { // fall through to emit continuation error below @@ -2711,7 +2995,9 @@ async function handleResponsesInner( alreadyAttempted: imageTierBias > 0, })) { imageTierBias = 1; + invalidateSameTargetRequest(); try { void response.body?.cancel().catch(() => {}); } catch { /* already closed */ } + nextContinuationRecoveryKind = "image-413"; continue; } break; diff --git a/src/types.ts b/src/types.ts index 0bbdc7e41..65a84061c 100644 --- a/src/types.ts +++ b/src/types.ts @@ -910,6 +910,29 @@ export interface ResponsesItemIdRepairConfig { repairMissingTerminalIds?: boolean; } +/** + * Same-target 429 wait-and-retry policy (`providers..retryOn429`). When present and not + * explicitly disabled, the proxy waits and replays the identical request on the same key before + * any key failover. All fields optional; the runtime applies defaults (attempts=3, + * intervalMs=5000, maxIntervalMs=60000, respectRetryAfter=true, enabled=true). + */ +export interface RateLimitRetryPolicy { + /** Master switch. The presence of the object also enables the policy (default true). */ + enabled?: boolean; + /** Extra replay attempts after the first 429 (1..20, default 3). */ + attempts?: number; + /** Fixed wait between attempts when the upstream sends no usable Retry-After (default 5000). */ + intervalMs?: number; + /** Cap for any single wait, including an upstream Retry-After (default 60000). */ + maxIntervalMs?: number; + /** Prefer the upstream Retry-After header when present and parseable (default true). */ + respectRetryAfter?: boolean; +} + +/** + * One configured provider entry. `authMode` (default `"key"`) decides whether same-target 429 + * retries are allowed; OAuth/forward credentials and local runtimes are never replayed. + */ export interface OcxProviderConfig { adapter: string; /** Cursor MCP compatibility bounds; positive integers when configured. */ @@ -1113,6 +1136,13 @@ export interface OcxProviderConfig { autoToolChoiceOnlyModels?: string[]; /** Model ids that expect prior assistant `reasoning_content` to be preserved in chat history. */ preserveReasoningContentModels?: string[]; + /** + * Opt-in same-target 429 retry policy. Codex itself never retries 429 (it retries 5xx only, + * openai/codex#30471), and single-key pools have no failover, so the proxy waits and replays + * the identical request on the same key before any failover. Pre-stream only: a 429 arrives + * before any response bytes are relayed, so the replay is lossless. + */ + retryOn429?: RateLimitRetryPolicy; /** * Model ids whose OpenAI-compatible chat endpoint accepts `reasoning_split: true` and returns * thinking separately in `reasoning_content` / `reasoning_details` instead of visible content. diff --git a/src/usage/log.ts b/src/usage/log.ts index 7e846bc82..9a66422b8 100644 --- a/src/usage/log.ts +++ b/src/usage/log.ts @@ -7,11 +7,16 @@ import type { OcxUsage } from "../types"; export type UsageStatus = "reported" | "unreported" | "unsupported" | "estimated"; +/** + * Recovery kinds recorded per attempt in the usage log; the GUI renders localized labels + * for these wire values. + */ export type AttemptRecoveryKind = | "transient-5xx" | "connection-reset" | "oauth-401" | "key-429" + | "rate-limit-429" | "anthropic-oauth-429" | "image-413"; @@ -173,6 +178,7 @@ const ATTEMPT_RECOVERY_KINDS = new Set([ "connection-reset", "oauth-401", "key-429", + "rate-limit-429", "anthropic-oauth-429", "image-413", ]); diff --git a/src/web-search/loop.ts b/src/web-search/loop.ts index b4a5e2797..c0eda72ce 100644 --- a/src/web-search/loop.ts +++ b/src/web-search/loop.ts @@ -1,13 +1,15 @@ import type { AdapterRequest, IncomingMeta, ProviderAdapter } from "../adapters/base"; -import type { AdapterEvent, OcxMessage, OcxParsedRequest, OcxProviderConfig, OcxThinkingContent, OcxUsage } from "../types"; +import type { AdapterEvent, OcxMessage, OcxParsedRequest, OcxProviderConfig, OcxThinkingContent, OcxUsage, RateLimitRetryPolicy } from "../types"; import { namespacedToolName } from "../types"; +import type { AttemptRecoveryKind } from "../usage/log"; import { bridgeToResponsesSSE } from "../bridge"; import { runWebSearch, type SidecarOutcome, type SidecarOutcomeRecorder, type SidecarSettings } from "./executor"; import { runAnthropicWebSearch } from "./anthropic-executor"; import { clearableDeadline } from "../lib/abort"; import { redactSecretString } from "../lib/redact"; import { readBoundedResponseBody } from "../lib/bounded-body"; -import { fetchWithResetRetry } from "../lib/upstream-retry"; +import { fetchWithResetRetry, prepareSameTarget429Wait } from "../lib/upstream-retry"; +import { rateLimitRetryDelayMs } from "../providers/key-failover"; import { isTranslatorBudgetExceededError, TRANSLATOR_MAX_TURN_BYTES, @@ -206,6 +208,10 @@ class LoopError extends Error { } } +/** + * Dependencies for one web-search loop iteration: parsed request, active adapter, + * incoming metadata, and the configured search executor. + */ export interface WebSearchLoopDeps { parsed: OcxParsedRequest; adapter: ProviderAdapter; @@ -239,11 +245,15 @@ export interface WebSearchLoopDeps { onUsage?: (usage: OcxUsage | undefined) => void; /** Observe the exact adapter request selected for each routed-model iteration. */ onRequestBuilt?: (request: AdapterRequest) => void; + /** Called before each routed-model dispatch in the loop, for attempt telemetry. Same-target 429 replays pass the `rate-limit-429` recovery kind. */ + onAttemptSend?: (recovery?: AttemptRecoveryKind) => void; /** * 429 key-failover hook: rotate the provider's active pool key and return a rebuilt adapter, * or null when the pool is exhausted (same semantics as the normal routed path). */ on429?: (retryAfterHeader: string | null) => ProviderAdapter | null; + /** Opt-in same-target 429 policy (key-auth providers). When present, 429 replays on the SAME key before on429 rotation. */ + retryOn429Policy?: Required | null; } /** @@ -260,6 +270,12 @@ export async function runWithWebSearch(deps: WebSearchLoopDeps): Promise 0 + ? Math.floor(deps.stallTimeoutSec * 1000) + : 300_000; + const messages: OcxMessage[] = [...parsed.context.messages]; const loopT0 = Date.now(); const allTools = parsed.context.tools ?? []; @@ -294,9 +310,20 @@ export async function runWithWebSearch(deps: WebSearchLoopDeps): Promise; + // Same-target 429 budget is per REQUEST, not per model iteration: later search rounds inherit + // what earlier rounds left of `attempts`, so a bounded multi-round turn can never exceed the + // configured replay count in total (a per-round reset would multiply it by maxSearches). + const rateLimitRetryPolicy = deps.retryOn429Policy ?? null; + let rateLimitRetries = 0; + // Acquire one iteration's final response headers. The first call is drained eagerly so an initial // connect/header/HTTP failure stays a non-2xx JSON response. Its successful BODY is deliberately // left unread until the downstream Responses SSE bridge exists. + /** + * Fetch one web-search iteration's final response headers, applying the response-header + * deadline and the same-target 429 retry policy (with awaited body release and deadline + * restart) before the `on429` key rotation. + */ const prepareIterationEvents = async function* (forceAnswer: boolean): AsyncGenerator { // On the forced-answer pass the synthetic web_search tool is gone, so the model MUST answer // from the results already in `messages`. A weak model can still produce a thin answer that @@ -313,30 +340,55 @@ export async function runWithWebSearch(deps: WebSearchLoopDeps): Promise => { - const request = await requestAdapter.buildRequest(iterParsed, { - headers: selectedForwardHeaders, - abortSignal: headerDeadline.signal, - translatorBudget, - }); - try { - deps.onRequestBuilt?.(request); - } catch { - // Diagnostics are best-effort and must never abort a web-search iteration. + /** + * Build and fetch one web-search iteration on the given adapter, under the iteration + * header deadline. The caller owns same-target 429 replays and key rotation around it. + * The outbound request is cached per adapter so a same-target replay reuses the EXACT + * URL, serialized body, and headers (builder runs once per target sequence). + */ + let cachedRequest: AdapterRequest | undefined; + let cachedAdapter: ProviderAdapter | undefined; + /** + * Build and fetch one web-search iteration on the given adapter, under the iteration + * header deadline. The caller owns same-target 429 replays and key rotation around it. + */ + const fetchOnce = async (requestAdapter: ProviderAdapter, recovery?: AttemptRecoveryKind): Promise => { + let request: AdapterRequest; + if (cachedRequest !== undefined && cachedAdapter === requestAdapter) { + request = cachedRequest; + } else { + request = await requestAdapter.buildRequest(iterParsed, { + headers: selectedForwardHeaders, + abortSignal: headerDeadline.signal, + translatorBudget, + }); + try { + deps.onRequestBuilt?.(request); + } catch { + // Diagnostics are best-effort and must never abort a web-search iteration. + } + cachedRequest = request; + cachedAdapter = requestAdapter; } let response: Response; try { - response = requestAdapter.fetchResponse - ? await requestAdapter.fetchResponse(request, { + if (requestAdapter.fetchResponse) { + deps.onAttemptSend?.(recovery); + response = await requestAdapter.fetchResponse(request, { abortSignal: headerDeadline.signal, timeoutMs: connectTimeoutMs, returnRawErrors: true, stream: true, - }) - : await fetchWithResetRetry( - () => { + }); + } else { + response = await fetchWithResetRetry( + (retryRecovery) => { + // Record every helper-driven send (the callback runs for the first attempt and + // each connection-reset replay); preserve the caller's recovery kind + // (rate-limit-429 / key-429) when the retry layer supplies none. + deps.onAttemptSend?.(retryRecovery ?? recovery); const h = new Headers(request.headers); if (!h.has("accept-encoding")) h.set("accept-encoding", "identity"); return fetch(request.url, { @@ -347,7 +399,8 @@ export async function runWithWebSearch(deps: WebSearchLoopDeps): Promise { process.off("unhandledRejection", onUnhandledRejection); } }); + + test("a throwing buildRequest is mapped to 400 invalid_request_error, not an unhandled rejection", async () => { + const unhandledRejections: unknown[] = []; + const onUnhandledRejection = (reason: unknown) => { unhandledRejections.push(reason); }; + process.on("unhandledRejection", onUnhandledRejection); + try { + adapterFactory = provider => ({ + name: "test-throw-build", + buildRequest: () => { throw new Error("fixture build failure"); }, + async *parseStream(): AsyncGenerator { + yield { type: "error", message: "unreachable" }; + }, + }); + + const response = await post("test-throw-build", false); + const body = await response.json() as { error?: { code?: string; message?: string } }; + + expect(response.status).toBe(400); + expect(body.error?.code).toBe("invalid_request_error"); + expect(body.error?.message).toContain("fixture build failure"); + expect(unhandledRejections).toEqual([]); + } finally { + process.off("unhandledRejection", onUnhandledRejection); + } + }); + + test("a client abort during buildRequest surfaces 499 client_cancelled instead of a 400", async () => { + const clientAbort = new AbortController(); + adapterFactory = provider => ({ + name: "test-abort-build", + buildRequest: () => { + clientAbort.abort(new DOMException("client disconnected", "AbortError")); + throw new Error("build interrupted by client disconnect"); + }, + async *parseStream(): AsyncGenerator { + yield { type: "error", message: "unreachable" }; + }, + }); + + const response = await handleResponses(new Request("http://localhost/v1/responses", { + method: "POST", + headers: { "content-type": "application/json" }, + body: JSON.stringify({ model: "fixture/model", input: "hello", stream: false }), + }), config("test-abort-build"), { model: "", provider: "" }, { abortSignal: clientAbort.signal }); + const body = await response.json() as { error?: { code?: string } }; + + expect(response.status).toBe(499); + expect(body.error?.code).toBe("client_cancelled"); + }); }); diff --git a/tests/config-user-edits.test.ts b/tests/config-user-edits.test.ts index 35d16d202..207833622 100644 --- a/tests/config-user-edits.test.ts +++ b/tests/config-user-edits.test.ts @@ -1,4 +1,4 @@ -import { afterEach, beforeEach, expect, test } from "bun:test"; +import { afterEach, beforeEach, expect, spyOn, test } from "bun:test"; import { mkdtempSync, readFileSync, rmSync, writeFileSync } from "node:fs"; import { tmpdir } from "node:os"; import { join } from "node:path"; @@ -6,10 +6,12 @@ import { armClaudeCodeBaseline, getConfigPath, loadConfig, + readConfigDiagnostics, reconcileLiveConfigFromDisk, saveConfig, saveConfigPreservingClaudeCode, } from "../src/config"; +import { rateLimitRetryPolicyFor } from "../src/providers/key-failover"; import type { OcxConfig } from "../src/types"; /** @@ -21,11 +23,13 @@ import type { OcxConfig } from "../src/types"; let home: string; let previousHome: string | undefined; +/** Merge a patch into the on-disk config.json, simulating a user hand-edit. */ function writeDiskConfig(patch: Record): void { const current = JSON.parse(readFileSync(getConfigPath(), "utf8")) as Record; writeFileSync(getConfigPath(), JSON.stringify({ ...current, ...patch }, null, 2) + "\n"); } +/** Read the current on-disk config.json as a plain record. */ function diskConfig(): Record { return JSON.parse(readFileSync(getConfigPath(), "utf8")) as Record; } @@ -109,6 +113,249 @@ test("an unrelated loadConfig does not refresh the armed baseline", () => { expect((diskConfig().claudeCode as Record).authMode).toBe("proxy"); }); +test("an invalid retryOn429 field degrades at load instead of discarding the config", () => { + writeDiskConfig({ + providers: { + test: { + adapter: "openai-chat", + baseUrl: "http://127.0.0.1:1/v1", + apiKey: "k", + allowPrivateNetwork: true, + retryOn429: { attempts: 0, attempt: 5, intervalMs: 120, respectRetryAfter: false }, + }, + }, + }); + const live = loadConfig(); + expect(live.providers.test).toBeDefined(); + // Invalid field (attempts: 0) and the misnamed key (attempt) dropped with warnings; + // valid fields kept; missing fields defaulted. + expect(live.providers.test.retryOn429).toEqual({ intervalMs: 120, respectRetryAfter: false }); +}); + +test("a non-object retryOn429 degrades at load instead of discarding the config", () => { + writeDiskConfig({ + providers: { + test: { + adapter: "openai-chat", + baseUrl: "http://127.0.0.1:1/v1", + apiKey: "k", + allowPrivateNetwork: true, + retryOn429: "enabled", + }, + }, + }); + const live = loadConfig(); + expect(live.providers.test).toBeDefined(); + expect(live.providers.test.retryOn429).toBeUndefined(); +}); + +test("an invalid retryOn429 master switch discards the policy instead of enabling it", () => { + writeDiskConfig({ + providers: { + test: { + adapter: "openai-chat", + baseUrl: "http://127.0.0.1:1/v1", + apiKey: "k", + allowPrivateNetwork: true, + retryOn429: { enabled: "false", intervalMs: 120 }, + }, + }, + }); + const live = loadConfig(); + expect(live.providers.test).toBeDefined(); + // A hand-edit that tried to disable retries must not become default-ENABLED. + expect(live.providers.test.retryOn429).toBeUndefined(); +}); + +test("a retryOn429 policy with every field invalid is dropped instead of enabling retries", () => { + writeDiskConfig({ + providers: { + test: { + adapter: "openai-chat", + baseUrl: "http://127.0.0.1:1/v1", + apiKey: "k", + allowPrivateNetwork: true, + // Every supplied field invalid: the sanitizer must NOT write back {} — presence + // would opt IN to retries with defaults, the opposite of a disable-oriented + // hand-edit like `attempts: 0`. + retryOn429: { attempts: 0 }, + }, + }, + }); + const live = loadConfig(); + expect(live.providers.test.retryOn429).toBeUndefined(); + expect(rateLimitRetryPolicyFor(live.providers.test)).toBeNull(); +}); + +test("an intentionally empty retryOn429 policy still resolves as enabled (presence = opt-in)", () => { + writeDiskConfig({ + providers: { + test: { + adapter: "openai-chat", + baseUrl: "http://127.0.0.1:1/v1", + apiKey: "k", + allowPrivateNetwork: true, + retryOn429: {}, + }, + }, + }); + const live = loadConfig(); + expect(live.providers.test.retryOn429).toEqual({}); + // Object presence is the opt-in contract: an explicit `retryOn429: {}` resolves to the + // enabled defaults, exactly like the documented hand-written config. + expect(rateLimitRetryPolicyFor(live.providers.test)).toEqual({ + enabled: true, + attempts: 3, + intervalMs: 5_000, + maxIntervalMs: 60_000, + respectRetryAfter: true, + }); +}); + +test("config diagnostics sanitize invalid retryOn429 before schema validation", () => { + writeDiskConfig({ + providers: { + test: { + adapter: "openai-chat", + baseUrl: "http://127.0.0.1:1/v1", + apiKey: "k", + allowPrivateNetwork: true, + retryOn429: { attempts: 0 }, + }, + }, + }); + const diagnostics = readConfigDiagnostics(); + // Without sanitization the schema rejects the config and the diagnostics path returns a + // default fallback, which the config command could persist over the user's providers. + expect(diagnostics.source).not.toBe("fallback"); + expect(diagnostics.config.providers.test).toBeDefined(); + expect(diagnostics.config.providers.test.retryOn429).toBeUndefined(); +}); + +test("invalid retryOn429 values never log the raw value", () => { + const warn = spyOn(console, "warn").mockImplementation(() => {}); + try { + writeDiskConfig({ + providers: { + test: { + adapter: "openai-chat", + baseUrl: "http://127.0.0.1:1/v1", + apiKey: "k", + allowPrivateNetwork: true, + retryOn429: "sk-super-secret-abc123", + }, + }, + }); + loadConfig(); + const logged = warn.mock.calls.map(call => call.join(" ")).join("\n"); + expect(logged).not.toContain("sk-super-secret-abc123"); + // Anchor the type-only diagnostic to the exact field so unrelated warnings can't satisfy it. + expect(logged).toContain('providers."test".retryOn429 (string) is invalid'); + } finally { + warn.mockRestore(); + } +}); + +test("unrecognized retryOn429 field names are redacted before logging", () => { + const warn = spyOn(console, "warn").mockImplementation(() => {}); + try { + writeDiskConfig({ + providers: { + test: { + adapter: "openai-chat", + baseUrl: "http://127.0.0.1:1/v1", + apiKey: "k", + allowPrivateNetwork: true, + retryOn429: { "sk-super-secret-9876": true, intervalMs: 120 }, + }, + }, + }); + const live = loadConfig(); + expect(live.providers.test.retryOn429).toEqual({ intervalMs: 120 }); + const logged = warn.mock.calls.map(call => call.join(" ")).join("\n"); + // The secret-shaped property NAME must never reach the log; the valid field survives. + expect(logged).not.toContain("sk-super-secret-9876"); + expect(logged).toContain("[REDACTED]"); + } finally { + warn.mockRestore(); + } +}); + +test("unrecognized retryOn429 field names are JSON-escaped before logging", () => { + const warn = spyOn(console, "warn").mockImplementation(() => {}); + try { + writeDiskConfig({ + providers: { + test: { + adapter: "openai-chat", + baseUrl: "http://127.0.0.1:1/v1", + apiKey: "k", + allowPrivateNetwork: true, + retryOn429: { "evil\nattempt": true, intervalMs: 120 }, + }, + }, + }); + const live = loadConfig(); + expect(live.providers.test.retryOn429).toEqual({ intervalMs: 120 }); + const logged = warn.mock.calls.map(call => call.join(" ")).join("\n"); + // The raw control character must never reach the log (no line forging); the escaped form + // still names the field for typo debugging. + expect(logged).not.toContain("evil\nattempt"); + expect(logged).toContain('"evil\\nattempt"'); + } finally { + warn.mockRestore(); + } +}); + +test("provider names are redacted before retryOn429 load warnings", () => { + const warn = spyOn(console, "warn").mockImplementation(() => {}); + try { + writeDiskConfig({ + providers: { + "sk-super-secret-9876": { + adapter: "openai-chat", + baseUrl: "http://127.0.0.1:1/v1", + apiKey: "k", + allowPrivateNetwork: true, + retryOn429: "enabled", + }, + }, + }); + loadConfig(); + const logged = warn.mock.calls.map(call => call.join(" ")).join("\n"); + // The sanitizer runs before schema validation, so a secret-shaped provider NAME must + // never reach the log either. + expect(logged).not.toContain("sk-super-secret-9876"); + expect(logged).toContain("[REDACTED]"); + } finally { + warn.mockRestore(); + } +}); + +test("provider names with control characters are JSON-escaped before retryOn429 load warnings", () => { + const warn = spyOn(console, "warn").mockImplementation(() => {}); + try { + writeDiskConfig({ + providers: { + "evil\nprovider": { + adapter: "openai-chat", + baseUrl: "http://127.0.0.1:1/v1", + apiKey: "k", + allowPrivateNetwork: true, + retryOn429: "enabled", + }, + }, + }); + loadConfig(); + const logged = warn.mock.calls.map(call => call.join(" ")).join("\n"); + // The raw newline must never forge a log line; the escaped form still names the provider. + expect(logged).not.toContain("evil\nprovider"); + expect(logged).toContain('"evil\\nprovider"'); + } finally { + warn.mockRestore(); + } +}); + // R4-1: the request path. A 429 mid-turn rotates a key and saves, with no user action. test("a 429 key rotation does not clobber the hand edit", async () => { const { rotateKeyOn429 } = await import("../src/providers/key-failover"); diff --git a/tests/images/loop.test.ts b/tests/images/loop.test.ts index 6795e8873..e77373c3c 100644 --- a/tests/images/loop.test.ts +++ b/tests/images/loop.test.ts @@ -95,6 +95,7 @@ const imageCallEvents: AdapterEvent[] = [ { type: "done" }, ]; +/** Run the image bridge with the given per-iteration event streams and return the client-facing SSE text. */ async function runAndGetSSE(streams: AdapterEvent[][], fulfill?: ImageCallResult): Promise { streamQueue = streams.map(s => [...s]); if (fulfill) fulfillResult = fulfill; @@ -177,6 +178,199 @@ describe("runWithImageBridge", () => { expect(buildRequestCalls).toBe(1); }); + test("retryOn429 replays on the same key before on429 rotation", async () => { + let sends = 0; + let rotations = 0; + let retrySends = 0; + const retryingAdapter: ProviderAdapter = { + ...mockAdapter, + fetchResponse: async () => { + sends += 1; + if (sends === 1) { + return new Response(JSON.stringify({ error: { message: "rate limited" } }), { + status: 429, + headers: { "content-type": "application/json" }, + }); + } + return new Response("{}", { status: 200, headers: { "content-type": "application/json" } }); + }, + }; + streamQueue = [[{ type: "text_delta" as const, text: "recovered" }, { type: "done" as const }]]; + const response = await runWithImageBridge({ + parsed: makeParsed(), + adapter: retryingAdapter, + plan, + retryOn429Policy: { enabled: true, attempts: 2, intervalMs: 120, maxIntervalMs: 60_000, respectRetryAfter: false }, + on429: () => { + rotations += 1; + return null; + }, + onAttemptSend: recovery => { + if (recovery === "rate-limit-429") retrySends += 1; + }, + }); + const sse = await response.text(); + expect(sse).toContain("recovered"); + expect(sends).toBe(2); + expect(rotations).toBe(0); + expect(retrySends).toBe(1); + // Same-target replay reuses the ONE built request (builder runs once per target sequence). + expect(buildRequestCalls).toBe(1); + }); + + test("retry wait longer than the stall budget still succeeds (heartbeats feed the watchdog)", async () => { + let sends = 0; + const retryingAdapter: ProviderAdapter = { + ...mockAdapter, + fetchResponse: async () => { + sends += 1; + if (sends === 1) { + return new Response(JSON.stringify({ error: { message: "rate limited" } }), { + status: 429, + headers: { "content-type": "application/json" }, + }); + } + return new Response("{}", { status: 200, headers: { "content-type": "application/json" } }); + }, + }; + streamQueue = [[{ type: "text_delta" as const, text: "recovered" }, { type: "done" as const }]]; + const response = await runWithImageBridge({ + parsed: makeParsed(), + adapter: retryingAdapter, + plan, + stallTimeoutSec: 1, + retryOn429Policy: { enabled: true, attempts: 1, intervalMs: 1_500, maxIntervalMs: 60_000, respectRetryAfter: false }, + }); + const sse = await response.text(); + // A 1.5s backoff under a 1s stall budget must not trip upstream_stall_timeout: the wait + // yields heartbeat events, the replay lands, and the turn completes. + expect(sends).toBe(2); + expect(sse).toContain("recovered"); + expect(sse).not.toContain("upstream_stall_timeout"); + }, 5_000); + + test("retry wait longer than connectTimeoutMs restarts the header deadline (no 504)", async () => { + let sends = 0; + const retryingAdapter: ProviderAdapter = { + ...mockAdapter, + fetchResponse: async () => { + sends += 1; + if (sends === 1) { + return new Response(JSON.stringify({ error: { message: "rate limited" } }), { + status: 429, + headers: { "content-type": "application/json" }, + }); + } + return new Response("{}", { status: 200, headers: { "content-type": "application/json" } }); + }, + }; + streamQueue = [[{ type: "text_delta" as const, text: "recovered" }, { type: "done" as const }]]; + const response = await runWithImageBridge({ + parsed: makeParsed(), + adapter: retryingAdapter, + plan, + connectTimeoutMs: 100, + retryOn429Policy: { enabled: true, attempts: 1, intervalMs: 150, maxIntervalMs: 60_000, respectRetryAfter: false }, + }); + const sse = await response.text(); + // The deliberate backoff must not consume the response-header deadline: a fresh deadline is + // armed after the wait, so the replay gets a new connect budget instead of a 504. + expect(sends).toBe(2); + expect(sse).toContain("recovered"); + expect(sse).not.toContain("504"); + }, 5_000); + + test("retryOn429 budget is shared across iterations (per request, not per round)", async () => { + let sends = 0; + let retrySends = 0; + let rotations = 0; + const retryingAdapter: ProviderAdapter = { + ...mockAdapter, + fetchResponse: async () => { + sends += 1; + if (sends === 1 || sends === 3) { + return new Response(JSON.stringify({ error: { message: "rate limited" } }), { + status: 429, + headers: { "content-type": "application/json" }, + }); + } + return new Response("{}", { status: 200, headers: { "content-type": "application/json" } }); + }, + }; + // Round 0: 429 -> one same-key replay (attempts=1) -> 200 carrying an image call. + // Round 1 (forced final): 429 with the request budget already spent -> no replay -> rotation. + streamQueue = [ + [...imageCallEvents], + [{ type: "text_delta" as const, text: "unused" }, { type: "done" as const }], + ]; + const response = await runWithImageBridge({ + parsed: makeParsed(), + adapter: retryingAdapter, + plan, + maxRounds: 1, + retryOn429Policy: { enabled: true, attempts: 1, intervalMs: 50, maxIntervalMs: 60_000, respectRetryAfter: false }, + on429: () => { + rotations += 1; + return null; + }, + onAttemptSend: recovery => { + if (recovery === "rate-limit-429") retrySends += 1; + }, + }); + const sse = await response.text(); + expect(sends).toBe(3); + expect(retrySends).toBe(1); + expect(rotations).toBe(1); + // The exhausted final 429 surfaces as the provider error, not a silent success. + expect(sse).toContain("Provider error 429"); + }); + + test("retryOn429 budget is not re-armed after on429 rotation returns a new adapter", async () => { + let sends = 0; + let retrySends = 0; + let rotations = 0; + const retryingAdapter: ProviderAdapter = { + ...mockAdapter, + fetchResponse: async () => { + sends += 1; + return new Response(JSON.stringify({ error: { message: "rate limited" } }), { + status: 429, + headers: { "content-type": "application/json" }, + }); + }, + }; + streamQueue = [[{ type: "text_delta" as const, text: "unused" }, { type: "done" as const }]]; + const response = await runWithImageBridge({ + parsed: makeParsed(), + adapter: retryingAdapter, + plan, + retryOn429Policy: { enabled: true, attempts: 1, intervalMs: 50, maxIntervalMs: 60_000, respectRetryAfter: false }, + on429: () => { + rotations += 1; + // First rotation returns a new adapter that also 429s; the exhausted budget must not + // re-arm for it. Second call returns null to terminate the pool. + return rotations === 1 + ? ({ + ...mockAdapter, + fetchResponse: async () => { + sends += 1; + return new Response("{}", { status: 429 }); + }, + } as ProviderAdapter) + : null; + }, + onAttemptSend: recovery => { + if (recovery === "rate-limit-429") retrySends += 1; + }, + }); + const sse = await response.text(); + // initial 429 + 1 same-key replay + 1 rotated send (no replay on the rotated adapter) = 3. + expect(sends).toBe(3); + expect(retrySends).toBe(1); + expect(rotations).toBe(2); + expect(sse).toContain("Provider error 429"); + }); + test("forced-final clears named image tool_choice", async () => { streamQueue = [ [{ type: "text_delta" as const, text: "done" }, { type: "done" as const }], diff --git a/tests/management-provider-validation.test.ts b/tests/management-provider-validation.test.ts index 7f4d9e67e..cf8195b0d 100644 --- a/tests/management-provider-validation.test.ts +++ b/tests/management-provider-validation.test.ts @@ -192,6 +192,50 @@ describe("provider management validation", () => { })).toContain("not supported on forward-auth"); }); + test("provider management validates retryOn429 bounds and unknown keys", () => { + const base = { adapter: "openai-chat", baseUrl: "https://api.openai.com/v1" }; + expect(providerManagementConfigError("custom", { + ...base, + retryOn429: { enabled: true, attempts: 3, intervalMs: 1_000, maxIntervalMs: 5_000, respectRetryAfter: false }, + })).toBeNull(); + expect(providerManagementConfigError("custom", { + ...base, + retryOn429: { attempts: 0 }, + })).toContain("retryOn429.attempts is invalid"); + expect(providerManagementConfigError("custom", { + ...base, + retryOn429: { attempts: 21 }, + })).toContain("retryOn429.attempts is invalid"); + expect(providerManagementConfigError("custom", { + ...base, + retryOn429: { intervalMs: "fast" }, + })).toContain("retryOn429.intervalMs is invalid"); + expect(providerManagementConfigError("custom", { + ...base, + retryOn429: { attempt: 3 }, + })).toContain("retryOn429 has unrecognized field"); + expect(providerManagementConfigError("custom", { + ...base, + retryOn429: "enabled", + })).toContain("retryOn429 is invalid"); + // A secret-shaped unknown field name must be redacted in the error, never echoed. + const secretError = providerManagementConfigError("custom", { + ...base, + retryOn429: { "sk-super-secret-9876": true }, + })!; + expect(secretError).toContain("retryOn429 has unrecognized field"); + expect(secretError).not.toContain("sk-super-secret-9876"); + expect(secretError).toContain("[REDACTED]"); + // A secret-shaped PROVIDER name must not be echoed by the retryOn429 error path either. + const secretNameError = providerManagementConfigError("sk-super-secret-9876", { + ...base, + retryOn429: { attempts: 0 }, + })!; + expect(secretNameError).toContain("retryOn429.attempts is invalid"); + expect(secretNameError).not.toContain("sk-super-secret-9876"); + expect(secretNameError).toContain("[REDACTED]"); + }); + test("provider discovery status is additive and omitted before an attempt", async () => { markProviderDiscoveryFailed("auth-broken", { reason: "http", httpStatus: 401 }); try { diff --git a/tests/provider-registry-parity.test.ts b/tests/provider-registry-parity.test.ts index a1be64d29..11093a5cf 100644 --- a/tests/provider-registry-parity.test.ts +++ b/tests/provider-registry-parity.test.ts @@ -209,6 +209,15 @@ describe("provider registry parity", () => { } }); + test("providerConfigSeed preserves the registry auth kind, including local", () => { + const local = PROVIDER_REGISTRY.find(entry => entry.authKind === "local"); + expect(local).toBeDefined(); + expect(providerConfigSeed(local!).authMode).toBe("local"); + const key = PROVIDER_REGISTRY.find(entry => entry.id === "deepseek"); + expect(key).toBeDefined(); + expect(providerConfigSeed(key!).authMode).toBe("key"); + }); + test("CN provider defaults and context windows match the audited registry refresh", () => { const deepseek = PROVIDER_REGISTRY.find(entry => entry.id === "deepseek"); expect(deepseek).toMatchObject({ diff --git a/tests/rate-limit-retry.test.ts b/tests/rate-limit-retry.test.ts new file mode 100644 index 000000000..244d244f1 --- /dev/null +++ b/tests/rate-limit-retry.test.ts @@ -0,0 +1,229 @@ +import { afterEach, describe, expect, test } from "bun:test"; +import { + rateLimitRetryDelayMs, + rateLimitRetryPolicyFor, +} from "../src/providers/key-failover"; +import { handleResponses } from "../src/server/responses"; +import type { OcxConfig, OcxProviderConfig } from "../src/types"; + +describe("rateLimitRetryPolicyFor", () => { + test("null when absent or explicitly disabled", () => { + expect(rateLimitRetryPolicyFor({} as OcxProviderConfig)).toBeNull(); + expect(rateLimitRetryPolicyFor({ retryOn429: { enabled: false } } as OcxProviderConfig)).toBeNull(); + }); + + test("null for OAuth, forward, local, and unknown auth modes (fail closed)", () => { + expect(rateLimitRetryPolicyFor({ + authMode: "oauth", + retryOn429: {}, + } as OcxProviderConfig)).toBeNull(); + expect(rateLimitRetryPolicyFor({ + authMode: "forward", + retryOn429: {}, + } as OcxProviderConfig)).toBeNull(); + expect(rateLimitRetryPolicyFor({ + authMode: "local", + retryOn429: {}, + } as OcxProviderConfig)).toBeNull(); + expect(rateLimitRetryPolicyFor({ + authMode: "custom-unknown", + retryOn429: {}, + } as OcxProviderConfig)).toBeNull(); + expect(rateLimitRetryPolicyFor({ + authMode: "key", + retryOn429: {}, + } as OcxProviderConfig)).not.toBeNull(); + }); + + test("applies defaults when the object is present", () => { + expect(rateLimitRetryPolicyFor({ retryOn429: {} } as OcxProviderConfig)).toEqual({ + enabled: true, + attempts: 3, + intervalMs: 5_000, + maxIntervalMs: 60_000, + respectRetryAfter: true, + }); + }); + + test("honors explicit values", () => { + expect(rateLimitRetryPolicyFor({ + retryOn429: { attempts: 10, intervalMs: 1_000, maxIntervalMs: 5_000, respectRetryAfter: false }, + } as OcxProviderConfig)).toEqual({ + enabled: true, + attempts: 10, + intervalMs: 1_000, + maxIntervalMs: 5_000, + respectRetryAfter: false, + }); + }); +}); + +describe("rateLimitRetryDelayMs", () => { + const policy = rateLimitRetryPolicyFor({ retryOn429: {} } as OcxProviderConfig)!; + + test("fixed interval when no header is present", () => { + expect(rateLimitRetryDelayMs(policy, null, 1_000_000)).toBe(5_000); + expect(rateLimitRetryDelayMs(policy, undefined, 1_000_000)).toBe(5_000); + }); + + test("honors Retry-After seconds and caps it at maxIntervalMs", () => { + expect(rateLimitRetryDelayMs(policy, "2", 1_000_000)).toBe(2_000); + expect(rateLimitRetryDelayMs(policy, "3600", 1_000_000)).toBe(60_000); + }); + + test("honors an HTTP-date Retry-After", () => { + const now = Date.parse("2026-10-21T07:27:30Z"); + expect(rateLimitRetryDelayMs(policy, "Wed, 21 Oct 2026 07:28:00 GMT", now)).toBe(30_000); + }); + + test("an already-expired HTTP-date Retry-After retries immediately", () => { + const now = Date.parse("2026-10-21T07:28:00Z"); + expect(rateLimitRetryDelayMs(policy, "Wed, 21 Oct 2026 07:27:30 GMT", now)).toBe(1); + expect(rateLimitRetryDelayMs(policy, "Wed, 21 Oct 2026 07:28:00 GMT", now)).toBe(1); + }); + + test("a far-future HTTP-date Retry-After is capped at maxIntervalMs", () => { + const now = Date.parse("2026-10-21T07:27:30Z"); + expect(rateLimitRetryDelayMs(policy, "Wed, 21 Oct 2027 07:28:00 GMT", now)).toBe(60_000); + }); + + test("Retry-After 0 retries immediately instead of falling back to the interval", () => { + expect(rateLimitRetryDelayMs(policy, "0", 1_000_000)).toBe(1); + }); + + test("fixed fallback is capped at maxIntervalMs (a single wait never exceeds the cap)", () => { + const p = rateLimitRetryPolicyFor({ + retryOn429: { intervalMs: 600_000, maxIntervalMs: 100 }, + } as OcxProviderConfig)!; + expect(rateLimitRetryDelayMs(p, null, 1_000_000)).toBe(100); + expect(rateLimitRetryDelayMs(p, "3600", 1_000_000)).toBe(100); + }); + + test("malformed Retry-After falls back to the fixed interval", () => { + expect(rateLimitRetryDelayMs(policy, "soon", 1_000_000)).toBe(5_000); + }); + + test("respectRetryAfter=false ignores the header", () => { + const p = rateLimitRetryPolicyFor({ + retryOn429: { respectRetryAfter: false, intervalMs: 111 }, + } as OcxProviderConfig)!; + expect(rateLimitRetryDelayMs(p, "2", 1_000_000)).toBe(111); + }); +}); + +describe("retry loop client-abort handling", () => { + const originalFetch = globalThis.fetch; + + afterEach(() => { + globalThis.fetch = originalFetch; + }); + + test("abort during the wait interrupts the sleep, cancels the 429 body, and returns 499 without replaying", async () => { + let sends = 0; + let upstreamBodyCancelled = false; + globalThis.fetch = (async (input, init) => { + const url = input instanceof Request ? input.url : String(input); + if (url === "https://llmapi.blsc.cn/chat/completions") { + sends += 1; + return new Response(new ReadableStream({ + start(controller) { + controller.enqueue(new TextEncoder().encode(JSON.stringify({ error: { message: "rate limited" } }))); + controller.close(); + }, + cancel() { + upstreamBodyCancelled = true; + }, + }), { status: 429, headers: { "content-type": "application/json" } }); + } + return originalFetch(input, init); + }) as typeof fetch; + + const config = { + port: 0, + defaultProvider: "blsc", + providers: { + blsc: { + adapter: "openai-chat", + baseUrl: "https://llmapi.blsc.cn", + authMode: "key", + apiKey: "key-alpha-000111222333", + retryOn429: { attempts: 3, intervalMs: 30_000, respectRetryAfter: false }, + }, + }, + } as OcxConfig; + + const abort = new AbortController(); + const pending = handleResponses(new Request("http://localhost/v1/responses", { + method: "POST", + headers: { "content-type": "application/json" }, + body: JSON.stringify({ model: "blsc/DeepSeek-V4-Flash", input: "hello", stream: false }), + }), config, { model: "blsc/DeepSeek-V4-Flash", provider: "blsc" }, { abortSignal: abort.signal }); + + // Wait until the first upstream 429 lands and the retry sleep is running. + for (let i = 0; i < 100 && sends === 0; i += 1) await Bun.sleep(10); + expect(sends).toBe(1); + + abort.abort(new DOMException("client disconnected", "AbortError")); + const response = await pending; + expect(response.status).toBe(499); + expect(sends).toBe(1); + expect(upstreamBodyCancelled).toBe(true); + const body = await response.json() as { error?: { code?: string } }; + expect(body.error?.code).toBe("client_cancelled"); + }); + + test("a never-settling 429 body cancel() cannot block the abort-aware backoff", async () => { + let sends = 0; + let cancelInitiated = false; + globalThis.fetch = (async (input, init) => { + const url = input instanceof Request ? input.url : String(input); + if (url === "https://llmapi.blsc.cn/chat/completions") { + sends += 1; + return new Response(new ReadableStream({ + start(controller) { + controller.enqueue(new TextEncoder().encode(JSON.stringify({ error: { message: "rate limited" } }))); + controller.close(); + }, + cancel() { + cancelInitiated = true; + // Never settles: the release must be bounded or the retry loop hangs here. + return new Promise(() => {}); + }, + }), { status: 429, headers: { "content-type": "application/json" } }); + } + return originalFetch(input, init); + }) as typeof fetch; + + const config = { + port: 0, + defaultProvider: "blsc", + providers: { + blsc: { + adapter: "openai-chat", + baseUrl: "https://llmapi.blsc.cn", + authMode: "key", + apiKey: "key-alpha-000111222333", + retryOn429: { attempts: 3, intervalMs: 30_000, respectRetryAfter: false }, + }, + }, + } as OcxConfig; + + const abort = new AbortController(); + const pending = handleResponses(new Request("http://localhost/v1/responses", { + method: "POST", + headers: { "content-type": "application/json" }, + body: JSON.stringify({ model: "blsc/DeepSeek-V4-Flash", input: "hello", stream: false }), + }), config, { model: "blsc/DeepSeek-V4-Flash", provider: "blsc" }, { abortSignal: abort.signal }); + + for (let i = 0; i < 100 && sends === 0; i += 1) await Bun.sleep(10); + expect(sends).toBe(1); + + abort.abort(new DOMException("client disconnected", "AbortError")); + const started = Date.now(); + const response = await pending; + expect(Date.now() - started).toBeLessThan(2_000); + expect(response.status).toBe(499); + expect(sends).toBe(1); + expect(cancelInitiated).toBe(true); + }); +}); diff --git a/tests/server-rate-limit-retry-e2e.test.ts b/tests/server-rate-limit-retry-e2e.test.ts new file mode 100644 index 000000000..6a6d7f4c6 --- /dev/null +++ b/tests/server-rate-limit-retry-e2e.test.ts @@ -0,0 +1,358 @@ +import { afterEach, beforeEach, describe, expect, test } from "bun:test"; +import { mkdtempSync, rmSync } from "node:fs"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import { saveConfig } from "../src/config"; +import { clearKeyCooldowns } from "../src/providers/key-failover"; +import { startServer } from "../src/server"; +import type { OcxConfig } from "../src/types"; +import { installIsolatedCodexHome, type IsolatedCodexHome } from "./helpers/isolated-codex-home"; + +let testDir = ""; +let previousHome: string | undefined; +let isolatedCodexHome: IsolatedCodexHome | null = null; + +beforeEach(() => { + previousHome = process.env.OPENCODEX_HOME; + isolatedCodexHome = installIsolatedCodexHome("ocx-ratelimit-e2e-codex-"); + testDir = mkdtempSync(join(tmpdir(), "ocx-ratelimit-e2e-")); + process.env.OPENCODEX_HOME = testDir; + clearKeyCooldowns(); +}); + +afterEach(() => { + if (previousHome === undefined) delete process.env.OPENCODEX_HOME; + else process.env.OPENCODEX_HOME = previousHome; + isolatedCodexHome?.restore(); + isolatedCodexHome = null; + if (testDir) rmSync(testDir, { recursive: true, force: true }); + clearKeyCooldowns(); +}); + +const okChatCompletion = JSON.stringify({ + id: "chatcmpl-ratelimit", + object: "chat.completion", + choices: [{ index: 0, message: { role: "assistant", content: "ok after retry" }, finish_reason: "stop" }], + usage: { prompt_tokens: 3, completion_tokens: 2, total_tokens: 5 }, +}); + +/** POST a non-streaming `/v1/responses` request to the proxy under test. */ +async function postResponses(serverUrl: string, model: string): Promise { + return fetch(new URL("/v1/responses", serverUrl), { + method: "POST", + headers: { "content-type": "application/json" }, + body: JSON.stringify({ model, input: "hello", stream: false }), + }); +} + +describe("server same-target 429 retry (end-to-end)", () => { + test("single-key provider replays the identical request until upstream succeeds", async () => { + const originalFetch = globalThis.fetch; + const seenBodies: string[] = []; + const seenHeaders: string[][] = []; + globalThis.fetch = (async (input, init) => { + const url = input instanceof Request ? input.url : String(input); + if (url === "https://llmapi.blsc.cn/chat/completions") { + seenBodies.push(String(init?.body)); + seenHeaders.push( + [...new Headers(init?.headers).entries()] + .sort(([a], [b]) => a.localeCompare(b)) + .flat(), + ); + if (seenBodies.length <= 2) { + return new Response(JSON.stringify({ error: { message: "rate limited" } }), { + status: 429, + headers: { "retry-after": "1", "content-type": "application/json" }, + }); + } + return new Response(okChatCompletion, { headers: { "content-type": "application/json" } }); + } + return originalFetch(input, init); + }) as typeof fetch; + + let server: ReturnType | null = null; + try { + const config = { + port: 0, + hostname: "127.0.0.1", + defaultProvider: "blsc", + providers: { + blsc: { + adapter: "openai-chat", + baseUrl: "https://llmapi.blsc.cn", + authMode: "key", + apiKey: "key-alpha-000111222333", + retryOn429: { attempts: 2, intervalMs: 120, respectRetryAfter: false }, + }, + }, + } as OcxConfig; + saveConfig(config); + server = startServer(0); + const res = await postResponses(server.url, "blsc/DeepSeek-V4-Flash"); + expect(res.status).toBe(200); + const json = await res.json() as { output?: { type: string; content?: { text?: string }[] }[] }; + expect(json.output?.find(o => o.type === "message")?.content?.[0]?.text).toBe("ok after retry"); + expect(seenBodies).toHaveLength(3); + expect(seenBodies[0]).toBe(seenBodies[1]); + expect(seenBodies[1]).toBe(seenBodies[2]); + // Same-target replays reuse the ONE built request: full header set identical too. + expect(seenHeaders).toHaveLength(3); + expect(seenHeaders[0]).toEqual(seenHeaders[1]); + expect(seenHeaders[1]).toEqual(seenHeaders[2]); + } finally { + server?.stop(true); + globalThis.fetch = originalFetch; + } + }); + + test("without retryOn429 the 429 surfaces immediately with Retry-After", async () => { + const originalFetch = globalThis.fetch; + let sends = 0; + globalThis.fetch = (async (input, init) => { + const url = input instanceof Request ? input.url : String(input); + if (url === "https://llmapi.blsc.cn/chat/completions") { + sends += 1; + return new Response(JSON.stringify({ error: { message: "rate limited" } }), { + status: 429, + headers: { "retry-after": "17", "content-type": "application/json" }, + }); + } + return originalFetch(input, init); + }) as typeof fetch; + + let server: ReturnType | null = null; + try { + const config = { + port: 0, + hostname: "127.0.0.1", + defaultProvider: "blsc", + providers: { + blsc: { + adapter: "openai-chat", + baseUrl: "https://llmapi.blsc.cn", + authMode: "key", + apiKey: "key-alpha-000111222333", + }, + }, + } as OcxConfig; + saveConfig(config); + server = startServer(0); + const res = await postResponses(server.url, "blsc/DeepSeek-V4-Flash"); + expect(res.status).toBe(429); + expect(res.headers.get("retry-after")).toBe("17"); + const json = await res.json() as { error?: { type?: string } }; + expect(json.error?.type).toBe("rate_limit_error"); + expect(sends).toBe(1); + } finally { + server?.stop(true); + globalThis.fetch = originalFetch; + } + }); + + test("exhausted attempts surface the 429", async () => { + const originalFetch = globalThis.fetch; + let sends = 0; + globalThis.fetch = (async (input, init) => { + const url = input instanceof Request ? input.url : String(input); + if (url === "https://llmapi.blsc.cn/chat/completions") { + sends += 1; + return new Response(JSON.stringify({ error: { message: "rate limited" } }), { + status: 429, + headers: { "retry-after": "1", "content-type": "application/json" }, + }); + } + return originalFetch(input, init); + }) as typeof fetch; + + let server: ReturnType | null = null; + try { + const config = { + port: 0, + hostname: "127.0.0.1", + defaultProvider: "blsc", + providers: { + blsc: { + adapter: "openai-chat", + baseUrl: "https://llmapi.blsc.cn", + authMode: "key", + apiKey: "key-alpha-000111222333", + retryOn429: { attempts: 1, intervalMs: 120, respectRetryAfter: false }, + }, + }, + } as OcxConfig; + saveConfig(config); + server = startServer(0); + const res = await postResponses(server.url, "blsc/DeepSeek-V4-Flash"); + expect(res.status).toBe(429); + expect(sends).toBe(2); + } finally { + server?.stop(true); + globalThis.fetch = originalFetch; + } + }); + + test("same-key retries run before multi-key failover, which still works after they exhaust", async () => { + const originalFetch = globalThis.fetch; + const seenAuth: string[] = []; + globalThis.fetch = (async (input, init) => { + const url = input instanceof Request ? input.url : String(input); + if (url === "https://llmapi.blsc.cn/chat/completions") { + const auth = new Headers(init?.headers).get("authorization") ?? ""; + seenAuth.push(auth); + if (auth.includes("key-beta")) { + return new Response(okChatCompletion, { headers: { "content-type": "application/json" } }); + } + return new Response(JSON.stringify({ error: { message: "rate limited" } }), { + status: 429, + headers: { "retry-after": "1", "content-type": "application/json" }, + }); + } + return originalFetch(input, init); + }) as typeof fetch; + + let server: ReturnType | null = null; + try { + const config = { + port: 0, + hostname: "127.0.0.1", + defaultProvider: "blsc", + providers: { + blsc: { + adapter: "openai-chat", + baseUrl: "https://llmapi.blsc.cn", + authMode: "key", + apiKey: "key-alpha-000111222333", + apiKeyPool: [ + { id: "k1", key: "key-alpha-000111222333", addedAt: 1 }, + { id: "k2", key: "key-beta-444555666777", addedAt: 2 }, + ], + retryOn429: { attempts: 1, intervalMs: 120, respectRetryAfter: false }, + }, + }, + } as OcxConfig; + saveConfig(config); + server = startServer(0); + const res = await postResponses(server.url, "blsc/DeepSeek-V4-Flash"); + expect(res.status).toBe(200); + expect(seenAuth).toEqual([ + "Bearer key-alpha-000111222333", + "Bearer key-alpha-000111222333", + "Bearer key-beta-444555666777", + ]); + } finally { + server?.stop(true); + globalThis.fetch = originalFetch; + } + }); + + test("key-auth openai-responses passthrough replays 429 on the same key", async () => { + const originalFetch = globalThis.fetch; + let sends = 0; + const seenAuth: string[] = []; + const seenBodies: string[] = []; + globalThis.fetch = (async (input, init) => { + const url = input instanceof Request ? input.url : String(input); + if (url === "https://passthrough.test/v1/responses") { + sends += 1; + const auth = new Headers(init?.headers).get("authorization") ?? ""; + seenAuth.push(auth); + seenBodies.push(String(init?.body)); + if (sends === 1) { + return new Response(JSON.stringify({ error: { message: "rate limited" } }), { + status: 429, + headers: { "content-type": "application/json" }, + }); + } + return new Response(JSON.stringify({ + id: "resp-ok", + object: "response", + status: "completed", + output: [{ type: "message", role: "assistant", content: [{ type: "output_text", text: "ok after retry" }] }], + }), { status: 200, headers: { "content-type": "application/json" } }); + } + return originalFetch(input, init); + }) as typeof fetch; + + let server: ReturnType | null = null; + try { + const config = { + port: 0, + hostname: "127.0.0.1", + defaultProvider: "passthrough", + providers: { + passthrough: { + adapter: "openai-responses", + baseUrl: "https://passthrough.test/v1", + authMode: "key", + apiKey: "key-alpha-000111222333", + retryOn429: { attempts: 2, intervalMs: 120, respectRetryAfter: false }, + }, + }, + } as OcxConfig; + saveConfig(config); + server = startServer(0); + const res = await postResponses(server.url, "passthrough/model"); + expect(res.status).toBe(200); + const text = await res.text(); + expect(text).toContain("ok after retry"); + expect(sends).toBe(2); + expect(seenAuth).toEqual([ + "Bearer key-alpha-000111222333", + "Bearer key-alpha-000111222333", + ]); + expect(seenBodies).toHaveLength(2); + expect(seenBodies[0]).toBe(seenBodies[1]); + } finally { + server?.stop(true); + globalThis.fetch = originalFetch; + } + }); + + test("retry budget stays per request across multi-key failover (never re-arms)", async () => { + const originalFetch = globalThis.fetch; + let sends = 0; + globalThis.fetch = (async (input, init) => { + const url = input instanceof Request ? input.url : String(input); + if (url === "https://llmapi.blsc.cn/chat/completions") { + sends += 1; + return new Response(JSON.stringify({ error: { message: "rate limited" } }), { + status: 429, + headers: { "content-type": "application/json" }, + }); + } + return originalFetch(input, init); + }) as typeof fetch; + + let server: ReturnType | null = null; + try { + const config = { + port: 0, + hostname: "127.0.0.1", + defaultProvider: "blsc", + providers: { + blsc: { + adapter: "openai-chat", + baseUrl: "https://llmapi.blsc.cn", + authMode: "key", + apiKey: "key-alpha-000111222333", + apiKeyPool: [ + { id: "k1", key: "key-alpha-000111222333", addedAt: 1 }, + { id: "k2", key: "key-beta-444555666777", addedAt: 2 }, + ], + retryOn429: { attempts: 1, intervalMs: 120, respectRetryAfter: false }, + }, + }, + } as OcxConfig; + saveConfig(config); + server = startServer(0); + const res = await postResponses(server.url, "blsc/DeepSeek-V4-Flash"); + expect(res.status).toBe(429); + // attempts(1) on the first key + one failover key = 3 sends; a re-armed budget would + // have replayed on the second key too (4+ sends). + expect(sends).toBe(3); + } finally { + server?.stop(true); + globalThis.fetch = originalFetch; + } + }); +}); diff --git a/tests/terminal-guard-server.test.ts b/tests/terminal-guard-server.test.ts index f85a3cddf..ba596cb80 100644 --- a/tests/terminal-guard-server.test.ts +++ b/tests/terminal-guard-server.test.ts @@ -14,6 +14,7 @@ const config = { }, } as unknown as OcxConfig; +/** Build an Anthropic SSE response from raw frames. */ function anthropicSse(body: string): Response { return new Response(body, { status: 200, headers: { "content-type": "text/event-stream" } }); } @@ -78,4 +79,242 @@ describe("server terminal guard integration", () => { expect(messages.at(-1)?.content?.[0]?.text).toContain("你刚才只描述了计划"); }); + test("terminal-guard continuation 429 replays on the same key before surfacing", async () => { + const retryConfig = { + ...config, + providers: { + "claude-se": { + adapter: "anthropic", + baseUrl: "https://example.test", + apiKey: "sk-test", + retryOn429: { attempts: 1, intervalMs: 120, respectRetryAfter: false }, + }, + }, + } as unknown as OcxConfig; + let sends = 0; + const requestBodies: string[] = []; + globalThis.fetch = (async (_input, init) => { + sends += 1; + requestBodies.push(String(init?.body ?? "")); + if (sends === 2) { + return new Response(JSON.stringify({ error: { message: "rate limited" } }), { + status: 429, + headers: { "content-type": "application/json" }, + }); + } + return anthropicSse(sends === 1 ? firstTurn : continuationTurn); + }) as typeof fetch; + + const response = await handleResponses(new Request("http://localhost/v1/responses", { + method: "POST", + headers: { "content-type": "application/json" }, + body: JSON.stringify({ + model: "se-claude-opus-4.8", + input: "请检查这个问题并修复代码", + stream: true, + tools: [{ type: "function", name: "exec_command", description: "run a command", parameters: { type: "object" } }], + }), + }), retryConfig, { model: "", provider: "" }); + + const text = await response.text(); + expect(response.status).toBe(200); + // initial turn + 429 continuation + replayed continuation + expect(sends).toBe(3); + expect(text).toContain("response.completed"); + expect(text).toContain("exec_command"); + // The 429 continuation (send 2) and its same-key replay (send 3) must be byte-identical. + expect(requestBodies).toHaveLength(3); + expect(requestBodies[1]).toBe(requestBodies[2]); + }); + + test("terminal-guard continuation retry budget stays per request across key failover", async () => { + const budgetConfig = { + ...config, + providers: { + "claude-se": { + adapter: "anthropic", + baseUrl: "https://example.test", + apiKey: "sk-test", + apiKeyPool: [ + { id: "k1", key: "sk-test", addedAt: 1 }, + { id: "k2", key: "sk-test-2", addedAt: 2 }, + ], + retryOn429: { attempts: 1, intervalMs: 120, respectRetryAfter: false }, + }, + }, + } as unknown as OcxConfig; + let sends = 0; + globalThis.fetch = (async (_input, init) => { + sends += 1; + if (sends === 1) { + return anthropicSse(firstTurn); + } + return new Response(JSON.stringify({ error: { message: "rate limited" } }), { + status: 429, + headers: { "content-type": "application/json" }, + }); + }) as typeof fetch; + + const response = await handleResponses(new Request("http://localhost/v1/responses", { + method: "POST", + headers: { "content-type": "application/json" }, + body: JSON.stringify({ + model: "se-claude-opus-4.8", + input: "请检查这个问题并修复代码", + stream: true, + tools: [{ type: "function", name: "exec_command", description: "run a command", parameters: { type: "object" } }], + }), + }), budgetConfig, { model: "", provider: "" }); + + await response.text(); + // initial turn + continuation retry (same key) + failover continuation (second key) = 4. + // A per-iteration budget would replay on the second key too (5+ sends). + expect(sends).toBe(4); + }); + + test("terminal-guard continuation shares the request-wide 429 budget with the main loop", async () => { + const budgetConfig = { + ...config, + providers: { + "claude-se": { + adapter: "anthropic", + baseUrl: "https://example.test", + apiKey: "sk-test", + retryOn429: { attempts: 1, intervalMs: 120, respectRetryAfter: false }, + }, + }, + } as unknown as OcxConfig; + let sends = 0; + globalThis.fetch = (async () => { + sends += 1; + if (sends === 1) { + // The main recovery loop consumes the only same-key replay... + return new Response(JSON.stringify({ error: { message: "rate limited" } }), { + status: 429, + headers: { "content-type": "application/json" }, + }); + } + if (sends === 2) { + // ...the replay succeeds and the terminal-guard continuation starts. + return anthropicSse(firstTurn); + } + // Continuation 429 with the request budget already spent: surfaces, never replays. + return new Response(JSON.stringify({ error: { message: "rate limited" } }), { + status: 429, + headers: { "content-type": "application/json" }, + }); + }) as typeof fetch; + + const response = await handleResponses(new Request("http://localhost/v1/responses", { + method: "POST", + headers: { "content-type": "application/json" }, + body: JSON.stringify({ + model: "se-claude-opus-4.8", + input: "请检查这个问题并修复代码", + stream: true, + tools: [{ type: "function", name: "exec_command", description: "run a command", parameters: { type: "object" } }], + }), + }), budgetConfig, { model: "", provider: "" }); + + const text = await response.text(); + // main 429 -> same-key replay -> continuation 429 (no budget left) = exactly 3 sends. + expect(sends).toBe(3); + expect(text).toContain("Provider continuation error 429"); + }); + + test("terminal-guard continuation abort during the 429 wait yields 499 without replaying", async () => { + const abortConfig = { + ...config, + providers: { + "claude-se": { + adapter: "anthropic", + baseUrl: "https://example.test", + apiKey: "sk-test", + retryOn429: { attempts: 3, intervalMs: 30_000, respectRetryAfter: false }, + }, + }, + } as unknown as OcxConfig; + let sends = 0; + globalThis.fetch = (async () => { + sends += 1; + if (sends === 1) { + return anthropicSse(firstTurn); + } + return new Response(JSON.stringify({ error: { message: "rate limited" } }), { + status: 429, + headers: { "content-type": "application/json" }, + }); + }) as typeof fetch; + + const abort = new AbortController(); + const pending = handleResponses(new Request("http://localhost/v1/responses", { + method: "POST", + headers: { "content-type": "application/json" }, + body: JSON.stringify({ + model: "se-claude-opus-4.8", + input: "请检查这个问题并修复代码", + stream: true, + tools: [{ type: "function", name: "exec_command", description: "run a command", parameters: { type: "object" } }], + }), + }), abortConfig, { model: "", provider: "" }, { abortSignal: abort.signal }); + + // The terminal-guard continuation runs inside the SSE producer, so consume the body to + // drive it, then wait until the continuation's 429 lands and its retry sleep is running. + const response = await pending; + const textPromise = response.text(); + for (let i = 0; i < 100 && sends < 2; i += 1) await Bun.sleep(10); + expect(sends).toBe(2); + abort.abort(new DOMException("client disconnected", "AbortError")); + const text = await textPromise; + expect(text).toContain("client closed request during terminal continuation"); + expect(sends).toBe(2); + }); + + test("terminal-guard 429 wait longer than the stall budget still succeeds (heartbeats)", async () => { + const stallConfig = { + ...config, + stallTimeoutSec: 1, + providers: { + "claude-se": { + adapter: "anthropic", + baseUrl: "https://example.test", + apiKey: "sk-test", + retryOn429: { attempts: 1, intervalMs: 1_500, respectRetryAfter: false }, + }, + }, + } as unknown as OcxConfig; + let sends = 0; + globalThis.fetch = (async () => { + sends += 1; + if (sends === 1) { + return anthropicSse(firstTurn); + } + if (sends === 2) { + return new Response(JSON.stringify({ error: { message: "rate limited" } }), { + status: 429, + headers: { "content-type": "application/json" }, + }); + } + return anthropicSse(continuationTurn); + }) as typeof fetch; + + const response = await handleResponses(new Request("http://localhost/v1/responses", { + method: "POST", + headers: { "content-type": "application/json" }, + body: JSON.stringify({ + model: "se-claude-opus-4.8", + input: "请检查这个问题并修复代码", + stream: true, + tools: [{ type: "function", name: "exec_command", description: "run a command", parameters: { type: "object" } }], + }), + }), stallConfig, { model: "", provider: "" }); + + const text = await response.text(); + // A 1.5s continuation backoff under a 1s stall budget must not trip upstream_stall_timeout: + // the wait yields heartbeat events, the replay lands, and the turn completes. + expect(sends).toBe(3); + expect(text).toContain("response.completed"); + expect(text).not.toContain("upstream_stall_timeout"); + }, 5_000); + }); diff --git a/tests/upstream-retry.test.ts b/tests/upstream-retry.test.ts index b1f4d4b65..db6a3df5b 100644 --- a/tests/upstream-retry.test.ts +++ b/tests/upstream-retry.test.ts @@ -2,7 +2,10 @@ import { afterEach, describe, expect, spyOn, test } from "bun:test"; import { fetchWithResetRetry, isConnectionResetError, + prepareSameTarget429Wait, + releaseResponseBodyBestEffort, retryBackoffDelayMs, + sleepWithHeartbeats, } from "../src/lib/upstream-retry"; function bunResetError(): Error { @@ -60,6 +63,86 @@ describe("isConnectionResetError", () => { }); }); +describe("sleepWithHeartbeats", () => { + test("a non-positive heartbeat interval is clamped instead of spinning forever", async () => { + const events: string[] = []; + for await (const event of sleepWithHeartbeats(3, undefined, 0)) { + events.push(event.type); + } + // 3ms of wait with a clamped 1ms step -> exactly 3 beats, then termination (no spin). + expect(events).toHaveLength(3); + }); + + test("a NaN heartbeat interval waits the full duration instead of aborting after one beat", async () => { + const started = Date.now(); + const events: string[] = []; + for await (const event of sleepWithHeartbeats(120, undefined, Number.NaN)) { + events.push(event.type); + } + // NaN falls back to the 1ms step: the full 120ms wait happens (120 beats), instead of the + // buggy NaN-chunk path that exited after one beat. + expect(events).toHaveLength(120); + expect(Date.now() - started).toBeGreaterThanOrEqual(110); + }); + + test("zero wait yields nothing", async () => { + const events: string[] = []; + for await (const event of sleepWithHeartbeats(0, undefined)) { + events.push(event.type); + } + expect(events).toEqual([]); + }); +}); + +describe("releaseResponseBodyBestEffort", () => { + test("a never-settling cancel() does not block past the bounded timeout", async () => { + const signal = new AbortController().signal; + const body = new ReadableStream({ + cancel() { + // Never settles — the release must still be bounded. + return new Promise(() => {}); + }, + }); + const started = Date.now(); + await releaseResponseBodyBestEffort(body, signal, 120); + const elapsed = Date.now() - started; + expect(elapsed).toBeGreaterThanOrEqual(100); + expect(elapsed).toBeLessThan(1_000); + }); + + test("a never-settling cancel() resolves immediately when the signal aborts", async () => { + const controller = new AbortController(); + const body = new ReadableStream({ + cancel() { + return new Promise(() => {}); + }, + }); + const pending = releaseResponseBodyBestEffort(body, controller.signal, 60_000); + controller.abort(new DOMException("client disconnected", "AbortError")); + const started = Date.now(); + await pending; + expect(Date.now() - started).toBeLessThan(500); + }); + + test("an already-aborted signal initiates cancellation without awaiting it", async () => { + const controller = new AbortController(); + controller.abort(); + let cancelInitiated = false; + const body = new ReadableStream({ + cancel() { + cancelInitiated = true; + return new Promise(() => {}); + }, + }); + await releaseResponseBodyBestEffort(body, controller.signal, 60_000); + expect(cancelInitiated).toBe(true); + }); + + test("null body is a no-op", async () => { + await expect(releaseResponseBodyBestEffort(null, new AbortController().signal, 10)).resolves.toBeUndefined(); + }); +}); + describe("fetchWithResetRetry", () => { test("retries a Bun-shaped reset and returns the second attempt's response", async () => { silenceWarn(); @@ -179,3 +262,44 @@ describe("retryBackoffDelayMs", () => { } }); }); + + +describe("prepareSameTarget429Wait", () => { + test("releases the body then waits without heartbeats when no interval is set", async () => { + let cancelled = false; + const body = new ReadableStream({ + cancel() { + cancelled = true; + }, + }); + const events: string[] = []; + const started = Date.now(); + for await (const event of prepareSameTarget429Wait({ + body, + delayMs: 40, + })) { + events.push(event.type); + } + expect(cancelled).toBe(true); + expect(events).toEqual([]); + expect(Date.now() - started).toBeGreaterThanOrEqual(30); + }); + + test("yields heartbeats when a heartbeat interval is provided", async () => { + const body = new ReadableStream({ + cancel() { + return; + }, + }); + const events: string[] = []; + for await (const event of prepareSameTarget429Wait({ + body, + delayMs: 30, + heartbeatIntervalMs: 10, + })) { + events.push(event.type); + } + expect(events.length).toBeGreaterThanOrEqual(2); + expect(events.every(type => type === "heartbeat")).toBe(true); + }); +}); diff --git a/tests/usage-log.test.ts b/tests/usage-log.test.ts index 499c2e490..370df85e0 100644 --- a/tests/usage-log.test.ts +++ b/tests/usage-log.test.ts @@ -18,6 +18,7 @@ import { usageTotalTokens, usageReadCacheStatsForTests, usageLogRevisionKey, + type PersistedUsageEntry, } from "../src/usage/log"; let testDir = ""; @@ -37,6 +38,32 @@ afterEach(() => { }); describe("usage log", () => { + test("persists the rate-limit-429 recovery kind on attempts", () => { + const entry: PersistedUsageEntry = { + requestId: "ocx-ratelimit-kind", + timestamp: 1, + provider: "blsc", + model: "blsc/DeepSeek-V4-Flash", + status: 429, + durationMs: 4, + usageStatus: "reported", + attempts: [{ + ordinal: 1, + provider: "blsc", + model: "blsc/DeepSeek-V4-Flash", + adapter: "openai-chat", + status: 429, + durationMs: 4, + sendCount: 2, + recoveryKinds: ["rate-limit-429", "rate-limit-429"], + usageStatus: "reported", + }], + }; + appendUsageEntry(entry); + expect(readUsageEntries()[0]?.attempts?.[0]?.recoveryKinds).toEqual(["rate-limit-429"]); + }); + + /** Build one minimal persisted-usage JSONL line for the given request id. */ const persistedLine = (requestId: string) => JSON.stringify({ requestId, timestamp: 1, diff --git a/tests/web-search.test.ts b/tests/web-search.test.ts index 6727fad22..1202815a7 100644 --- a/tests/web-search.test.ts +++ b/tests/web-search.test.ts @@ -11,6 +11,7 @@ import type { OcxMessage, OcxParsedRequest } from "../src/types"; import { fakeChatGptJwt } from "./helpers/fake-chatgpt-jwt"; import { createTestTranslatorBudget } from "./helpers/translator-budget"; +/** Run the web-search loop with a default test translator budget. */ function runWithWebSearch( deps: Omit & { incomingMeta?: WebSearchLoopDeps["incomingMeta"] }, ): Promise { @@ -653,6 +654,218 @@ describe("web-search sidecar native web_search_call emission", () => { ]); }); + test("retryOn429 replays on the same key before on429 rotation", async () => { + globalThis.fetch = (() => Promise.resolve(new Response( + 'event: response.completed\ndata: {"type":"response.completed"}\n\n', + { headers: { "Content-Type": "text/event-stream" } }, + ))) as typeof fetch; + + let sends = 0; + let rotations = 0; + let retrySends = 0; + let builds = 0; + const retryingAdapter: ProviderAdapter = { + name: "mock-retry429", + buildRequest: () => { + builds += 1; + return { + url: "https://routed.test/v1", + method: "POST", + headers: {}, + body: "{}", + }; + }, + fetchResponse: async () => { + sends += 1; + if (sends === 1) { + return new Response("rate limited", { status: 429, headers: { "retry-after": "30" } }); + } + return new Response("{}", { status: 200 }); + }, + async *parseStream() { + yield { type: "text_delta", text: "answer after same-key retry" }; + yield { type: "done" }; + }, + async parseResponse() { throw new Error("parseResponse must be unreachable"); }, + }; + + const response = await runWithWebSearch({ + parsed: parseRequest({ model: "routed/model", input: "hi", stream: true, tools: [{ type: "web_search" }] }), + adapter: retryingAdapter, + forwardProvider, + hostedTool: { type: "web_search" }, + selectedForwardHeaders: new Headers({ authorization: "Bearer token" }), + settings: { model: "gpt-5.4-mini", reasoning: "low", timeoutMs: 30_000 }, + maxSearches: 1, + retryOn429Policy: { enabled: true, attempts: 2, intervalMs: 120, maxIntervalMs: 60_000, respectRetryAfter: false }, + on429: () => { + rotations += 1; + return null; + }, + onAttemptSend: recovery => { + if (recovery === "rate-limit-429") retrySends += 1; + }, + }); + expect(response.status).toBe(200); + const frames = await collectSse(response.body!); + const completed = frames.find(f => f.event === "response.completed")?.data.response as Record; + const output = completed.output as { type: string; content?: { text?: string }[] }[]; + expect(output.find(o => o.type === "message")?.content?.[0]?.text).toBe("answer after same-key retry"); + expect(sends).toBe(2); + expect(rotations).toBe(0); + expect(retrySends).toBe(1); + // Same-target replay reuses the ONE built request (builder runs once per target sequence). + expect(builds).toBe(1); + }); + + test("retry wait longer than the stall budget still succeeds (heartbeats feed the watchdog)", async () => { + globalThis.fetch = (() => Promise.resolve(new Response( + 'event: response.completed\ndata: {"type":"response.completed"}\n\n', + { headers: { "Content-Type": "text/event-stream" } }, + ))) as typeof fetch; + + let sends = 0; + const retryingAdapter: ProviderAdapter = { + name: "mock-retry429", + buildRequest: () => ({ url: "https://routed.test/v1", method: "POST", headers: {}, body: "{}" }), + fetchResponse: async () => { + sends += 1; + if (sends === 1) { + return new Response("rate limited", { status: 429, headers: { "retry-after": "30" } }); + } + return new Response("{}", { status: 200 }); + }, + async *parseStream() { + yield { type: "text_delta", text: "answer after long backoff" }; + yield { type: "done" }; + }, + async parseResponse() { throw new Error("parseResponse must be unreachable"); }, + }; + + const response = await runWithWebSearch({ + parsed: parseRequest({ model: "routed/model", input: "hi", stream: true, tools: [{ type: "web_search" }] }), + adapter: retryingAdapter, + forwardProvider, + hostedTool: { type: "web_search" }, + selectedForwardHeaders: new Headers({ authorization: "Bearer token" }), + settings: { model: "gpt-5.4-mini", reasoning: "low", timeoutMs: 30_000 }, + maxSearches: 1, + stallTimeoutSec: 1, + retryOn429Policy: { enabled: true, attempts: 1, intervalMs: 1_500, maxIntervalMs: 60_000, respectRetryAfter: false }, + }); + const frames = await collectSse(response.body!); + // A 1.5s backoff under a 1s stall budget must not trip upstream_stall_timeout. + expect(sends).toBe(2); + expect(frames.find(f => f.event === "response.completed")).toBeDefined(); + expect(frames.find(f => f.event === "response.failed")).toBeUndefined(); + }, 5_000); + + test("retry wait longer than connectTimeoutMs restarts the header deadline (no 504)", async () => { + globalThis.fetch = (() => Promise.resolve(new Response( + 'event: response.completed\ndata: {"type":"response.completed"}\n\n', + { headers: { "Content-Type": "text/event-stream" } }, + ))) as typeof fetch; + + let sends = 0; + const retryingAdapter: ProviderAdapter = { + name: "mock-retry429", + buildRequest: () => ({ url: "https://routed.test/v1", method: "POST", headers: {}, body: "{}" }), + fetchResponse: async () => { + sends += 1; + if (sends === 1) { + return new Response("rate limited", { status: 429, headers: { "retry-after": "30" } }); + } + return new Response("{}", { status: 200 }); + }, + async *parseStream() { + yield { type: "text_delta", text: "answer after deadline restart" }; + yield { type: "done" }; + }, + async parseResponse() { throw new Error("parseResponse must be unreachable"); }, + }; + + const response = await runWithWebSearch({ + parsed: parseRequest({ model: "routed/model", input: "hi", stream: true, tools: [{ type: "web_search" }] }), + adapter: retryingAdapter, + forwardProvider, + hostedTool: { type: "web_search" }, + selectedForwardHeaders: new Headers({ authorization: "Bearer token" }), + settings: { model: "gpt-5.4-mini", reasoning: "low", timeoutMs: 30_000 }, + maxSearches: 1, + connectTimeoutMs: 100, + retryOn429Policy: { enabled: true, attempts: 1, intervalMs: 150, maxIntervalMs: 60_000, respectRetryAfter: false }, + }); + const frames = await collectSse(response.body!); + // The deliberate backoff must not consume the response-header deadline: a fresh deadline is + // armed after the wait, so the replay gets a new connect budget instead of a 504. + expect(sends).toBe(2); + expect(frames.find(f => f.event === "response.completed")).toBeDefined(); + expect(frames.find(f => f.event === "response.failed")).toBeUndefined(); + }, 5_000); + + test("retryOn429 budget is shared across iterations (per request, not per round)", async () => { + globalThis.fetch = ((input) => { + const url = String(input); + if (url.startsWith("https://routed.test/")) return Promise.resolve(new Response("{}", { status: 200 })); + // sidecar /responses: return a minimal completed SSE so the search round advances. + return Promise.resolve(new Response( + 'event: response.completed\ndata: {"type":"response.completed"}\n\n', + { headers: { "Content-Type": "text/event-stream" } }, + )); + }) as typeof fetch; + + let sends = 0; + let retrySends = 0; + let rotations = 0; + const retryingAdapter: ProviderAdapter = { + name: "mock-retry429", + buildRequest: () => ({ url: "https://routed.test/v1", method: "POST", headers: {}, body: "{}" }), + fetchResponse: async () => { + sends += 1; + if (sends === 1 || sends === 3) { + return new Response("rate limited", { status: 429, headers: { "retry-after": "30" } }); + } + return new Response("{}", { status: 200 }); + }, + async *parseStream() { + if (sends === 2) { + // Round 0 success carries a web_search call so the loop advances to a forced-answer round. + yield { type: "tool_call_start", id: "call_1", name: "web_search" }; + yield { type: "tool_call_delta", arguments: JSON.stringify({ query: "current docs" }) }; + yield { type: "tool_call_end" }; + } else { + yield { type: "text_delta", text: "unused" }; + } + yield { type: "done" }; + }, + async parseResponse() { throw new Error("parseResponse must be unreachable"); }, + }; + + const response = await runWithWebSearch({ + parsed: parseRequest({ model: "routed/model", input: "hi", stream: true, tools: [{ type: "web_search" }] }), + adapter: retryingAdapter, + forwardProvider, + hostedTool: { type: "web_search" }, + selectedForwardHeaders: new Headers({ authorization: "Bearer token" }), + settings: { model: "gpt-5.4-mini", reasoning: "low", timeoutMs: 30_000 }, + maxSearches: 1, + retryOn429Policy: { enabled: true, attempts: 1, intervalMs: 50, maxIntervalMs: 60_000, respectRetryAfter: false }, + on429: () => { + rotations += 1; + return null; + }, + onAttemptSend: recovery => { + if (recovery === "rate-limit-429") retrySends += 1; + }, + }); + const frames = await collectSse(response.body!); + expect(sends).toBe(3); + expect(retrySends).toBe(1); + expect(rotations).toBe(1); + const failed = frames.find(f => f.event === "response.failed")?.data.response as { error?: { message?: string } } | undefined; + expect(failed?.error?.message ?? "").toContain("429"); + }); + test("loop 429 with exhausted pool (on429 null) surfaces the provider error", async () => { const firstAdapter: ProviderAdapter = { name: "mock-429", diff --git a/tests/xai-transport.test.ts b/tests/xai-transport.test.ts index ed71080f8..e999f1763 100644 --- a/tests/xai-transport.test.ts +++ b/tests/xai-transport.test.ts @@ -294,12 +294,16 @@ describe("xAI outbound compatibility headers", () => { ]) expect(seen[0].has(name)).toBe(false); }); - test("same resolved transport refreshes req-id but keeps conv-id stable", async () => { + test("same resolved transport pins one req-id per logical request (identical replays) and keeps conv-id stable", async () => { const { seen } = await capture("oauth", 2); expect(seen).toHaveLength(2); expect(seen[0].get("x-grok-req-id")).toMatch(UUID_V4); expect(seen[1].get("x-grok-req-id")).toMatch(UUID_V4); - expect(seen[1].get("x-grok-req-id")).not.toBe(seen[0].get("x-grok-req-id")); + // A same-target 429 replay must be byte-identical, including x-grok-req-id; a new resolve + // (e.g. after key rotation) produces a fresh transport and therefore a fresh id. + expect(seen[1].get("x-grok-req-id")).toBe(seen[0].get("x-grok-req-id")); + const freshTransport = await capture("oauth", 1); + expect(freshTransport.seen[0].get("x-grok-req-id")).not.toBe(seen[0].get("x-grok-req-id")); expect(seen[0].get("x-grok-conv-id")).toBe(deriveXaiConvId("codex-session-abc")); expect(seen[1].get("x-grok-conv-id")).toBe(seen[0].get("x-grok-conv-id")); expect(seen[1].get("x-grok-session-id")).toBe(seen[0].get("x-grok-session-id"));