Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
39 changes: 36 additions & 3 deletions devlog/_plan/260804_router_intelligence/001_pr_stack_status.md
Original file line number Diff line number Diff line change
Expand Up @@ -48,8 +48,8 @@ other; closing one is a maintainer decision and neither is stale.
| RI-06 | `feat/ri-06-health-aware-routing` | `dev` (post-#1012 merge) | `af692bb7a` | #1013 | https://github.com/lidge-jun/opencodex/pull/1013 | MERGED |
| RI-07 | `feat/ri-07-quota-aware-routing` | `dev` (post-#1013 merge) | `1f07c00b8` | #1014 | https://github.com/lidge-jun/opencodex/pull/1014 | MERGED |
| RI-08 | `feat/ri-08-cost-aware-routing` | `dev` (post-#1014 merge) | `410db97e4` | #1015 | https://github.com/lidge-jun/opencodex/pull/1015 | MERGED |
| RI-09 | `feat/ri-09-route-explainability-api` | `dev` (post-#1015 merge) | `d887c1202` | #1016 | https://github.com/lidge-jun/opencodex/pull/1016 | rebased / review in progress |
| RI-10 | `feat/ri-10-routing-intelligence-ui` | `feat/ri-09` head | pending | pending | pending | queued |
| RI-09 | `feat/ri-09-route-explainability-api` | `dev` (post-#1015 merge) | `68d3aa083` | #1016 | https://github.com/lidge-jun/opencodex/pull/1016 | MERGED |
| RI-10 | `feat/ri-10-routing-intelligence-ui` | `dev` (post-#1016 merge; rebasing) | pending | #1018 | https://github.com/lidge-jun/opencodex/pull/1018 | OPEN |

## Per-PR acceptance log

Expand Down Expand Up @@ -270,7 +270,7 @@ other; closing one is a maintainer decision and neither is stale.
(5) assembleCandidateEvidence typed as `OcxConfig` after the RI-08 health/
quota/cost merge.
- Final commit: `d887c120282c8d381f92cd11c1eebe2373a27281`
- PR: #1016 https://github.com/lidge-jun/opencodex/pull/1016
- PR: #1016 https://github.com/lidge-jun/opencodex/pull/1016 (MERGED)
- Verification:
- `bun x tsc --noEmit`: PASSED (0 errors)
- `bun run test tests/route-explainability.test.ts`: 10/10 pass -
Expand All @@ -282,6 +282,39 @@ other; closing one is a maintainer decision and neither is stale.
- `bun run privacy:scan`: passed
- Remaining Low findings: none

### RI-10 - feat/ri-10-routing-intelligence-ui

- Base SHA: `68d3aa0836648ae1d7b592fc5aa3b30146c0886d` (`dev` after #1016 merge)
- Reviewed commit: same as final (author self-review before push)
- Findings (self-review): 4 fixed pre-push -
1. GUI lint: hardcoded "profiles" in an error literal (i18n rule) - now a
key-less status code;
2. GUI lint: setState-in-effect for the initial load - deferred via
setTimeout(0);
3. missing `.checkbox` CSS class - added to styles.css; dry-run/analytics
tables reuse the existing `.tbl` grammar;
4. dev-mode GUI session bootstrap cannot authenticate through the Vite
proxy - the screenshot is captured same-origin against the production
GUI served by the backend instead.
- Final commit: pending (recorded after commit)
- PR: #1018 https://github.com/lidge-jun/opencodex/pull/1018
- Verification:
- `bun x tsc --noEmit`: PASSED (0 errors)
- `bun run lint:gui`: PASSED (0 errors)
- `bun run build:gui`: PASSED (production build + prepare:package)
- `bun run test` (12 routing suites): 258/258 pass
- `bun run privacy:scan`: passed
- docs-site `bun run build`: 216 pages built, PASSED
- Locale parity: compile-checked TKey set (all six locales updated)
- Screenshot: live same-origin capture of `#routing` (profiles + dry-run +
analytics) against a temporary config; uploaded to the PR via comment
attachment
- Environment note: the temporary screenshot backend briefly rewrote the
Codex/Grok fence to port 10200; restored with `ocx ensure` to the live
proxy (10100) and verified. Test processes and temp files cleaned up.
- Remaining Low findings: none
- Remaining Low findings: none
Comment thread
coderabbitai[bot] marked this conversation as resolved.

## Baseline note

The full-suite baseline on this Windows machine did not complete within the
Expand Down
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Original file line number Diff line number Diff line change
Expand Up @@ -84,3 +84,26 @@ selector の後には bare native OpenAI-family id だけを指定できます
- 空ではない `inputModalities` 交差。省略されたメンバー値を `["text"]` として扱います。

コンテキスト メタデータのない裸のリレー ID、または接続されていないモダリティを持つターゲットは、カタログからコンボを削除します。同期によって概要の警告が表示され、ダッシュボードで **注意が必要** とマークされます。コンテキスト メタデータを追加し、モダリティを調整したり、検出可能な互換性のある機能を備えたターゲット モデルを追加したりできます。

## ルーティングポリシープロファイル(`config.routingProfiles`)

明示的に要求された `policy/<id>`(または設定されたエイリアス)が、固定された候補許可リストの中から、ハードな能力要件と決定的で説明可能なスコアリングで選択します。既存のモデル ID が暗黙的にプロファイルを通ることはありません。`candidates`(明示的な許可リスト)、オプションの `alias`、`require`(`minContextWindow`、`minQuotaHeadroom`、`tools`、`imageInput`、`structuredOutput`、`localOnly`、`remoteAllowed`、`encryptedCodexTasks`、`reasoningEffort`、`serviceTier`)、`optimize`(latency/health/cost/quota の重み)、`limits.maxEstimatedCostUsd`、`unknownEvidence`(allow/penalize/exclude)をサポートします。未知はゼロや無料にはなりません。

CLI: `ocx route policy list`、`ocx route policy show <id>`、`ocx route policy dry-run <id> --model-context <tokens> --tools`、`ocx route policy evaluate <id>`。

コンボは明示的な順序・重み付きターゲットのルーティングとフェイルオーバーです。ポリシープロファイルは、候補間の証拠に基づく選択です。

## リクエスト履歴とルーティング分析

- `GET /api/request-history` - 派生インデックス(`routing-history.sqlite`)からのカーソルページング付き全履歴。フィルタ: `provider`、`model`、`requestedModel`、`status`、`conversationId`、`surface`、`inboundProtocol`、`apiKeyId`、`profileId`、`fallback`、`from`、`to`。
- `GET /api/request-history/:requestId/route-decision` - このルートが選ばれた理由(トレース、候補、除外、スコア、プロファイル+リビジョン、実行試行、結果)。
- `GET /api/routing-analytics` - 成功率・失敗率・フォールバック率、p50/p95/p99 の所要時間と TTFT、不完全ストリーム率、クールダウン失敗数、成功あたりの推定コスト、カバレッジ、信頼度、切り捨てフラグ。
- `GET /api/routing-profiles`、`POST /api/routing-profiles/dry-run` - プロファイル参照とドライラン評価(上流への送信なし)。

返される履歴とルート決定ペイロードは、マスク済みのリクエストメタデータのみを公開します(例: 不透明な `apiKeyId` ラベル)。資格情報、生のプロンプト本文、プロバイダのシークレットは含みません。

CLI: `ocx logs explain <request-id>`、`ocx logs rebuild-index`、`ocx logs index-status`。

## 移行

`routingProfiles` は任意の追加設定です。既存の設定ファイルと古い `usage.jsonl` 行はそのまま読み込めます。インデックスは使い捨てで、削除すると次回クエリ時に `usage.jsonl` から自動再構築されます。自動チューニングは行われません。
Original file line number Diff line number Diff line change
Expand Up @@ -82,3 +82,26 @@ combo는 목록에 오를 수 없더라도 계속 직접 라우팅할 수 있습
- 비어 있지 않은 `inputModalities` 교집합. 생략된 member value는 `["text"]`로 취급합니다.

context metadata가 없는 bare relay id이거나 modalities가 서로 겹치지 않는 target이 있으면 combo가 catalog에서 빠집니다. sync는 summary warning을 내고 dashboard는 이를 **Needs attention**으로 표시합니다. context metadata를 추가하거나, modalities를 맞추거나, 발견 가능한 호환 capability를 가진 model을 대상으로 삼으십시오.

## 라우팅 정책 프로필 (`config.routingProfiles`)

명시적으로 요청된 `policy/<id>`(또는 설정된 별칭)가 고정된 후보 허용 목록에서 하드 능력 요구사항과 결정적·설명 가능한 점수로 선택합니다. 기존 모델 ID가 암시적으로 프로필을 통과하지 않습니다. `candidates`(명시적 허용 목록), 선택적 `alias`, `require`(`minContextWindow`, `minQuotaHeadroom`, `tools`, `imageInput`, `structuredOutput`, `localOnly`, `remoteAllowed`, `encryptedCodexTasks`, `reasoningEffort`, `serviceTier`), `optimize`(latency/health/cost/quota 가중치), `limits.maxEstimatedCostUsd`, `unknownEvidence`(allow/penalize/exclude)를 지원합니다. 알 수 없음은 0이나 무료가 되지 않습니다.

CLI: `ocx route policy list`, `ocx route policy show <id>`, `ocx route policy dry-run <id> --model-context <tokens> --tools`, `ocx route policy evaluate <id>`.

콤보는 명시적인 순서·가중치 대상 라우팅 및 장애 조치입니다. 정책 프로필은 후보 간 증거 기반 선택입니다.

## 요청 기록 및 라우팅 분석

- `GET /api/request-history` - 파생 인덱스(`routing-history.sqlite`)에서 커서 페이지네이션으로 전체 기록을 조회. 필터: `provider`, `model`, `requestedModel`, `status`, `conversationId`, `surface`, `inboundProtocol`, `apiKeyId`, `profileId`, `fallback`, `from`, `to`.
- `GET /api/request-history/:requestId/route-decision` - 이 경로가 선택된 이유(추적, 후보, 제외, 점수, 프로필+리비전, 실행 시도, 결과).
- `GET /api/routing-analytics` - 성공/실패/취소/폴백 비율, p50/p95/p99 소요 시간 및 TTFT, 불완전 스트림 비율, 쿨다운 실패 수, 성공 요청당 추정 비용, 커버리지, 신뢰도, 잘림 플래그.
- `GET /api/routing-profiles`, `POST /api/routing-profiles/dry-run` - 프로필 조회와 드라이런 평가(업스트림 전송 없음).

반환되는 히스토리와 라우트 결정 페이로드는 마스킹된 요청 메타데이터만 노출합니다(예: 불투명한 `apiKeyId` 라벨). 자격 증명, 원본 프롬프트 본문, 공급자 시크릿은 포함하지 않습니다.

CLI: `ocx logs explain <request-id>`, `ocx logs rebuild-index`, `ocx logs index-status`.

## 마이그레이션

`routingProfiles`는 선택적 추가 설정입니다. 기존 설정 파일과 이전 `usage.jsonl` 행은 그대로 읽힙니다. 인덱스는 일회용이며 삭제 시 다음 쿼리에서 `usage.jsonl`로 자동 재구축됩니다. 자동 튜닝은 없습니다.
33 changes: 33 additions & 0 deletions docs-site/src/content/docs/reference/configuration/routing.md
Original file line number Diff line number Diff line change
Expand Up @@ -189,3 +189,36 @@ Codex picker list it only when every target exposes capabilities that can be int
A bare relay id with no context metadata or targets with disjoint modalities removes the combo from
the catalog. Sync emits a summary warning and the dashboard marks it **Needs attention**. Add context
metadata, align modalities, or target models with discoverable compatible capabilities.

## Request history and routing analytics

- `GET /api/request-history` - cursor-paginated full history from the derived
index (`routing-history.sqlite`), with filters (`provider`, `model`,
`requestedModel`, `status`, `conversationId`, `surface`, `inboundProtocol`,
`apiKeyId`, `profileId`, `fallback`, `from`, `to`) and opaque `cursor`
pagination. `GET /api/request-history/:requestId` returns one canonical row.
- `GET /api/request-history/:requestId/route-decision` - the why-this-route
explanation: trace (candidates, exclusions, score components, profile +
revision), execution attempt sequence, and final outcome.
Comment thread
Wibias marked this conversation as resolved.
- `GET /api/routing-analytics` - success/failure/cancelled/fallback rates,
p50/p95/p99 duration and TTFT, incomplete-stream rate, cooldown-triggering
failures, cost per successful request, coverage, confidence, and an
explicit truncation flag.
- `GET /api/routing-profiles`, `POST /api/routing-profiles/dry-run` - profile
inspection and dry-run evaluation (no upstream dispatch).
Comment thread
coderabbitai[bot] marked this conversation as resolved.

Returned history and route-decision payloads expose only masked request metadata
(for example opaque `apiKeyId` labels). They do not include credentials, raw
prompt bodies, or provider secrets.

CLI: `ocx logs explain <request-id>`, `ocx logs rebuild-index`,
`ocx logs index-status`, `ocx route policy list | show | dry-run | evaluate`.
Comment thread
Wibias marked this conversation as resolved.

## Migration

`routingProfiles` is optional and additive: existing config files load
unchanged. Old `usage.jsonl` rows without `routeDecision` parse unchanged.
The history index is disposable - deleting `routing-history.sqlite` triggers
an automatic rebuild from `usage.jsonl` on the next query; `ocx logs
rebuild-index` forces one. Nothing in this system auto-tunes weights,
budgets, or candidate sets.
Original file line number Diff line number Diff line change
Expand Up @@ -99,3 +99,26 @@ Combo остаётся доступной для прямой маршрутиз
из каталога. Sync выводит итоговое предупреждение, а дашборд помечает её как **Needs attention**.
Добавьте метаданные контекста, согласуйте модальности или выберите модели с обнаруживаемыми
совместимыми возможностями.

## Профили маршрутизации (`config.routingProfiles`)

Явно запрошенный `policy/<id>` (или настроенный псевдоним) выбирает среди фиксированного разрешённого списка кандидатов по жёстким требованиям к возможностям и детерминированной объяснимой оценке. Существующие идентификаторы моделей никогда не проходят через профиль неявно. Поддерживаются: `candidates` (явный список), необязательный `alias`, `require` (`minContextWindow`, `minQuotaHeadroom`, `tools`, `imageInput`, `structuredOutput`, `localOnly`, `remoteAllowed`, `encryptedCodexTasks`, `reasoningEffort`, `serviceTier`), `optimize` (веса latency/health/cost/quota), `limits.maxEstimatedCostUsd`, `unknownEvidence` (allow/penalize/exclude). Неизвестное не становится нулём или бесплатным.

CLI: `ocx route policy list`, `ocx route policy show <id>`, `ocx route policy dry-run <id> --model-context <tokens> --tools`, `ocx route policy evaluate <id>`.

Комбо — это явная маршрутизация с порядком/весами и отказоустойчивостью. Профиль — это выбор на основе доказательств среди кандидатов.

## История запросов и аналитика маршрутизации

- `GET /api/request-history` - полная история с курсорной пагинацией из производного индекса (`routing-history.sqlite`). Фильтры: `provider`, `model`, `requestedModel`, `status`, `conversationId`, `surface`, `inboundProtocol`, `apiKeyId`, `profileId`, `fallback`, `from`, `to`.
- `GET /api/request-history/:requestId/route-decision` - объяснение выбора маршрута (трасса, кандидаты, исключения, оценки, профиль+ревизия, попытки, результат).
- `GET /api/routing-analytics` - доли успеха/отказа/отмены/фолбэка, p50/p95/p99 длительность и TTFT, доля неполных потоков, сбои кулдауна, оценка стоимости за успешный запрос, покрытие, доверие, флаг усечения.
- `GET /api/routing-profiles`, `POST /api/routing-profiles/dry-run` - просмотр профилей и пробная оценка (без отправки запросов).

Возвращаемые записи истории и решений маршрута содержат только маскированные метаданные запроса (например, непрозрачные метки `apiKeyId`). Учётные данные, сырые тела промптов и секреты провайдеров не включаются.

CLI: `ocx logs explain <request-id>`, `ocx logs rebuild-index`, `ocx logs index-status`.

## Миграция

`routingProfiles` — необязательная аддитивная настройка. Существующие конфиги и старые строки `usage.jsonl` загружаются без изменений. Индекс одноразовый: при удалении он автоматически перестраивается из `usage.jsonl` при следующем запросе. Автонастройки нет.
Original file line number Diff line number Diff line change
Expand Up @@ -89,3 +89,26 @@ selector 校验、冲突规则和隐私说明见[提供方配置](/reference/con
如果是一个没有上下文元数据的裸 relay id,或者目标之间的模态互不相交,combo 就会从
目录中移除。同步时会输出一条汇总警告,仪表板会将其标记为 **Needs attention**。
补充上下文元数据、对齐模态,或者把目标模型切换为可发现且兼容的能力。

## 路由策略配置文件(`config.routingProfiles`)

显式请求的 `policy/<id>`(或配置的别名)会在固定的候选白名单中,根据硬性能力要求与确定性、可解释的评分进行选择。现有模型 ID 永远不会隐式经过配置文件。支持 `candidates`(显式白名单)、可选 `alias`、`require`(`minContextWindow`、`minQuotaHeadroom`、`tools`、`imageInput`、`structuredOutput`、`localOnly`、`remoteAllowed`、`encryptedCodexTasks`、`reasoningEffort`、`serviceTier`)、`optimize`(latency/health/cost/quota 权重)、`limits.maxEstimatedCostUsd`、`unknownEvidence`(allow/penalize/exclude)。未知不会被当作零或免费。

CLI:`ocx route policy list`、`ocx route policy show <id>`、`ocx route policy dry-run <id> --model-context <tokens> --tools`、`ocx route policy evaluate <id>`。

组合是显式的有序/加权目标路由与故障转移;策略配置文件是基于证据在候选之间进行选择。

## 请求历史与路由分析

- `GET /api/request-history` - 从派生索引(`routing-history.sqlite`)进行游标分页的全历史查询。过滤器:`provider`、`model`、`requestedModel`、`status`、`conversationId`、`surface`、`inboundProtocol`、`apiKeyId`、`profileId`、`fallback`、`from`、`to`。
- `GET /api/request-history/:requestId/route-decision` - 为什么选择此路由(跟踪、候选、排除、分数、配置文件+版本、执行尝试、结果)。
- `GET /api/routing-analytics` - 成功/失败/取消/回退率、p50/p95/p99 耗时与 TTFT、不完整流率、冷却失败数、每次成功请求的估算成本、覆盖率、置信度、截断标志。
- `GET /api/routing-profiles`、`POST /api/routing-profiles/dry-run` - 配置文件查看与试运行评估(不发送上游请求)。

返回的历史记录与路由决策负载仅暴露已脱敏的请求元数据(例如不透明的 `apiKeyId` 标签)。不包含凭证、原始提示正文或提供商密钥。

CLI:`ocx logs explain <request-id>`、`ocx logs rebuild-index`、`ocx logs index-status`。

## 迁移

`routingProfiles` 是可选的增量配置:现有配置文件与旧 `usage.jsonl` 行均可原样加载。索引是一次性的——删除后会在下次查询时从 `usage.jsonl` 自动重建。系统不会自动调优。
Loading
Loading