Skip to content
56 changes: 54 additions & 2 deletions devlog/_plan/260804_router_intelligence/001_pr_stack_status.md
Original file line number Diff line number Diff line change
Expand Up @@ -45,8 +45,8 @@ other; closing one is a maintainer decision and neither is stale.
| RI-03 | `feat/ri-03-routing-analytics` | `dev` (post-#1004 merge) | `a594938c5` | #1005 | https://github.com/lidge-jun/opencodex/pull/1005 | MERGED |
| RI-04 | `feat/ri-04-policy-profile-core` | `dev` (post-#1005 merge) | `31c9f0b28` | #1011 | https://github.com/lidge-jun/opencodex/pull/1011 | MERGED |
| RI-05 | `feat/ri-05-capability-aware-routing` | `dev` (post-#1011 merge) | `088194a3a` | #1012 | https://github.com/lidge-jun/opencodex/pull/1012 | MERGED |
| RI-06 | `feat/ri-06-health-aware-routing` | `dev` (post-#1012 merge) | `af692bb7a` | #1013 | https://github.com/lidge-jun/opencodex/pull/1013 | in progress |
| RI-07 | `feat/ri-07-quota-aware-routing` | `feat/ri-06` head | pending | pending | pending | queued |
| RI-06 | `feat/ri-06-health-aware-routing` | `dev` (post-#1012 merge) | `af692bb7a` | #1013 | https://github.com/lidge-jun/opencodex/pull/1013 | MERGED |
| RI-07 | `feat/ri-07-quota-aware-routing` | `dev` (post-#1013 merge) | `0c5100271` (pre-restack) | #1014 | https://github.com/lidge-jun/opencodex/pull/1014 | in progress |
| RI-08 | `feat/ri-08-cost-aware-routing` | `feat/ri-07` head | pending | pending | pending | queued |
| RI-09 | `feat/ri-09-route-explainability-api` | `feat/ri-08` head | pending | pending | pending | queued |
| RI-10 | `feat/ri-10-routing-intelligence-ui` | `feat/ri-09` head | pending | pending | pending | queued |
Expand Down Expand Up @@ -214,3 +214,55 @@ other; closing one is a maintainer decision and neither is stale.
green; `privacy:scan` passed.
- Sync: merged `dev` (post-#1012, `088194a3a`) into the branch so the head is
mergeable and CI can run; base sync of the stack continues with RI-07.

### RI-07 - feat/ri-07-quota-aware-routing

- Base SHA: `909ce21d4ffe4dbcad9c9baa2efb1c94e1e7dcd6` (RI-06 head)
- Reviewed commit: same as final (author self-review before push)
- Findings (self-review): 2 fixed pre-push - (1) `minQuotaHeadroom` was
missing from the schema REQUIRE_KEYS so normalization silently dropped it;
(2) `minQuotaHeadroom` was missing from `OcxRoutingProfileRequirements`
(types.ts) - typecheck caught it.
- Final commit: `480f1578` (review-thread fixes; earlier commits `b562fdb9`,
`288dd8a9`, `e74a211a`, `7aebbd36`)
- PR: #1014
- Verification:
- `bun x tsc --noEmit`: PASSED (0 errors)
- `bun run test tests/quota-scoring.test.ts`: 9/9 pass - codex-pool and
anthropic evidence, unknown-stays-unknown, unknown-quota policy
(exclude/penalize/allow), headroom preference, minQuotaHeadroom gating,
account-selection boundary, plan-aware window selection
- Focused regression suites: 203/203 pass across 9 files
- `bun run privacy:scan`: passed
- Remaining Low findings: none

### RI-07 review round (full-review #1014 + simplify)

- Base SHA: `909ce21d4` (RI-06 head) -> rebased onto the reviewed RI-06 head
`a9c6f8e8` so the stack stays aligned; PR head `9dbdce2ab` before fixes.
- Simplify (approved): misindented `require.minContextWindow` check fixed in
`profile.ts`; shared RI-06 simplify carried in by the rebase.
- Bot-thread + own findings fixed in this PR:
1. evaluator: `minQuotaHeadroom` now gates only KNOWN headroom; unknown
quota is governed by `unknownEvidence.quota` (labeled `unknown-quota`,
never `unknown-capability`);
2. router: quota evidence receives the active codex account id so runtime
quota scoring reflects cached quota when a deterministic account exists;
3. dry-run API: omitted `candidates` populate quota evidence alongside
capability + health;
4. shared RI-06 review-round fixes (indexer tail, nested image evidence,
request-side requirements, alias namespace collision, policy fallthrough,
adapter tool inference) land here via the rebase.
- Verification: `tsc --noEmit` 0 errors; focused suites green (252/252 across
the routing set); `privacy:scan` passed.
- Base sync deferred: waiting for RI-05 (#1012) to merge before updating
these branches from `dev`.

## Baseline note

The full-suite baseline on this Windows machine did not complete within the
available window (background run, >3h, no summary emitted; the suite is
~8k tests and this machine is heavily loaded). Focused suites, typecheck and
privacy:scan pass per PR; the upstream PR #966 verification report records
~7941 pass / 10 environmental failures on clean dev. A final full-suite
attempt is scheduled at stack end.
27 changes: 20 additions & 7 deletions docs-site/src/content/docs/reference/configuration/routing.md
Original file line number Diff line number Diff line change
Expand Up @@ -104,13 +104,13 @@ namespace, or reserved bare native families (`gpt-*`, `o1-*`, `o3-*`, `o4-*`, `c
| `candidates` | `{ provider: string; model: string }[]` | required | Explicit allowlist of `provider/model` refs. No implicit expansion. |
| `alias?` | `string` | — | Optional public model id in place of `policy/<id>`. |
| `require?` | object | `{}` | Hard capability requirements evaluated before scoring (see below). |
| `optimize?` | object | latency 0.55, health 0.25, cost 0.10, quota 0.10 | Scoring weights; normalized deterministically. |
| `optimize?` | object | latency 0.55, health 0.25, cost 0.10, quota 0.10 | Scoring weights, normalized deterministically. Only `health` and `quota` have score dimensions; the configured-priority share is `1 - health - quota` (default 0.65), and `latency`/`cost` fold into that priority share rather than scoring independently. |
| `limits?` | object | — | Hard limits, e.g. `maxEstimatedCostUsd` (enforced by the dry-run evaluator when candidate cost evidence is known). |
| `unknownEvidence?` | object | capability `exclude`, health/quota/cost `penalize` | How unknown evidence is treated per dimension: `allow`, `penalize`, or `exclude`. Unknown never becomes zero. |

`require` supports: `minContextWindow` (positive integer), and the booleans `tools`, `imageInput`,
`structuredOutput`, `localOnly`, `remoteAllowed`, `encryptedCodexTasks`; plus `reasoningEffort` and
`serviceTier` strings.
`require` supports: `minContextWindow` (positive integer), `minQuotaHeadroom` (0..1 fraction),
and the booleans `tools`, `imageInput`, `structuredOutput`, `localOnly`, `remoteAllowed`,
`encryptedCodexTasks`; plus `reasoningEffort` and `serviceTier` strings.

For `unknownEvidence.capability`, `penalize` currently behaves like `allow`: scoring has only a
configured-priority component until a capability score dimension ships (planned with RI-06+), so
Expand Down Expand Up @@ -152,6 +152,17 @@ CLI: `ocx route policy list [--json]`, `ocx route policy show <id> [--json]`, an
`ocx route policy dry-run <id> [--model-context <tokens>] [--tools] [--image] [--structured-output] [--json]`.
Dry-run evaluates candidates without sending any upstream request.

Quota evidence (`optimize.quota`, `require.minQuotaHeadroom`, `unknownEvidence.quota`) comes from
the local Codex pool and Anthropic account quota caches, which are keyed by account. In **Pool** mode
the canonical `openai` provider preserves its existing account selection, then reads quota for the
selected account; **Direct** mode reads quota only from the current (caller/main) account. For other
providers (e.g. Anthropic), runtime candidates use the provider's active account. Quota evidence
never changes account selection, session affinity, cooldowns, or switching behavior — it only feeds
policy scoring. To see quota-aware behavior in a dry-run, supply account refs through the dry-run/API
candidate evidence: `candidates[].codexAccountId` (Codex pool, provider `openai`) or
`candidates[].accountRef` (Anthropic) derives the matching cached account quota; an explicit
`candidates[].quota` object is echoed as given.

Comment thread
Wibias marked this conversation as resolved.
### Combos vs policy profiles

- A **combo** is explicit ordered/weighted target routing and failover: the configured order (or
Expand All @@ -160,9 +171,11 @@ Dry-run evaluates candidates without sending any upstream request.
requirements filter first, then deterministic scoring ranks the survivors.

Both are virtual namespaces with aliases and collision validation; they differ in *how* a candidate
is chosen. Profile scoring currently uses the configured-priority component only; health (RI-06),
quota (RI-07), and cost (RI-08) score dimensions are planned. Per-request route-decision traces are
recorded when a policy profile executes.
is chosen. Profile scoring combines the configured-priority component with the health (RI-06) and
quota (RI-07) score dimensions where evidence is present; the `latency` and `cost` weights fold into
the priority share rather than scoring independently, and cost is also enforced through the
`limits.maxEstimatedCostUsd` cap. Per-request route-decision traces are recorded when a policy
profile executes.

### Catalog eligibility

Expand Down
25 changes: 25 additions & 0 deletions src/router.ts
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,8 @@ import {
import { decodeRoutedModelId, encodeRoutedModelId } from "./providers/slug-codec";
import { getStaleCached } from "./codex/model-cache";
import { codexAccountNamespaceEntries } from "./codex/account-namespaces";
import { getEffectiveActiveCodexAccountId } from "./codex/routing";
import { getAccountSet } from "./oauth/store";
import {
buildRouteDecisionTrace,
type RouteDecisionKind,
Expand All @@ -32,6 +34,7 @@ import { getRoutingProfile, resolvePolicyProfileId } from "./routing/profile";
import { evaluatePolicyProfile, type PolicyRequestEvidence } from "./routing/evaluator";
import { candidateCapabilityEvidence } from "./routing/capability";
import { policyCandidateHealthEvidence } from "./routing/health";
import { quotaEvidenceForCandidate } from "./routing/quota";

export class NoEligiblePolicyCandidateError extends Error {
/** Evaluation trace (with per-candidate exclusions) when nothing qualified. */
Expand Down Expand Up @@ -506,6 +509,28 @@ function routeModelInternal(
model: candidate.model,
capability: candidateCapabilityEvidence(config, candidate.provider, candidate.model),
health: policyCandidateHealthEvidence(config, candidate, now),
quota: quotaEvidenceForCandidate({
provider: candidate.provider,
model: candidate.model,
...(candidate.provider === OPENAI_CODEX_PROVIDER_ID
&& providerCodexAccountMode(
OPENAI_CODEX_PROVIDER_ID,
config.providers[OPENAI_CODEX_PROVIDER_ID],
) === "pool"
? (() => {
const codexAccountId = getEffectiveActiveCodexAccountId(config);
return {
codexAccountId,
codexAccountPlan: codexAccountId
? config.codexAccounts?.find(account => account.id === codexAccountId)?.plan
: undefined,
};
})()
: {}),
accountRef: candidate.provider === "anthropic"
? getAccountSet("anthropic")?.activeAccountId
: undefined,
}),
Comment thread
Wibias marked this conversation as resolved.
Comment thread
Wibias marked this conversation as resolved.
}));
const evaluation = evaluatePolicyProfile(config, policyId, policyEvidence ?? {}, candidateEvidence, now);
if (evaluation.selectedIndex === null) {
Expand Down
54 changes: 51 additions & 3 deletions src/routing/evaluator.ts
Original file line number Diff line number Diff line change
Expand Up @@ -22,11 +22,14 @@ import {
} from "./trace";
import { getRoutingProfile, policyModelId, type NormalizedRoutingProfile } from "./profile";
import { healthScore } from "./health";
import { quotaScore } from "./quota";

/** Unknown health under "penalize": a low-but-not-zero deterministic floor. */
export const HEALTH_UNKNOWN_PENALTY_SCORE = 0.3;
/** Unknown health under "allow": neutral midpoint of the [0,1] health scale. */
export const HEALTH_UNKNOWN_NEUTRAL_SCORE = 0.5;
/** Unknown quota under "penalize": deterministic low floor. */
export const QUOTA_UNKNOWN_PENALTY_SCORE = 0.3;

export interface PolicyRequestEvidence {
/** Required context window for this request (tokens). */
Expand All @@ -43,6 +46,8 @@ export interface PolicyCandidateEvidence {
provider: string;
model: string;
accountRef?: string;
/** Codex pool account id (provider "openai"); used to derive account-scoped quota evidence. */
codexAccountId?: string;
capability?: RouteCapabilityEvidence;
health?: RouteHealthEvidence;
quota?: RouteQuotaEvidence;
Expand Down Expand Up @@ -93,6 +98,7 @@ function booleanRequirement(
function requirementFor(
require: NormalizedRoutingProfile["require"],
capability: RouteCapabilityEvidence | undefined,
quota: RouteQuotaEvidence | undefined,
): RouteRequirementEvidence[] {
const requirements: RouteRequirementEvidence[] = [];
if (require.minContextWindow !== undefined) {
Expand All @@ -108,6 +114,22 @@ function requirementFor(
requirements.push({ id: "min-context-window", expected: require.minContextWindow, outcome: "unknown" });
}
}
// minQuotaHeadroom gates only KNOWN quota via the normalized score, which
// maps exhaustion to zero and unknown/incomplete evidence to null. Unknown
// quota is governed by the profile's `unknownEvidence.quota` policy
// (exclude / penalize / allow) via the quota score path - never by the
// capability unknown policy.
if (require.minQuotaHeadroom !== undefined) {
const quotaHeadroom = quotaScore(quota);
if (quotaHeadroom !== null) {
requirements.push({
id: "min-quota-headroom",
expected: require.minQuotaHeadroom,
actual: quotaHeadroom,
outcome: quotaHeadroom >= require.minQuotaHeadroom ? "satisfied" : "unsatisfied",
});
}
}
const tools = booleanRequirement("tools", require.tools, capability?.tools);
if (tools) requirements.push(tools);
const image = booleanRequirement("image-input", require.imageInput, capability?.image);
Expand Down Expand Up @@ -258,7 +280,7 @@ export function evaluatePolicyProfile(
candidate => candidate.provider === declared.provider && candidate.model === declared.model,
) ?? { provider: declared.provider, model: declared.model };
const requirements = [
...requirementFor(profile.require, evidence.capability),
...requirementFor(profile.require, evidence.capability, evidence.quota),
...requestRequirementFor(requestEvidence, evidence.capability),
];
const exclusions: RouteExclusionReason[] = [];
Expand Down Expand Up @@ -307,14 +329,40 @@ export function evaluatePolicyProfile(
healthValue = HEALTH_UNKNOWN_NEUTRAL_SCORE;
}

// Quota scoring (RI-07): unknown quota follows the profile policy;
// exhausted or low-headroom evidence lowers the score.
const quota = evidence.quota;
let quotaValue = quota ? quotaScore(quota) : null;
if (quotaValue === null) {
if (profile.unknownEvidence.quota === "exclude") {
exclusions.push({ code: "unknown-quota" });
eligible = false;
} else if (profile.unknownEvidence.quota === "penalize") {
quotaValue = QUOTA_UNKNOWN_PENALTY_SCORE;
}
}

const priorityScore = configuredPriorityScore(index, profile.candidates.length);
const healthWeight = profile.optimize.health;
const quotaWeight = profile.optimize.quota;
// Only spend a dimension's weight when a value is actually present:
// "allow" leaves missing health/quota components null, so subtracting
// their weights would shrink the priority share for evidence the profile
// explicitly permits to be absent. Renormalize those weights back into
// priority instead of silently changing the ranking semantics.
const spentHealth = healthValue !== null ? healthWeight : 0;
const spentQuota = quotaValue !== null ? quotaWeight : 0;
const priorityWeight = Math.max(0, 1 - spentHealth - spentQuota);
const components: RouteScoreEvidence["components"] = { configuredPriority: priorityScore };
let total = priorityScore;
let total = priorityWeight * priorityScore;
if (healthWeight > 0 && healthValue !== null) {
total = priorityScore * (1 - healthWeight) + healthValue * healthWeight;
total += healthWeight * healthValue;
components.health = healthValue;
}
if (quotaWeight > 0 && quotaValue !== null) {
total += quotaWeight * quotaValue;
components.quota = quotaValue;
}
const score: RouteScoreEvidence = { total, components };
const evaluated: PolicyEvaluationCandidate = {
provider: evidence.provider,
Expand Down
10 changes: 10 additions & 0 deletions src/routing/profile.ts
Original file line number Diff line number Diff line change
Expand Up @@ -41,6 +41,8 @@ export interface RoutingProfileValidationIssue {

export interface NormalizedRoutingProfileRequirements {
minContextWindow?: number;
/** Minimum remaining quota headroom fraction (0..1). */
minQuotaHeadroom?: number;
tools?: boolean;
imageInput?: boolean;
structuredOutput?: boolean;
Expand All @@ -64,6 +66,7 @@ export interface NormalizedRoutingProfile {

const REQUIRE_KEYS = [
"minContextWindow",
"minQuotaHeadroom",
"tools",
"imageInput",
"structuredOutput",
Expand Down Expand Up @@ -253,6 +256,13 @@ export function routingProfileIssues(
|| require.minContextWindow < 1)) {
issues.push({ path: ["require", "minContextWindow"], message: "minContextWindow must be a positive integer" });
}
if (require.minQuotaHeadroom !== undefined
&& (typeof require.minQuotaHeadroom !== "number"
|| !Number.isFinite(require.minQuotaHeadroom)
|| require.minQuotaHeadroom < 0
|| require.minQuotaHeadroom > 1)) {
issues.push({ path: ["require", "minQuotaHeadroom"], message: "minQuotaHeadroom must be a number from 0 to 1" });
}
for (const key of ["tools", "imageInput", "structuredOutput", "localOnly", "remoteAllowed", "encryptedCodexTasks"] as const) {
if (require[key] !== undefined && typeof require[key] !== "boolean") {
issues.push({ path: ["require", key], message: `${key} must be a boolean` });
Expand Down
Loading
Loading