feat(routing): add quota-aware policy scoring - #1014
Conversation
|
Warning Review limit reached
Next review available in: 18 minutes Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available. How can I continue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews. How do review limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please refer docs for additional details. Review details⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: ASSERTIVE Plan: Pro Plus Run ID: 📒 Files selected for processing (1)
📝 WalkthroughWalkthroughRouting policies now use cached quota headroom for requirements, trace evidence, candidate scoring, and account-aware routing. The PR also updates tests, routing documentation, and stack records. ChangesQuota-aware routing
Estimated code review effort: 4 (Complex) | ~45 minutes Sequence Diagram(s)sequenceDiagram
participant Router
participant QuotaEvidence
participant PolicyEvaluator
participant RoutingProfile
Router->>QuotaEvidence: Build account-aware quota evidence
QuotaEvidence-->>Router: Return evidence or unknown
Router->>PolicyEvaluator: Evaluate candidate
PolicyEvaluator->>RoutingProfile: Check minQuotaHeadroom
RoutingProfile-->>PolicyEvaluator: Return eligibility
PolicyEvaluator-->>Router: Return weighted score
Possibly related PRs
Suggested reviewers: 🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 9dbdce2ab2
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
9dbdce2 to
dce85b7
Compare
[GD] Verdict: changes-requestedTLDR
Full verdictSemantic propagation
UsefulnessFixes a real gap: policy routing now scores candidates on remaining quota headroom and can hard-gate on Bugs / correctness
Security
Spec / standards
Reviews
Base / CI
Simplification
Gatenone (not draft/WIP/do-not-merge); ship-gate blocked only on Bottom lineThe review loop is closed: every confirmed finding and bot thread is fixed with evidence and regression coverage, and local gates are green. The single remaining item is the base sync after #1012 merges (deliberately deferred per your instruction) — once synced and CI is green on the mergeable head, this PR is ready for merge. |
0c51002 to
b3a1a1e
Compare
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 0c51002715
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
b3a1a1e to
7aebbd3
Compare
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 6
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@devlog/_plan/260804_router_intelligence/001_pr_stack_status.md`:
- Around line 226-235: Update the RI-07 acceptance metadata in the verification
record: replace the pending PR entry with PR `#1014` and correct the quota-scoring
test count from 7/7 to 9/9, preserving the remaining verification details.
In `@docs-site/src/content/docs/reference/configuration/routing.md`:
- Around line 155-162: The routing quota documentation incorrectly describes
runtime quota as always unknown and says quota scoring is planned. Update the
runtime candidate and quota-scoring sections to document that Pool mode
preserves its existing account selection, then reads quota for the selected
account, while Direct mode reads quota only from the current account; clarify
that quota evidence does not affect account selection, session affinity,
cooldowns, or switching behavior, and align the wording with the
Pool-versus-Direct CLI/API behavior.
In `@src/router.ts`:
- Around line 511-521: Update quotaEvidenceForCandidate calls in src/router.ts
lines 511-521 and src/server/management/routing-profile-routes.ts lines 129-139
to provide the resolved Anthropic accountRef, using the existing
account-reference resolution path alongside the pooled Codex account handling.
Apply the same resolution in both live routing and automatic dry-run paths so
Anthropic quota can be loaded.
In `@src/routing/evaluator.ts`:
- Around line 120-126: Update the minQuotaHeadroom requirement logic in the
evaluator to use quotaScore(quota) as the normalized actual value instead of
directly checking quota.headroom. Emit the requirement only when the normalized
score is non-null, so exhausted evidence without headroom evaluates as zero
while unknown or incomplete evidence is excluded. Add regressions covering
exhausted quota without headroom and unknown quota with malformed headroom.
In `@src/routing/quota.ts`:
- Around line 33-49: Update quotaEvidenceForCandidate() to obtain and use the
Codex account plan when processing codexAccountId. Select only the plan-specific
percentage and reset timestamp—monthly for go/free accounts and the existing
applicable value for other plans—then compute headroom and resetAtMs from those
values and pass plan to isCodexQuotaExhausted(quota, plan), preserving the
existing evidence shape.
In `@tests/quota-scoring.test.ts`:
- Around line 87-115: Extend the test around evaluatePolicyProfile to add an
unknownEvidence.quota: "allow" configuration and evaluation. Assert the
candidate remains eligible and its quota score is not
QUOTA_UNKNOWN_PENALTY_SCORE, completing coverage of the exclude, penalize, and
allow branches.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: ASSERTIVE
Plan: Pro Plus
Run ID: a201279c-2937-4fa0-9b88-6d84fc5273f3
📒 Files selected for processing (14)
devlog/_plan/260804_router_intelligence/001_pr_stack_status.mddocs-site/src/content/docs/reference/configuration/routing.mdsrc/router.tssrc/routing/evaluator.tssrc/routing/profile.tssrc/routing/quota.tssrc/routing/trace.tssrc/server/management/routing-profile-routes.tssrc/types.tstests/policy-execution.test.tstests/quota-scoring.test.tstests/request-evidence.test.tstests/request-history-index.test.tstests/routing-profile.test.ts
There was a problem hiding this comment.
Actionable comments posted: 1
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
docs-site/src/content/docs/reference/configuration/routing.md (1)
155-165: 🎯 Functional Correctness | 🟡 Minor | ⚡ Quick winDocument missing quota as unknown.
If the cache has no usable reading, or the reading is incomplete, quota remains unknown. The evaluator then applies
unknownEvidence.quotaasexclude,penalize, orallow. Do not describe missing data as exhaustion or zero headroom.Suggested wording
Quota evidence ... is keyed by account. +Missing or incomplete cached readings remain unknown. `unknownEvidence.quota` +controls whether the candidate is excluded, penalized, or allowed.As per path instructions, quota coverage must be described explicitly and missing readings must not be treated as usage.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@docs-site/src/content/docs/reference/configuration/routing.md` around lines 155 - 165, Update the quota evidence documentation around the local Codex and Anthropic quota caches to state explicitly that absent, unusable, or incomplete readings remain unknown and are evaluated through unknownEvidence.quota as exclude, penalize, or allow. Clarify that missing data must not be interpreted as quota exhaustion, zero headroom, or usage, while preserving the existing account-selection and candidate-evidence behavior.Source: Path instructions
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@docs-site/src/content/docs/reference/configuration/routing.md`:
- Around line 174-176: Remove the inactive cost scoring entry from the
scoring-weight table in the routing configuration documentation, or explicitly
mark it as reserved until RI-08; keep the documented scoring weights aligned
with the behavior implemented by the evaluator and retain cost only as the
limits.maxEstimatedCostUsd hard cap.
---
Outside diff comments:
In `@docs-site/src/content/docs/reference/configuration/routing.md`:
- Around line 155-165: Update the quota evidence documentation around the local
Codex and Anthropic quota caches to state explicitly that absent, unusable, or
incomplete readings remain unknown and are evaluated through
unknownEvidence.quota as exclude, penalize, or allow. Clarify that missing data
must not be interpreted as quota exhaustion, zero headroom, or usage, while
preserving the existing account-selection and candidate-evidence behavior.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: ASSERTIVE
Plan: Pro Plus
Run ID: 58275fd3-14f2-487f-9db7-da1d4eb9082d
📒 Files selected for processing (7)
devlog/_plan/260804_router_intelligence/001_pr_stack_status.mddocs-site/src/content/docs/reference/configuration/routing.mdsrc/router.tssrc/routing/evaluator.tssrc/routing/quota.tssrc/server/management/routing-profile-routes.tstests/quota-scoring.test.ts
There was a problem hiding this comment.
Actionable comments posted: 1
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (2)
docs-site/src/content/docs/reference/configuration/routing.md (2)
160-161: 🎯 Functional Correctness | 🟡 Minor | ⚡ Quick winDo not describe quota as scoring-only.
require.minQuotaHeadroomis evaluated before scoring insrc/routing/evaluator.tsLines 98-176. A candidate with insufficient known headroom becomes ineligible.unknownEvidence.quota: "exclude"can also exclude a candidate.Change the sentence to state that quota affects policy-profile eligibility and scoring. Keep the existing clarification that quota does not change Codex account selection, session affinity, cooldowns, or switching.
As per path instructions, quota must not be described as changing account selection or switching behavior, but its eligibility effect must remain documented.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@docs-site/src/content/docs/reference/configuration/routing.md` around lines 160 - 161, Update the quota behavior description near the dry-run/API clarification to state that quota affects policy-profile eligibility and scoring, while preserving that it does not change Codex account selection, session affinity, cooldowns, or switching behavior.Source: Path instructions
155-159: 🎯 Functional Correctness | 🟡 Minor | ⚡ Quick winDocument that API-key routes never use Codex accounts.
Lines 155-159 distinguish Pool and Direct modes, but they do not state the API-key boundary. Add that API-key routes use their configured provider credentials and never resolve Codex pool or main accounts for quota evidence.
As per path instructions, Pool, Direct, and API-key account behavior must be documented separately.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@docs-site/src/content/docs/reference/configuration/routing.md` around lines 155 - 159, Update the quota evidence section in the routing documentation to explicitly state that API-key routes use their configured provider credentials and never resolve Codex pool or main accounts for quota evidence. Keep Pool, Direct, and API-key account behavior described as separate cases.Source: Path instructions
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@docs-site/src/content/docs/reference/configuration/routing.md`:
- Line 107: Update the optimize configuration documentation at the entries
corresponding to lines 107 and 138 to match the scoring behavior implemented by
evaluator.ts: only health, quota, and configuredPriority are scored directly,
while normalized latency and cost weights are folded into configuredPriority,
yielding a default configuredPriority share of 0.65. Remove the claim that cost
does not affect scoring, or instead change evaluator.ts to score latency and
cost independently.
---
Outside diff comments:
In `@docs-site/src/content/docs/reference/configuration/routing.md`:
- Around line 160-161: Update the quota behavior description near the
dry-run/API clarification to state that quota affects policy-profile eligibility
and scoring, while preserving that it does not change Codex account selection,
session affinity, cooldowns, or switching behavior.
- Around line 155-159: Update the quota evidence section in the routing
documentation to explicitly state that API-key routes use their configured
provider credentials and never resolve Codex pool or main accounts for quota
evidence. Keep Pool, Direct, and API-key account behavior described as separate
cases.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: ASSERTIVE
Plan: Pro Plus
Run ID: 221fa4e6-4c3b-429a-9d3a-e77faa4f1b12
📒 Files selected for processing (1)
docs-site/src/content/docs/reference/configuration/routing.md
Why this helpsPR #1014 (RI-07) adds quota-aware policy scoring to the routing-profile evaluator. It lets profiles weight |
1 participant
Summary
RI-07 of the Router Intelligence / Routing Control Plane programme. Adds
quota-aware policy scoring: candidates are scored on known remaining
headroom, exhausted state, reset time, and (configurable) minimum-headroom
requirements - using the existing privacy-safe quota caches.
Unknown quota stays unknown (never zero); the profile's
unknownEvidence.quotapolicy decides how unknown evidence affects eligibility and score.
Scope
src/routing/quota.ts:quotaEvidenceForCandidate()- Codex pool accounts(
getAccountQuota, usage percents only) and Anthropic per-account quotacache;
headroomas a 0..1 fraction,exhausted, nearest futureresetAtMs, and a stablesourcecode.quotaScore()- deterministic: headroom fraction (larger is better),exhausted -> 0, unknown -> null.
src/routing/trace.ts-RouteQuotaEvidencegainsheadroom(fraction);bounded parse.
src/types.ts/src/routing/profile.ts- new hard requirementrequire.minQuotaHeadroom(0..1), validated and normalized like everyother requirement.
src/routing/evaluator.ts- quota scoring folded into the composite:total = priority*priorityWeight + health*healthWeight + quota*quotaWeightwhere implemented weights come from
profile.optimize. Unknown quota:exclude->unknown-quotaexclusion (ineligible),penalize->deterministic 0.3 floor,
allow-> priority-only.minQuotaHeadroomgates eligibility as a hard requirement.
src/router.ts- execution assembles quota evidence per candidate.require.minQuotaHeadroomdocumented in the routing reference.tests/quota-scoring.test.ts- 7 tests.Where policy selection ends and account-pool selection begins
accounts: exact account selectors and existing pool strategies
(quota/round-robin/fill-first, cooldown, affinity) remain authoritative in
their existing scope.
openaiaccounts, Anthropic accountrefs) is consumed when a candidate carries an account reference
(dry-run/evaluate with
candidates[].accountRef/codexAccountId).Execution-time candidates without account refs get honest unknown quota.
silently switch a pinned account.
Privacy / security
existing caches - no raw quota responses, account emails, or tokens.
bun run privacy:scanpasses.Compatibility
minQuotaHeadroom); existing configs load unchanged.optimize.quota: 0andunknownEvidence.quota: allow.Dependency
base:
feat/ri-06-health-aware-routinghead909ce21d4.Non-goals
or transport attribution ([Bug]: DNS and network reachability failures incorrectly rotate Codex pool accounts #914).
Local verification (exact)
bun x tsc --noEmit-> PASSED (0 errors)bun run test tests/quota-scoring.test.ts-> 7/7 passbun run privacy:scan-> passedSummary by CodeRabbit
New Features
Bug Fixes
Documentation