docs(stage-router): correct the decision flow and rewrite threshold tuning - #288
Open
sabhatinas wants to merge 1 commit into
Open
docs(stage-router): correct the decision flow and rewrite threshold tuning#288sabhatinas wants to merge 1 commit into
sabhatinas wants to merge 1 commit into
Conversation
…uning Signed-off-by: sabhatinas <sabhatinas@nvidia.com>
|
WalkthroughThe routing documentation now defines signed confidence scoring, capable and efficient routing bands, hard overrides, replay-based calibration, provider-specific configuration, handoff behavior, and startup logging requirements. ChangesStage router routing
Estimated code review effort: 2 (Simple) | ~10 minutes Poem
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 3
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@docs/routing_algorithms/stage_router_routing.md`:
- Around line 68-76: Update the Mermaid flowchart’s de-escalation decision to
require tests_passed, a recent write or edit, and severity <= 0.0 before
selecting the efficient tier. Update the classifier path so an invalid or absent
classifier tier falls through to the picker’s default fall_open tier, matching
the runtime behavior in the stage-router picker.
- Around line 200-204: Update the “Caveat on efficient outcomes” section to
remove the claim that stage-router efficient performance is always at least as
good as standalone efficient performance or constitutes a conservative lower
bound. State that inherited conversation history may help or hurt, and treat the
two runs as non-equivalent baselines unless matched benchmark results establish
a lower bound.
- Around line 258-263: Update the routing-log example in the documentation to
use a path writable by the server user, or explicitly document the required
ownership and permissions for /var/lib/switchyard. In the request example, set
the model value to "switchyard" so it matches routes.stage.id and selects the
intended route.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Enterprise
Run ID: e43c89fd-971a-4c60-ae0e-1ece77160281
📒 Files selected for processing (1)
docs/routing_algorithms/stage_router_routing.md
Comment on lines
+68
to
+76
| h -->|no| dz{"tests passed<br/>+ recent write?"} | ||
| dz -->|yes| eff["efficient (tests_passed)"] | ||
| dz -->|no| sc["raw = 0.10 × (severity/0.7 + spinning + exploring − production)<br/>score = tanh(5 × raw)"] | ||
| sc --> g{"|score| >= threshold?"} | ||
| g -->|"yes, score > 0"| cap2["capable (dimensions)"] | ||
| g -->|"yes, score < 0"| eff2["efficient (dimensions)"] | ||
| g -->|no| c{"classifier set?"} | ||
| c -->|yes| k["classifier picks capable/efficient"] | ||
| c -->|no| d["use picker default tier"] | ||
| c -->|yes| k["classifier picks tier"] | ||
| c -->|no| d["picker default tier (fall_open)"] |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟠 Major | ⚡ Quick win
Align the flowchart with the runtime fallbacks.
The de-escalation branch must also require no windowed error (severity <= 0.0) and must accept a recent edit as well as a recent write. The classifier branch must show a fall-open path when the classifier returns no valid tier. Update Lines 68-76 to match switchyard/lib/processors/stage_router/picker.py:54-88.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@docs/routing_algorithms/stage_router_routing.md` around lines 68 - 76, Update
the Mermaid flowchart’s de-escalation decision to require tests_passed, a recent
write or edit, and severity <= 0.0 before selecting the efficient tier. Update
the classifier path so an invalid or absent classifier tier falls through to the
picker’s default fall_open tier, matching the runtime behavior in the
stage-router picker.
Comment on lines
+200
to
+204
| **Caveat on efficient outcomes.** In stage-router the efficient model inherits | ||
| conversation history up to the escalation point, whereas a pure-efficient run | ||
| starts fresh. So efficient performs at least as well inside stage-router as it | ||
| does alone, and any comparison against a standalone efficient run is a | ||
| conservative lower bound. |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟠 Major | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -eu
printf '%s\n' '--- target section ---'
sed -n '175,220p' docs/routing_algorithms/stage_router_routing.md
printf '%s\n' '--- related stage-router references ---'
rg -n -i -C 3 'stage-router|standalone|pure-efficient|efficient model|lower bound|escalat' docs tests .github 2>/dev/null | head -n 300
printf '%s\n' '--- repository files relevant to routing and benchmarks ---'
git ls-files | rg -i 'routing|router|benchmark|stage'Repository: NVIDIA-NeMo/Switchyard
Length of output: 29090
🏁 Script executed:
#!/bin/bash
set -eu
printf '%s\n' '--- stage-router implementation outlines ---'
ast-grep outline crates/switchyard-components/src/stage_router.rs --view compact 2>/dev/null || true
ast-grep outline crates/libsy/src/algorithms/stage.rs --view compact 2>/dev/null || true
ast-grep outline switchyard/lib/processors/stage_router_request_processor.py --view compact 2>/dev/null || true
printf '%s\n' '--- request-processing and tier-switching code ---'
rg -n -C 5 'conversation|history|handoff|tier|EFFICIENT|CAPABLE|request|messages|transcript' \
crates/switchyard-components/src/stage_router.rs \
crates/libsy/src/algorithms/stage.rs \
switchyard/lib/processors/stage_router_request_processor.py \
switchyard/lib/processors/stage_router/*.py \
tests/test_stage_router_request_processor.py \
tests/test_stage_router_profile.py
printf '%s\n' '--- benchmark instructions and staged scoring ---'
rg -n -C 5 -i 'stage.router|efficient|capable|baseline|standalone|history|fresh|comparison' \
benchmark docs/routing_algorithms/stage_router_routing.md \
.agents/skills/switchyard-stage-router-scorer/SKILL.mdRepository: NVIDIA-NeMo/Switchyard
Length of output: 50379
🏁 Script executed:
#!/bin/bash
set -eu
printf '%s\n' '--- stage-router documentation context ---'
sed -n '1,175p' docs/routing_algorithms/stage_router_routing.md
printf '%s\n' '--- benchmark references to staged and standalone runs ---'
rg -n -C 4 -i 'stage.?router|pure.?efficient|standalone|baseline|capable|efficient|fresh|history|handoff' \
benchmark/README.md benchmark/run-baseline.sh benchmark/run_manifest.py benchmark/score_staged_run.py \
.agents/skills/switchyard-stage-router-scorer/SKILL.md \
docs/routing_algorithms/stage_router_routing.md
printf '%s\n' '--- tests that assert request/history preservation or tier handoff ---'
rg -n -C 4 -i 'not mutated|messages|handoff|selected_target|selected_model|stage_router' \
tests/test_stage_router_request_processor.py tests/test_stage_router_handoff_notes.py \
tests/test_stage_router_classifier.pyRepository: NVIDIA-NeMo/Switchyard
Length of output: 50380
Remove the unsupported lower-bound claim. Inherited conversation history can help or hurt the efficient model. Treat stage-router and standalone efficient runs as non-equivalent baselines unless matched benchmark results establish a lower bound.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@docs/routing_algorithms/stage_router_routing.md` around lines 200 - 204,
Update the “Caveat on efficient outcomes” section to remove the claim that
stage-router efficient performance is always at least as good as standalone
efficient performance or constitutes a conservative lower bound. State that
inherited conversation history may help or hurt, and treat the two runs as
non-equivalent baselines unless matched benchmark results establish a lower
bound.
Comment on lines
+258
to
+263
| Add `--routing-log-file /var/lib/switchyard/routing_requests.jsonl` to record | ||
| per-request routing decisions for later analysis. | ||
|
|
||
| Keep the route `id` aligned with whatever model alias your agent sends — that | ||
| string is what selects this route. | ||
|
|
There was a problem hiding this comment.
🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -euo pipefail
rg -n -C 4 \
'routing-log-file|route.*id|model.*alias|switchyard-server|argparse|clap' \
--glob '*.py' --glob '*.rs' --glob '*.toml' .Repository: NVIDIA-NeMo/Switchyard
Length of output: 50379
🏁 Script executed:
#!/bin/bash
set -euo pipefail
printf '%s\n' '--- server CLI definition ---'
cat -n crates/switchyard-server/src/cli.rs | sed -n '1,140p'
printf '%s\n' '--- routing log references ---'
rg -n -C 3 'routing[_-]log|routing log|routing_requests|log_file' .
printf '%s\n' '--- stage-router documentation and request examples ---'
rg -n -C 5 'stage_router|stage-router|routes\.stage|model alias|model_alias|model:' \
docs switchyard tests --glob '*.md' --glob '*.py' --glob '*.toml' --glob '*.yaml' --glob '*.yml' \
| head -n 500
printf '%s\n' '--- server startup and config references ---'
rg -n -C 4 'switchyard-server|--config|/var/lib|routes\.toml|routing' \
docs README.md crates/switchyard-server --glob '*.md' --glob '*.toml' --glob '*.rs' \
| head -n 500Repository: NVIDIA-NeMo/Switchyard
Length of output: 50381
🏁 Script executed:
#!/bin/bash
set -euo pipefail
printf '%s\n' '--- reviewed document ---'
cat -n docs/routing_algorithms/stage_router_routing.md | sed -n '1,290p'
printf '%s\n' '--- route resolution and request model selection ---'
rg -n -C 8 'resolve_route|selected_model|model.*header|body.*model|metadata.*model|request.*model|route.*model' \
crates/switchyard-server/src switchyard/cli switchyard/lib \
--glob '*.rs' --glob '*.py' | head -n 600
printf '%s\n' '--- launcher request/model configuration ---'
rg -n -C 6 'model|route|alias|OPENAI|ANTHROPIC|CODEX|CLAUDE' \
switchyard/cli/launchers switchyard/cli/launch_command.py docs \
--glob '*.py' --glob '*.md' | head -n 800
printf '%s\n' '--- deployment users and filesystem paths ---'
rg -n -C 5 'User=|useradd|systemd|/var/lib/switchyard|WorkingDirectory|runuser|sudo|permissions|writable' \
. --glob '*.service' --glob '*.yml' --glob '*.yaml' --glob '*.toml' --glob '*.md' --glob '*.sh' --glob 'Dockerfile*' \
| head -n 800Repository: NVIDIA-NeMo/Switchyard
Length of output: 50381
🏁 Script executed:
#!/bin/bash
set -euo pipefail
printf '%s\n' '--- exact Rust route lookup ---'
cat -n crates/switchyard-server/src/lib.rs | sed -n '570,635p'
printf '%s\n' '--- route-bundle registration for stage_router ---'
cat -n switchyard/cli/route_bundle.py | sed -n '1030,1085p'
cat -n switchyard/cli/route_bundle.py | sed -n '420,490p'
printf '%s\n' '--- launcher model/default route flow ---'
rg -n -C 8 'display_model|default_route|profile_routes|default_model|route.*model|model.*route' \
switchyard/cli/launchers switchyard/cli/launch_command.py \
--glob '*.py' | head -n 500
printf '%s\n' '--- deployment-related files and path references ---'
git ls-files | rg -i '(^|/)(dockerfile[^/]*|.*systemd.*|.*service$|deploy|deployment|container|helm|compose|install|readme|operations)'
rg -n -C 4 '/var/lib/switchyard|User=|useradd|systemd|docker run|WORKDIR|chmod|chown|routing_requests\.jsonl' \
$(git ls-files) 2>/dev/null | head -n 500Repository: NVIDIA-NeMo/Switchyard
Length of output: 50378
🏁 Script executed:
#!/bin/bash
set -euo pipefail
printf '%s\n' '--- native TOML route loading and route-ID contract ---'
rg -n -C 8 'routes|route.*id|id.*route|algorithm_for_model|load_server_state|Route' \
crates/switchyard-server/src/config.rs crates/switchyard-server/src/lib.rs \
crates/switchyard-server/README.md --glob '*.rs' --glob '*.md' | head -n 500
printf '%s\n' '--- server README startup and route examples ---'
cat -n crates/switchyard-server/README.md | sed -n '1,115p'
printf '%s\n' '--- standalone source-data verifier ---'
python3 - <<'PY'
from pathlib import Path
import re
cli = Path("crates/switchyard-server/src/cli.rs").read_text()
log = Path("crates/switchyard-server/src/routing_log.rs").read_text()
server = Path("crates/switchyard-server/src/lib.rs").read_text()
doc = Path("docs/routing_algorithms/stage_router_routing.md").read_text()
checks = {
"clap flag field": bool(re.search(r'#\[arg\(long, value_name = "PATH"\)\]\s+routing_log_file:\s+Option<PathBuf>', cli)),
"CLI applies routing log": "with_routing_log(path)" in cli,
"log creates parent": "fs::create_dir_all(parent)" in log,
"log opens append/create": all(x in log for x in ("OpenOptions::new()", ".create(true)", ".append(true)")),
"server exact model lookup": "state.algorithm_for_model(&requested_model)" in server,
"document route id": bool(re.search(r'\[routes\.stage\].*?id\s*=\s*"switchyard"', doc, re.S)),
}
for name, result in checks.items():
print(f"{name}: {'PASS' if result else 'FAIL'}")
PYRepository: NVIDIA-NeMo/Switchyard
Length of output: 44771
Use a writable routing-log path. --routing-log-file is supported, and the request must send "model": "switchyard" to select routes.stage.id. The server creates parent directories but does not change permissions, so document directory ownership or use a path writable by the server user instead of assuming /var/lib/switchyard.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@docs/routing_algorithms/stage_router_routing.md` around lines 258 - 263,
Update the routing-log example in the documentation to use a path writable by
the server user, or explicitly document the required ownership and permissions
for /var/lib/switchyard. In the request example, set the model value to
"switchyard" so it matches routes.stage.id and selects the intended route.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Docs only. No code change.
Decision flow was wrong
confidence >= thresholdand jumped straight to "signals pick capable/efficient", which hid both hard rules and the sign check. It also implied the score is the confidence.Show where the score comes from
Added the actual formula and what it implies, rather than asserting "~0.46":
with a table of what 1 / 1.5 / 2 / 3 corroborating signals actually score.
Threshold tuning rewritten
0.0 / 0.5 / 0.7 / 1.0recommendation table — it read as a menu of portable settings, which they are not.efficient_firstsends[-1, t)to efficient and[t, +1]to capable;capable_firstsends[-1, -t]to efficient and(-t, +1]to capable.tinto "maxed signals needed to leave the default tier".0.3and said plainly that swapping either model changes the trajectories, so the score distribution moves and the sametbuys a different split. Recalibrate per tier pair.Calibration rewritten
switchyard-stage-router-scorerskill, histogram the scores, put the cut line where the mass tells you to, sweep, and check escalations land where the efficient tier actually fails.recent_turn_window— short windows react fast and flap, long windows need sustained trouble and react late — and noted it is not independent oft.Route configuration
Replaced the single-provider OpenRouter gpt-4o/4o-mini example with the two-provider shape we actually benchmark: Anthropic Messages capable tier, OpenAI Chat efficient tier, per-target
extra_body, handoff notes, and the--routing-log-fileflag.Verified with
mkdocs build --strict.Summary by CodeRabbit