Skip to content

docs(stage-router): correct the decision flow and rewrite threshold tuning #288

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
sabhatinas wants to merge 1 commit into
base: main
Choose a base branch
from
+164 −121
Open

docs(stage-router): correct the decision flow and rewrite threshold tuning #288

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
285 changes: 164 additions & 121 deletions docs/routing_algorithms/stage_router_routing.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,8 +6,9 @@ is to spend the capable model on the turns that need it (exploration, error
recovery, hard reasoning) and let the efficient model carry the routine,
mechanical work. Which tier a turn defaults to depends on the picker you choose
(`capable_first` or `efficient_first`); the signals then move individual turns
off that default. You configure it with a single knob, `confidence_threshold`,
plus an optional LLM classifier.
off that default. Two knobs shape that behaviour — `confidence_threshold` (how
much evidence it takes to leave the default tier) and `recent_turn_window` (how
much history the signals see) — plus an optional LLM classifier.

If the selected target exceeds its context window, the router tries the next
eligible target until one succeeds or all configured targets have been tried. See
Expand All @@ -29,38 +30,57 @@ For each LLM call, stage-router estimates which stage the agent is in from the
- **PROGRESS → efficient**: `recent_production_intensity` (writes and edits
landing over the recent window) pushes toward the efficient tier.

The axes are **corroborative**: the signed score is `tanh`-squashed to a
confidence in `[0, 1]`, so one full signal alone scores ~`0.46` and a second
corroborating signal is what pushes it decisively past a `0.5` threshold. A
critical-error severity is a hard override that escalates on its own. The router
then routes:
### How the score is computed

- the **capable** tier for uncertain, exploratory, or error-recovery turns, and
- the **efficient** tier for settled, mechanical turns.
The four signals are summed on one axis — error evidence minus production
evidence — then squashed:

`confidence_threshold` sets how sure that estimate must be before the router acts
on the signal alone. Below it, the turn stays on the picker's default tier (or,
if you added the optional classifier, goes to it first). A turn with no
tool-result history yet has no stage to estimate, so it takes the default tier.
```text
raw = 0.10 × ( severity / 0.7 + spinning + exploring − production_intensity )
score = tanh( 5.0 × raw ) → [-1, +1]
```

Each maxed signal contributes one unit of `0.10`, so no single axis can peg the
score on its own. The `tanh` keeps the result bounded and makes the axes
**corroborative** — agreement between signals is what moves the score decisively:

| Maxed signals agreeing | `raw` | `score` |
|---|---|---|
| 1 | `0.10` | `±0.462` |
| 1.5 | `0.15` | `±0.635` |
| 2 | `0.20` | `±0.762` |
| 3 (all error signals) | `0.30` | `±0.905` |

**Sign is direction, magnitude is confidence.** A positive score points at the
capable tier, negative at the efficient tier, and `confidence = |score|`.

Two hard rules run *before* the scorer and ignore the threshold entirely: a
critical-error severity (or a context compaction) forces capable, and a settled
turn — tests passed, a recent write, no windowed error — forces efficient.

The routing decision for one turn:

```mermaid
%%{init: {"flowchart": {"nodeSpacing": 18, "rankSpacing": 26}}}%%
flowchart LR
t["turn"] --> g{"confidence >= threshold?"}
g -->|yes| s["signals pick capable/efficient"]
flowchart TB
t["turn"] --> h{"critical error<br/>or compaction?"}
h -->|yes| cap["capable (override)"]
h -->|no| dz{"tests passed<br/>+ recent write?"}
dz -->|yes| eff["efficient (tests_passed)"]
dz -->|no| sc["raw = 0.10 × (severity/0.7 + spinning + exploring − production)<br/>score = tanh(5 × raw)"]
sc --> g{"|score| >= threshold?"}
g -->|"yes, score > 0"| cap2["capable (dimensions)"]
g -->|"yes, score < 0"| eff2["efficient (dimensions)"]
g -->|no| c{"classifier set?"}
c -->|yes| k["classifier picks capable/efficient"]
c -->|no| d["use picker default tier"]
c -->|yes| k["classifier picks tier"]
c -->|no| d["picker default tier (fall_open)"]
Comment on lines +68 to +76

@coderabbitai coderabbitai Bot Aug 4, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🎯 Functional Correctness | 🟠 Major | ⚡ Quick win

Align the flowchart with the runtime fallbacks.

The de-escalation branch must also require no windowed error (severity <= 0.0) and must accept a recent edit as well as a recent write. The classifier branch must show a fall-open path when the classifier returns no valid tier. Update Lines 68-76 to match switchyard/lib/processors/stage_router/picker.py:54-88.

🤖 Prompt for AI Agents 
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@docs/routing_algorithms/stage_router_routing.md` around lines 68 - 76, Update
the Mermaid flowchart’s de-escalation decision to require tests_passed, a recent
write or edit, and severity <= 0.0 before selecting the efficient tier. Update
the classifier path so an invalid or absent classifier tier falls through to the
picker’s default fall_open tier, matching the runtime behavior in the
stage-router picker.


classDef box font-family:monospace,fill:none,stroke:#9aa0a6,stroke-width:1px;
class t,s,k,d,g,c box;
class t,h,dz,sc,g,c,cap,eff,cap2,eff2,k,d box;
```

With `capable_first`, the default is capable, so a turn only reaches the cheaper
efficient model on a confident efficient signal (or an efficient verdict from
the classifier). Raising the threshold shrinks that path; lowering it widens it.
A turn with no tool-result history yet has no stage to estimate, so it takes the
default tier.

## Pickers

Expand All @@ -76,140 +96,157 @@ Both pickers read the same signals; only the default tier differs.

## Tuning `confidence_threshold`

The scorer rates each turn from `0` (signals are neutral) to `1` (signals point
hard at one tier). `confidence_threshold` is the bar that rating has to clear
before the router will switch off the picker's default tier. Clear it and the
router routes to the tier the signals indicate; fall short and the turn stays on
the default.
Scores live on `[-1, +1]`. The threshold `t` carves that line into three bands,
and the picker decides who owns the middle one.

With the default `capable_first` picker, every turn starts on the capable tier
and only drops to the efficient tier when the signals say "efficient" and clear
the threshold. So the threshold sets how much evidence it takes to switch to the
cheaper tier:
**`efficient_first`** — the middle band falls to efficient:

- Raise it and only strong, decisive signals drop a turn to efficient, so the
router stays on capable longer (more quality, more cost).
- Lower it and weaker signals are enough to drop to efficient, so more turns go
cheap (more savings, more risk).
```text
efficient │ capable
├───────────────────────────────────────────┼───────────┤
-1 t +1
(confident efficient + fall_open) │ (confident escalation)
```

`efficient_first` is the mirror: turns start on efficient and need a signal that
clears the threshold to escalate to capable.
**`capable_first`** — the middle band falls to capable:

(If you add the optional classifier, sub-threshold turns go to it instead of
staying on the default tier.)
```text
efficient │ capable
├───────────┼───────────────────────────────────────────┤
-1 -t +1
(confident │ (fall_open + confident capable)
drop) │
```

**Set `0.5` explicitly.** `confidence_threshold` is required by the TOML schema;
`0.5` is the recommended starting point and what the example below uses.
So for `efficient_first`, `[-1, t)` routes efficient and `[t, +1]` routes
capable. For `capable_first`, `[-1, -t]` routes efficient and `(-t, +1]` routes
capable. Both pickers read the same scores; only the ownership of the
low-confidence middle differs. (With a classifier configured, the middle band
goes to the classifier instead of straight to the default tier.)

| `confidence_threshold` | Include `classifier:` block? | Typical use |
|---|---|---|
| `0.0` | no | Cost/latency-sensitive. Every signal-based verdict is accepted; no per-turn LLM call. Critical-error signals still escalate to capable. |
| `0.5` | no | Recommended starting point. The scorer is corroborative — one full wrong signal scores ~`0.46`, just under `0.5` — so a decisive escalation takes a strong signal plus corroboration, while a critical error overrides regardless. Derived from SWE-Bench Pro Python-75 calibration. |
| `0.7` - `0.9` | yes | Classifier-assisted. Low-confidence turns go to the LLM classifier before falling back to the default tier. |
| `1.0` | yes (required) | Classifier-driven. Tool signals only apply hard overrides; other turns reach the classifier. |
Because the score is `tanh(5 × raw)`, the threshold translates directly into
"how many maxed signals must agree":

The signal-vs-classifier split is dataset-dependent. Measure it in
production: `/v1/stats` reports traffic by tier and model, while response headers
and structured decision logs explain individual selections.
| `t` | Maxed signals needed to leave the default tier |
|---|---|
| `0.2` | `0.41` — a fraction of one signal |
| `0.3` | `0.62` — most of one signal |
| `0.5` | `1.10` — one signal plus corroboration |
| `0.7` | `1.73` — nearly two signals |

Raising `t` widens the middle band, so more turns sit on the default tier;
lowering it narrows the band and lets weaker evidence move a turn. Critical-error
overrides fire regardless of `t`.

**Start at `0.3` and sweep.** `confidence_threshold` is required by the TOML
schema. `0.3` is a good opening value because it leaves the band narrow enough
that real signals move turns, which gives a sweep something to measure. Do not
treat any single number as portable: **swap either model and the trajectories
change shape**, so the score distribution moves and the same `t` buys a different
routing split. Recalibrate whenever the tier pair changes.

### Calibrating the threshold from run data

The recommended `0.5` starting point was derived from SWE-Bench Pro Python-75
calibration. To tune for a different task set or model pair, follow this
minimum-data path.

**What you need**

| Run | Coverage | Purpose |
|---|---|---|
| Pure-capable | ~40–75 representative tasks | Baseline outcomes + signal features |
| Pure-efficient | ~20 tasks (sampled from capable results) | Counterfactual outcomes |

Neither run needs to cover the full task set. A few dozen capable tasks gives
enough outcome diversity; the efficient probe only needs to cover the interesting
quadrant candidates identified from those capable results.

**How to sample the efficient probe set**

Stratify the pure-capable results across four quadrant candidates before running efficient:
Calibration is a sweep against your own score distribution, not a lookup. Our
published values were calibrated on Terminal-Bench 2.1 and, more recently,
SWE-Bench Pro — which is exactly why you should re-derive them for your task set
and tier pair rather than adopting them.

| Category | Criterion | Count | Value |
|---|---|---|---|
| Easy + clean | Capable passes, small diff, clear spec | ~5 | Establishes SAFE floor |
| Easy + tricky | Capable passes, subtle logic | ~5 | Catches LOSS false-positives |
| Hard + structural | Capable fails, large multi-file diff | ~5 | HARD noise baseline |
| Hard + localized | Capable fails, small targeted fix | ~5 | Best RESCUE signal |
**1. Get a run to replay.** Any completed run with real tool traffic works; you
do not need a matched capable/efficient pair to pick a threshold. A few dozen
tasks is enough, because every turn in every task is a scored sample.

Sample across repos and diff sizes. Don't over-represent one project.

**Building RESCUE / LOSS quadrants**

From the overlap tasks (those with both capable and efficient results):

- `RESCUE` = capable-fail ∩ efficient-pass → escalation is beneficial here
- `LOSS` = capable-pass ∩ efficient-fail → do NOT escalate here
- `SAFE` = both pass
- `HARD` = both fail

**Running the sweep**

Replay your runs through the real Rust scorer and picker with
`benchmark/score_staged_run.py` (the `switchyard-stage-router-scorer` skill). It emits
per-turn scores and per-task routing splits at a given threshold and window —
the actual `pick_capable_first` / `pick_efficient_first` decisions, not a
**2. Replay it through the real scorer.** `benchmark/score_staged_run.py` (the
`switchyard-stage-router-scorer` skill) runs the actual Rust scorer and picker,
so you get the decisions the router would really have made — not a
counterfactual:

```bash
# Score a probe run at a candidate threshold
uv run python benchmark/score_staged_run.py --run benchmark/tb_runs/<your_run> \
--threshold 0.5 --window 3
--threshold 0.3 --window 3
# → /tmp/<run>-scores.jsonl (per turn: score, confidence, pick_cf, pick_ef)
# → /tmp/<run>-per-task.csv (per task: routing split, mean score/confidence)
```

Sweep a few candidate thresholds and read the routing split and pass rate off
the per-task CSV; the lowest threshold that rescues the RESCUE quadrant without
over-escalating the LOSS quadrant is your calibrated value. Because the scorer
is corroborative, a `0.5` threshold takes ~1.5 signals of agreement — a policy
that escalates ~20% of tasks maps roughly to `confidence_threshold: 0.5` with
`capable_first`.

Signals come from the actual picker replay, so even 15–20 probe tasks give a
stable result.

**Caveat on efficient outcomes in stage-router vs. pure-efficient**
**3. Look at the score distribution before picking `t`.** Histogram the `score`
column from the JSONL. The threshold is a cut line on that histogram, so where
the mass sits tells you what any given `t` will buy:

```text
turns
│ ▁▃▅█▅▃▁ ← most turns cluster near 0
│ ▁▂▄██████▄▂▁ (ambiguous, fall_open)
│ ▁▂▄████████████▄▂▁
└────┴────┴────┴────┴────┴────┴────┴──
-1 -0.5 0 +0.5 +1
↑t=0.3 ↑t=0.5
wider band ──┘ └── narrower escalation path
```

In stage-router, the efficient model may inherit partial context from the capable arm
(conversation history up to the escalation point). Pure-efficient runs start
fresh, so RESCUE is a conservative lower bound. Efficient performs at least as
well in stage-router as it does alone.
**4. Sweep and read the split off the CSV.** Re-run at several `t` values and
compare the routing split against what you want. If you have both capable and
efficient outcomes for the same tasks, check that escalations land on the tasks
the efficient tier actually fails — the point is to escalate where it changes the
result, not to hit a target percentage.

**Tuning the signal window.** `recent_turn_window` sets how many trailing tool
results the signals are computed over, and it moves the distribution as much as
`t` does. A short window (`3`) reacts fast — a couple of bad results escalate
quickly, and the router drops back just as fast once work resumes. A longer
window (`5`+) smooths over isolated failures and needs sustained trouble to
escalate, which cuts flapping at the cost of reacting late. Sweep it alongside
`t`; they are not independent.

**Caveat on efficient outcomes.** In stage-router the efficient model inherits
conversation history up to the escalation point, whereas a pure-efficient run
starts fresh. So efficient performs at least as well inside stage-router as it
does alone, and any comparison against a standalone efficient run is a
conservative lower bound.
Comment on lines +200 to +204

@coderabbitai coderabbitai Bot Aug 4, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🎯 Functional Correctness | 🟠 Major | ⚡ Quick win

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
set -eu
printf '%s\n' '--- target section ---'
sed -n '175,220p' docs/routing_algorithms/stage_router_routing.md
printf '%s\n' '--- related stage-router references ---'
rg -n -i -C 3 'stage-router|standalone|pure-efficient|efficient model|lower bound|escalat' docs tests .github 2>/dev/null | head -n 300
printf '%s\n' '--- repository files relevant to routing and benchmarks ---'
git ls-files | rg -i 'routing|router|benchmark|stage'

Repository: NVIDIA-NeMo/Switchyard

Length of output: 29090

🏁 Script executed:

#!/bin/bash
set -eu
printf '%s\n' '--- stage-router implementation outlines ---'
ast-grep outline crates/switchyard-components/src/stage_router.rs --view compact 2>/dev/null || true
ast-grep outline crates/libsy/src/algorithms/stage.rs --view compact 2>/dev/null || true
ast-grep outline switchyard/lib/processors/stage_router_request_processor.py --view compact 2>/dev/null || true
printf '%s\n' '--- request-processing and tier-switching code ---'
rg -n -C 5 'conversation|history|handoff|tier|EFFICIENT|CAPABLE|request|messages|transcript' \
  crates/switchyard-components/src/stage_router.rs \
  crates/libsy/src/algorithms/stage.rs \
  switchyard/lib/processors/stage_router_request_processor.py \
  switchyard/lib/processors/stage_router/*.py \
  tests/test_stage_router_request_processor.py \
  tests/test_stage_router_profile.py
printf '%s\n' '--- benchmark instructions and staged scoring ---'
rg -n -C 5 -i 'stage.router|efficient|capable|baseline|standalone|history|fresh|comparison' \
  benchmark docs/routing_algorithms/stage_router_routing.md \
  .agents/skills/switchyard-stage-router-scorer/SKILL.md

Repository: NVIDIA-NeMo/Switchyard

Length of output: 50379

🏁 Script executed:

#!/bin/bash
set -eu
printf '%s\n' '--- stage-router documentation context ---'
sed -n '1,175p' docs/routing_algorithms/stage_router_routing.md
printf '%s\n' '--- benchmark references to staged and standalone runs ---'
rg -n -C 4 -i 'stage.?router|pure.?efficient|standalone|baseline|capable|efficient|fresh|history|handoff' \
  benchmark/README.md benchmark/run-baseline.sh benchmark/run_manifest.py benchmark/score_staged_run.py \
  .agents/skills/switchyard-stage-router-scorer/SKILL.md \
  docs/routing_algorithms/stage_router_routing.md
printf '%s\n' '--- tests that assert request/history preservation or tier handoff ---'
rg -n -C 4 -i 'not mutated|messages|handoff|selected_target|selected_model|stage_router' \
  tests/test_stage_router_request_processor.py tests/test_stage_router_handoff_notes.py \
  tests/test_stage_router_classifier.py

Repository: NVIDIA-NeMo/Switchyard

Length of output: 50380

Remove the unsupported lower-bound claim. Inherited conversation history can help or hurt the efficient model. Treat stage-router and standalone efficient runs as non-equivalent baselines unless matched benchmark results establish a lower bound.

🤖 Prompt for AI Agents 
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@docs/routing_algorithms/stage_router_routing.md` around lines 200 - 204,
Update the “Caveat on efficient outcomes” section to remove the claim that
stage-router efficient performance is always at least as good as standalone
efficient performance or constitutes a conservative lower bound. State that
inherited conversation history may help or hurt, and treat the two runs as
non-equivalent baselines unless matched benchmark results establish a lower
bound.


## Route configuration

A working two-provider config — the shape we benchmark with. The capable tier
speaks Anthropic Messages and the efficient tier speaks OpenAI Chat Completions;
the router translates between them per turn.

```toml
schema_version = 1

[llm_clients.openrouter]
[llm_clients.anthropic]
format = "anthropic_messages"
base_url = "https://api.anthropic.com"
api_key_env = "ANTHROPIC_API_KEY"

[llm_clients.efficient_provider]
format = "openai_chat"
base_url = "https://openrouter.ai/api/v1"
api_key_env = "OPENROUTER_API_KEY"
base_url = "https://your-efficient-endpoint/v1"
api_key_env = "EFFICIENT_API_KEY"

[targets.strong]
id = "openai/gpt-4o"
llm_client = "openrouter"
[targets.capable]
id = "claude-opus-4-5"
llm_client = "anthropic"

[targets.weak]
id = "openai/gpt-4o-mini"
llm_client = "openrouter"
# Per-target extra_body is merged into the outbound request. Use it to pin
# provider-specific options, e.g. reasoning effort on the capable tier.
[targets.capable.extra_body.output_config]
effort = "medium"

[targets.efficient]
id = "your-efficient-model"
llm_client = "efficient_provider"

[routes.stage]
id = "switchyard/stage"
id = "switchyard"
type = "stage_router"
capable_target = "strong"
efficient_target = "weak"
capable_target = "capable"
efficient_target = "efficient"
picker = "efficient_first"
confidence_threshold = 0.5
confidence_threshold = 0.3 # calibrate for your tier pair — see above
recent_turn_window = 3 # optional, defaults to 3

[routes.stage.handoff_notes]
escalation_note = "[router-guidance] A weaker model was handling this task and showed signs of stalling, looping, or repeated errors on the preceding steps, so control was escalated to you, a stronger model. Re-examine the current state directly and do not simply repeat the previous approach."
only_on_wrong_signal_escalation = true
```

Save as `routes.toml` and start the server:
Expand All @@ -218,6 +255,12 @@ Save as `routes.toml` and start the server:
switchyard-server --config routes.toml --port 4000
```

Add `--routing-log-file /var/lib/switchyard/routing_requests.jsonl` to record
per-request routing decisions for later analysis.

Keep the route `id` aligned with whatever model alias your agent sends — that
string is what selects this route.

Comment on lines +258 to +263

@coderabbitai coderabbitai Bot Aug 4, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
set -euo pipefail

rg -n -C 4 \
  'routing-log-file|route.*id|model.*alias|switchyard-server|argparse|clap' \
  --glob '*.py' --glob '*.rs' --glob '*.toml' .

Repository: NVIDIA-NeMo/Switchyard

Length of output: 50379

🏁 Script executed:

#!/bin/bash
set -euo pipefail

printf '%s\n' '--- server CLI definition ---'
cat -n crates/switchyard-server/src/cli.rs | sed -n '1,140p'

printf '%s\n' '--- routing log references ---'
rg -n -C 3 'routing[_-]log|routing log|routing_requests|log_file' .

printf '%s\n' '--- stage-router documentation and request examples ---'
rg -n -C 5 'stage_router|stage-router|routes\.stage|model alias|model_alias|model:' \
  docs switchyard tests --glob '*.md' --glob '*.py' --glob '*.toml' --glob '*.yaml' --glob '*.yml' \
  | head -n 500

printf '%s\n' '--- server startup and config references ---'
rg -n -C 4 'switchyard-server|--config|/var/lib|routes\.toml|routing' \
  docs README.md crates/switchyard-server --glob '*.md' --glob '*.toml' --glob '*.rs' \
  | head -n 500

Repository: NVIDIA-NeMo/Switchyard

Length of output: 50381

🏁 Script executed:

#!/bin/bash
set -euo pipefail

printf '%s\n' '--- reviewed document ---'
cat -n docs/routing_algorithms/stage_router_routing.md | sed -n '1,290p'

printf '%s\n' '--- route resolution and request model selection ---'
rg -n -C 8 'resolve_route|selected_model|model.*header|body.*model|metadata.*model|request.*model|route.*model' \
  crates/switchyard-server/src switchyard/cli switchyard/lib \
  --glob '*.rs' --glob '*.py' | head -n 600

printf '%s\n' '--- launcher request/model configuration ---'
rg -n -C 6 'model|route|alias|OPENAI|ANTHROPIC|CODEX|CLAUDE' \
  switchyard/cli/launchers switchyard/cli/launch_command.py docs \
  --glob '*.py' --glob '*.md' | head -n 800

printf '%s\n' '--- deployment users and filesystem paths ---'
rg -n -C 5 'User=|useradd|systemd|/var/lib/switchyard|WorkingDirectory|runuser|sudo|permissions|writable' \
  . --glob '*.service' --glob '*.yml' --glob '*.yaml' --glob '*.toml' --glob '*.md' --glob '*.sh' --glob 'Dockerfile*' \
  | head -n 800

Repository: NVIDIA-NeMo/Switchyard

Length of output: 50381

🏁 Script executed:

#!/bin/bash
set -euo pipefail

printf '%s\n' '--- exact Rust route lookup ---'
cat -n crates/switchyard-server/src/lib.rs | sed -n '570,635p'

printf '%s\n' '--- route-bundle registration for stage_router ---'
cat -n switchyard/cli/route_bundle.py | sed -n '1030,1085p'
cat -n switchyard/cli/route_bundle.py | sed -n '420,490p'

printf '%s\n' '--- launcher model/default route flow ---'
rg -n -C 8 'display_model|default_route|profile_routes|default_model|route.*model|model.*route' \
  switchyard/cli/launchers switchyard/cli/launch_command.py \
  --glob '*.py' | head -n 500

printf '%s\n' '--- deployment-related files and path references ---'
git ls-files | rg -i '(^|/)(dockerfile[^/]*|.*systemd.*|.*service$|deploy|deployment|container|helm|compose|install|readme|operations)'
rg -n -C 4 '/var/lib/switchyard|User=|useradd|systemd|docker run|WORKDIR|chmod|chown|routing_requests\.jsonl' \
  $(git ls-files) 2>/dev/null | head -n 500

Repository: NVIDIA-NeMo/Switchyard

Length of output: 50378

🏁 Script executed:

#!/bin/bash
set -euo pipefail

printf '%s\n' '--- native TOML route loading and route-ID contract ---'
rg -n -C 8 'routes|route.*id|id.*route|algorithm_for_model|load_server_state|Route' \
  crates/switchyard-server/src/config.rs crates/switchyard-server/src/lib.rs \
  crates/switchyard-server/README.md --glob '*.rs' --glob '*.md' | head -n 500

printf '%s\n' '--- server README startup and route examples ---'
cat -n crates/switchyard-server/README.md | sed -n '1,115p'

printf '%s\n' '--- standalone source-data verifier ---'
python3 - <<'PY'
from pathlib import Path
import re

cli = Path("crates/switchyard-server/src/cli.rs").read_text()
log = Path("crates/switchyard-server/src/routing_log.rs").read_text()
server = Path("crates/switchyard-server/src/lib.rs").read_text()
doc = Path("docs/routing_algorithms/stage_router_routing.md").read_text()

checks = {
    "clap flag field": bool(re.search(r'#\[arg\(long, value_name = "PATH"\)\]\s+routing_log_file:\s+Option<PathBuf>', cli)),
    "CLI applies routing log": "with_routing_log(path)" in cli,
    "log creates parent": "fs::create_dir_all(parent)" in log,
    "log opens append/create": all(x in log for x in ("OpenOptions::new()", ".create(true)", ".append(true)")),
    "server exact model lookup": "state.algorithm_for_model(&requested_model)" in server,
    "document route id": bool(re.search(r'\[routes\.stage\].*?id\s*=\s*"switchyard"', doc, re.S)),
}
for name, result in checks.items():
    print(f"{name}: {'PASS' if result else 'FAIL'}")
PY

Repository: NVIDIA-NeMo/Switchyard

Length of output: 44771

Use a writable routing-log path. --routing-log-file is supported, and the request must send "model": "switchyard" to select routes.stage.id. The server creates parent directories but does not change permissions, so document directory ownership or use a path writable by the server user instead of assuming /var/lib/switchyard.

🤖 Prompt for AI Agents 
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@docs/routing_algorithms/stage_router_routing.md` around lines 258 - 263,
Update the routing-log example in the documentation to use a path writable by
the server user, or explicitly document the required ownership and permissions
for /var/lib/switchyard. In the request example, set the model value to
"switchyard" so it matches routes.stage.id and selects the intended route.

This is the recommended default: routing on tool signals alone, no classifier.

### Optional: handoff notes
Expand Down
Loading