Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 6 additions & 6 deletions .agents/skills/switchyard-stage-router-scorer/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -34,12 +34,12 @@ trajectory step (tool_use + tool_result)
→ dc.process(ctx, request) # DimensionCollector — one per task, accumulates state
→ get_tool_result_signal(ctx) # read signal from ctx
→ stage_score_signal(signal) # raw (score, confidence) from the Rust scorer (for analysis)
pick_capable_first(ctx, threshold) # actual cf decision (same ctx, no re-process)
pick_efficient_first(ctx, threshold) # actual ef decision (same ctx, no re-process)
stage_pick_tier(signal, "capable_first", threshold) # actual cf decision
stage_pick_tier(signal, "efficient_first", threshold) # actual ef decision
```

**Key:** `dc.process()` is called **once per turn** on a single ctx. Both pickers read the signal
already stored in that ctx — no duplicate processing.
**Key:** `dc.process()` is called **once per turn** on a single context. Both picker modes use the
same extracted signal, with no duplicate processing.

**What the picker does beyond raw score:**
- **escalate** (`should_escalate`): `compacted` OR `severity >= 1.0` → force CAPABLE
Expand Down Expand Up @@ -78,7 +78,7 @@ opus_pct_cf, nemotron_pct_cf, opus_pct_ef, nemotron_pct_ef`
|------|-------------|------------|------------|
| strong_clear | ≥ T | Opus | Opus |
| strong_uncertain | (0, T) | Opus (fall_open) | Nemotron (fall_open) |
| weak_uncertain | (-T, 0) | Nemotron (fall_open) | Nemotron (fall_open) |
| weak_uncertain | (-T, 0) | Opus (fall_open) | Nemotron (fall_open) |
| weak_clear | ≤ -T | Nemotron | Nemotron |

Overrides can change any band. Use `pick_cf`/`pick_ef` for the true decision.
Expand All @@ -92,6 +92,6 @@ Overrides can change any band. Use `pick_cf`/`pick_ef` for the true decision.

## Anti-patterns

- Don't call `dc.process()` multiple times per turn for different pickers — both pickers read from the same ctx. One `dc.process()` call per turn is correct.
- Don't call `dc.process()` multiple times per turn for different picker modes. Extract one signal per turn and reuse it.
- Don't infer routing split from score bands alone — overrides and fall_open change the actual decision. Always use `pick_cf` / `pick_ef`.
- Don't compare cost directly across configs: Nemotron has ~39% cache hit rate vs ~92% for Opus.
4 changes: 2 additions & 2 deletions .github/ISSUE_TEMPLATE/bug_report.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,11 +12,11 @@ What happened? One or two sentences.

## Reproduction

Minimal steps to reproduce. Include the command line, the inbound request shape, and the configured recipe.
Minimal steps to reproduce. Include the command line, the inbound request shape, and the configured route.

```bash
# example
switchyard serve --config profiles.yaml --port 4000
switchyard serve --routes examples/route.yaml --port 4000
curl -s http://localhost:4000/v1/chat/completions -d '{"model":"...","messages":[...]}'
```

Expand Down
3 changes: 0 additions & 3 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -185,9 +185,6 @@ jobs:
uv run --isolated --no-project --python 3.14 \
--with "${SWITCHYARD_DEFAULT_PACKAGE}" \
python -c "import switchyard; print('import OK, version:', switchyard.__version__)"
uv run --isolated --no-project --python 3.12 \
--with "${SWITCHYARD_DEFAULT_PACKAGE}" \
python -c "from switchyard import RandomRoutingPresets; print('preset import OK:', sorted(RandomRoutingPresets.PRESETS))"
- name: Assert heavy packages absent from slim install
working-directory: /tmp
run: |
Expand Down
10 changes: 5 additions & 5 deletions .github/workflows/perf.yml
Original file line number Diff line number Diff line change
Expand Up @@ -56,7 +56,7 @@ jobs:

# ------------------------------------------------------------------
# Start a local zero-latency mock OpenAI upstream. This replaces the
# removed noop route: the proxy serves a real `type: model` chain that
# The proxy serves a real `type: passthrough` chain that
# forwards to this loopback stub, which returns a fixed completion
# instantly. The extra loopback hop adds a small constant overhead.
# ------------------------------------------------------------------
Expand Down Expand Up @@ -127,7 +127,7 @@ jobs:
echo "STUB_PID=$!" >> "$GITHUB_ENV"

# ------------------------------------------------------------------
# Start the proxy: a `type: model` route pointed at the local stub.
# Start the proxy: a `type: passthrough` route pointed at the local stub.
# ------------------------------------------------------------------
- name: Start switchyard proxy
run: |
Expand All @@ -138,10 +138,10 @@ jobs:
format: openai
routes:
mock-model:
type: model
model: mock-model
type: passthrough
target: mock-model
YAML
uv run switchyard serve --routing-profiles bench.yaml --port $PROXY_PORT &
uv run switchyard serve --routes bench.yaml --port $PROXY_PORT &
echo "PROXY_PID=$!" >> "$GITHUB_ENV"

- name: Wait for proxy to be ready
Expand Down
27 changes: 16 additions & 11 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -167,9 +167,7 @@ switchyard/
│ ├── roles.py # Python LLMBackend re-export and translation aliases
│ ├── switchyard.py # Switchyard — chain executor
│ ├── proxy_context.py # ProxyContext — per-request state carrier
│ ├── profiles/ # Profile configs/runtimes for pre-built routing behavior
│ ├── route_table.py # RouteTable — model-id dispatch to runnable chains
│ ├── route_table_builders.py # Shared profile-backed table builders
│ ├── llm_client.py # OpenAILLMClient
│ ├── cost_estimator.py # Token-cost bookkeeping
│ ├── stats_accumulator.py # Stats accumulation helpers
Expand All @@ -189,7 +187,6 @@ switchyard/
│ │ └── backend_format_resolver.py # BackendFormatResolver
│ ├── processors/ # Request-side / response-side component implementations
│ │ ├── format_translate.py
│ │ ├── random_routing_request_processor.py
│ │ ├── stats_request_processor.py
│ │ └── stats_response_processor_accumulator.py
│ ├── endpoints/ # FastAPI endpoint wrappers (require `nemo-switchyard[server]`)
Expand Down Expand Up @@ -245,8 +242,8 @@ and their transitives never appear in downstream vulnerability scans.
```bash
export OPENROUTER_API_KEY="sk-or-..."

# Serve a routing bundle. Routes live in YAML; see docs/routing_algorithms/overview.md.
switchyard serve --routing-profiles routes.yaml --port 4000
# Serve the minimal Python YAML bundle (noop and passthrough only).
switchyard serve --routes examples/route.yaml --port 4000

# Launch against the packaged OpenRouter deployment.
switchyard launch claude --model switchyard
Expand All @@ -264,7 +261,7 @@ uv run pytest tests/ -v

# Single test file / function
uv run pytest tests/test_switchyard.py -v
uv run pytest tests/test_random_routing_llm_backend.py::test_server_config -v
uv run pytest tests/test_route_bundle.py::test_noop_route_returns_ok_without_an_upstream -v

# Live end-to-end tests are not part of the public test suite; if you write
# one, set the provider key explicitly and run it directly, e.g.:
Expand All @@ -280,7 +277,7 @@ uv run mypy switchyard
1. Pick the right stage: request component (pre-call), response component (post-call), `LLMBackend` (rare), or Rust translation codec work.
2. Create a file with the explicit name (`snake_case` of the class name), one class per file.
3. Implement the async method for that stage (`process` for components, `call` for backends).
4. Wire into the owning programmatic profile config or route-bundle builder.
4. Wire it into the owning explicit chain or route-bundle builder.
5. Add tests under `tests/`.
6. Export from the relevant `__init__.py` and from `switchyard/__init__.py`'s `__all__`.

Expand All @@ -297,16 +294,24 @@ class MyRequestComponent:
return request
```

### Profiles and app factory
### App factory

```python
from switchyard import PassthroughProfileConfig, ProfileSwitchyard, build_switchyard_app
from switchyard import BackendFormat, LlmTarget, Switchyard, TranslationEngine
from switchyard.lib.backends import OpenAiNativeBackend
from switchyard import build_switchyard_app
import uvicorn

switchyard = ProfileSwitchyard(PassthroughProfileConfig(
target = LlmTarget(
model="gpt-4o",
format=BackendFormat.OPENAI,
api_key="sk-...",
base_url="https://api.openai.com/v1",
).build())
)
switchyard = Switchyard(
backend=OpenAiNativeBackend(target),
translator=TranslationEngine(),
)
uvicorn.run(build_switchyard_app(switchyard), port=4000)
```

Expand Down
10 changes: 5 additions & 5 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -17,11 +17,11 @@ adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
KV-cache-aware routing with request failover) or an external load balancer
such as [Traefik](https://doc.traefik.io/traefik/reference/routing-configuration/http/load-balancing/service/)
or HAProxy.
- **Public `type: noop` and `type: passthrough` route types** — removed from
route bundles. Use `type: model` to register a single explicit model target.
Catalog auto-discovery via a bare `type: passthrough` route is gone; there is
no `type: model` equivalent, so list the model ids you want as explicit
`type: model` routes.
- **Python routing profiles** — the `Profile` framework, typed profile configs,
and advanced Python route types are removed. The compatibility Python server
accepts only explicit `noop` and single-target `passthrough` routes through
`switchyard serve --routes`; routing algorithms remain available through
libsy and the Rust server.

### Fixed

Expand Down
13 changes: 10 additions & 3 deletions INSTALLATION.md
Original file line number Diff line number Diff line change
Expand Up @@ -109,13 +109,20 @@ pip install nemo-switchyard[all]
Embed Switchyard with minimal overhead:

```python
from switchyard import PassthroughProfileConfig, ProfileSwitchyard
from switchyard import LlmTarget, OpenAiNativeBackend, Switchyard, TranslationEngine

# Core library only — no server/CLI dependencies
switchyard = ProfileSwitchyard(PassthroughProfileConfig(
target = LlmTarget(
id="direct",
model="gpt-4o-mini",
format="openai",
api_key="sk-...",
base_url="https://api.openai.com/v1",
).build())
)
switchyard = Switchyard(
backend=OpenAiNativeBackend(target),
translator=TranslationEngine(),
)
```

## Troubleshooting
Expand Down
24 changes: 12 additions & 12 deletions benchmark/score_staged_run.py
Original file line number Diff line number Diff line change
Expand Up @@ -4,13 +4,13 @@
"""Score a benchmark run directory via the stage-router Rust scorer.

Reads trajectory.json from each completed task, feeds each tool-use turn through:
DimensionCollector.process(ctx, request) → pick_capable_first / pick_efficient_first
DimensionCollector.process(ctx, request) → stage_pick_tier

The picker functions replicate live routing exactly: overrides, scorer, and fall_open.

Usage:
uv run python benchmark/score_run.py --run benchmark/tb_runs/<run-name>
uv run python benchmark/score_run.py --run benchmark/tb_runs/<run-name> \\
uv run python benchmark/score_staged_run.py --run benchmark/tb_runs/<run-name>
uv run python benchmark/score_staged_run.py --run benchmark/tb_runs/<run-name> \\
--output /tmp/scores.jsonl --threshold 0.20 --window 3
"""
import argparse
Expand All @@ -22,19 +22,16 @@
from pathlib import Path
from statistics import mean

from switchyard.lib.processors.stage_router.picker import (
CAPABLE,
pick_capable_first,
pick_efficient_first,
)
from switchyard_rust.components import (
DimensionCollector,
get_tool_result_signal,
stage_pick_tier,
stage_score_signal,
)
from switchyard_rust.core import ChatRequest, ProxyContext

RECENT_WINDOW = 3
CAPABLE = 1


def _tool_use_id(message: str) -> str:
Expand Down Expand Up @@ -119,10 +116,13 @@ async def score_trajectory(
# histogram shows the capable/efficient separation directly.
score, confidence = stage_score_signal(signal)

# Actual picker decisions on the same ctx — both just read the signal,
# no extra dc.process() calls needed
tier_cf = await pick_capable_first(ctx, confidence_threshold)
tier_ef = await pick_efficient_first(ctx, confidence_threshold)
# Apply both picker modes to the same signal without reprocessing the turn.
outcome_cf = stage_pick_tier(signal, "capable_first", confidence_threshold)
outcome_ef = stage_pick_tier(signal, "efficient_first", confidence_threshold)
tier_cf_name = outcome_cf.tier if outcome_cf.resolved else outcome_cf.default_tier
tier_ef_name = outcome_ef.tier if outcome_ef.resolved else outcome_ef.default_tier
tier_cf = int(tier_cf_name == "capable")
tier_ef = int(tier_ef_name == "capable")

rows.append({
"task_name": task_name,
Expand Down
2 changes: 1 addition & 1 deletion crates/switchyard-components/src/backends/multi.rs
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@
//! This backend owns the mechanical part of routing once a processor or caller
//! has selected a target: rewrite the request model, stamp typed context, and
//! delegate to the configured backend for that target. Selection policy stays
//! outside this type; processors and profiles decide which target should run.
//! outside this type; processors and algorithms decide which target should run.

use std::collections::HashSet;
use std::fmt;
Expand Down
2 changes: 1 addition & 1 deletion crates/switchyard-components/src/stage_router.rs
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@
//! Its input, a [`ToolResultSignal`][crate::dimension_collector::ToolResultSignal],
//! is already libsy's `ToolSignals`, so no request adaptation is needed here — this
//! module simply re-exports the API so the crate's processors decide a turn's tier
//! through the same implementation the libsy profile uses.
//! through the same implementation as the libsy algorithm.

pub use switchyard_libsy::{
CodingAgentDimensions, DecisionSource, PickOutcome, PickerMode, ScoreResult, StageClassifier,
Expand Down
4 changes: 2 additions & 2 deletions crates/switchyard-components/src/stats/accumulator.rs
Original file line number Diff line number Diff line change
Expand Up @@ -138,7 +138,7 @@ impl StatsAccumulator {

/// Records usage after the corresponding success call was already attributed.
///
/// `StatsLlmBackend` and profile runtimes record success before the response
/// `StatsLlmBackend` and routing runtimes record success before the response
/// processor records tokens. Those internal paths must not consume a legacy
/// pending untiered success that belongs to some other direct accumulator caller.
pub fn record_usage_after_success_attribution(
Expand Down Expand Up @@ -281,7 +281,7 @@ impl StatsAccumulator {
Ok(())
}

/// Records one routing decision source for a profile family.
/// Records one routing decision source for a routing algorithm.
///
/// This is intentionally separate from model/tier accounting: a stage-router can
/// choose `efficient` because of an override, a dimensions score, an LLM-classifier
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -35,7 +35,7 @@ fn openai_request(model: &str) -> ChatRequest {
}))
}

// Runs one request through the engine and applies the profile-local model rewrite.
// Runs one request through the engine and applies the route-local model rewrite.
fn route_once(engine: &RandomRoutingEngine, mut request: ChatRequest) -> Result<ChatRequest> {
let decision = engine.select(request.model().map(str::to_owned))?;
request.set_model(decision.selected_model.as_str());
Expand Down
5 changes: 2 additions & 3 deletions crates/switchyard-py/src/component_bindings/stage_router.rs
Original file line number Diff line number Diff line change
Expand Up @@ -5,9 +5,8 @@
//!
//! Exposes [`switchyard_components::stage_router::pick_tier`] as
//! `stage_pick_tier(signal, picker_mode, confidence_threshold) -> PickOutcome`
//! so the Python `processor.py` runs the exact same routing decision as the Rust
//! profile. The async classifier and the `no_signal` case stay in Python: this
//! returns a resolved decision, or a request to consult the classifier.
//! for Python analysis tools. This returns a resolved decision or a request to
//! consult a classifier.

use pyo3::exceptions::PyValueError;
use pyo3::prelude::*;
Expand Down
4 changes: 2 additions & 2 deletions crates/switchyard-py/src/interop/subagent.rs
Original file line number Diff line number Diff line change
Expand Up @@ -3,10 +3,10 @@

//! Python binding for the canonical sub-agent detection policy.
//!
//! Profiles must not sniff lineage headers themselves: the fact (explicit
//! Routing implementations must not sniff lineage headers themselves: the fact (explicit
//! `x-switchyard-is-subagent`, Claude Code agent lineage, Codex/relay markers) and the
//! work-vs-maintenance policy both live in the protocol crate, so every engine — the libsy
//! classifier and the serve-path profile alike — answers "is this delegated work?"
//! classifier and the server alike — answers "is this delegated work?"
//! identically.

use std::collections::BTreeMap;
Expand Down
4 changes: 2 additions & 2 deletions docs/getting_started.md
Original file line number Diff line number Diff line change
Expand Up @@ -124,8 +124,8 @@ binaries are not published yet.

### Configure

The Rust server reads an explicit TOML file. It does not use the legacy Python
CLI's saved configuration or YAML routing profiles.
The Rust server reads an explicit TOML file. It does not use the Python
server's minimal YAML route bundle.

Create `routes.toml` with an LLM-classifier route:

Expand Down
3 changes: 2 additions & 1 deletion docs/internal/metrics_reference.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,8 @@ a drop-in scrape config and starter alert rules.
| Auth | None |
| Default scrape interval | 15s |

`GET /metrics` is served by `switchyard-server`.
`GET /metrics` is served by the Python route-bundle server started with
`switchyard serve --routes PATH` and by the native Rust server.

A JSON summary of the same traffic lives at `GET /v1/stats`.

Expand Down
Loading
Loading