Skip to content

getting_started.md

Latest commit

 

History

History
244 lines (176 loc) · 7.42 KB

getting_started.md

File metadata and controls

244 lines (176 loc) · 7.42 KB
Raw

Getting Started with Switchyard

Switchyard has two native Rust execution paths:

  • Launcher path: install the Python-distributed CLI and launch Claude Code, Codex, or OpenClaw through the packaged Rust server binding.
  • Server path: build and run the standalone Rust server for API clients and custom deployments.

Launcher Path

Use this path when you want Switchyard to start and configure a supported coding agent for you.

Install the CLI

Install uv if it is not already available:

curl -LsSf https://astral.sh/uv/install.sh | sh
source "$HOME/.local/bin/env"

Then install the published Switchyard tool:

uv tool install "nemo-switchyard[cli,server]"

This creates an isolated Python tool environment containing the switchyard CLI, its CLI and server dependencies, and the packaged PyO3 Rust extension. switchyard launch starts the native Rust server through that extension; this path does not install or run the standalone switchyard-server binary.

Install the coding agent you want to launch, then verify both commands are on your PATH.

Launch with the packaged OpenRouter deployment

The packaged deployment exposes the route ID switchyard. Export an OpenRouter key and choose an agent:

export OPENROUTER_API_KEY="your-openrouter-key"  # pragma: allowlist secret
switchyard launch claude --model switchyard

Codex and OpenClaw use the same deployment:

switchyard launch codex --model switchyard
switchyard launch openclaw --model switchyard

Launch with a custom deployment

Pass a native TOML deployment and select one of its route IDs:

switchyard launch codex --model my-route --config routes.toml

The launcher manages the native server lifecycle and points the selected coding agent at it. The server configuration format is the same TOML schema used by the standalone server below.

Server Path

Use this path when you want a standalone proxy for API clients or need to operate the Rust server directly.

Prerequisites

  • Git, a native build toolchain, and Rust with Cargo
  • An API key for OpenRouter, OpenAI, Anthropic, or another OpenAI-compatible endpoint. To use OpenRouter, create an account at openrouter.ai and generate a key from the OpenRouter keys page.

On Ubuntu or WSL, install the build prerequisites and Rust with rustup:

sudo apt-get update
sudo apt-get install -y build-essential curl git
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
source "$HOME/.cargo/env"

On macOS or native Windows, follow the official Rust installation instructions. The Rust installer includes rustc, Cargo, and rustup.

Install uv for the repository's Python-based tooling and CI checks. It is not required to build or run the Rust server:

curl -LsSf https://astral.sh/uv/install.sh | sh

If either installer updates your shell configuration, restart the shell before continuing. Verify the tools:

git --version
rustc --version
cargo --version
uv --version

Build the server

Build the Rust server from source:

git clone https://github.com/NVIDIA-NeMo/Switchyard.git
cd Switchyard
cargo build --locked --release -p switchyard-server
./target/release/switchyard-server --help

The repository pins Rust 1.96.1 in rust-toolchain.toml; rustup selects and installs it automatically when Cargo runs from the repository. Prebuilt Rust binaries are not published yet.

Configure

The Rust server reads an explicit TOML file. It does not use the legacy Python CLI's saved configuration or YAML routing profiles.

Create routes.toml with an LLM-classifier route:

schema_version = 1

[llm_clients.openrouter]
format = "openai_chat"
base_url = "https://openrouter.ai/api/v1"
api_key_env = "OPENROUTER_API_KEY"

[targets.weak]
id = "openai/gpt-4o-mini"
llm_client = "openrouter"

[targets.strong]
id = "openai/gpt-4o"
llm_client = "openrouter"

[routes.smart]
id = "switchyard"
type = "llm_classifier"
mode = "capability"
classifier_target = "weak"
strong_target = "strong"
weak_target = "weak"
base_threshold = 0.5

format selects the upstream protocol and must be openai_chat, openai_responses, or anthropic_messages. api_key_env names the environment variable the server reads; the secret does not belong in the TOML file.

Run the server

Export the provider credential, validate the configuration without binding a socket, then start the release binary:

export OPENROUTER_API_KEY="your-openrouter-key"  # pragma: allowlist secret
./target/release/switchyard-server --config routes.toml --dry-run
./target/release/switchyard-server --config routes.toml \
  --host 127.0.0.1 --port 4000

Any client that speaks OpenAI Chat Completions, Anthropic Messages, or OpenAI Responses API can connect. The route id is the model name clients use.

In another terminal:

curl http://localhost:4000/health
curl http://localhost:4000/v1/models
curl http://localhost:4000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model":"switchyard","messages":[{"role":"user","content":"hello"}]}'

Routing algorithms

Choose a route type

This guide uses llm_classifier, which asks a classifier target whether each request should use the weak or strong target. The Rust server also supports:

Algorithm Use it when Config
Random You need a weighted split for A/B tests or baselines. random
LLM classifier Request content should decide whether to use the weak or strong target. llm_classifier
Stage router Tool-result and progress signals should select an efficient or capable target. stage_router

A single TOML file can declare multiple routes. The table key, such as routes.smart, is a local configuration name; each route's id is exposed as a model on GET /v1/models.

See Routing Overview to compare strategies, and the switchyard-server guide for the complete TOML schema, route options, TLS, and metrics.

Troubleshooting

No API key / auth error

test -n "$OPENROUTER_API_KEY" && echo "key is set" || echo "key is missing"
./target/release/switchyard-server --config routes.toml --dry-run

Confirm that api_key_env in routes.toml names the environment variable you exported. The dry run validates the schema, environment lookup, target references, and route construction without starting the server.

Connection refused

Check health: curl http://localhost:4000/health

Telemetry header opt-out

Switchyard adds an X-Switchyard-Version header to outbound LLM calls for release attribution. No request or response content is included. To disable:

export SWITCHYARD_TELEMETRY_OPT_OUT=1

Next steps