[Skill proposal] building-vllm-mixture-of-models

[Xunzhuo]

Proposed skill name

building-vllm-mixture-of-models

Does something like this already exist?

Yes — as documentation, a runbook, or internal guide

Closest prior art:

• vLLM Semantic Router provides the routing runtime and documentation.
• lemonade-router-builder generates Lemonade-specific router policy JSON for client/local use.
• serving-llms-on-instinct serves individual models with vLLM on AMD Instinct.

The proposed skill is complementary: it helps an agent turn a set of heterogeneous model endpoints and workload goals into a validated, runnable Mixture-of-Models configuration using vLLM Semantic Router on AMD infrastructure.

Is AMD the only credible institution to create this skill?

Yes

AMD is uniquely positioned to provide first-party ROCm/Instinct deployment defaults, hardware-aware model-pool guidance, and validated performance recipes. AMD also sponsors vLLM Semantic Router with GPU resources and ROCm support. The general router semantics should continue to come from the upstream project; this skill should encode the AMD-specific, tested workflow.

Where should this skill live?

Path A: incubated in this repo (may move to a product repo and continbue to be registered here)

It can graduate to an appropriate AMD-owned product repository once ownership and release coupling are clear.

Catalog focus area

Application integration

Skill description

User goal

Help a developer build a working Mixture-of-Models system on AMD infrastructure with vLLM Semantic Router. The agent should translate natural-language goals such as quality, latency, cost, privacy, safety, and model specialization into a model pool and routing policy, then generate and validate the configuration and smoke tests.

A focused MVP should end with one measurable outcome: a schema-valid router configuration plus reproducible commands/tests showing that representative prompts reach the intended model paths. Deployment, benchmarking, and automatic policy tuning can be follow-on work if they make the first version too broad.

Trigger surface

Use when the user mentions:

• vLLM Semantic Router, vllm-sr, semantic routing, or programmable routing
• Mixture-of-Models / MoM, model pools, model cascades, or heterogeneous inference
• routing prompts by domain, complexity, modality, privacy, safety, latency, cost, or user preference
• selecting or composing models served by vLLM on ROCm / AMD Instinct
• router configuration, signals, decisions, route traces, fusion, fallback, or policy validation

Do not use for configuring a Lemonade collection.router policy; use lemonade-router-builder instead. Do not use merely to launch one vLLM endpoint; use serving-llms-on-instinct instead.

Platform scope

• Primary: server deployments on AMD Instinct GPUs with ROCm
• Development: local configuration generation and offline validation
• Model backends: existing OpenAI-compatible/vLLM endpoints across heterogeneous AMD-backed model pools
• Orchestration: start with the upstream-supported deployment path; avoid inventing a second control plane

Inputs

• Available models/endpoints and their capabilities
• Target AMD GPU/ROCm environment and deployment constraints
• Routing objectives and priority order (quality, latency, cost, privacy, safety)
• Representative prompts or an evaluation dataset
• Optional existing vLLM Semantic Router configuration

If required model IDs, endpoints, or success criteria are missing, the skill should ask rather than invent them. Secrets must remain in environment variables and must not be written into generated files.

Outputs

• A concise model-pool and routing-policy design with assumptions called out
• A generated vLLM Semantic Router configuration using upstream-supported schema and defaults
• Validation results and actionable diagnostics for invalid configuration
• Reproducible launch/apply commands appropriate for the detected AMD/ROCm environment
• A small smoke-test set with expected routes and route-trace checks
• A summary of unresolved trade-offs and the next benchmark to run

Proposed opinionated workflow

1. Detect and verify ROCm, GPU architecture, container/runtime, router version, and reachable model endpoints.
2. Inventory each model's capabilities and operational constraints; never guess unsupported hardware or model compatibility.
3. Convert the user's objectives into explicit routing signals, decisions, fallbacks, and failure behavior.
4. Generate the smallest viable upstream-compatible configuration.
5. Run deterministic schema/static validation before showing or applying it.
6. With user confirmation before external changes, launch/apply the configuration and run representative route-trace smoke tests.
7. Report which requests routed where, whether expectations passed, and what should be benchmarked next.

Suggested skill contents

• SKILL.md: decision workflow, AMD prerequisites, failure recovery, and boundaries with adjacent skills
• references/: pinned upstream configuration concepts plus AMD Instinct/ROCm deployment notes
• scripts/validate.py (or upstream validator wrapper): deterministic config and endpoint checks
• examples/: small/large model cascade, domain specialist pool, and privacy-aware routing
• evals/evals.py: positive/negative trigger tests and behavior checks that reject invented model IDs, leaked secrets, or unvalidated configs
• skill-card.md: ownership, license, risks, and supported environment

Acceptance criteria for the first version

• Produces an upstream-schema-valid configuration from a documented example prompt.
• Validates before presenting or applying configuration.
• Demonstrates at least three representative requests routing to expected model paths on an AMD Instinct/ROCm setup.
• Handles unreachable endpoints and invalid policies without leaving partial state.
• States supported router/ROCm versions and GPU architectures explicitly.
• Includes behavioral tests for triggering, non-triggering, missing inputs, validation failure, and route verification.