diff --git a/skills/hyperloom-workload-optimizer/SKILL.md b/skills/hyperloom-workload-optimizer/SKILL.md new file mode 100644 index 0000000..deae531 --- /dev/null +++ b/skills/hyperloom-workload-optimizer/SKILL.md @@ -0,0 +1,401 @@ +--- +name: hyperloom-workload-optimizer +description: >- + Autonomously optimizes end-to-end LLM inference throughput on AMD Instinct GPUs + and reports a validated gain, using the Hyperloom multi-agent optimizer. Given a + model, framework, workload (TP/EP, concurrency, ISL/OSL, precision), an objective + and a time budget, it explores per-workload which levers to pull (serving/config + parameters and env, framework enablement and source patches, and hot GPU-kernel + rewrites), benchmarks each candidate, and returns the optimization stack that + produced the gain. Use when the user wants to make a model serve faster, raise + tokens/sec or throughput, optimize or tune vLLM or SGLang on MI300X/MI325X/MI350X/MI355X, + run Hyperloom, run the kernel-agent, quantize-then-optimize with Quark, set up + Hyperloom from scratch, or resume a Hyperloom session. Do not use to stand up a + server for plain serving, diagnose a broken ROCm install, or run a one-off + kernel/benchmark or trace analysis without the optimization loop. +--- + + + +# Hyperloom Workload Optimizer + +You are the catalog entry point for Hyperloom optimization on AMD Instinct GPUs. +Bootstrap the workspace, prepare the runtime environment, collect workload +parameters, then install, launch, and monitor the optimizer. This skill owns the +orchestration and the launcher gates; environment prep and workload intake are +delegated to the skills the Hyperloom wheel installs, and +`@${HYPERLOOM_SKILL_PATH}` (`inference_optimizer`) is the execution baseline. + +Do not manually optimize inside chat unless debugging. + +## Prerequisites + +- AMD Instinct GPU host (MI300X / MI325X / MI350X / MI355X) with ROCm +- `/dev/kfd` and `/dev/dri` present; `amd-smi` or `rocm-smi` works +- Python 3.10+ and network access to install the Hyperloom wheel +- Anthropic (or compatible) LLM credentials for agent backends +- A dedicated agent workspace directory + +Every command in this skill runs on that GPU host. Confirm the shell you are in +is on it before Phase 0, so a bootstrap does not land on a machine with no GPU. + +The Hyperloom **runtime** ships via `pip install` of the published wheel. + +## What Hyperloom runs + +The CLI starts a Python Coordinator that coordinates: + +- **Orchestration** — baseline, explore, specialist, integrate_patch, sweep +- **Kernel** — trace_analyze, run_optimization, integrate +- **Critic** — proposal review (default `--critic-agent`) +- **Robustness** — health monitoring and RCA (default `--robustness-agent`) + +State lives under a **session directory** per run; run-state root is +`$USER_DATA_PATH` (default `/workspace/hyperloom`), independent of the install +directory (`INSTALL_DIR`, where the wheel and `.env` live) and may point to +shared storage. Layout: `$USER_DATA_PATH/runtime/` (install.sh outputs, +`kernel-agent.env.sh`), `logs/`, and `//` per session +holding `manifest.json`, `state.json`, `runs/`, `reports/`, `optimizer_runs/`. + +## Workflow overview + +Match `hyperloom-custom-advanced` section order — do **not** ask workload +questions while writing `.env` or during `/hyperloom-setup`. + +- **Phase 0 Bootstrap** — `pip install`, `/hyperloom-setup` → `.env` (credentials + run mode only) +- **Phase 1 Environment** — custom-advanced §Setup Configuration (baremetal: confirm host; docker: start container + setup inside, contract in [setup.md](setup.md)) +- **Phase 2 Workload intake** — custom-advanced §Advanced Configuration → Model Resolution → show launch plan → user confirms +- **Phase 3 Execute** — install.sh → preflight → launch → monitor → report + +Load `hyperloom-custom-advanced` at Phase 1 and follow its sections in order +(discovery: `.cursor/` / `.claude/` / `.agents/skills/hyperloom-custom-advanced/SKILL.md`). +If it is not on disk, stop and tell the user to restart the agent so the newly +installed skills are picked up — do not improvise the environment or workload +sections from memory, since the wheel is the source of truth for both. +For deeper optimizer behavior read `@${HYPERLOOM_SKILL_PATH}` (`inference_optimizer`); +Iron Rules + CLI reference: [reference.md](reference.md). + +## Iron Rules (launcher gates) + +Run order is always **IR-2 → IR-1 → launch**. Full text in [reference.md](reference.md). + +- **IR-1 — GPU unoccupied.** Before every `optimize` (fresh or `--resume`), every + visible GPU must have zero foreign serving PIDs (`sglang.launch_server` / + `vllm.entrypoints` / `Magpie`) and ≲ 500 MiB VRAM in use. +- **IR-2 — install.sh before launch.** Run `install.sh` and source + `kernel-agent.env.sh` in the **same shell** that spawns `optimize`. +- **Resume carve-out:** `--resume` may skip install only when `install.sh` exited + 0 earlier in the same shell, `kernel-agent.env.sh` is still sourced, and the + session's `manifest.json` exists. Any failure → re-run `install.sh`. + +## Phase discipline (do not skip) + +One phase at a time. Each phase asks only its own questions, waits for the +user's answers, completes its exit condition, then moves on. Never batch +questions from different phases into one prompt. In particular, never ask +workload questions (model, framework, TP/EP, precision, ISL/OSL, hours…) during +Phase 0 or Phase 1 — those belong to Phase 2 only. + +## Phase 0 — Bootstrap + +Skip completed steps (idempotent). Ask only about the install directory and +credentials/run mode here. Do not ask about the model or workload yet. + +### Confirm the install directory + +The wheel installs into a target directory with `pip install --target `, +which also holds `.env` and runtime artifacts. Do not silently use the current +directory. Show the resolved current directory (`pwd`) and confirm it with the +user, or let them choose another dedicated path. Wait for the answer, then `cd` +into the chosen directory before installing. + +### Install the Hyperloom wheel + +Skip when `hyperloom/` (wheel) or `src/hyperloom/` (source) already exists in the +confirmed directory. + +Find the latest release of +[AMD-AGI/Hyperloom](https://github.com/AMD-AGI/Hyperloom/releases), tell the user +that version, and ask whether to install it or a version they name. The wheel +asset is `hyperloom_inference_optimizer--py3-none-any.whl`. + +```bash +cd "$INSTALL_DIR" # the directory confirmed above +python3 -m pip install "" --target . +``` + +Confirm `hyperloom/inference_optimizer/assets/install.sh` exists. Restart the +agent if wheel skills are not visible. + +### Credentials and run mode + +Run `/hyperloom-setup` (installed to `.cursor/skills/hyperloom-setup/`). It +writes `.env`, sets `USER_DATA_PATH`, `HYPERLOOM_RUN_MODE`, and +`HYPERLOOM_SKILL_PATH`, and on bare metal runs `install_baremetal.sh`. + +**Phase 0 is done when all hold:** + +- `hyperloom/inference_optimizer/assets/install.sh` exists +- `.env` exists with non-placeholder LLM secrets +- `USER_DATA_PATH`, `HYPERLOOM_RUN_MODE`, and `HYPERLOOM_SKILL_PATH` are set + +More bootstrap detail: [setup.md](setup.md). + +## Phase 1 — Environment prep + +Load `hyperloom-custom-advanced` and follow its **Setup Configuration** +section only. + +**Baremetal (`HYPERLOOM_RUN_MODE=baremetal`):** confirm `install_baremetal.sh` +finished and the serving framework from setup is importable. Do not ask workload +questions yet. + +**Docker (`HYPERLOOM_RUN_MODE=docker`):** image choice, `docker run`, and the +in-container setup are owned entirely by custom-advanced Setup Configuration — +follow it, do not restate its commands or flags here. Do not ask workload +questions until the container is up and in-container setup succeeded, and never +run `optimize` on the host. + +Phase 1 is done when the target environment (host or container) is ready. + +## Phase 2 — Workload intake + +Enter only after Phase 0 and Phase 1 exit conditions hold. This is the first and +only phase that asks workload questions. + +Now follow custom-advanced **Advanced Configuration**, **Default Values**, and +**Model Resolution**. Use the agent's structured question UI when available. +Never copy API keys into chat output. + +| Field | CLI flag | Default | Notes | +|---|---|---|---| +| Model path | `--model` | required | Local dir with `config.json`, or HF cache | +| Framework | `--framework` | `sglang` | or `vllm`; prefer `.env` `FRAMEWORK` when set | +| TP / EP | `--tp` / `--ep` | `1` / `1` | tensor / expert parallel | +| CONC | `--conc` | `64` | client concurrency | +| ISL / OSL | `--isl` / `--osl` | `1024` / `1024` | input / output seq lengths | +| PRECISION | `--precision` | `bf16` | match checkpoint; `fp8` for FP8 models | +| MAX_HOURS | `--max-hours` | CLI `2.0` | skill recommends `8`; `0.5`–`3` for smoke | +| TARGET_GAIN | `--target-gain` | `30` | desired % gain | + +**Optional:** `--no-kernel`, `--no-explore`, `--no-framework-agent`, +`--no-enable-conc-sweep`, `--no-enable-roofline`, `--gpu-type`, `--server-args`, +`--compare-against-gpu`, phase budget percentages, `--quantize` prelude. + +Infer `PRECISION` from the model name when obvious (e.g. an `FP8` model implies +`--precision fp8`) and confirm it — do not silently keep the `bf16` default. + +### Confirmation gate (required before Phase 3) + +The Coordinator has no in-loop `setup` / `classify` — a value not asked here is +silently lost to its default. Before running any Phase 3 command, present the +full launch plan (including defaulted fields) and get explicit user confirmation. + +Print the plan in the reply body as this aligned block: + +```text +Launch plan — please confirm: + MODEL_PATH /wekafs/models/Qwen3-14B-FP8 + FRAMEWORK vllm + TP=1 EP=1 CONC=64 + ISL=1024 OSL=1024 + PRECISION=fp8 + MAX_HOURS=0.5 TARGET_GAIN=20% + phases --no-kernel (smoke) + RUN_MODE baremetal +``` + +Never put the plan inside the confirmation prompt itself. A prompt renders as one +wrapped paragraph, which collapses the alignment above into an unreadable blob the +user has to search for `MAX_HOURS` in. Keep the prompt to a single short question +such as `Approve this launch plan?`, and if you offer a "change something" option, +name the field to change rather than making the user retype it as free text. + +Do **not** run `install.sh` or launch `optimize` until the user approves this plan. + +### Persist the plan (required — shells do not share exports) + +Agent shells do not persist exports between calls, so write the confirmed values +to `$RUN_DIR/workload.env` right after approval. Every Phase 3 block sources it; +without this, launch silently falls back to `${TP:-1}` / `${CONC:-64}` defaults +and `--model ""`. Fill each value from the approved plan. + +```bash +export USER_DATA_PATH="${USER_DATA_PATH:?run /hyperloom-setup first}" +export RUN_DIR="${USER_DATA_PATH}/optimizer_runs" +mkdir -p "$RUN_DIR" +# Quoted heredoc (<<'EOF'): values are written literally, so a MODEL_PATH with +# spaces, $, or $(...) is not expanded or executed. Edit each value to the plan. +cat > "$RUN_DIR/workload.env" <<'EOF' +export MODEL_PATH=/wekafs/models/Qwen3-14B-FP8 +export FRAMEWORK=vllm +export TP=1 +export EP=1 +export CONC=64 +export ISL=1024 +export OSL=1024 +export PRECISION=fp8 +export MAX_HOURS=0.5 +export TARGET_GAIN=20 +export OPT_FLAGS="--no-kernel" # optional Phase 2 flags, space-separated; empty if none +EOF +``` + +## Phase 3 — Install (IR-2) + +Resolve paths for wheel or source layout: + +```bash +export REPO_ROOT="$(pwd -P)" +set -a; . "${REPO_ROOT}/.env"; set +a +export USER_DATA_PATH="${USER_DATA_PATH:?USER_DATA_PATH missing}" +. "${USER_DATA_PATH}/optimizer_runs/workload.env" # confirmed Phase 2 values +export PYTHONPATH="${REPO_ROOT}:${PYTHONPATH:-}" +ulimit -Sn 65536 || true + +INSTALL_SH="${REPO_ROOT}/hyperloom/inference_optimizer/assets/install.sh" +[ -f "$INSTALL_SH" ] || INSTALL_SH="${REPO_ROOT}/src/hyperloom/inference_optimizer/assets/install.sh" + +bash "$INSTALL_SH" +. "${KERNEL_AGENT_ENV:-${USER_DATA_PATH}/runtime/kernel-agent.env.sh}" +export PYTHONPATH="${REPO_ROOT}:${PYTHONPATH:-}" +``` + +In Docker mode, run this inside the container. + +## Phase 3 — Preflight (IR-1) + +`install.sh` exports `$PYTHON`; the fallback below covers agent sandboxes that do +not persist exports between shell calls. + +```bash +export SKILL_DIR="${SKILL_DIR:?absolute path of the directory holding this SKILL.md}" +. "${USER_DATA_PATH}/optimizer_runs/workload.env" # confirmed Phase 2 values +export PYTHON="${PYTHON:-$(command -v python3)}" +"$PYTHON" "${SKILL_DIR}/scripts/preflight.py" +``` + +The gate exits non-zero — do not launch — when `MODEL_PATH` is missing or has no +`config.json`, torch sees no GPU, a foreign serving process still holds a card, +or any GPU holds more than `IR1_VRAM_LIMIT_MIB` (default 500) MiB. + +It also blocks when VRAM cannot be read at all: no `amd-smi`/`rocm-smi` on +`PATH`, a probe that exits non-zero, or output it cannot parse. An unreadable +probe cannot rule out a busy GPU, and a foreign process holding VRAM under a +different name would slip through. Confirm the GPUs are idle by hand before +re-running with `IR1_ALLOW_UNVERIFIED_VRAM=1`. + +Never print API keys or tokens. `scripts/tests/test_preflight.py` covers the +probe shapes this gate must reject. + +## Phase 3 — Launch + +After IR-2 and IR-1 pass, launch. `setsid nohup` is required for runs longer than +5 minutes, so the run outlives the agent shell. + +```bash +export REPO_ROOT="$(pwd -P)" +export SKILL_DIR="${SKILL_DIR:?absolute path of the directory holding this SKILL.md}" +bash "${SKILL_DIR}/scripts/launch.sh" +``` + +Every workload value comes from the confirmed `workload.env`; the script has no +`${VAR:-default}` fallbacks, so a missing value fails loudly instead of launching +a different config. Put any optional Phase 2 flags (`--no-kernel`, `--no-explore`, +`--gpu-type`, `--model-class`, `--server-args`, `--compare-against-gpu`, +`--quantize`, phase budget flags) into `OPT_FLAGS` in `workload.env`. `OPT_FLAGS` +is word-split, so quote any flag value that contains spaces, e.g. +`export OPT_FLAGS='--server-args "--foo bar"'`. + +### Launch health check (30 s after start) + +Required after every launch and resume. The PID recorded at launch is the +**setsid wrapper**, which exits immediately — it is NOT the optimizer. This reads +the real `.pid` and `.session_dir` from the launch-info JSON, rewrites the PID +file so the monitor watches the right process, and records both in +`$RUN_DIR/last_launch.env` for the later phases. + +```bash +export REPO_ROOT="$(pwd -P)" +export SKILL_DIR="${SKILL_DIR:?absolute path of the directory holding this SKILL.md}" +bash "${SKILL_DIR}/scripts/launch_health.sh" +``` + +It exits non-zero when the launch-info JSON never appeared, no optimizer process +can be found, or `session_dir` is still unset — inspect the reported run log in +those cases. Never guess `session_dir` from a timestamp; concurrent sessions +share `USER_DATA_PATH`. + +## Phase 3 — Monitor + +Poll at most every 5 minutes unless debugging a startup failure. Use the state +reader the wheel ships rather than parsing `state.json` by hand — it also prints +the recent lifecycle events. + +```bash +export REPO_ROOT="$(pwd -P)" +. "${USER_DATA_PATH}/optimizer_runs/last_launch.env" # SESSION_DIR from launch +STATE_TOOL="${REPO_ROOT}/hyperloom/inference_optimizer/tools/read_optimizer_state.py" +[ -f "$STATE_TOOL" ] || STATE_TOOL="${REPO_ROOT}/src/hyperloom/inference_optimizer/tools/read_optimizer_state.py" +"${PYTHON:-python3}" "$STATE_TOOL" "$SESSION_DIR" +``` + +For recent action counts grouped by category, the wheel also ships +`tools/event_counts.py`, invoked the same way. + +Report session id + log path, `baseline_tput` / `current_best` / +`cumulative_gain`, explore accepted/rejected, last kernel opt (correctness, +speedup, KEEP/REVERT), and process-alive vs `stop_reason`. See +[reference.md](reference.md) Report fields. + +## Resume + +Resume runs in a fresh shell. Re-run the IR-2 and IR-1 gates first, exactly as for +a fresh launch — the script does not re-check them. + +```bash +export REPO_ROOT="$(pwd -P)" +export SKILL_DIR="${SKILL_DIR:?absolute path of the directory holding this SKILL.md}" +bash "${SKILL_DIR}/scripts/resume.sh" +bash "${SKILL_DIR}/scripts/launch_health.sh" +``` + +It resumes the session recorded in `last_launch.env` and always passes +`--resume-from` explicitly, because a bare `--resume` auto-picks the newest +session and can target the wrong run. Resume writes its own log +(`run_resume-*.log`) so the original run log is preserved. Reuse the IR-2 +carve-out rules; re-run `install.sh` if the shell or env changed. + +| `stop_reason` | Action | +|---|---| +| `time_exhausted` | `--resume` same session | +| `no_more_leverage` | stop; resume only if user changes strategy | +| `policy_loop` | inspect `policy_denial_history`; clear stale prunes | + +## Expected optimizer flow + +1. Establish `baseline_tput`. +2. Coordinator runs roofline/profile analysis after baseline. +3. `explore` tests serving parameters incrementally. +4. Kernel-agent runs on hot paths with compile + correctness evidence. +5. `sweep` validates concurrency around the best candidate. +6. Final report under `$SESSION_DIR/reports/`. + +## When to defer + +- **Plain serving only** — use `serving-llms-on-instinct`. +- **ROCm driver broken** — diagnose the ROCm stack first (e.g. a `rocm-doctor` + skill if published); do not start the optimizer on a broken driver. +- **Edge cases** — read `@${HYPERLOOM_SKILL_PATH}` for multi-node, atom + framework (IR-8), critic/robustness backends, cache topology, and the + full failure matrix. + +## Further reading + +- Bootstrap detail: [setup.md](setup.md) +- Iron Rules + CLI reference: [reference.md](reference.md) +- Authoritative runtime skill: `hyperloom/inference_optimizer/SKILL.md` diff --git a/skills/hyperloom-workload-optimizer/evals/evals.py b/skills/hyperloom-workload-optimizer/evals/evals.py new file mode 100644 index 0000000..ca5888d --- /dev/null +++ b/skills/hyperloom-workload-optimizer/evals/evals.py @@ -0,0 +1,95 @@ +# Copyright (c) 2026 Advanced Micro Devices, Inc. All rights reserved. +# +# See LICENSE for license information. + +"""Behavioral smoke tests for the `hyperloom-workload-optimizer` skill. + +Run locally (needs the `claude` CLI authenticated): + + pip install -r eval/behavioral/requirements.txt + cd eval/behavioral + python -m pytest -c pytest.ini -p conftest \ + ../../skills/hyperloom-workload-optimizer/evals/evals.py +""" + +from harness import claude + + +def test_routes_optimize_vllm_throughput_request(): + with claude("opus", skill="hyperloom-workload-optimizer") as agent: + run = agent.prompt( + "I want to optimize vLLM inference throughput on an MI300X. " + "What are the first steps before launching Hyperloom?" + ) + + run.logs_contains("hyperloom-workload-optimizer") + + run.should( + "Mention installing the Hyperloom wheel or checking for the hyperloom package" + ) + run.should( + "Mention workspace bootstrap such as hyperloom-setup or setup.md" + ) + run.should( + "Describe a phased flow where workload parameters (model path, " + "framework, TP, concurrency, ISL, OSL, precision, time budget) are " + "collected in a later workload-intake phase, not during bootstrap" + ) + run.should( + "Say it will present a launch plan and get user confirmation before " + "launching the optimizer" + ) + run.should( + "Mention running install.sh and sourcing kernel-agent.env.sh (IR-2) " + "before launching the optimizer" + ) + run.should( + "Mention a GPU preflight check for stale serving processes or VRAM " + "in use (IR-1)" + ) + run.should( + "Explain that confirmed workload values are persisted (e.g. to a " + "workload.env file) and sourced at launch, since agent shells do not " + "keep exports between calls" + ) + + run.should_not( + "Start a plain vLLM docker serve as the primary answer without the optimization loop" + ) + run.should_not( + "Launch the optimizer immediately after setup without collecting TP, " + "concurrency, ISL, OSL, and precision or confirming a launch plan" + ) + + +def test_phase_discipline_bootstrap_first(): + with claude("opus", skill="hyperloom-workload-optimizer") as agent: + run = agent.prompt( + "I have a fresh empty workspace. Help me get Hyperloom set up from " + "scratch so I can optimize a model later." + ) + + run.should( + "Focus on bootstrap first: confirm the install directory, install " + "the wheel, and run hyperloom-setup for credentials and run mode" + ) + run.should_not( + "Ask for workload parameters like model path, TP, ISL, OSL, or " + "precision in the same turn as install-directory or run-mode setup" + ) + run.should_not( + "Launch hyperloom.inference_optimizer.cli optimize before the " + "environment is prepared and a launch plan is confirmed" + ) + + +def test_declines_plain_serving_request(): + with claude("opus", skill="hyperloom-workload-optimizer") as agent: + run = agent.prompt( + "Just start a vLLM server on MI300X for Qwen3-8B, no optimization." + ) + + run.should( + "Decline plain serving or redirect to serving-llms-on-instinct or a serving workflow" + ) + run.should_not("Launch hyperloom.inference_optimizer.cli optimize for plain serving") diff --git a/skills/hyperloom-workload-optimizer/reference.md b/skills/hyperloom-workload-optimizer/reference.md new file mode 100644 index 0000000..221bfdb --- /dev/null +++ b/skills/hyperloom-workload-optimizer/reference.md @@ -0,0 +1,139 @@ +# Hyperloom Workload Optimizer Reference + +Iron Rules, CLI flags, and launcher contracts for +[SKILL.md](SKILL.md). The packaged Hyperloom optimizer skill +(`hyperloom/inference_optimizer/SKILL.md` after wheel install) is the +authoritative source for edge cases. + +## Table of contents + +1. [Iron Rules](#iron-rules) +2. [CLI workload flags](#cli-workload-flags) +3. [Critic and robustness backends](#critic-and-robustness-backends) +4. [Framework selection](#framework-selection) +5. [Failure signals](#failure-signals) +6. [Report fields](#report-fields) + +## Iron Rules + +Launcher gates that must hold before `python -m hyperloom.inference_optimizer.cli optimize`. + +### IR-1 — GPU unoccupied before every launch + +Before every `optimize` (fresh or `--resume`), verify every visible GPU has +**zero foreign serving PIDs and ≲ 500 MiB VRAM in use**. Leftover +`sglang.launch_server` / `vllm.entrypoints` / `Magpie` processes silently +degrade the next baseline. + +### IR-2 — install.sh before every launch + +Run `bash "$INSTALL_SH"` and source +`${KERNEL_AGENT_ENV:-${USER_DATA_PATH}/runtime/kernel-agent.env.sh}` in the +**same shell** that spawns `optimize`. Skipping install fails after baseline: +missing TraceLens/GEAK, hung Ray tasks, or `401` on kernel-opt gateway calls. + +**Resume carve-out:** `--resume` may skip install only when all hold: + +1. `install.sh` exited 0 earlier in the *same shell* +2. `kernel-agent.env.sh` is still sourced +3. The resumed session's `manifest.json` exists + +Any failure → treat as a fresh launch and re-run `install.sh`. + +### IR-3 — KB + PR Monitor (soft degrade) + +`_preflight()` runs `preflight_kb.sh`. Exit `1` auto-enables `--degraded-kb` / +`--degraded-pr`; launch continues. IR-3 never aborts. + +### IR-4 / IR-6 — EXPLORE contracts (Coordinator-internal) + +- **IR-4:** EXPLORE is specialist-informed; GPU specialists lease cards via + `gpu_research_lane` and must not touch production serving on port 8888. +- **IR-6:** EXPLORE force-exits when wall-clock remaining < + `--explore-force-exit-hours-remaining` (default 3 h) or phase budget < + `--explore-force-exit-budget-pct` (default 20%). +- Plateau signals are advisory; IR-6 and per-phase budgets are hard gates. + +### IR-8 — `--framework atom` is single-node only + +`--framework atom` rejects `--nodes >= 2` with exit code 2. + +## CLI workload flags + +Pass workload values as CLI flags — they are the source of truth for the +Coordinator. + +| Surface | CLI flag | Notes | +|---|---|---| +| Model path | `--model` | required | +| Framework | `--framework` | `sglang` (default) / `vllm` / `atom` / `xdit` | +| GPU type | `--gpu-type` | rocm-smi auto-detect when unset | +| Model class | `--model-class` | categorical key for seed grids and recipes | +| Input seq length | `--isl` | default `1024` | +| Output seq length | `--osl` | default `1024` | +| Concurrency | `--conc` | default `64`; use `--conc-sweep-concs` for a ladder | +| Tensor parallel | `--tp` | default `1` | +| Expert parallel | `--ep` | default `1` (MoE) | +| Precision | `--precision` | `bf16` default / `fp8` / ... | +| Budget | `--max-hours` | CLI parser default `2.0`; this skill's Phase 2 workflow recommends `8` (aligns with hyperloom-custom-advanced) — launch always passes it explicitly | +| Max model len | `--max-model-len` | auto-derived from ISL+OSL when omitted | +| Reference GPU | `--compare-against-gpu` | optional external baseline | +| Quantization prelude | `--quantize` | runs quantization-agent once before the loop | + +### Quantization prelude + +When the user asks to quantize then optimize: + +```bash +python3 -m hyperloom.inference_optimizer.cli optimize \ + --model "$MODEL_PATH" \ + --framework vllm \ + --quantize "fp8 global scheme, fp8 kv_cache, exclude lm_head" \ + --max-hours 4 +``` + +Ignored on `--resume`. + +## Critic and robustness backends + +| Mode | Flag | When | +|---|---|---| +| Live critic | `--critic-agent` (default) | production runs | +| Mock critic | `--critic-mock` | offline / smoke | +| Live robustness | `--robustness-agent` (default) | single-node production | +| Mock robustness | `--robustness-mock` | multi-node auto-downgrade or smoke | + +Multi-node (`--nodes >= 2`): CLI auto-downgrades robustness to mock (heartbeat +only) because local probes false-positive across pods. + +## Framework selection + +| Framework | Serving | Multi-node | Notes | +|---|---|---|---| +| `sglang` | yes | yes | default | +| `vllm` | yes | yes | | +| `atom` | yes | **no** | Magpie atom entrypoint | +| `xdit` | no | varies | diffusion `img/s` | + +## Failure signals + +| Symptom | Action | +|---|---| +| `stop_reason=no_more_leverage` | stop and report; resume only if user changes strategy | +| `stop_reason=time_exhausted` | `--resume` same session | +| `stop_reason=policy_loop` | inspect `policy_denial_history`; clear stale prunes before retry | +| `correctness_passed=false` | do not integrate kernel patch | +| `No accelerator` (Magpie) | fix `PATH` / `ROCR_VISIBLE_DEVICES` | +| Optimizer died before launch-info JSON | inspect `RUN_LOG`; never guess `session_dir` by timestamp | + +## Report fields + +Report back: + +- session id (`manifest.json`) and log path +- `cumulative_gain`, `current_best`, `baseline_tput` +- explore accepted/rejected summary +- last kernel opt: correctness, micro speedup, E2E gain, KEEP/REVERT +- process alive vs `stop_reason` + +Never print API keys or tokens. diff --git a/skills/hyperloom-workload-optimizer/scripts/_env.sh b/skills/hyperloom-workload-optimizer/scripts/_env.sh new file mode 100644 index 0000000..4441cf1 --- /dev/null +++ b/skills/hyperloom-workload-optimizer/scripts/_env.sh @@ -0,0 +1,52 @@ +# Copyright (c) 2026 Advanced Micro Devices, Inc. All rights reserved. +# +# See LICENSE for license information. + +# Shared Phase 3 preamble, sourced by launch.sh, launch_health.sh and resume.sh. +# Agent shells do not persist exports, so every entry point rebuilds the same +# environment here in a fixed order: .env, then workload.env, then the +# kernel-agent env written by install.sh. +# +# Sourced, not executed: a failure here exits the calling script. + +: "${REPO_ROOT:="$(pwd -P)"}" +cd "$REPO_ROOT" || exit 1 + +if [ -f "$REPO_ROOT/.env" ]; then + # Values containing spaces must be double-quoted in .env, otherwise this + # source fails with exit 127. hyperloom-setup writes them quoted. + set -a + . "$REPO_ROOT/.env" + set +a +fi + +: "${USER_DATA_PATH:?USER_DATA_PATH missing -- run the Hyperloom setup skill first}" +export REPO_ROOT USER_DATA_PATH +export RUN_DIR="${USER_DATA_PATH}/optimizer_runs" +mkdir -p "$RUN_DIR" + +WORKLOAD_ENV="${RUN_DIR}/workload.env" +if [ ! -f "$WORKLOAD_ENV" ]; then + echo "ERROR: $WORKLOAD_ENV missing -- re-run the Phase 2 'Persist the plan' step" >&2 + exit 1 +fi +# Confirmed Phase 2 values. Deliberately no ${VAR:-default} fallbacks so a +# missing value fails loudly instead of launching a different config. +. "$WORKLOAD_ENV" +: "${MODEL_PATH:?MODEL_PATH empty -- re-run the Phase 2 'Persist the plan' step}" + +KERNEL_AGENT_ENV="${KERNEL_AGENT_ENV:-${USER_DATA_PATH}/runtime/kernel-agent.env.sh}" +if [ ! -f "$KERNEL_AGENT_ENV" ]; then + echo "ERROR: $KERNEL_AGENT_ENV missing -- run IR-2 (install.sh) first" >&2 + exit 1 +fi +export KERNEL_AGENT_ENV +. "$KERNEL_AGENT_ENV" + +export PYTHON="${PYTHON:-$(command -v python3)}" +export PATH="$(dirname "$PYTHON"):/usr/local/bin:$PATH" +export PYTHONPATH="${REPO_ROOT}:${PYTHONPATH:-}" + +# Run handles are recorded here because the health check, monitor and resume +# steps each run in a fresh shell. +LAST_LAUNCH_ENV="${RUN_DIR}/last_launch.env" diff --git a/skills/hyperloom-workload-optimizer/scripts/launch.sh b/skills/hyperloom-workload-optimizer/scripts/launch.sh new file mode 100644 index 0000000..0c7f8d5 --- /dev/null +++ b/skills/hyperloom-workload-optimizer/scripts/launch.sh @@ -0,0 +1,56 @@ +#!/usr/bin/env bash +# Copyright (c) 2026 Advanced Micro Devices, Inc. All rights reserved. +# +# See LICENSE for license information. +# +# Start a fresh optimize run in the background. Run IR-2 (install.sh) and IR-1 +# (preflight.py) first; this script does not re-check the gates. +# +# setsid nohup is required: runs outlive the agent shell, which can die on an +# SSH disconnect. Every workload value comes from workload.env. + +set -euo pipefail + +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd -P)" +# shellcheck source=./_env.sh +. "${SCRIPT_DIR}/_env.sh" + +RUN_TAG="$(basename "$MODEL_PATH")-$(date +%Y%m%d_%H%M%S)" +RUN_LOG="${RUN_DIR}/run_${RUN_TAG}.log" +PID_FILE="${RUN_DIR}/run_${RUN_TAG}.pid" +LAUNCH_INFO_FILE="${RUN_DIR}/launch_${RUN_TAG}.json" + +# OPT_FLAGS holds the optional Phase 2 flags and is left unquoted on purpose so +# it word-splits into separate arguments. +# shellcheck disable=SC2086 +setsid nohup "$PYTHON" -m hyperloom.inference_optimizer.cli --verbose optimize \ + --model "$MODEL_PATH" \ + --framework "$FRAMEWORK" \ + --tp "$TP" \ + --ep "$EP" \ + --conc "$CONC" \ + --isl "$ISL" \ + --osl "$OSL" \ + --precision "$PRECISION" \ + --max-hours "$MAX_HOURS" \ + --target-gain "$TARGET_GAIN" \ + --tick-interval-sec 30 \ + --launch-info-file "$LAUNCH_INFO_FILE" \ + ${OPT_FLAGS:-} \ + > "$RUN_LOG" 2>&1 < /dev/null & + +# This is the setsid wrapper pid, which exits immediately; launch_health.sh +# replaces it with the real optimizer pid from the launch-info JSON. +echo $! > "$PID_FILE" + +cat > "$LAST_LAUNCH_ENV" <&2 + exit 1 +fi +. "$LAST_LAUNCH_ENV" + +sleep "${LAUNCH_HEALTH_DELAY_SEC:-30}" + +if [ ! -f "$LAUNCH_INFO_FILE" ]; then + echo "ERROR: $LAUNCH_INFO_FILE not written; inspect $RUN_LOG" >&2 + exit 1 +fi + +read_json() { + "$PYTHON" -c 'import json,sys;print(json.load(open(sys.argv[1])).get(sys.argv[2],""))' \ + "$1" "$2" 2>/dev/null || true +} + +REAL_PID="$(read_json "$LAUNCH_INFO_FILE" pid)" +if [ -z "$REAL_PID" ]; then + REAL_PID="$(pgrep -f 'hyperloom.inference_optimizer.cli .*optimize' | head -1 || true)" +fi +if [ -z "$REAL_PID" ]; then + echo "ERROR: optimizer pid not found; inspect $RUN_LOG" >&2 + exit 1 +fi +echo "$REAL_PID" > "$PID_FILE" + +if [ -d "/proc/${REAL_PID}" ]; then + echo "optimizer_alive=true pid=${REAL_PID}" +else + echo "ERROR: pid ${REAL_PID} is already gone; inspect $RUN_LOG" >&2 + exit 1 +fi + +SESSION_DIR="$(read_json "$LAUNCH_INFO_FILE" session_dir)" +if [ -z "$SESSION_DIR" ]; then + echo "ERROR: no session_dir yet in $LAUNCH_INFO_FILE; inspect $RUN_LOG" >&2 + exit 1 +fi + +cat > "$LAST_LAUNCH_ENV" < None: + print(f"IR-1 BLOCK: {message}", file=sys.stderr) + + +def _warn(message: str) -> None: + print(f"IR-1 WARNING: {message}", file=sys.stderr) + + +def check_model_path() -> bool: + path = os.environ.get("MODEL_PATH", "").strip() + if not path: + _fail("MODEL_PATH is empty; re-run the Phase 2 'Persist the plan' step") + return False + if not os.path.isdir(path): + _fail(f"MODEL_PATH is not a directory: {path}") + return False + if not os.path.isfile(os.path.join(path, "config.json")): + _fail(f"MODEL_PATH has no config.json: {path}") + return False + print(f"model_path_ok={path}") + return True + + +def check_torch_gpus() -> bool: + try: + import torch + except Exception as exc: + _fail(f"torch is not importable ({type(exc).__name__}); run install.sh first") + return False + try: + available = torch.cuda.is_available() + count = torch.cuda.device_count() if available else 0 + except Exception as exc: + _fail(f"torch GPU probe raised {type(exc).__name__}: {str(exc)[:200]}") + return False + print(f"torch_cuda_available={available} torch_cuda_device_count={count}") + if not available or count == 0: + _fail("torch sees no GPU; check ROCm, /dev/kfd and /dev/dri") + return False + return True + + +def check_foreign_processes() -> bool: + # Report the matched pattern and pid only; a serving cmdline can carry tokens. + found = [] + for pid in filter(str.isdigit, os.listdir("/proc")): + try: + with open(f"/proc/{pid}/cmdline", "rb") as handle: + raw = handle.read() + except OSError: + continue + text = raw.replace(b"\0", b" ").decode("utf-8", "ignore") + if not text: + continue + for pattern in FOREIGN_SERVING_PATTERNS: + if pattern in text: + found.append((pid, pattern)) + break + for pid, pattern in found: + print(f"foreign_serving_process pid={pid} matched={pattern}") + if found: + _fail(f"{len(found)} foreign serving process(es) still hold the GPUs") + return False + print("foreign_serving_processes=0") + return True + + +def _to_mib(value: object, unit: object) -> float: + if isinstance(value, bool) or not isinstance(value, (int, float, str)): + raise VramUnreadable(f"unsupported VRAM value type: {type(value).__name__}") + try: + number = float(value) + except (TypeError, ValueError): + raise VramUnreadable(f"VRAM value is not numeric: {value!r}") from None + key = str(unit or "mib").strip().lower() + if key not in UNIT_TO_MIB: + raise VramUnreadable(f"unknown VRAM unit: {unit!r}") + return number * UNIT_TO_MIB[key] + + +def _extract_used_vram(entry: object) -> float: + if not isinstance(entry, dict): + raise VramUnreadable(f"GPU entry is {type(entry).__name__}, expected an object") + containers = [entry] + for key in MEM_CONTAINER_KEYS: + nested = entry.get(key) + if isinstance(nested, dict): + containers.append(nested) + for container in containers: + for key in USED_VRAM_KEYS: + if key not in container: + continue + reading = container[key] + if isinstance(reading, dict): + if "value" not in reading: + raise VramUnreadable(f"{key} object has no 'value' field") + return _to_mib(reading.get("value"), reading.get("unit")) + return _to_mib(reading, "mib") + raise VramUnreadable(f"no used-VRAM field found; keys={sorted(entry)[:8]}") + + +def _run_probe(argv: list[str]) -> str: + try: + completed = subprocess.run( + argv, capture_output=True, text=True, timeout=PROBE_TIMEOUT_SEC + ) + except subprocess.TimeoutExpired: + raise VramUnreadable(f"{argv[0]} timed out after {PROBE_TIMEOUT_SEC}s") from None + except OSError as exc: + raise VramUnreadable(f"{argv[0]} could not be executed: {exc}") from None + if completed.returncode != 0: + detail = (completed.stderr or completed.stdout or "").strip()[:200] + raise VramUnreadable(f"{argv[0]} exited {completed.returncode}: {detail}") + if not completed.stdout.strip(): + raise VramUnreadable(f"{argv[0]} produced no output") + return completed.stdout + + +def _parse_json(payload: str, tool: str) -> object: + try: + return json.loads(payload) + except ValueError as exc: + raise VramUnreadable(f"{tool} output is not valid JSON: {exc}") from None + + +def _iter_gpu_entries(parsed: object, tool: str) -> list[tuple[object, object]]: + if isinstance(parsed, list): + return list(enumerate(parsed)) + if isinstance(parsed, dict): + # A known container key wins over the per-key scan below, so a sibling + # metadata object is not mistaken for a GPU entry. + for key in GPU_LIST_KEYS: + nested = parsed.get(key) + if isinstance(nested, list): + return list(enumerate(nested)) + entries = [(key, value) for key, value in parsed.items() if isinstance(value, dict)] + if entries: + return entries + raise VramUnreadable(f"{tool} output has no recognizable GPU list") + + +def read_vram_usage() -> list[tuple[object, float]]: + if shutil.which("amd-smi"): + tool = "amd-smi" + payload = _run_probe(["amd-smi", "metric", "-m", "--json"]) + elif shutil.which("rocm-smi"): + tool = "rocm-smi" + payload = _run_probe(["rocm-smi", "--showmeminfo", "vram", "--json"]) + else: + raise VramUnreadable("neither amd-smi nor rocm-smi is on PATH") + + entries = _iter_gpu_entries(_parse_json(payload, tool), tool) + readings: list[tuple[object, float]] = [] + for device, entry in entries: + try: + readings.append((device, _extract_used_vram(entry))) + except VramUnreadable as exc: + raise VramUnreadable(f"{tool} gpu {device}: {exc}") from None + if not readings: + raise VramUnreadable(f"{tool} reported no GPUs") + return readings + + +def check_vram() -> bool: + limit = int(os.environ.get("IR1_VRAM_LIMIT_MIB", DEFAULT_VRAM_LIMIT_MIB)) + allow_unverified = os.environ.get("IR1_ALLOW_UNVERIFIED_VRAM", "").strip() == "1" + try: + readings = read_vram_usage() + except VramUnreadable as exc: + if allow_unverified: + _warn( + f"VRAM unreadable ({exc}); proceeding because " + "IR1_ALLOW_UNVERIFIED_VRAM=1. Confirm the GPUs are idle by hand." + ) + return True + _fail( + f"VRAM unreadable ({exc}). A busy GPU cannot be ruled out, so the " + "launch is blocked. Fix the probe, or set " + "IR1_ALLOW_UNVERIFIED_VRAM=1 after confirming the GPUs are idle." + ) + return False + + over_limit = [] + for device, mib in readings: + marker = "OVER_LIMIT" if mib > limit else "ok" + print(f"gpu {device}: used_vram_mib={mib:.0f} ({marker})") + if mib > limit: + over_limit.append(device) + if over_limit: + _fail(f"GPU(s) {over_limit} hold more than {limit} MiB; stop them first") + return False + return True + + +def main() -> int: + checks = ( + check_model_path, + check_torch_gpus, + check_foreign_processes, + check_vram, + ) + passed = True + for check in checks: + if not check(): + passed = False + print(f"IR1_RESULT={'PASS' if passed else 'BLOCK'}") + return 0 if passed else 1 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/skills/hyperloom-workload-optimizer/scripts/resume.sh b/skills/hyperloom-workload-optimizer/scripts/resume.sh new file mode 100644 index 0000000..f93b80c --- /dev/null +++ b/skills/hyperloom-workload-optimizer/scripts/resume.sh @@ -0,0 +1,58 @@ +#!/usr/bin/env bash +# Copyright (c) 2026 Advanced Micro Devices, Inc. All rights reserved. +# +# See LICENSE for license information. +# +# Resume the session recorded at launch. Re-run IR-2 (install.sh) and IR-1 +# (preflight.py) first, exactly as for a fresh launch; this script does not +# re-check the gates. +# +# --resume-from is always passed explicitly: a bare --resume auto-picks the +# newest session and can target the wrong run. Resume writes its own log so the +# original run log is preserved. + +set -euo pipefail + +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd -P)" +# shellcheck source=./_env.sh +. "${SCRIPT_DIR}/_env.sh" + +if [ -z "${SESSION_DIR:-}" ] && [ -f "$LAST_LAUNCH_ENV" ]; then + . "$LAST_LAUNCH_ENV" +fi +if [ -z "${SESSION_DIR:-}" ]; then + echo "ERROR: SESSION_DIR unknown -- read .session_dir from the launch-info JSON" >&2 + exit 1 +fi +if [ ! -f "${SESSION_DIR}/state.json" ]; then + echo "ERROR: ${SESSION_DIR} has no state.json; the CLI refuses to resume it" >&2 + exit 1 +fi + +RESUME_TAG="resume-$(date +%Y%m%d_%H%M%S)" +RUN_LOG="${RUN_DIR}/run_${RESUME_TAG}.log" +PID_FILE="${RUN_DIR}/run_${RESUME_TAG}.pid" +LAUNCH_INFO_FILE="${RUN_DIR}/launch_${RESUME_TAG}.json" + +# shellcheck disable=SC2086 +setsid nohup "$PYTHON" -m hyperloom.inference_optimizer.cli --verbose optimize \ + --resume --resume-from "$SESSION_DIR" \ + --tick-interval-sec 30 \ + --launch-info-file "$LAUNCH_INFO_FILE" \ + ${OPT_FLAGS:-} \ + > "$RUN_LOG" 2>&1 < /dev/null & + +echo $! > "$PID_FILE" + +cat > "$LAST_LAUNCH_ENV" </dev/null || true + done + rm -rf "$WORK" +} +trap cleanup EXIT + +fail() { + echo "[FAIL] $1" >&2 + exit 1 +} + +# --- stub optimizer ------------------------------------------------------- +# Only "-m hyperloom..." is simulated; every other invocation (the launch-info +# JSON reader inside launch_health.sh) goes to the real interpreter. +cat > "$WORK/stub_python" < "\${STUB_ARGV_OUT}" + +mkdir -p "\${STUB_SESSION_DIR}" +echo '{}' > "\${STUB_SESSION_DIR}/manifest.json" +echo '{"phase": "PRELUDE"}' > "\${STUB_SESSION_DIR}/state.json" + +# Report a forked child as the optimizer pid. The launcher's \$! is the wrapper, +# which is a different process -- that gap is what the health check must close. +sleep 30 & +child=\$! + +if [ -n "\$info" ]; then + printf '{"pid": %s, "session_dir": "%s"}\n' "\$child" "\${STUB_SESSION_DIR}" > "\$info" +fi + +wait "\$child" +STUB +chmod +x "$WORK/stub_python" + +# --- fixture -------------------------------------------------------------- +# Each fixture is self-contained: .env and workload.env hold absolute paths, so a +# variant must be generated fresh rather than copied from another root. +make_fixture() { + local root="$1" + mkdir -p "$root/model" "$root/data/runtime" "$root/data/optimizer_runs" + + # A value with a space and a colon, double-quoted the way hyperloom-setup + # writes it. Unquoted, the .env source in _env.sh would fail with exit 127. + cat > "$root/.env" < "$root/model/config.json" + echo 'export KERNEL_AGENT_MARKER=1' > "$root/data/runtime/kernel-agent.env.sh" + + cat > "$root/data/optimizer_runs/workload.env" < "$WORK/launch.out" + +LAST_LAUNCH="$ROOT/data/optimizer_runs/last_launch.env" +[ -f "$LAST_LAUNCH" ] || fail "launch.sh did not write last_launch.env" +# shellcheck disable=SC1090 +. "$LAST_LAUNCH" +WRAPPER_PID="$(cat "$PID_FILE")" +STARTED_PIDS+=("$WRAPPER_PID") + +grep -q "^export LAUNCH_INFO_FILE=" "$LAST_LAUNCH" \ + || fail "last_launch.env is missing LAUNCH_INFO_FILE" +echo "[ok] launch.sh recorded the run handles on disk" + +for expected in --model --framework --tp --ep --conc --isl --osl --precision \ + --max-hours --target-gain; do + grep -qx -- "$expected" "$STUB_ARGV_OUT" || fail "$expected not passed to the CLI" +done +grep -qx -- "--no-kernel" "$STUB_ARGV_OUT" \ + || fail "OPT_FLAGS did not word-split into a separate argument" +grep -qx -- "$ROOT/model" "$STUB_ARGV_OUT" || fail "MODEL_PATH not passed" +echo "[ok] every workload.env value reached the CLI, OPT_FLAGS word-split" + +# --- health check --------------------------------------------------------- +bash "$SCRIPTS_DIR/launch_health.sh" > "$WORK/health.out" +grep -q "optimizer_alive=true" "$WORK/health.out" || fail "health check reported no live optimizer" + +REPORTED_PID="$(cat "$PID_FILE")" +STARTED_PIDS+=("$REPORTED_PID") +JSON_PID="$("$REAL_PYTHON" -c 'import json,sys;print(json.load(open(sys.argv[1]))["pid"])' "$LAUNCH_INFO_FILE")" +[ "$WRAPPER_PID" != "$JSON_PID" ] \ + || fail "fixture is not exercising the gap: wrapper pid equals the optimizer pid" +[ "$REPORTED_PID" = "$JSON_PID" ] \ + || fail "pid file holds $REPORTED_PID, expected the optimizer pid $JSON_PID" +echo "[ok] pid file holds the optimizer pid, not the setsid wrapper" + +grep -q "^export SESSION_DIR=" "$LAST_LAUNCH" || fail "SESSION_DIR was not persisted" +echo "[ok] session_dir persisted for the monitor and resume steps" + +# --- health check failure path ------------------------------------------- +# A run whose launch-info JSON never appeared: the handles exist but the file +# does not, which is what a crash during startup looks like. +BROKEN="$WORK/broken" +make_fixture "$BROKEN" +cat > "$BROKEN/data/optimizer_runs/last_launch.env" < "$WORK/health_fail.out" 2>&1; then + fail "health check passed even though the launch-info JSON was missing" +fi +grep -q "launch_broken.json not written" "$WORK/health_fail.out" \ + || fail "health check did not name the missing launch-info JSON" +echo "[ok] health check blocks when the launch-info JSON never appeared" + +# --- resume --------------------------------------------------------------- +export STUB_ARGV_OUT="$WORK/argv_resume.txt" +bash "$SCRIPTS_DIR/resume.sh" > "$WORK/resume.out" +# shellcheck disable=SC1090 +. "$LAST_LAUNCH" +STARTED_PIDS+=("$(cat "$PID_FILE")") + +grep -qx -- "--resume-from" "$WORK/argv_resume.txt" \ + || fail "resume.sh did not pass --resume-from explicitly" +if grep -qx -- "--model" "$WORK/argv_resume.txt"; then + fail "resume.sh passed --model, which resume must omit" +fi +case "$RUN_LOG" in + *run_resume-*) ;; + *) fail "resume.sh reused the original run log: $RUN_LOG" ;; +esac +echo "[ok] resume targets the recorded session and writes its own log" + +# --- missing gate inputs -------------------------------------------------- +NOPLAN="$WORK/noplan" +make_fixture "$NOPLAN" +rm -f "$NOPLAN/data/optimizer_runs/workload.env" +if (cd "$NOPLAN" && REPO_ROOT="$NOPLAN" bash "$SCRIPTS_DIR/launch.sh") \ + > "$WORK/noplan.out" 2>&1; then + fail "launch.sh started without workload.env" +fi +grep -q "workload.env missing" "$WORK/noplan.out" \ + || fail "launch.sh did not name the missing workload.env" +echo "[ok] launch.sh refuses to start without the confirmed plan" + +NOINSTALL="$WORK/noinstall" +make_fixture "$NOINSTALL" +rm -f "$NOINSTALL/data/runtime/kernel-agent.env.sh" +if (cd "$NOINSTALL" && REPO_ROOT="$NOINSTALL" bash "$SCRIPTS_DIR/launch.sh") \ + > "$WORK/noinstall.out" 2>&1; then + fail "launch.sh started without the kernel-agent env from install.sh" +fi +grep -q "run IR-2" "$WORK/noinstall.out" \ + || fail "launch.sh did not point at IR-2 when the kernel-agent env was missing" +echo "[ok] launch.sh refuses to start before IR-2 has run" + +echo +echo "all launch-flow checks passed" diff --git a/skills/hyperloom-workload-optimizer/scripts/tests/test_preflight.py b/skills/hyperloom-workload-optimizer/scripts/tests/test_preflight.py new file mode 100644 index 0000000..d0caa02 --- /dev/null +++ b/skills/hyperloom-workload-optimizer/scripts/tests/test_preflight.py @@ -0,0 +1,232 @@ +# Copyright (c) 2026 Advanced Micro Devices, Inc. All rights reserved. +# +# See LICENSE for license information. + +"""Regression tests for the IR-1 VRAM gate in ../preflight.py. + +Each case installs a fake amd-smi that reports a busy GPU in a different output +shape, then asserts the gate blocks the launch. Before the fail-closed rewrite, +an unexpected shape or a non-zero probe exit let a GPU holding ~140 GiB pass as +idle, so these cases are the reproduction for that bug. + +Run standalone; no pytest or third-party dependency required: + + python3 scripts/tests/test_preflight.py +""" + +from __future__ import annotations + +import importlib.util +import os +import stat +import sys +import tempfile +from pathlib import Path + +BUSY_MIB = 143360 +SCRIPT = Path(__file__).resolve().parents[1] / "preflight.py" + + +def _load_preflight(): + spec = importlib.util.spec_from_file_location("preflight_under_test", SCRIPT) + module = importlib.util.module_from_spec(spec) + spec.loader.exec_module(module) + return module + + +preflight = _load_preflight() + + +def _install_fake_smi(directory: Path, name: str, body: str) -> None: + path = directory / name + path.write_text(f"#!/bin/sh\n{body}\n") + path.chmod(path.stat().st_mode | stat.S_IEXEC | stat.S_IXGRP | stat.S_IXOTH) + + +def _emit(payload: str) -> str: + quoted = payload.replace("'", "'\\''") + return f"printf '%s' '{quoted}'" + + +CASES: list[tuple[str, str | None, bool, dict[str, str]]] = [ + ( + "expected shape, GPU busy", + _emit( + '[{"gpu": 0, "mem_usage": {"total_vram": {"value": 196592, "unit": "MB"},' + ' "used_vram": {"value": %d, "unit": "MB"}}}]' % BUSY_MIB + ), + False, + {}, + ), + ( + "expected shape, GPU idle", + _emit('[{"gpu": 0, "mem_usage": {"used_vram": {"value": 283, "unit": "MB"}}}]'), + True, + {}, + ), + ( + "top-level object instead of list, GPU busy", + _emit('{"gpu_0": {"mem_usage": {"used_vram": {"value": %d, "unit": "MB"}}}}' % BUSY_MIB), + False, + {}, + ), + ( + "gpu list wrapped in a top-level container key, GPU busy", + _emit( + '{"gpu_data": [{"gpu": 0, "mem_usage": {"used_vram":' + ' {"value": %d, "unit": "MB"}}}]}' % BUSY_MIB + ), + False, + {}, + ), + ( + "gpu list wrapped in a top-level container key, GPU idle", + _emit( + '{"gpu_data": [{"gpu": 0, "mem_usage": {"used_vram":' + ' {"value": 283, "unit": "MB"}}}]}' + ), + True, + {}, + ), + ( + "wrapped gpu list alongside a sibling metadata key, GPU idle", + _emit( + '{"gpu_data": [{"gpu": 0, "mem_usage": {"used_vram":' + ' {"value": 283, "unit": "MB"}}}],' + ' "metadata": {"version": "26.2.2"}}' + ), + True, + {}, + ), + ( + "scalar used_vram instead of value/unit map, GPU busy", + _emit('[{"gpu": 0, "mem_usage": {"used_vram": %d}}]' % BUSY_MIB), + False, + {}, + ), + ( + "container renamed mem_usage -> mem, GPU busy", + _emit('[{"gpu": 0, "mem": {"used_vram": {"value": %d, "unit": "MB"}}}]' % BUSY_MIB), + False, + {}, + ), + ( + "used_vram key absent (renamed vram_used), GPU busy", + _emit('[{"gpu": 0, "mem_usage": {"vram_used": {"value": %d, "unit": "MB"}}}]' % BUSY_MIB), + False, + {}, + ), + ( + "no used-VRAM field at all", + _emit('[{"gpu": 0, "mem_usage": {"total_vram": {"value": 196592, "unit": "MB"}}}]'), + False, + {}, + ), + ( + "amd-smi exits non-zero (driver error)", + 'echo "Unable to communicate with the amdgpu driver" >&2\nexit 1', + False, + {}, + ), + ( + "warning banner before JSON, GPU busy", + 'echo "WARNING: amdgpu version mismatch"\n' + + _emit('[{"gpu": 0, "mem_usage": {"used_vram": {"value": %d, "unit": "MB"}}}]' % BUSY_MIB), + False, + {}, + ), + ( + "used_vram explicitly null", + _emit('[{"gpu": 0, "mem_usage": {"used_vram": null}}]'), + False, + {}, + ), + ( + "empty GPU list", + _emit("[]"), + False, + {}, + ), + ( + "GiB unit, GPU busy", + _emit('[{"gpu": 0, "mem_usage": {"used_vram": {"value": 140, "unit": "GiB"}}}]'), + False, + {}, + ), + ( + "MB unit just under the MiB ceiling", + _emit('[{"gpu": 0, "mem_usage": {"used_vram": {"value": 500, "unit": "MB"}}}]'), + True, + {}, + ), + ( + "unknown unit", + _emit('[{"gpu": 0, "mem_usage": {"used_vram": {"value": 12, "unit": "furlongs"}}}]'), + False, + {}, + ), + ( + "second GPU unreadable while first is idle", + _emit( + '[{"gpu": 0, "mem_usage": {"used_vram": {"value": 100, "unit": "MB"}}},' + ' {"gpu": 1, "mem_usage": {}}]' + ), + False, + {}, + ), + ( + "no probe tool on PATH", + None, + False, + {}, + ), + ( + "no probe tool on PATH with explicit override", + None, + True, + {"IR1_ALLOW_UNVERIFIED_VRAM": "1"}, + ), + ( + "unreadable probe with explicit override", + 'exit 1', + True, + {"IR1_ALLOW_UNVERIFIED_VRAM": "1"}, + ), +] + + +def run_case(body: str | None, extra_env: dict[str, str]) -> bool: + saved_env = dict(os.environ) + with tempfile.TemporaryDirectory() as tmp: + fake_dir = Path(tmp) + if body is not None: + _install_fake_smi(fake_dir, "amd-smi", body) + try: + # Isolate PATH so only the fake probe (if any) is discoverable. + os.environ["PATH"] = str(fake_dir) + os.environ.pop("IR1_ALLOW_UNVERIFIED_VRAM", None) + os.environ["IR1_VRAM_LIMIT_MIB"] = "500" + os.environ.update(extra_env) + return preflight.check_vram() + finally: + os.environ.clear() + os.environ.update(saved_env) + + +def main() -> int: + failures = 0 + for name, body, expected_pass, extra_env in CASES: + actual_pass = run_case(body, extra_env) + ok = actual_pass == expected_pass + if not ok: + failures += 1 + want = "PASS" if expected_pass else "BLOCK" + got = "PASS" if actual_pass else "BLOCK" + print(f"[{'ok' if ok else 'FAIL'}] {name}: want {want}, got {got}") + print() + print(f"{len(CASES) - failures}/{len(CASES)} cases behaved as expected") + return 1 if failures else 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/skills/hyperloom-workload-optimizer/setup.md b/skills/hyperloom-workload-optimizer/setup.md new file mode 100644 index 0000000..f2a4131 --- /dev/null +++ b/skills/hyperloom-workload-optimizer/setup.md @@ -0,0 +1,112 @@ +# Hyperloom Workspace Bootstrap + +Use this file from [SKILL.md](SKILL.md) Phase 0. The optimizer runtime ships in the +Hyperloom Python wheel; this skill does not bundle it. + +Steps 0–2 only prepare the workspace. Return to [SKILL.md](SKILL.md) and continue +with Phase 1 (environment prep), Phase 2 (workload intake), then Phase 3 +(install, launch, monitor). + +## Step 0 — Confirm the install directory + +The wheel installs into a target directory (`pip install --target `) that +also holds `.env` and runtime artifacts. Do not silently use the current +directory. Show the resolved `pwd` and confirm it with the user, or let them +pick another dedicated path. `cd` into the chosen directory before installing. + +## Step 1 — Install the Hyperloom wheel + +Skip when `hyperloom/` already exists in the directory (wheel layout) or +`src/hyperloom/` exists (source checkout). + +Find the latest release of +[AMD-AGI/Hyperloom](https://github.com/AMD-AGI/Hyperloom/releases), tell the user +that version, and ask whether to install it or a version they name. The wheel +asset is `hyperloom_inference_optimizer--py3-none-any.whl`. + +```bash +cd "$INSTALL_DIR" # the directory confirmed in Step 0 +python3 -m pip install "" --target . +``` + +After install, confirm: + +- `hyperloom/inference_optimizer/assets/install.sh` exists +- `.cursor/skills/hyperloom-custom-advanced/SKILL.md` exists (or `.claude/` / + `.agents/` equivalent) + +Restart the agent if the new skills are not visible. + +## Step 2 — Credentials and run mode + +**Preferred:** run the bundled setup skill installed by the wheel: + +- Cursor / Codex: `$hyperloom-setup` or load `hyperloom-setup` from + `.cursor/skills/hyperloom-setup/SKILL.md` +- Claude Code: `/hyperloom-setup` + +That skill owns the credential questions, the run-mode question, and the bare-metal +host setup backend (`install_baremetal.sh`). It is the source of truth for which +LLM providers are accepted and which variable names they use, so do not +reimplement it here or guess variable names from memory. + +If it is not available, restart the agent and re-check that +`.cursor/skills/hyperloom-setup/SKILL.md` (or the `.claude/` / `.agents/` +equivalent) exists. If it is still missing, the wheel install is incomplete — +stop and report that, rather than hand-writing `.env`. + +When it returns, verify `.env` before continuing. It must define +`HYPERLOOM_RUN_MODE` (`baremetal` or `docker`), `USER_DATA_PATH`, +`HYPERLOOM_SKILL_PATH`, and one complete LLM credential set. Check the values are +real, not placeholders. `HYPERLOOM_SKILL_PATH` must point at the packaged +optimizer skill that exists on disk — `hyperloom/inference_optimizer/SKILL.md` +for a wheel install, `src/hyperloom/inference_optimizer/SKILL.md` for a source +checkout. + +Inspect `.env` with `grep` rather than echoing it, and never print a key value +into chat. Phase 3 loads `.env` with a shell `source`, so any value containing a +space must stay double-quoted — `hyperloom-setup` writes +`ANTHROPIC_CUSTOM_HEADERS="Ocp-Apim-Subscription-Key: ${ANTHROPIC_API_KEY}"` for +exactly that reason. A hand-edited `.env` that drops those quotes makes the launch +shell fail with exit 127. + +Do not write your own live probe of the LLM endpoint. Hyperloom's `optimize` +preflight already probes `/models` with the operator's auth and custom +headers, and refuses to start on an auth failure, so a wrong key fails there in +the first seconds rather than mid-run. Two of its outcomes are only warnings, and +both must be surfaced to the user instead of scrolled past: + +- `gateway has no /models route (HTTP 404/405) ... Proceeding` — the key is + unverified, not verified. Expected when `ANTHROPIC_BASE_URL` is native + `https://api.anthropic.com`, which has no `/models` at that path. +- `gateway catalog unreachable ... Proceeding with custom orchestration model + support enabled` — this is the auth failure downgraded to a warning because a + custom model id was allowed. A bad key looks like this instead of an error. + +## Docker run mode — the Phase 1 contract + +When `HYPERLOOM_RUN_MODE=docker`, `hyperloom-custom-advanced` owns the container +steps (image, `docker run` flags, setup inside the container). Follow it rather +than inventing flags. Regardless of how it gets there, all of the following must +hold before Phase 3 launches anything, and every later command in this skill runs +*inside* the container: + +- the container is running and you have a shell in it +- `/dev/kfd` and `/dev/dri` are mapped, and `amd-smi` or `rocm-smi` works inside +- the install directory from Step 0 and `USER_DATA_PATH` are mounted at the same + absolute paths inside the container, so `.env` and run state resolve identically +- `MODEL_PATH` resolves inside the container +- the setup backend has been run inside the container + +If any of these is unmet, fix it before Phase 3. Launching with a half-prepared +container produces failures that look like optimizer bugs. + +## Step 3 — Return to SKILL.md + +Bootstrap is complete. Go back to [SKILL.md](SKILL.md) and continue with Phase 1 +(environment prep), Phase 2 (workload intake), then Phase 3 (install, preflight, +launch, monitor). + +Read `@${HYPERLOOM_SKILL_PATH}` only for edge cases the catalog skill does not +cover: multi-node, `--framework atom`, critic/robustness backend selection, +aiter cache topology, and the full failure matrix. diff --git a/skills/hyperloom-workload-optimizer/skill-card.md b/skills/hyperloom-workload-optimizer/skill-card.md new file mode 100644 index 0000000..d408a99 --- /dev/null +++ b/skills/hyperloom-workload-optimizer/skill-card.md @@ -0,0 +1,13 @@ +# Skill Card + +## Description + +Autonomously optimizes end-to-end LLM inference throughput on AMD Instinct GPUs with Hyperloom and reports a validated gain. + +## Owner + +AMD AGI / Hyperloom + +## License + +MIT diff --git a/walkthroughs/README.md b/walkthroughs/README.md index 39976d4..9bd7a1b 100644 --- a/walkthroughs/README.md +++ b/walkthroughs/README.md @@ -10,4 +10,5 @@ Please choose a skill to get started. * [local-ai-use](./local-ai-use.md): Teach your agent how to run image generation locally. * [local-ai-app-integration](./local-ai-app-integration.md): Add a local AI mode to a cloud-only app. -* [tracelens-analysis-orchestrator](./tracelens-analysis-orchestrator.md): Run agentic PyTorch profiler trace analysis and produce a prioritized performance report. \ No newline at end of file +* [tracelens-analysis-orchestrator](./tracelens-analysis-orchestrator.md): Run agentic PyTorch profiler trace analysis and produce a prioritized performance report. +* [hyperloom-workload-optimizer](./hyperloom-workload-optimizer.md): Set up Hyperloom and autonomously optimize LLM inference throughput on AMD Instinct GPUs. diff --git a/walkthroughs/hyperloom-workload-optimizer.md b/walkthroughs/hyperloom-workload-optimizer.md new file mode 100644 index 0000000..939763d --- /dev/null +++ b/walkthroughs/hyperloom-workload-optimizer.md @@ -0,0 +1,123 @@ +# AMD Skills Walkthroughs: `hyperloom-workload-optimizer` + +This skill teaches your AI agent to set up a Hyperloom workspace and autonomously +optimize end-to-end LLM inference throughput on AMD Instinct GPUs (MI300X / +MI325X / MI350X / MI355X). + +**What you'll end up with:** a running Hyperloom optimization session with +`manifest.json`, `state.json`, benchmark runs under `runs/`, and a final report +under `reports/` showing validated throughput gain over baseline. + +Expect a full optimization run to take hours depending on `--max-hours` and model +size. This walkthrough covers workspace bootstrap; a short smoke run is possible +with a small model and `--max-hours 0.5 --no-kernel`. + +## Prerequisites + +**Hardware** + +- AMD Instinct GPU with ROCm (`/dev/kfd`, `/dev/dri`) +- Sufficient VRAM for the target model at the chosen TP degree + +**Software** + +- Python 3.10+ +- An agentic runner: **Cursor**, **Claude Code**, or **Codex** +- Anthropic API access (or AMD LLM gateway) for Hyperloom agent backends +- Docker (recommended) or bare-metal ROCm + serving framework + +**Workspace** + +- A dedicated empty directory opened as the agent workspace + +## Step 1 — Enable the skill + +**Claude Code:** + +```bash +npx skills add amd/skills --skill hyperloom-workload-optimizer --agent claude-code +``` + +**Cursor:** install the `amd-skills` plugin from the AMD skills marketplace, or +copy `skills/hyperloom-workload-optimizer/` into your project's +`.cursor/skills/` directory. + +Confirm the skill is visible: + +```text +Which skills do you see? +``` + +You should see `hyperloom-workload-optimizer` in the list. + +## Step 2 — Bootstrap the workspace + +In the dedicated workspace, ask the agent: + +```text +Set up Hyperloom and prepare to optimize vLLM throughput on MI300X. +``` + +The agent works through four phases in order and should not ask workload +questions before the environment is ready: + +1. **Phase 0 — Bootstrap:** confirm the install directory, find the latest + Hyperloom wheel on GitHub releases and `pip install --target .`, then run + `/hyperloom-setup` to write `.env` (credentials + run mode only). +2. **Phase 1 — Environment prep:** load `hyperloom-custom-advanced` Setup + Configuration. On bare metal, confirm the host is ready; in Docker, start a + long-running container and run the in-container setup first. +3. **Phase 2 — Workload intake:** only now ask for workload parameters (model, + framework, TP, conc, ISL/OSL, precision, budget) and confirm a launch plan. +4. **Phase 3 — Execute:** run `install.sh` (IR-2) and the GPU preflight (IR-1), + then launch and monitor. + +Verify: + +```bash +ls hyperloom/inference_optimizer/assets/install.sh +test -f .env && grep -q HYPERLOOM_SKILL_PATH .env +``` + +## Step 3 — Launch a short optimization + +Provide a local model path and workload when the agent asks, for example: + +```text +Optimize /path/to/Qwen3-8B with vLLM on MI300X: TP=1, conc=64, ISL=1024, OSL=1024, +max-hours 0.5, no kernel agent. Launch and monitor. +``` + +Once workload values are resolved, the agent should run `install.sh` (IR-2), +the GPU preflight (IR-1), launch `hyperloom.inference_optimizer.cli optimize` +with those values as CLI flags, then poll `state.json`. + +## Step 4 — Read results + +When the session stops or the budget expires, ask: + +```text +Report Hyperloom status: baseline, current best, cumulative gain, stop_reason. +``` + +Check artifacts: + +```bash +ls "$USER_DATA_PATH"/*/*/manifest.json +ls "$USER_DATA_PATH"/*/*/reports/ +``` + +## Troubleshooting + +- **`/hyperloom-setup` not found** — confirm `pip install --target .` ran in the + workspace and restart the agent. +- **`install.sh` fails** — check network access for Magpie / TraceLens clones; + see the [Hyperloom install guide](https://github.com/AMD-AGI/Hyperloom/blob/main/docs/install/install.md). +- **GPU occupied** — kill stale `vllm` / `sglang` / `Magpie` processes (IR-1). +- **Plain serving request** — use `serving-llms-on-instinct` instead. + +## Next steps + +- Resume: `Resume the latest Hyperloom session for .` +- Full run: increase `--max-hours`, enable kernel-agent (drop `--no-kernel`). +- Advanced flags: see `reference.md` in the skill folder.