New Skill: hyperloom-workload-optimizer - #114
Conversation
Add the Hyperloom workload optimizer catalog skill that bootstraps a workspace, prepares the runtime, collects workload parameters, then installs, launches, and monitors the optimizer on AMD Instinct GPUs. Register it in the Claude/Codex/Cursor plugin manifests and walkthroughs.
Remove the primussafe container guard from the behavioral eval; it referenced an internal platform name. Docker images will standardize on ROCm, so the generic plain-serving guard is sufficient.
Stop restating custom-advanced docker commands and flags in Phase 1; the image choice, docker run, and in-container setup are owned by custom-advanced Setup Configuration. Keep only the catalog's orchestration guardrails.
Persist confirmed Phase 2 values to workload.env and source it in every Phase 3 block so launch no longer silently falls back to TP/CONC/model defaults; guard empty MODEL_PATH. Add vllm.entrypoints to the IR-1 process scan, parse the launch-info JSON with python3 instead of jq, align the max-hours default column, fix the reference IR heading and table of contents, and add a trailing newline to the walkthrough index.
Read the real optimizer .pid from the launch-info JSON (setsid wrapper $! exits immediately) and rewrite the pid file. Rebuild the resume path to re-source .env/workload.env, re-run IR-2/IR-1, resume the explicit session with --resume-from, and use $PYTHON. Fail preflight when CUDA is unavailable, write workload.env with a quoted heredoc, launch via $PYTHON, guard MODEL_PATH, and resolve the monitor session dir from launch-info. Trim SKILL.md under the 500-line limit.
|
|
||
| ## Phase 1 — Environment prep | ||
|
|
||
| Load `hyperloom-custom-advanced` and follow its **Setup Configuration** |
There was a problem hiding this comment.
The intro (line 28-30) says this skill "inlines the operator workflow so you can run end-to-end without bouncing across files," but Phase 1/2 delegate the actual setup (image choice, docker run, in-container setup, workload intake) to hyperloom-custom-advanced, which isn't in this PR (it only lands after the wheel installs) and is the demo skill rather than core inference_optimizer. That contradicts the self-contained claim and makes the Docker path non-reviewable if that skill isn't loaded. Can we inline the minimal self-contained setup into this skill's own setup.md, based on core inference_optimizer?
| export PYTHON="${PYTHON:-$(command -v python3)}" | ||
| export IR1_VRAM_LIMIT_MIB="${IR1_VRAM_LIMIT_MIB:-500}" | ||
|
|
||
| "$PYTHON" - <<'PY' |
There was a problem hiding this comment.
The preflight (:275-335), launch (:349-382), monitor (:424-437), and resume (:450-474) blocks are many many lines of inline bash/Python. Per the authoring standard, fragile order-dependent code should live in scripts/ and be invoked, not pasted into the body for the agent to reconstruct each run. Can we move these into scripts/ (e.g. preflight.sh, launch.sh, monitor.sh) and have the body call them? That also shrinks the body well under the 500-line cap and makes the steps testable.
| export IR1_VRAM_LIMIT_MIB="${IR1_VRAM_LIMIT_MIB:-500}" | ||
|
|
||
| "$PYTHON" - <<'PY' | ||
| import os, shutil, subprocess, sys |
There was a problem hiding this comment.
IR-1 is meant to block launch when a GPU is busy, but the VRAM check can miss that. If amd-smi/rocm-smi returns JSON in a shape the parser doesn't expect, the reading at line 313 comes back empty and gets skipped, so a GPU that's actually occupied looks free and the launch proceeds. Since this is a safety gate, could we make it stop and warn the user when it can't read VRAM, instead of continuing? Right now an unexpected amd-smi output is enough to let a run start on a busy GPU.
| Confirm `hyperloom/inference_optimizer/assets/install.sh` exists. Restart the | ||
| agent if wheel skills are not visible. | ||
|
|
||
| ### Credentials and run mode |
There was a problem hiding this comment.
Two credential gaps:
- The skill doesn't help the user choose between Hyperloom's two credential shapes: single gateway (SAFE_API_KEY + OPENAI_BASE_URL) vs the Anthropic split (ANTHROPIC_API_KEY + ANTHROPIC_BASE_URL + CLAUDE_MODEL). Can it ask which one they have and recommend a known-good model, claude-opus-4-8 (preferred) or claude-opus-4-7, since both pass Hyperloom's model gate?
- We only check the secret isn't <PLEASE_FILL_IN> (setup.md:91), but Hyperloom does no live key validation, so a wrong/expired key fails late as a 401 mid-run. Can we add a cheap preflight (one small call against the configured base URL) that confirms the key works before the long optimize run? Turns a wasted multi-hour run into a 5-second early failure.
|
|
||
| ```bash | ||
| cd "$INSTALL_DIR" # the directory confirmed above | ||
| wheel_url="$(curl -fsSL "https://api.github.com/repos/AMD-AGI/Hyperloom/releases?per_page=20" \ |
There was a problem hiding this comment.
This resolver takes the newest matching asset across all releases with no prerelease filter, so it currently resolves to 1.0.0a2 (an alpha) and drifts as new releases publish, non-reproducible and can pull a prerelease onto a user's box. Can we skip prereleases (rel.get("prerelease")) and/or pin a known-good version (e.g. latest stable 0.9.0, or an env-overridable HYPERLOOM_WHEEL_VERSION) so bootstraps are deterministic?
|
|
||
| ## Prerequisites | ||
|
|
||
| - AMD Instinct GPU host (MI300X / MI325X / MI350X / MI355X) with ROCm |
There was a problem hiding this comment.
Can we make the execution-location assumption explicit? Prerequisites and Phase 1 assume the agent already has a shell on the GPU host with local /dev/kfd+/dev/dri. The common case is a remote Instinct server the user SSHes into, which isn't addressed. Suggest one line in Prerequisites ("this skill assumes it runs on the GPU host; for a remote host, SSH in first") plus a pointer to Hyperloom's remote/multi-node support (docs/how-to/multi-node/, remote sandbox via SAFE_API_URL).
|
|
||
| - **IR-1 — GPU unoccupied.** Before every `optimize` (fresh or `--resume`), every | ||
| visible GPU must have zero foreign serving PIDs (`sglang.launch_server` / | ||
| `vllm.entrypoints` / `Magpie`) and ≲ 500 MiB VRAM in use. |
There was a problem hiding this comment.
The fallback .env hardcodes HYPERLOOM_RUN_MODE=docker with no prompt. On the /hyperloom-setup path the mode is chosen for the user (fine), but this fallback silently forces docker even on a baremetal host with ROCm already set up. Can the fallback ask docker vs baremetal (one line on the tradeoff: docker = isolated/reproducible but needs an image + /dev/kfd,/dev/drimapping; baremetal = simpler when ROCm is present but touches the host env), and do a quick feasibility check (docker: daemon reachable + devices mappable; baremetal: ROCm + serving framework importable) before writing.env`?
Rewrite the IR-1 VRAM gate as scripts/preflight.py and make it fail closed: an unreadable probe now blocks the launch instead of being skipped, since an empty reading was indistinguishable from idle GPUs. Normalize MB/MiB so the ceiling comparison no longer mixes units, tolerate the amd-smi/rocm-smi output shapes that vary across ROCm releases, and cover the shapes that used to pass silently in scripts/tests/test_preflight.py. Drop the inline .env fallback, which hardcoded docker mode and duplicated hyperloom-setup's decisions. Replace it with the key contract, a diagnosis path when the setup skill is missing, and a rule against sourcing .env, whose values are unquoted and may contain spaces. Point at the optimize preflight catalog probe rather than adding a second live key check, and require its warning-only outcomes to be surfaced. Document the docker delegation contract, stop and report when hyperloom-custom-advanced is absent, state that every command runs on the GPU host, and resolve the wheel by reporting the newest release and asking the user instead of taking it silently.
…eader The launch, health-check and resume blocks were order-dependent shell pasted into the body for the agent to reassemble on every run: .env, then workload.env, then the kernel-agent env, then a PATH derived from $PYTHON, and only then setsid nohup. They now live in scripts/launch.sh, scripts/launch_health.sh and scripts/resume.sh over a shared scripts/_env.sh, which also removes the preamble that launch and resume duplicated line for line. The run handles are recorded in optimizer_runs/last_launch.env. The health check, monitor and resume steps run in fresh shells, so LAUNCH_INFO_FILE and SESSION_DIR were previously only reachable when the agent happened to keep one shell alive. The health check now fails loudly when the launch-info JSON never appeared or no optimizer process can be found, instead of reporting an empty pid. Drop the inline state.json parser in the monitor step and call the read_optimizer_state.py that the wheel ships, which prints the same fields plus the recent lifecycle events. Resolve it for both the wheel and source layouts, matching how install.sh is resolved. Correct the .env guidance in setup.md: values are sourced by the Phase 3 shell, and hyperloom-setup double-quotes any value containing a space precisely so that source cannot fail. The previous text claimed the opposite.
Exercises launch.sh, launch_health.sh and resume.sh against a stub optimizer, so no GPU, wheel or network is needed. Covers the happy path (workload.env values reaching the CLI, OPT_FLAGS word-splitting, run handles persisted to last_launch.env, session_dir recorded) and four refusal paths: a launch-info JSON that never appeared, a missing workload.env, a missing kernel-agent env from IR-2, and resume without --model. The stub reports a forked child as the optimizer pid, because setsid execs directly when it need not fork -- with a single-process stub the wrapper pid and the optimizer pid coincide and the check cannot tell whether launch_health.sh rewrote the pid file at all. The fixture now asserts the two differ before comparing, so removing the rewrite fails the test. Each fixture is generated rather than copied: .env and workload.env hold absolute paths, so a copied root would still point at the original directories and the refusal paths would silently pass.
The confirmation gate showed an aligned plan block but never said where it goes, so an agent using a structured question tool put the whole plan into the prompt field. That field renders as one wrapped paragraph: the column alignment collapses and the user has to hunt for MAX_HOURS inside a blob. Require the plan in the reply body and keep the prompt to a single short question. Also name the field to change instead of offering a free-text "change something" option, which makes the user retype parameter names.
94c3d70 to
8de0f02
Compare
|
@Mahdi-CV Thanks for the review — all seven are addressed across 8de0f02, d644585, 7218f06, 644052e.
Removed the "inlines the operator workflow without bouncing across files" line. The skill is now described as what it is: it orchestrates, and hands environment prep to hyperloom-setup and workload intake to hyperloom-custom-advanced. Both ship inside the wheel and are version-locked to the runtime, so this repo keeps no copy of them. Missing delegates now fail explicitly instead of degrading — if hyperloom-custom-advanced isn't present, Phase 1 stops and reports it.
All four are out: scripts/preflight.py, launch.sh, launch_health.sh, resume.sh, with the shared env preamble in _env.sh. The monitor step no longer parses state.json itself; it calls tools/read_optimizer_state.py, which ships in the wheel, so it can't drift from the format the optimizer actually writes. Each step in SKILL.md is now a 3–5 line call, and the body went from 509 to 401 lines. Added scripts/tests/test_launch_flow.sh: it runs the real scripts against a stub optimizer over 8 scenarios, covering the happy path plus four refusals — missing workload.env, missing install output, launch-info.json never appearing, and the optimizer process not being present. For the PID-rewrite scenario the stub forks, and the test asserts the setsid wrapper PID differs from the optimizer PID before comparing against what landed in PID_FILE.
It's fail-closed now: the launch proceeds only if every check passed, and an unreadable VRAM probe blocks. The single escape hatch is IR1_ALLOW_UNVERIFIED_VRAM=1, which has to be set explicitly. Three concrete defects that caused silent passes are fixed as well: amd-smi / rocm-smi JSON shape differences across versions (list vs dict, different field spellings) made the parse come back empty; some fields report MB while the threshold compared MiB, so 96432 MB read as idle; and parse exceptions were swallowed. scripts/tests/test_preflight.py covers 18 cases built around those shapes, and I verified the assertions by removing them one at a time. The foreign-process check now prints only the PID and the matched pattern name rather than the full cmdline, so a token on the command line doesn't reach the log.
On the choice of shape: not adding one. hyperloom-setup already prompts for CLAUDE_MODEL and defaults it to claude-opus-4-8 (SKILL.md:102), and what it writes is "the Anthropic keys plus the common keys" (:195-197). Credential collection is its job and this skill follows the shape it produces; offering a different shape here would conflict with it rather than merely duplicate it. The runtime-side allowlist and the preferred/fallback hint are in cli/init.py:718-725. On validation: Hyperloom's optimize preflight already runs a live probe with the bearer header and the custom headers, and exits 2 on an auth failure, so there's no second probe here. The gap is that two of its outcomes only warn instead of blocking, so setup.md points at that probe and requires the agent to report those two warning-only results explicitly.
The alpha line is expected for now — v1.0.0a1/a2/a3 is the current release line, and this will be adjusted once we publish a formal 1.0.0. Filtering on GitHub's prerelease field wouldn't work in any case: all 11 releases have it set to false, alphas included, since it's set by hand. What changed is that it no longer installs silently. It reports the resolved version to the user and waits for confirmation, and the user can name a specific version instead.
Added to Prerequisites: this skill runs on the target GPU host. setup.md now carries a docker-mode contract listing what must hold before Phase 3 — container running, GPU devices mapped, workspace mounted at the same absolute path, MODEL_PATH resolvable inside, and setup already run inside the container.
The template is gone. hyperloom-setup is the single source for LLM providers and variable names; if the setup skill is missing, setup.md gives a diagnostic path (restart the agent, confirm the wheel installed completely) rather than a local fallback template. How to inspect .env is now spelled out: read it line by line, don't source it, don't echo the key. ANTHROPIC_CUSTOM_HEADERS and similar values contain spaces and a colon, and hyperloom-setup writes values double-quoted for that reason, so sourcing breaks them. |
amd-smi 26.x on ROCm 7.x returns {"gpu_data": [...]} instead of a bare
list, so _iter_gpu_entries found no GPU entries and the fail-closed gate
blocked every launch with "no recognizable GPU list" even on an idle GPU
holding 270 MiB. The known container keys are tried before the
dict-of-dicts scan so a sibling metadata object is not read as a GPU.
e01a7b3 to
6abcf72
Compare
danielholanda
changed the title
hyperloom-workload-optimizer
2 participants
Summary
Adds
hyperloom-workload-optimizer, a skill that autonomously optimizesend-to-end LLM inference throughput on AMD Instinct GPUs using the Hyperloom
multi-agent optimizer, and reports a validated gain.
Given a model, framework, workload shape (TP/EP, concurrency, ISL/OSL,
precision), an objective and a time budget, the skill explores which levers to
pull per workload — serving/config parameters and environment, framework
enablement and source patches, and hot GPU-kernel rewrites — benchmarks each
candidate, and returns the optimization stack that produced the gain.
Contribution Path A (skill authored in this repository).
Contents
skills/hyperloom-workload-optimizer/SKILL.md— skill bodyskills/hyperloom-workload-optimizer/setup.md— environment setup (bare metal and Docker)skills/hyperloom-workload-optimizer/reference.md— configuration and flag referenceskills/hyperloom-workload-optimizer/skill-card.md— governance cardskills/hyperloom-workload-optimizer/evals/evals.py— behavioral testwalkthroughs/hyperloom-workload-optimizer.md— walkthrough, linked fromwalkthroughs/README.md.claude-plugin/marketplace.json; derived manifests regeneratedvia
./.github/scripts/publish.shValidation
./.github/scripts/check.shpasses locally: 0 errors across 7 skills, and theCursor and Codex manifests are reported up to date.