Skip to content

Commit fdd2fa6

Browse files
committed
Publish rocm-doctor skill (thin driver over rocm examine/diagnose/fix)
Add the rocm-doctor skill to the catalog: diagnoses ROCm/HIP/PyTorch/ llama.cpp failures on AMD GPUs against the closed catalog now shipped in the rocm CLI (rocm examine / diagnose / fix), fixing with consent or routing upstream. A Phase 0 step installs the rocm CLI (with consent) when it's missing so the skill is self-sufficient. Register it in the marketplace, regenerate the derived manifests, mark it in-repo in the README, drop the superseded staging copy, and remove the now-dead skillspector suppressions for the deleted staging scripts. Signed-off-by: Eugene Volen <Eugene.Volen@amd.com>
1 parent f2681c0 commit fdd2fa6

16 files changed

Lines changed: 335 additions & 4300 deletions

File tree

.claude-plugin/marketplace.json

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -16,6 +16,7 @@
1616
"skills": [
1717
"./skills/local-ai-use",
1818
"./skills/local-ai-app-integration",
19+
"./skills/rocm-doctor",
1920
"./skills/serving-llms-on-instinct",
2021
"./skills/tracelens-analysis-orchestrator"
2122
],

.codex-plugin/plugin.json

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -20,6 +20,7 @@
2020
"skills": [
2121
"./skills/local-ai-use",
2222
"./skills/local-ai-app-integration",
23+
"./skills/rocm-doctor",
2324
"./skills/serving-llms-on-instinct",
2425
"./skills/tracelens-analysis-orchestrator"
2526
],

.cursor-plugin/marketplace.json

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -15,6 +15,7 @@
1515
"skills": [
1616
"./skills/local-ai-use",
1717
"./skills/local-ai-app-integration",
18+
"./skills/rocm-doctor",
1819
"./skills/serving-llms-on-instinct",
1920
"./skills/tracelens-analysis-orchestrator"
2021
],

.github/skillspector-allow.yml

Lines changed: 0 additions & 53 deletions
Original file line numberDiff line numberDiff line change
@@ -17,59 +17,6 @@
1717
# Add entries sparingly and only when the finding is demonstrably benign.
1818

1919
suppressions:
20-
- skill: rocm-doctor
21-
rule: YR1
22-
file: scripts/apply_fix.py
23-
match: backdoor_persistence
24-
reason: >-
25-
False positive. The 'backdoor_persistence' YARA rule's $bashrc_persist
26-
string matches any `echo ... >> ~/.bashrc`. Here it is the documented
27-
remediation that appends `export PATH="/opt/rocm/bin:$PATH"` so ROCm
28-
binaries land on PATH after install. Standard ROCm setup guidance, not a
29-
persistence backdoor or payload.
30-
- skill: rocm-doctor
31-
rule: YR1
32-
file: scripts/diagnose.py
33-
match: backdoor_persistence
34-
reason: >-
35-
False positive. Same $bashrc_persist match: diagnose.py prints the
36-
remediation command `echo 'export PATH=<bin>:$PATH' >> ~/.bashrc` (or
37-
~/.zshrc) for the user to add ROCm/HIP to PATH. No payload, no SSH key
38-
injection, no hidden user.
39-
- skill: rocm-doctor
40-
rule: OH1
41-
file: scripts/apply_fix.py
42-
match: Unvalidated Output Injection
43-
reason: >-
44-
False positive. The flag is on the generic `_run(cmd: list[str], ...)`
45-
helper, which calls `subprocess.run(cmd, ..., shell defaults to False)`
46-
with a list-form argv, so there is no shell interpolation. Every `cmd`
47-
is a hardcoded argv list assembled in-script (e.g.
48-
`["usermod","-a","-G","render,video",user]`, `["modprobe","amdgpu"]`);
49-
the only dynamic pieces are the local username from `$USER`/`$LOGNAME`
50-
and binary paths resolved via `shutil.which`. No LLM/model output ever
51-
reaches this sink, so there is nothing to validate or sanitize.
52-
- skill: rocm-doctor
53-
rule: OH1
54-
file: scripts/examine.py
55-
match: Unvalidated Output Injection
56-
reason: >-
57-
False positive. Same generic `_run(cmd: list[str], ...)` helper as in
58-
apply_fix.py: list-form `subprocess.run` with no shell=True. The read-only
59-
probes only ever pass fixed argv lists (`["rocminfo"]`,
60-
`["lspci","-nn","-D"]`, the PowerShell/CIM `Get-CimInstance` probes, the
61-
framework binary from `shutil.which`). No model output flows into the
62-
command, and there is no shell to inject into.
63-
- skill: rocm-doctor
64-
rule: PE3
65-
file: scripts/examine.py
66-
match: Credential Access
67-
reason: >-
68-
False positive. Line 493 is a code comment ("Resolve uid/gid to names via
69-
/etc/passwd & /etc/group") describing how `_stat_device` maps a device's
70-
owner uid/gid to names. The actual resolution uses the stdlib `pwd`/`grp`
71-
modules (`pwd.getpwuid` / `grp.getgrgid`), not any read of /etc/passwd,
72-
/etc/shadow, .env, or token files. No credential material is accessed.
7320
- skill: local-ai-use
7421
rule: SC2
7522
file: SKILL.md

README.md

Lines changed: 4 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -80,7 +80,7 @@ Cross-stack skills, from client to cloud.
8080

8181
| Skill | What it does | Source |
8282
| --- | --- | --- |
83-
| `rocm-doctor` | Diagnose ROCm / PyTorch / llama.cpp failures on AMD GPUs against a fixed list of known misconfigurations. | _planned_ |
83+
| [`rocm-doctor`](skills/rocm-doctor/SKILL.md) | Diagnose ROCm / HIP / PyTorch / llama.cpp failures on AMD GPUs (Linux and Windows) against a closed list of known misconfigurations, then fix with consent or route upstream. Thin driver over the `rocm` CLI (`examine` / `diagnose` / `fix`). | in-repo |
8484
| `hyperloom-kernel-optimizer` | Autonomously optimizes LLM inference on AMD GPUs. | _planned_ |
8585
| `vllm-semantic-router` | Setup a vLLM router that semantically maps your request to the best available platform. | _planned_ |
8686

@@ -101,11 +101,11 @@ A skill is a self-contained folder that bundles everything an agent needs to per
101101

102102
```
103103
skills/
104-
rocm-doctor/
104+
<skill-name>/
105105
SKILL.md
106106
skill-card.md
107-
scripts/
108-
references/
107+
scripts/ # optional
108+
references/ # optional
109109
```
110110

111111
When an agent decides a skill is relevant (or you invoke it explicitly), it loads that `SKILL.md` and follows the instructions inside. Descriptions stay in context cheaply; the full body of a skill only loads when the task actually matches.

skills/rocm-doctor/SKILL.md

Lines changed: 139 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,139 @@
1+
---
2+
name: rocm-doctor
3+
description: >-
4+
Diagnoses why ROCm, the HIP SDK, PyTorch, or llama.cpp is broken on an AMD GPU
5+
on Linux or Windows, then applies a low-risk fix with consent or hands back the
6+
exact next step. Also routes Lemonade, LM Studio, and Ollama problems to the
7+
right upstream channel. Use when the user reports that ROCm or HIP "isn't
8+
working", torch.cuda.is_available() is False, rocminfo / hipInfo can't see the
9+
GPU, or hits hipErrorNoBinaryForGpu, HSA_STATUS_ERROR_INVALID_ISA, "invalid
10+
device function", "no kernel image is available", cannot open /dev/kfd,
11+
permission denied on /dev/kfd, "ROCk module is NOT loaded", a missing
12+
libamdhip64.so / amdhip64_6.dll / hipblas.dll / vcruntime140_1.dll, an
13+
HSA_OVERRIDE_GFX_VERSION page fault, an iGPU+dGPU crash, a container that can't
14+
see the GPU, or an amdgpu-install / DKMS failure. Backed by the `rocm` CLI
15+
(`rocm examine` / `rocm diagnose` / `rocm fix`); this skill is a thin driver
16+
over those commands, not a re-implementation.
17+
---
18+
19+
# ROCm Doctor
20+
21+
Given a "ROCm / PyTorch / llama.cpp isn't working on my AMD GPU" complaint,
22+
identify which **known misconfiguration** is the cause and either fix it (with
23+
consent) or hand back the exact next step.
24+
25+
This skill does **not** probe or reason on its own. The `rocm` CLI owns the
26+
probe, the closed failure-mode catalog, and the fixes; the skill just drives it
27+
and relays the results. The catalog is a **closed list** — if the symptom
28+
doesn't match a known mode, route the user upstream instead of guessing.
29+
30+
## Workflow
31+
32+
0. **Ensure the `rocm` CLI is present.** Everything below shells out to it, so
33+
check first and install it if missing:
34+
35+
```
36+
rocm --version
37+
```
38+
39+
If that succeeds, skip to step 1. If it's not found, install it **with the
40+
user's consent** (this fetches and runs an installer that drops the `rocm` and
41+
`rocmd` binaries into `~/.local/bin`). Only nightly builds are published
42+
today, so install from the `nightly` channel:
43+
44+
- **Linux / macOS:**
45+
```
46+
curl -fsSL https://raw.githubusercontent.com/ROCm/rocm-cli/main/install.sh | sh -s -- nightly
47+
```
48+
- **Windows (PowerShell):**
49+
```
50+
$env:ROCM_CLI_CHANNEL = "nightly"
51+
irm https://raw.githubusercontent.com/ROCm/rocm-cli/main/install.ps1 | iex
52+
```
53+
54+
(Once rocm-cli cuts a stable release, drop the `nightly` channel — `sh` /
55+
`iex` alone will pull the latest stable build.)
56+
57+
After install, confirm `~/.local/bin` is on `PATH` and re-run `rocm --version`.
58+
If it still isn't available, hand the user the install page
59+
(https://github.com/ROCm/rocm-cli) and stop.
60+
61+
1. **Diagnose.** Pass the user's error text as the symptom:
62+
63+
```
64+
rocm diagnose --symptom "<paste the exact error>" --json
65+
```
66+
67+
Read the JSON:
68+
- `matched[]` — ranked causes, each with `id`, `title`, `score` (0–100),
69+
`evidence[]`, and a `fix` (with `fix_id`, `summary`, `commands`, `verify`,
70+
`notes`, and the `needs_sudo` / `needs_reboot` / `needs_relogin` /
71+
`auto_applicable` flags). `score >= 75` = high confidence; `50–74` = likely
72+
(confirm one more piece of evidence with the user first).
73+
- `out_of_scope` — when set (e.g. WSL2), do **not** diagnose. First, if the
74+
user's symptom clearly names an app that ships its own runtime (Lemonade,
75+
Ollama, LM Studio), route them to that app's tracker (see
76+
[Framework routing](#framework-routing)) — those trackers apply regardless
77+
of platform. Otherwise relay the `out_of_scope` message and stop (see
78+
[Out of scope](#out-of-scope)).
79+
- `route_when_no_match` — when `matched` is empty, hand the user this
80+
upstream tracker; **do not speculate**. Note the CLI picks this target from
81+
the *host-detected* framework, not from the symptom text — so for an app
82+
named only in the symptom, route it yourself per
83+
[Framework routing](#framework-routing).
84+
85+
2. **Propose the fix.** Show the top match's `title`, `evidence`, plan, and
86+
`verify` command. Only propose applying it when the user is on board.
87+
88+
3. **Apply with consent.** For an auto-applicable fix:
89+
90+
```
91+
rocm fix <fix-id> # auto fixes: prompt before changing anything
92+
rocm fix <fix-id> --dry-run # show the exact change, touch nothing
93+
rocm fix <fix-id> --yes # required to apply in a non-interactive shell
94+
```
95+
96+
Only the four auto-applicable fixes prompt and mutate. The other 11 are
97+
**print-only** (bootloader, kernel, reinstall, Windows driver, …): `rocm fix
98+
<id>` just prints the plan for the user to run themselves — no prompt, and the
99+
CLI never performs those.
100+
101+
4. **Verify.** Have the user run the `verify` command from the diagnosis.
102+
103+
Use `rocm examine` (or `rocm examine --json`) when you only need the host state
104+
(GPU, driver, ROCm install, groups, framework) without a diagnosis.
105+
106+
## Framework routing
107+
108+
`rocm diagnose` covers frameworks that build against the **system** ROCm/HIP:
109+
110+
- **PyTorch**, **llama.cpp** — in scope; diagnose normally.
111+
112+
Apps that ship their **own** ROCm runtime aren't diagnosed here — route the user
113+
to the right tracker. (The CLI's `route_when_no_match` also targets these, but
114+
only when the host probe *detects* that app; when the app is named only in the
115+
symptom, do the routing yourself using the list below.)
116+
117+
- **Lemonade** → https://github.com/lemonade-sdk/lemonade/issues
118+
- **Ollama** → https://github.com/ollama/ollama/issues
119+
- **LM Studio** → in-app support (no public repo)
120+
- Anything else with no catalog match → ROCm core:
121+
https://github.com/ROCm/ROCm/issues (this is what `route_when_no_match`
122+
returns by default).
123+
124+
## Out of scope
125+
126+
- **WSL2** — a distinct platform (`/dev/dxg` + the Windows host driver, not the
127+
in-tree `amdgpu` module or `/dev/kfd`). `rocm examine`/`diagnose` detect it and
128+
route out; relay that guidance and point at AMD's ROCm-on-WSL guide.
129+
- **NVIDIA / Intel / Apple Silicon GPUs**, and **fresh installs on a clean
130+
machine** (a setup task, not a diagnosis). Exit cleanly and say so.
131+
132+
## Rules
133+
134+
- Never invent a fix. If `rocm diagnose` returns no match, route upstream.
135+
- Never run a mutating fix without the user's explicit OK; prefer `--dry-run`
136+
first. New failure modes are added to the CLI catalog, not improvised here.
137+
138+
See `reference.md` for the full closed catalog and the CLI command/exit-code
139+
reference.

skills/rocm-doctor/reference.md

Lines changed: 91 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,91 @@
1+
# ROCm Doctor — reference
2+
3+
The closed failure-mode catalog and the CLI it drives. The catalog is
4+
authoritative in the `rocm` CLI (`crates/rocm-core`); this file mirrors it for
5+
humans. To add or change a failure mode, change the CLI catalog — not this doc.
6+
7+
## CLI commands
8+
9+
### `rocm examine [--json]`
10+
11+
Inspect the host: GPU + gfx target, driver, ROCm install, render/video groups,
12+
`/dev/kfd` + render devices, kernel modules, framework introspection, recent
13+
amdgpu kernel-log evidence. `--json` emits the **Examination** document for
14+
tooling.
15+
16+
- It's a general system inspector, so it **always exits 0**. The verdict is the
17+
`status` field: `ok` · `no-amd-gpu` · `wsl` · `unsupported-os` · `degraded`.
18+
19+
### `rocm diagnose [--symptom "<text>"] [--top N] [--json]`
20+
21+
Match the host + symptom against the closed catalog. **Always exits 0**; read
22+
the result from `--json`:
23+
24+
- `matched[]` — ranked `{ id, title, score, evidence[], fix }`. Tiers:
25+
`>= 75` high confidence, `50–74` likely, `< 50` weak.
26+
- `min_score_for_match` (50), `high_confidence_threshold` (75).
27+
- `out_of_scope` — set when the host is off-catalog (e.g. WSL2); `matched` is empty.
28+
- `route_when_no_match``{ target, url }` upstream tracker to use when nothing matched.
29+
30+
### `rocm fix [<id>] [--yes] [--dry-run] [--device-index N]`
31+
32+
Apply a fix by id (run with no id to list). Exit codes:
33+
34+
| code | meaning |
35+
| --- | --- |
36+
| 0 | applied / dry-run / print-only plan / list |
37+
| 1 | internal error |
38+
| 2 | usage error (incl. unknown fix-id) |
39+
| 3 | not applicable on this host (OS mismatch, missing/negative `--device-index`) — nothing changed |
40+
| 4 | attempted but the command failed |
41+
| 5 | user declined at the prompt |
42+
43+
Only four fixes are auto-applicable — `fix-2-unset-override`,
44+
`fix-4-render-group`, `fix-6-path`, `fix-9-igpu-dgpu` — and the rest print their
45+
plan for the user to run. Pass the **full** id (`rocm fix fix-2-unset-override`,
46+
not `rocm fix fix-2`; a short id returns exit 2, unknown fix-id). Auto fixes
47+
print the exact command, honor `--dry-run`, refuse on a non-interactive shell
48+
without `--yes`, and confirm before mutating.
49+
50+
## Closed catalog (15 failure modes)
51+
52+
| id | OS | Failure mode | Typical signal | Auto-fix |
53+
| --- | --- | --- | --- | --- |
54+
| `fix-1-arch` | both | GPU gfx target not in the framework's build arch list | `hipErrorNoBinaryForGpu`, `HSA_STATUS_ERROR_INVALID_ISA`, "invalid device function" | no |
55+
| `fix-2-unset-override` | both | `HSA_OVERRIDE_GFX_VERSION` set on a GPU that now has a native wheel | page faults / `OUT_OF_REGISTERS`, override set in env | yes |
56+
| `fix-3-rocm-kernel` | linux | ROCm + distro/kernel form an unsupported triple | ROCm installed but `amdgpu` not loaded; DKMS build failure | no |
57+
| `fix-4-render-group` | linux | User not in `render`/`video` group (or `/dev/kfd` owned by the other group) | cannot open `/dev/kfd`, permission denied | yes |
58+
| `fix-5-amdgpu-load` | linux | `amdgpu` module not loaded (or blacklisted) | "ROCk module is NOT loaded", blacklist entry, Secure Boot | no |
59+
| `fix-6-path` | both | ROCm/HIP binaries not on PATH after install | `rocminfo: command not found`, `hipInfo` missing from PATH | yes |
60+
| `fix-7-stale-repos` | linux | Stale/conflicting APT/DNF repos from prior installer runs | apt 404 `repo.radeon.com`, unmet deps, ≥2 ROCm repo files | no |
61+
| `fix-8-wheel-rocm` | both | Framework wheel built for a different ROCm major than the system | `libamdhip64.so.X` / `amdhip64_X.dll` load failure | no |
62+
| `fix-9-igpu-dgpu` | both | iGPU enumerated alongside dGPU, destabilising the runtime | APU + discrete AMD present, `HIP_VISIBLE_DEVICES` unset, crash/segfault | yes |
63+
| `fix-10-container` | linux | Container can't see `/dev/kfd` or `/dev/dri/renderD*` | running in docker/podman, kfd/render devices missing | no |
64+
| `fix-11-iommu` | linux | Multi-GPU hang with IOMMU enabled | ≥2 AMD GPUs, `iommu=` not `pt`, hang/deadlock/timeout | no |
65+
| `fix-12-installer` | linux | `amdgpu-install` left a broken DKMS / repo state | dpkg half-configured, DKMS failed, `--accept-eula` | no |
66+
| `fix-13-hip-sdk-missing` | windows | HIP SDK not installed | no HIP SDK under Program Files, `hipInfo` not recognized | no |
67+
| `fix-14-adrenalin-too-old` | windows | Adrenalin / kernel-mode driver too old for the HIP SDK | `hipInfo` can't enumerate, "driver too old", HSA "no agents found" | no |
68+
| `fix-15-msvc-redist` | windows | MSVC runtime missing (HIP DLLs can't load) | `vcruntime140.dll` / `vcruntime140_1.dll` missing | no |
69+
70+
Linux-only: fix-3, -4, -5, -7, -10, -11, -12. Windows-only: fix-13, -14, -15.
71+
Cross-platform: fix-1, -2, -6, -8, -9.
72+
73+
## Framework routing
74+
75+
`rocm diagnose` diagnoses frameworks built against the **system** ROCm/HIP:
76+
77+
- **PyTorch**, **llama.cpp** — in scope.
78+
79+
Apps that ship their own runtime are routed upstream (via `route_when_no_match`):
80+
81+
- **Lemonade**https://github.com/lemonade-sdk/lemonade/issues
82+
- **Ollama**https://github.com/ollama/ollama/issues
83+
- **LM Studio** → in-app support (no public repo)
84+
- Otherwise → ROCm core: https://github.com/ROCm/ROCm/issues
85+
86+
## Out of scope
87+
88+
- **WSL2** — distinct platform (`/dev/dxg` + Windows host driver). Detected and
89+
routed out; point at AMD's ROCm-on-WSL guide:
90+
https://rocm.docs.amd.com/projects/radeon-ryzen/en/latest/docs/install/installryz/wsl/howto_wsl.html
91+
- NVIDIA / Intel / Apple Silicon GPUs; fresh installs on a clean machine.

skills/rocm-doctor/skill-card.md

Lines changed: 17 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,17 @@
1+
# Skill Card
2+
3+
## Description
4+
5+
Diagnoses ROCm / HIP SDK / PyTorch / llama.cpp failures on AMD GPUs (Linux and
6+
Windows) against a closed list of known misconfigurations, and applies a
7+
low-risk fix with consent or routes the user to the right upstream channel. A
8+
thin driver over the `rocm` CLI (`rocm examine` / `rocm diagnose` / `rocm fix`) —
9+
the probe, catalog, and fixes live in the CLI, versioned with the binary.
10+
11+
## Owner
12+
13+
rocm-cli team (AMD)
14+
15+
## License
16+
17+
MIT

0 commit comments

Comments
 (0)