New Skill: hrr-replay-analysis - #117
Open
andraghetti wants to merge 2 commits into
Open
Conversation
andraghetti
force-pushed
the
hrr-replay-analysis
branch
2 times, most recently
from
August 5, 2026 09:53
ed9f08e to
017bc10
Compare
Ships the HRR replay skill the catalog has advertised as planned, under the published catalog name so the skill and the catalog row agree, and publishes it in the Claude, Cursor and Codex manifests. The skill replays a HIP Record and Replay archive and reports what went wrong as a structured finding: fault class, fault address, failing event, and the implicated kernel. It stops at the finding; kernel patching, rocgdb and fix verification are out of scope and handed off. Behavioral evals come in two tiers. Four tests run against replay logs checked into evals/fixtures, so the reasoning is graded on a runner with no GPU, and they include an unreadable-archive guard and a false-activation screen. A fifth replays a real archive and skips unless /dev/kfd, an hrr-playback binary and HRR_EVAL_ARCHIVE are all present. Validated on a gfx950 host against a real offline vLLM inference capture (185,469 events, 13,233 kernels), a crash capture, and an archive whose format the playback build cannot read. Defects fixed along the way: - A clean replay still reported a hipBLASLt GEMM as the kernel, matched out of the archive's own kernel listing; next to a PASS that reads as a culprit. The kernel fields are now cleared for replay_pass. - A memory fault tears the process down before HRR attributes the failing dispatch, so the finding named no kernel even for a single-kernel archive. The --info kernel table is now parsed and used as a fallback when the archive holds exactly one kernel, marked as inferred. Names truncated by the table's column width are ignored, since a truncated symbol cannot be looked up. - Classification ran per input and the archive --info pass overwrote the replay verdict with UNKNOWN; it now runs once over every input. - A GPU memory fault also trips the generic abort line, which was tested first, so a read-only page fault reported replay_fatal_api. - The Complete: matcher accepted only YES/NO while --info prints "yes (clean shutdown)", so a clean archive never matched. - Format mismatches were unclassified and read as a workload result; they now report archive_version_mismatch. - The GPU picker's awk emitted free VRAM before reading the used-memory line, so "most free VRAM" always selected the largest card, and GPU selection ran before the --info early exit, which must work on a host with no GPU. - The replay masked the device with ROCR_VISIBLE_DEVICES, which re-indexes devices underneath the HIP mask and can land the replay on a different card.
andraghetti
force-pushed
the
hrr-replay-analysis
branch
from
August 5, 2026 09:55
017bc10 to
7382263
Compare
andraghetti
marked this pull request as ready for review
August 5, 2026 09:57
Review against the HRR runbooks found three gaps. Two are applied as proposed; the third is applied as a caveat rather than a new fault class. - RE_MEM_FAULT_ERR anchors on the two fields actually consumed. The bracket's leading fields vary by ROCm build, and on builds that omit `host:` the kernel name was dropped entirely, which is the one field the report exists to give. - `--info` kernel rows tolerate a missing id column, and the launch total is recovered from the API call-count block when no `Kernels:` line is printed. - run_hrr_replay.sh adds --sync-after-launch by default, with --no-sync to opt out. Without it the GPU is serialized once at the end, so a fault is reported but never attributed: on a real gfx950 crash log this fills failing_call_index, which was empty before. --timing opts out on its own, since serializing every launch is exactly what a timing run must not do. - Per-event progress lines are parsed, so the last launch to start before a fault stands in when the runtime's fault line named no kernel. A fault on a <<<>>>-launched ATen kernel stays illegal_memory_access rather than becoming a separate class. Capture records pointers embedded in by-value struct args and replay translates them with a defensive rescan, so such a fault can be genuine, and reclassifying it would suppress a real finding. The translation is a value-based heuristic and archives predating it carry no recorded offsets, so the finding carries a note and the skill asks for the original failure signature rather than resolving the ambiguity either way.
4 tasks
hrr-replay-analysis
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds
hrr-replay-analysis, the skill the catalog has advertised as_planned_since #94. It replays a HIP Record and Replay (HRR) archive on the host GPU and reports what went wrong as a structured finding: fault class, fault address, failing event index, and the implicated kernel.An HRR archive is a deterministic record of a real HIP workload, so a customer failure can be reproduced from the recording alone, without their application, source or data. The skill stops at the finding and hands off; kernel patching, rocgdb and fix verification are deliberately out of scope.
Layout
Testing
Structural gates:
./.github/scripts/check.sh: 7 skills, 0 errors, Cursor and Codex manifests in syncsubprocess, both advisory)End to end on a gfx950 host, against three real archives:
That validation changed the skill. The clean workload was reporting a hipBLASLt GEMM as its kernel, matched out of the archive's own kernel listing, which next to a pass reads as a culprit; a clean replay now implicates nothing. The crashing workload was reporting no kernel at all, because a memory fault tears the process down before the failing dispatch is attributed, so the archive's kernel table is now used as an explicitly-marked fallback when it holds exactly one kernel. Both cases are covered by fixtures. The full list of parser and device-masking fixes is in the commit message.
Behavioral evals come in two tiers. Four tests run against replay logs checked into
evals/fixtures/, so they grade the agent's reasoning (right fault class, right address, right kernel) on a runner with no GPU, and they include an unreadable-archive guard and a false-activation screen. A fifth test replays a real archive end to end and skips unless/dev/kfd, anhrr-playbackbinary andHRR_EVAL_ARCHIVEare all present, so it stays opt-in rather than failing where no GPU is available.