fix(kernel): honor graph under-recording in the TraceLens idle gate - #1121
Merged
Conversation
The high-idle gate suppresses the hot-kernel list (and thus skips the whole forge kernel lane) whenever GPU idle exceeds the threshold. The bypass trace route already skips that gate when the trace is a cuda/HIP-graph under-recorded capture -- continuous graph replay overflows the profiler activity buffer, so only ~1 of N replays is recorded and idle% is inflated. The TraceLens route (deterministic + LLM-orchestrator) applied the idle gate with no such guard, so a graph-mode workload that is really compute/GEMM-bound (rocm-smi shows full saturation) was reported as ~87% idle and had every hot kernel suppressed, wrongly skipping forge-loop. Reuse the tested _bypass_trace_reader graph-coverage detection on both TraceLens idle-gate sites: when the raw trace is graph under-recorded, skip the high-idle suppression and surface the existing bypass_graph_under_recorded warning instead (candidates stay ranked by recorded-kernel GPU share). Best-effort and fail-open -- any probe failure falls back to the plain idle gate, never worse than today. Adds unit tests for the guard (under-recorded skip, plain-gate passthrough, below-threshold no-op, and never-raises on a bad trace path).
BaoYunkai
requested review from
a team,
devalshahamd and
tsrikris
as code owners
August 7, 2026 08:19
…lighten probe Addresses review of #1121. P1 (correctness): _graph_under_recorded treated any graph-mode trace with busy_fraction < 0.5 (or >=4 launches and busy < 0.9) as under-recorded, so a genuinely idle / sparse / host-bound graph workload -- whose replays are all fully recorded -- was wrongly cleared through the idle gate, defeating the high-idle suppression. Rekey the decision on *recorded-launch coverage*: the share of graph-launch correlations that actually recorded a kernel. Activity- buffer overflow drops whole replays so coverage collapses toward ~1/launches, whereas a fully-recorded idle workload keeps kernels on essentially every launch (coverage ~1.0) and stays gated. The reader now counts distinct graph-launch correlations with recorded kernels and exposes graph_launches_with_kernels. P2 (perf): the TraceLens graph-coverage probe called analyze_trace with top_k=0 + emit_launches=True, materializing every kernel-launch row and full top-N lists on large --skip-split traces. graph_coverage is independent of both, so probe with top_k=1 + emit_launches=False. Log fix: after skipping the idle gate for graph under-recording, both TraceLens routes no longer also log the contradictory "below gate" line. Tests: counter-example (4 launches all recorded, ~100% idle -> NOT under-recorded, idle gate still suppresses) on the bypass route; existing under-recorded cases (coverage 0.25/0.33) stay flagged.
Addresses the second review round of #1121. A (wrong trace on the default split path): cli_trace_path is reassigned to the steady-state chunk before the idle gate runs, and the graph-coverage probe used it. A chunk can drop the hipGraphLaunch runtime events, so detection reported non-graph and the original high-idle misfire returned on the default (non --skip-split) path. Capture the un-split raw_trace_path = trace_files[0] and probe graph coverage from it at both TraceLens idle-gate sites; add a raw-vs-launch-stripped-chunk regression showing detection needs the launch events. B (numerator/denominator scope mismatch under steady state): _finalize filtered k_events to the steady window before counting launches-with-kernels, while graph_launch_count stayed whole-trace, so a fully-recorded N-launch trace whose window held one replay read as 1/N and false-flagged under-recorded (breaking xDiT / explicit steady-state bypass). Compute recorded-launch coverage over the full event stream BEFORE the window filter so numerator and denominator share whole-trace scope; add a _finalize steady-window regression. Tests: +3 (whole-trace scope under window, raw-vs-chunk detection); full suite 277 passed.
CI E2E report — ✅ Succeeded
|
ZhengGong-amd
added a commit
that referenced
this pull request
Aug 8, 2026
One conflict, both sides adding imports to the workload-env builder: main brought `deps_cache_root` and `GENERIC_FRAMEWORK_ROOT_ENV`, this branch the two FlyDSL source-root helpers. Resolved as the union; every name is used. Nothing from main needed mirroring into the rewrite route. Its kernel-source work -- rejecting non-path sentinels (#1107) and honoring graph under-recording in the idle gate (#1121) -- lands in trace analysis, upstream of where this branch picks a candidate up, and only ever hands the route fewer bogus candidates. The integrate hardening applies to the `integrate_patch` executor, which is a separate path from the kernel `integrate` handler an apply-back flows through. Co-authored-by: Cursor <cursoragent@cursor.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The high-idle gate suppresses the hot-kernel candidate list (skipping the forge kernel lane entirely) whenever GPU idle % exceeds the threshold (default 80%). The bypass trace route already skips that gate when the trace is a cuda/HIP-graph under-recorded capture — under continuous graph replay the profiler activity buffer overflows and only ~1 of N replays is recorded, so idle% is inflated (
bypass_graph_under_recorded). The TraceLens route (deterministic + LLM-orchestrator) applied the idle gate with no such guard.Result: a graph-mode workload that is actually compute/GEMM-bound (rocm-smi shows full saturation) gets reported as ~87% idle from the roofline capture, every hot kernel is suppressed, and forge-loop is wrongly skipped as "host/scheduling-bound".
Observed on Qwen3-8B sglang (
.../safe-opt-claw-top-models-forge-qwen-sgla-b0c92b15-a1/.../20260806T082051Z):busy_of_wall=0.093roofline artifact →report_source=skipped:high_gpu_idle_pct, while the session's own scouts flagged the 87% idle as a "cuda-graph capture artifact" and the real workload as GEMM-bound.Changes
tracelens_analysis.py:_graph_coverage_from_raw_trace()— reuse the tested_bypass_trace_reader.analyze_trace(..., emit_launches=True)graph-coverage detection; fail-open to{}._evaluate_idle_gate_with_graph_guard()— when idle exceeds the threshold, first check graph under-recording; if under-recorded, skip suppression and return thebypass_graph_under_recordedwarning instead ofhigh_gpu_idle_pct.Test plan
test_tracelens_csv.py: under-recorded → skip suppression; not-under-recorded → plain gate applies; below-threshold → no probe; bad trace path → never raises.pytest test_tracelens_csv.py test_idle_gate.py test_bypass_trace_analysis.py— 273 passed, no regressions.