fix(kernel): stop sentinel strings from short-circuiting source resolution - #1106
Closed
zhanglei-amd wants to merge 1 commit into
Closed
fix(kernel): stop sentinel strings from short-circuiting source resolution#1106zhanglei-amd wants to merge 1 commit into
zhanglei-amd wants to merge 1 commit into
Conversation
…ution TraceLens writes "Not found" / "AITER (vendor)" into source_file whenever it cannot attribute a Synthetic Op. Those are non-empty strings, so every downstream "already resolved?" check read them as a resolved source and skipped the resolution tiers entirely -- 75% of hot kernels were dropped before reaching a backend, and the misleading skip_reason blamed the framework root instead of the missing lookup. Resolve by shape rather than truthiness: a real source_file always carries a source extension, a producer placeholder never does. On top of that, add a trace-based resolver that reads the runtime python_function stack (exact to file, line and function, and immune to the grep fallback's test-file and CPU-implementation false positives), blacklist bare runtime API names that are not kernels at all, and keep an opt-in LLM fallback for cases the deterministic tiers cannot reach. Measured on MiniMax-M3-MXFP8: candidate reusable rate 24.8% -> 71.4%, with all three MXFP8 hot kernels resolving to their real Triton sources.
zhanglei-amd
requested review from
a team,
devalshahamd and
tsrikris
as code owners
August 6, 2026 07:04
| import json | ||
| import os | ||
| import re | ||
| from pathlib import Path |
| import os | ||
| import re | ||
| from pathlib import Path | ||
| from typing import Any, Callable |
| import json | ||
| import os | ||
| import re | ||
| from pathlib import Path |
| import os | ||
| import re | ||
| from pathlib import Path | ||
| from typing import Any, Callable |
Contributor
Author
|
Reopened as #1107 on a correctly named branch: this change carries only the source-resolution work, while feat/zhanglei/collective-forge-driver is reserved for the collective lane. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
TraceLens writes
"Not found"/"AITER (vendor)"intosource_filewhenever it cannot attribute a Synthetic Op (every hand-written Triton kernel launched outside the ATen dispatcher). Those are non-empty strings, so every downstream "already resolved?" check read them as a resolved source and skipped the resolution tiers entirely.The damage was not just a missing path.
classify_patchabilitythen reported the nonsensicalsource not under a reusable framework root: Not found, which pointed debugging at the framework root instead of the missing lookup.Measured across 19 MiniMax-M3-MXFP8 sessions: 76 of 101 hot-kernel candidates (75%) were dropped before reaching a backend, including every MXFP8 hot kernel in the model.
Approach
Resolve by shape rather than truthiness — a real
source_filealways carries a source extension, a producer placeholder never does. This holds for placeholder spellings nobody enumerated in advance (AITER (vendor)showed up 3 times in real data and was never on any list).Four layers, deterministic first, LLM only as the last resort:
_SOURCE_EXT_RE+looks_like_source_path(), three call sitesNot foundat table-parse time_LAUNCHER_PATH_PLACEHOLDERSintracelens_skill_runner.py_trace_launcher_resolver.py_RUNTIME_API_NAMES_llm_source_fallback.py, off by defaultThe trace resolver reads
python_functionframes via correlation with the runtime launch event. It is exact to file, line and function, and structurally immune to the two false positives the grep fallback produces (test files and CPU implementations) because it reads runtime fact rather than matching text.Results
Measured on a live MiniMax-M3-MXFP8 run:
source_fileNot foundAll three MXFP8 hot kernels now resolve:
_mxfp8_grouped_gemm_kernel->kernels/ops/moe/mxfp8_moe_amd_gfx95.py_mxfp8_linear_kernel->kernels/ops/quantization/mxfp8_amd_gfx95.py_gqa_share_sparse_fwd_kernel->attention/minimax_sparse/prefill/topk_sparse.pyruns/kernel_optwent from empty across all 19 historical sessions to actually dispatching forge attempts.Scope
This PR carries only the source-resolution work. The collective-lane changes developed alongside it are deliberately excluded:
tracelens_analysis.pywas split hunk by hunk, dropping the three hunks that inject collective candidates (one of them a barefrom _nccl_summary_candidates import ...that wouldImportErrorwithout the unshipped module).Verified before commit: the staged snapshot references no unshipped module, and all four touched modules pass
py_compile.Not addressed
E2E throughput. This unblocks hot kernels from reaching a backend; whether that converts to gain depends on the quality of what forge/GEAK generates. Observed so far: candidates do reach forge, but the variants produced were only 1.38% faster and fell inside the 2% noise floor.
Test plan
pytest src/hyperloom/agents/kernel/tests/test_trace_launcher_resolver.pypytest src/hyperloom/agents/kernel/tests/test_source_resolution_guards.pypytest src/hyperloom/agents/kernel/tests/test_llm_source_fallback.pyanalysis.mdfiles and confirm hot-kernel rows no longer carry a placeholdersource_fileruns/kernel_optis non-empty and bare runtime API names are never routedMade with Cursor