Skip to content

fix(kernel): stop sentinel strings from short-circuiting source resolution - #1106

Closed
zhanglei-amd wants to merge 1 commit into
mainfrom
feat/zhanglei/collective-forge-driver
Closed

fix(kernel): stop sentinel strings from short-circuiting source resolution#1106
zhanglei-amd wants to merge 1 commit into
mainfrom
feat/zhanglei/collective-forge-driver

Conversation

@zhanglei-amd

Copy link
Copy Markdown
Contributor

Problem

TraceLens writes "Not found" / "AITER (vendor)" into source_file whenever it cannot attribute a Synthetic Op (every hand-written Triton kernel launched outside the ATen dispatcher). Those are non-empty strings, so every downstream "already resolved?" check read them as a resolved source and skipped the resolution tiers entirely.

The damage was not just a missing path. classify_patchability then reported the nonsensical source not under a reusable framework root: Not found, which pointed debugging at the framework root instead of the missing lookup.

Measured across 19 MiniMax-M3-MXFP8 sessions: 76 of 101 hot-kernel candidates (75%) were dropped before reaching a backend, including every MXFP8 hot kernel in the model.

Approach

Resolve by shape rather than truthiness — a real source_file always carries a source extension, a producer placeholder never does. This holds for placeholder spellings nobody enumerated in advance (AITER (vendor) showed up 3 times in real data and was never on any list).

Four layers, deterministic first, LLM only as the last resort:

Purpose Where
Stop placeholders from posing as resolved sources _SOURCE_EXT_RE + looks_like_source_path(), three call sites
Kill Not found at table-parse time _LAUNCHER_PATH_PLACEHOLDERS in tracelens_skill_runner.py
Extract the source location from the runtime call stack new _trace_launcher_resolver.py
Keep bare runtime API names out of the queue _RUNTIME_API_NAMES
Fallback for cases the deterministic tiers cannot reach new _llm_source_fallback.py, off by default

The trace resolver reads python_function frames via correlation with the runtime launch event. It is exact to file, line and function, and structurally immune to the two false positives the grep fallback produces (test files and CPU implementations) because it reads runtime fact rather than matching text.

Results

Measured on a live MiniMax-M3-MXFP8 run:

before after
candidates detected 101 (19 sessions) 7
reusable 25 (24.8%) 5 (71.4%)
hot-kernel source_file Not found real Triton paths

All three MXFP8 hot kernels now resolve:

  • _mxfp8_grouped_gemm_kernel -> kernels/ops/moe/mxfp8_moe_amd_gfx95.py
  • _mxfp8_linear_kernel -> kernels/ops/quantization/mxfp8_amd_gfx95.py
  • _gqa_share_sparse_fwd_kernel -> attention/minimax_sparse/prefill/topk_sparse.py

runs/kernel_opt went from empty across all 19 historical sessions to actually dispatching forge attempts.

Scope

This PR carries only the source-resolution work. The collective-lane changes developed alongside it are deliberately excluded: tracelens_analysis.py was split hunk by hunk, dropping the three hunks that inject collective candidates (one of them a bare from _nccl_summary_candidates import ... that would ImportError without the unshipped module).

Verified before commit: the staged snapshot references no unshipped module, and all four touched modules pass py_compile.

Not addressed

E2E throughput. This unblocks hot kernels from reaching a backend; whether that converts to gain depends on the quality of what forge/GEAK generates. Observed so far: candidates do reach forge, but the variants produced were only 1.38% faster and fell inside the 2% noise floor.

Test plan

  • pytest src/hyperloom/agents/kernel/tests/test_trace_launcher_resolver.py
  • pytest src/hyperloom/agents/kernel/tests/test_source_resolution_guards.py
  • pytest src/hyperloom/agents/kernel/tests/test_llm_source_fallback.py
  • Replay historical analysis.md files and confirm hot-kernel rows no longer carry a placeholder source_file
  • Full session on 8x MI355X: confirm runs/kernel_opt is non-empty and bare runtime API names are never routed

Made with Cursor

…ution

TraceLens writes "Not found" / "AITER (vendor)" into source_file whenever
it cannot attribute a Synthetic Op. Those are non-empty strings, so every
downstream "already resolved?" check read them as a resolved source and
skipped the resolution tiers entirely -- 75% of hot kernels were dropped
before reaching a backend, and the misleading skip_reason blamed the
framework root instead of the missing lookup.

Resolve by shape rather than truthiness: a real source_file always carries
a source extension, a producer placeholder never does. On top of that, add
a trace-based resolver that reads the runtime python_function stack (exact
to file, line and function, and immune to the grep fallback's test-file and
CPU-implementation false positives), blacklist bare runtime API names that
are not kernels at all, and keep an opt-in LLM fallback for cases the
deterministic tiers cannot reach.

Measured on MiniMax-M3-MXFP8: candidate reusable rate 24.8% -> 71.4%, with
all three MXFP8 hot kernels resolving to their real Triton sources.
@zhanglei-amd
zhanglei-amd requested review from a team, devalshahamd and tsrikris as code owners August 6, 2026 07:04
import json
import os
import re
from pathlib import Path
import os
import re
from pathlib import Path
from typing import Any, Callable
import json
import os
import re
from pathlib import Path
import os
import re
from pathlib import Path
from typing import Any, Callable
@zhanglei-amd

Copy link
Copy Markdown
Contributor Author

Reopened as #1107 on a correctly named branch: this change carries only the source-resolution work, while feat/zhanglei/collective-forge-driver is reserved for the collective lane.

@zhanglei-amd
zhanglei-amd deleted the feat/zhanglei/collective-forge-driver branch August 6, 2026 07:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants