fix(kernel): stop sentinel strings from short-circuiting source resolution

[zhanglei-amd]

Problem

TraceLens writes a placeholder into source_file whenever it cannot resolve a
launcher — Not found for every Synthetic Op, AITER (vendor) for vendor
kernels, N/A elsewhere. Downstream code tested that field for truthiness, so a
placeholder counted as a resolved source: the grep fallback was skipped and
classify_patchability rejected the candidate with "source not under a
reusable framework root: Not found".

This hit the hottest kernels precisely because they are the ones TraceLens
cannot attribute — hand-written Triton kernels launched straight from Python
have no cpu_op parent, so they surface as Synthetic Ops.

Resolution ladder

curated dictionary → trace-derived launcher → name grep → model fallback
                                                        → model review

Placeholders are normalized to empty before any of it, since every later tier
is gated on an empty source_file. looks_like_source_path() replaces the
truthiness test, and a rejected value is preserved as source_file_rejected
rather than silently dropped. Bare runtime API names (hipModuleLaunchKernel
and friends) are excluded from grep, which would otherwise match unrelated call
sites.

Trace-derived resolution

_trace_launcher_resolver walks kernel correlation → runtime launch → the
enclosing python_function frame. Three properties make it trustworthy:

Deterministic. Profiler timestamps are microsecond-granular, so adjacent
frames on a fast call chain share a ts. Ordering by start alone left the tie
to trace write order — the same call chain resolved differently depending on
which frame was serialised first. Ordering by (ts, -dur) lets nesting decide.

Process-aware. Kernels, launches and frames are keyed with pid. A merged
multi-process trace restarts correlation and thread ids per rank, so without it
one rank's launch overwrites another's and a kernel binds to a different
process's launcher.

Bounded. Kernels launched from C++ or only replayed from a graph never
resolve, so "stop once everything resolves" would read an entire capture folder
at two streaming passes per file. The file count is capped, scanning stops when
a file adds nothing, and the probe map is sized by launches rather than by every
correlated runtime event.

A tie at the top of the frame vote now leaves the kernel unresolved rather than
picking whichever the counter saw first — a plurality is not agreement, and the
grep tier is a better answer than an arbitrary one.

The two model tiers

Both run unconditionally; there is no environment switch. They differ in scope
and authority, and docs/reference/environment-variables.md documents each
separately.

Fallback sees a candidate only after all three deterministic tiers come up
empty, and only above 5% GPU share. It may return one of the exact grep
shortlist strings and nothing else; an invented path is rejected, as is anything
below 0.7 confidence.

Review exists because the deterministic tiers do not fail by coming up
empty — they fail by coming up confidently wrong. Measured across historical
sessions, only 59% of verifiable resolutions mention the kernel they claim to
define, and aten::fill_ alone resolves to four unrelated business files, each
a real root-resident path passing every mechanical check. A tier gated on an
empty source_file can never see any of them, so this one inspects the whole
table and may keep, rewrite or unresolve any entry. Two guard rails bound it: a
rewritten path must exist on disk under a known framework root, and every
revision records previous_source_file and previous_method so a bad review is
reversible.

Neither tier can fail a run. No model configured, a gateway error, a timeout or
an unparseable reply all leave the deterministic result standing.

Observability

Every non-trivial outcome is recorded as source_resolution_reason, so a skip
is distinguishable from a failure:

Value	Meaning
(absent)	Resolved deterministically
llm_fallback_skipped: gpu_pct ...	Below the 5% floor; no call made
llm_fallback_no_shortlist	Grep found nothing; no call made
llm_fallback_declined: ...	Answered but rejected
llm_fallback_error: ...	Import error, gateway rejection, or timeout
trace_resolver_error: ...	Trace unreadable or resolver raised
rejected_non_path_sentinel	A placeholder was zeroed

Per-file resolver errors reach the caller through a file_errors list instead
of surfacing as an ordinary "0 resolved". The resolved/total tally goes to the
run log, where an operator actually looks.

New artifact

kernel_source_resolution.json records, per hot kernel, the identity, GPU
share, resolved location and deciding tier, behind a schema_version consumers
can gate on. Previously this existed only as fields mutated onto candidate rows
by a dozen functions, with no single place to read "where does this kernel live,
and how do we know".

Scope: it is an audit view, not the stage contract. kernel_candidates.json
remains the contract, since it alone carries shapes, eligibility, backends and
task groups. Review-tier revisions reach the pipeline because
apply_resolution_entries_to_candidates() folds them back onto the candidates
and re-runs classification — a rewrite changes source_type, which decides
whether the kernel is dispatched at all.

On a completed session the artifact reported 37 entries, 36 resolved: 29 by
grep, 6 by the trace tier, 1 by the curated dictionary.

Files

New:

• common/kernel_source_contract.py — versioned artifact schema and validation
• tools/_trace_launcher_resolver.py
• tools/_llm_source_fallback.py
• tools/_llm_source_review.py
• tests/test_trace_launcher_resolver.py
• tests/test_source_resolution_guards.py
• tests/test_source_resolution_artifact.py
• tests/test_llm_source_fallback.py

Modified:

• tools/tracelens_analysis.py
• tools/tracelens_skill_runner.py
• docs/reference/environment-variables.md
• inference_optimizer/tests/test_setup_cli.py — unrelated CI fix, see below

Testing

Full kernel-agent suite: 1321 passed. The one failure
(test_bypass_report.py::test_render_surfaces_source_dispatchability_and_task_groups)
reproduces on a clean origin/main worktree and is unrelated.

The test_setup_cli.py change is a CI fix, not part of this feature. The
DeepSeek-only preflight test inherited the host environment, so a runner with
OPENAI_BASE_URL set tripped the cross-provider check and failed a test whose
premise is that only DeepSeek is configured. It reproduces on a clean
origin/main worktree by exporting that variable, so it is a pre-existing
sensitivity rather than a regression — but it was blocking this PR's checks.

Not included

Roofline patch-set repairs and the SGLang patch vendoring were part of an
earlier revision of this branch and have been removed. Consumer-side
consolidation is also out of scope: request_handlers.py still carries its own
copy of the source-path classification logic, which can disagree with the
producer's. Unifying that touches eight consumers and belongs in its own PR.

[github-actions]

CI E2E report — ❌ Failed

item	value
result	❌ Failed
model	Qwen/Qwen3-0.6B (dense)
resources	1× GPU, TP=1
PR branch	fix/zhanglei/kernel-source-resolution
commit	300cfb1ef4c84fc333f241e836da37fd28657610
session_id	8b18608a-22db-4f7d-93c2-8ed0a11ec7c9
queue → dispatch	675m 40s
run time	124m 24s
total	800m 4s

details

[tsrikris]

@zhanglei-amd Note that @Ahmedhasssan-aig is developing an alternate to the JSON lookup #1113