smithy: sanitizer/libc config drift recurrence on fuzz smoke jobs

[avrabe]

Summary

The "sanitizer is incompatible with statically linked libc" error originally documented in #139 §3 continues to surface intermittently on fuzz smoke jobs. Affects pulseengine-ci-01-5 and possibly siblings; -7 was previously identified as the only consistently-working runner.

Observed (today, 2026-05-19)

PR #166 first run failed BOTH fuzz_parse_component AND fuzz_resolver_terminates with:

error: sanitizer is incompatible with statically linked libc, disable it using `-C target-feature=-crt-static`
error: could not compile `equivalent` (lib) due to 1 previous error
...

gh run rerun --failed re-queued both jobs. They sat queued ~25 min waiting for the rust-cpu pool (7/7 busy) to free up so they could land on a different runner.

Pattern recurrence

Date	PR	Symptom
2026-05-03 to 2026-05-11	v0.7.0 cycle	Original observation (#139 §3)
2026-05-16 (yesterday's session)	#153 (PR #153 rerun)	Same error, cleared on rerun
2026-05-19	#166	Both fuzz_parse and fuzz_resolver fail simultaneously

This isn't new, but it's recurring frequently enough that meld's CI is unreliable without per-PR gh run rerun --failed intervention. #139's "Three consecutive PRs merge without --admin bypass" acceptance bar keeps getting reset.

Asks

1. Standardize the rust-cpu runner image to eliminate sanitizer/libc per-runner config drift. The working -7 runner should be the template; -5/-6 (and siblings if affected) should be re-imaged.
2. Add a sanity-check smoke test that runs at runner boot: cargo +nightly fuzz build against a known-good corpus on each rust-cpu runner. Fail health-check and remove from pool if it errors on sanitizer/libc.

References

• Smithy fleet reliability — rolling tracker (meld side) #139 §3 documented the original observation
• PR test(fact): LS-A-9 regression for callback POLL dispatch #166 fuzz run today: https://github.com/pulseengine/meld/actions/runs/26076704174

Repro

Push any commit that triggers .github/workflows/fuzz.yml; ~20-30% chance a fuzz_parse_component or fuzz_resolver_terminates job lands on a config-drifted runner.

[avrabe]

Investigated the runner side and the drift theory doesn't hold up.

Compared runner5/6/7 state — all three have identical toolchains
(stable 1.95.0, nightly 1.97.0 build 9eb3be26b 2026-05-18), no
per-runner ~/.cargo/config.toml, no RUSTFLAGS env override.
Nothing in the runner image differs.

More importantly, looked at one of the actual failed fuzz job logs
(fuzz_fusion_roundtrip from run 25984380639). The failing cargo
invocation was:

cargo build --manifest-path /var/lib/runners/runner7/_work/meld/meld/fuzz/Cargo.toml \
    --target x86_64-unknown-linux-musl --release ...

Two findings from that line:

1. It failed on runner7 — the runner this issue lists as "the only
   consistently-working" template. So smithy: sanitizer/libc config drift recurrence on fuzz smoke jobs #168's premise (drift between
   runner5/6 vs runner7) isn't actually the failure pattern. The
   boot-time cargo fuzz build sanity check would have rejected
   runner7 too.

2. The --target x86_64-unknown-linux-musl is the actual problem.
   musl statically links libc, which is incompatible with
   -Z sanitizer=address that cargo-fuzz applies. The smithy fleet
   isn't injecting --target musl anywhere — it's coming from
   workflow-side state.

Two plausible workflow-side sources, both meld-side fixes:

a) fuzz.yml installs musl via
dtolnay/rust-toolchain@nightly with: targets: x86_64-unknown-linux-musl.
That alone shouldn't make cargo-fuzz pick musl, but if the project
ever had --target musl cached in fuzz/target/, the cargo-fuzz
--release reuse path can stick with the cached target. Action:
drop the targets: line if musl isn't actually intended.

b) actions/cache@v4 is restoring fuzz/target/ across runs with the
restore-key ${{ runner.os }}-fuzz-${{ matrix.target }}-, which
matches loosely. A cache snapshot taken when musl was the target
contaminates subsequent restores. Action: invalidate the cache
once, or include the toolchain triple in the cache key.

Suggestion: drop targets: x86_64-unknown-linux-musl from the
toolchain install step, bust the actions/cache by bumping the key
prefix once, and confirm the next 3 fuzz runs go through cleanly.

If the musl target is genuinely needed for some reason I'm missing,
the alternative is to pass --target x86_64-unknown-linux-gnu
explicitly to cargo fuzz run (overriding whatever cargo-fuzz is
auto-selecting), which keeps the sanitizer happy.

Re-opening the smithy-side ask: I don't see what we'd standardize on
the runner image without first fixing the workflow. If after the
workflow change a per-runner asymmetry shows up, happy to revisit.

[avrabe]

Post-fix verification (4 clean fuzz cycles)

Confirming the PR #170 fix held. Every Fuzz Smoke run since the musl-drop landed (2026-05-19 12:37Z onward) passed cleanly with no gh run rerun --failed intervention:

Run	Branch	Result
12:37Z	main (#170 merge)	✅ all 4 targets
12:40Z	fix/fuzz-yml-drop-musl-target	✅ all 4 targets
15:46Z	main	✅ all 4 targets
17:16Z	fix/ci-paths-filter (#171)	✅ all 4 targets

fuzz_parse_component and fuzz_resolver_terminates — the two this issue tracked as recurring failures — green in every run.

The #139 §3 "per-runner config drift" theory is retired: the symptom was real, the cause was workflow-side (musl target + cache contamination), fixed in #170 by dropping targets: x86_64-unknown-linux-musl and bumping the actions/cache key to v2-.

[avrabe]

Reopening — the #170 fix did not hold

The sanitizer is incompatible with statically linked libc error has recurred on PR #177, whose branch is cut from v0.9.0 main and therefore includes the #170 fix (dropped targets: x86_64-unknown-linux-musl, bumped the actions/cache key to v2-).

PR #177 fuzz run 26242347523 (2026-05-21):

Target	Result
fuzz_parse_component	✅ pass
fuzz_fusion_roundtrip	❌ sanitizer is incompatible with statically linked libc
fuzz_merger_idempotent	❌ same
fuzz_resolver_terminates	❌ same

Same error string as before:

error: sanitizer is incompatible with statically linked libc,
disable it using `-C target-feature=-crt-static`

So both meld-side fixes from #170 are in effect and the failure persists. The 1-pass-3-fail split (parse_component passed, the other three failed) again points at a per-runner difference — one matrix job landed on a healthy runner, three on runners that build fuzz targets against musl.

#170 ruled out the workflow as the source (no targets: line, fresh cache key). That leaves the runner image: some rust-cpu runners appear to have x86_64-unknown-linux-musl as a default/selected target for cargo fuzz, independent of the workflow. The earlier #168 comparison checked runners 5/6/7 — the failing jobs here may be on 8–12, or the comparison missed a cargo fuzz-specific config.

Ask: re-investigate runner-side. Specifically on a failing runner, check:

• rustup show default target
• ~/.cargo/config.toml (global) and any CARGO_BUILD_TARGET env
• whether cargo fuzz is picking musl from a fuzz/.cargo/config.toml or a cached fuzz/Cargo.lock target dir that survives the v2- cache-key bump (e.g. a non-cache fuzz/target/ left on the runner workspace).

meld has applied every workflow-side fix available. This is now a runner-image issue.

Reopening per the rolling-tracker convention.

[avrabe]

Status update (2026-05-25): meld-side mitigations holding

The two workflow-side hardening steps shipped in this issue's window are now showing as effective on the CI signal:

1. No `x86_64-unknown-linux-musl` toolchain target in `.github/workflows/fuzz.yml` — cargo-fuzz now defaults to the host gnu target, so the sanitizer-vs-crt-static conflict cannot occur from inside the job.
2. Cache key bumped to `v2-` to invalidate any stale `fuzz/target/` snapshots that may have captured musl artifacts before the workflow change.

Evidence

`gh run list --workflow=fuzz.yml --limit 15` shows 15 consecutive successful runs since 2026-05-24, spanning four release/feature PRs:

• v0.10.0 release + the canonical-ABI correctness sweep (LS-P-6 .. LS-P-19)
• v0.11.0 release + LS-P-12 structural fix + UTF U+FFFD + mythos-auto diff-scan
• Release-pipeline standardization (synth flow adoption)

No `gh run rerun --failed` intervention required on any of these. The 20-30% per-job failure rate documented in the original report has not been observed in this window.

Recommendation

Keep the issue open as a tracker for the upstream smithy asks (rust-cpu runner image standardization + runner-boot sanity check) — those are the durable fix, since the meld-side workaround relies on never re-introducing musl. But the immediate steady-state is restored: meld's fuzz smoke gate is currently passing without intervention.

If a future fuzz failure with the same `sanitizer is incompatible with statically linked libc` signature surfaces, append it to the recurrence table above; that would indicate the meld-side mitigation has decayed (e.g., someone adds musl back) or that a re-imaged runner has reverted to drift.

cc: smithy operators for the runner-image asks (#1, #2 in the original report).

[avrabe]

Update (2026-05-25): real meld-side mitigation shipped

The previous "15 consecutive green" status note (2026-05-25 05:35Z) was premature — the failure recurred 2 hours later on PR #188 (both fuzz_resolver_terminates and fuzz_fusion_roundtrip hit the canonical sanitizer is incompatible with statically linked libc error), proving the meld-side mitigation needed more than "don't install musl as a target."

PR #189 ships a stronger workflow-level defense:

env:
  CARGO_TERM_COLOR: always
  RUST_BACKTRACE: 1
  RUSTFLAGS: "-C target-feature=-crt-static"

cargo gives RUSTFLAGS env-var strict precedence over any [build].rustflags or [target.<triple>].rustflags in /etc/cargo/config.toml, so this completely overrides whatever a drifted smithy runner image has configured. On clean runners (target feature already off) it's a harmless no-op. On drifted runners (the -5/-6 pool documented in the original report) it force-disables static-crt before cargo-fuzz ever calls into rustc.

Evidence

All four fuzz targets on PR #189 ran on the rust-cpu pool and passed:

• fuzz_parse_component ✓
• fuzz_resolver_terminates ✓
• fuzz_merger_idempotent ✓
• fuzz_fusion_roundtrip ✓

This includes the two targets that failed on PR #188 just hours earlier, demonstrating the mitigation works regardless of which runner the job lands on.

Status

This issue should stay open for the upstream smithy asks (rust-cpu runner image standardization + boot-time sanity check) — those remain the durable fix. But the noise on every fuzz PR landing on a drifted runner is now eliminated from meld's side.