fix(build): macOS test binaries link the toolchain's own libc++ (A1) + unpin llvm (#202)

* fix(build): macOS test binaries link the toolchain's own libc++ (A1 root fix)

TestBinary previously linked the SYSTEM -lc++ while compiling against the
toolchain's libc++ HEADERS — a header/dylib version split that detonated
when libc++ 22 moved string hashing out of line (__hash_memory undefined
against Apple's older dylib; every gtest link on macOS+llvm-22.1.8 failed).

Tests are host-only by definition, so an rpath into the toolchain registry
is acceptable for them in a way it isn't for distributables. ldStdlibTest
becomes: -nostdlib++ -L<llvm>/lib -lc++ -lc++abi -Wl,-rpath,<llvm>/lib
(when the toolchain ships a libc++ dylib; system -lc++ fallback otherwise).
Same-version headers and dylib by construction — future libc++
out-of-lining is self-consistent. Dynamic teardown also dissolves the
static-destruction SIGABRT that originally motivated the system-lib
exception. Distributables keep the static -load_hidden LLVM libc++.

ci-macos returns to latest-first llvm install (the 20.1.7 pin's exit
criterion): the job now PROVES each new llvm default self-consistent.
New e2e 94_macos_test_stdlib (requires: macos): unordered_map<string>
hashing under mcpp test + otool -L asserts no /usr/lib/libc++ reference.

Design: .agents/docs/2026-07-08-root-cause-remediation-design.md A1.

* fix(build): drop explicit -lc++abi from the test stdlib link

libc++.1.dylib reexports its abi; linking libc++abi.dylib explicitly
alongside the system abi (loaded transitively via macOS frameworks)
doubled the __cxa_* runtime state — every test binary aborted before
main (exit 6) on the first macOS CI round.

* fix(build): restore -lc++abi (this llvm's libc++ doesn't reexport abi); add macOS failure forensics

Round 2 proved the abi symbols are NOT reexported by this distribution's
libc++.dylib (undefined __cxa_*/operator new at link). Round 1's
runtime SIGABRT therefore has a different cause — the new forensics step
captures otool -L, LC_RPATHs and a direct binary run on failure so the
next round names it.

* ci(macos): sandbox uses LLVM_ROOT's exact version; forensics captures crash reports

The version-glob pick chose a stale cached 20.1.7 beside the freshly
installed 22.1.8 — rounds 1-3 never actually tested the new llvm. Also
capture the newest .ips crash report after a direct failing run (the
SIGABRT is silent on stderr).

* build(macos): toolchain pin llvm@20.1.7 -> 22.1.8; forensics extracts crash termination

Rounds 1-4 never tested llvm 22: mcpp.toml's macos pin kept resolving
20.1.7 (whose lib/ dylibs silently SIGABRT at load — they were never
exercised in the static-archive era). Pin the current llvm so mcpp's own
macOS build+tests run on it; forensics now parses the .ips crash report
(termination + triggered-thread frames) instead of dumping raw JSON.

* fix(build): link the toolchain libc++/abi dylibs by path (round-5 crash: dual libc++ states)

Crash report named it: exit-time locale::~locale freeing a pointer
malloc never allocated — two libc++ copies in one process, from -l
resolution mixing the sibling libc++.a into the link. By-path dylib
linking removes the ambiguity. Forensics now also counts statically
embedded libc++ text symbols (nm) to prove the binary is archive-free.

* fix(build): test binaries use the same static -load_hidden libc++ as distributables

Forensics rounds 5-6 closed the dynamic route: this llvm distribution's
libc++abi/libunwind dylibs upward-link /usr/lib/libc++, so the system
libc++ ALWAYS loads beside the toolchain's — gtest's static initializers
then construct a stringstream in one copy and destroy it in the other
(BUG_IN_CLIENT_OF_LIBMALLOC_POINTER_BEING_FREED in locale::~locale).
Static hidden archives keep exactly one libc++ inside the binary — the
shape mcpp's own shipped binaries have used green for months. If the
historical gtest exit-teardown SIGABRT resurfaces it will now be caught
by this same CI lane with forensics in place.

* docs(remediation): A1 addendum — forensics-driven revision + xim llvm dylib self-containment item

* test(e2e): 94 asserts 'N passed; 0 failed' — mcpp new's template adds test_smoke (2 passed)

Author: Sunrisepeak

.agents/docs/2026-07-08-root-cause-remediation-design.md

@@ -52,6 +52,33 @@ dylib reference at all).
 `xlings install llvm -y` first); delete the "TestBinary → system -lc++"
 comment block. LLVM 22.1.8 becomes usable on macOS.
 
+### A1 addendum — evidence-driven revision (CI forensics, rounds 1–7)
+
+The dynamic-dylib design above was falsified by crash-report forensics:
+this llvm distribution's libc++abi/libunwind dylibs upward-link
+/usr/lib/libc++, so the SYSTEM libc++ always loads beside the toolchain's
+(.ips image list), and gtest's initializers constructed a stringstream in
+one copy and destroyed it in the other
+(BUG_IN_CLIENT_OF_LIBMALLOC_POINTER_BEING_FREED at locale::~locale).
+
+Process invariant: exactly one ACTIVE libc++ state per process. Options:
+system -lc++ (the original workaround — falsified by __hash_memory);
+toolchain dylib + rpath (falsified above); **static -load_hidden archives
+— adopted**: the already-proven distributable model, now uniform across
+ALL mcpp-built binaries (tests were the only exception, and the exception
+was the landmine); fixing the distribution itself — see below.
+
+**Ecosystem root fix (new item, xim llvm packaging)**: rewrite the
+packaged lib/*.dylib install names/deps at packaging time (@rpath
+inter-references, no /usr/lib/libc++ upward link) so the dylib set is
+self-contained. Owned by the xlings-res/llvm builder; the C.1 consumer
+smoke gate keeps non-self-contained packages out of the index. Until
+then, dynamic toolchain-libc++ linking stays unsupported on macOS.
+
+Also part of the exit criteria: mcpp.toml's own macos toolchain pin moves
+with the current llvm (20.1.7 → 22.1.8) so mcpp builds itself on what
+users get.
+
 ## A2 — GCC 15 module instantiation: raise the cross-toolchain floor (repos `xlings-res` packaging + `mcpp` CI)
 
 **Defect anchor**: `src/manifest/types.cppm` `force_template_instantiations()`

.github/workflows/ci-macos.yml

@@ -58,10 +58,10 @@ jobs:
 
       - name: Install LLVM via xlings
         run: |
-          # Pin 20.1.7: llvm 22.1.8's libc++ headers reference __hash_memory,
-          # absent from the libc++ the link resolves on macOS (ld64.lld
-          # undefined symbol; hermetic-link follow-up tracks the real fix).
-          xlings install llvm@20.1.7 -y || xlings install llvm -y
+          # latest-first: test binaries now link the toolchain's own libc++
+          # (A1 root fix in flags.cppm), so new llvm releases are
+          # self-consistent — this job is the proof for each new default.
+          xlings install llvm -y || xlings install llvm@20.1.7 -y
           # Verify clang++ is available
           LLVM_ROOT=$(find "$HOME/.xlings" -path "*/xpkgs/xim-x-llvm/*/bin/clang++" | head -1 | xargs dirname | xargs dirname)
           echo "LLVM_ROOT=$LLVM_ROOT"
@@ -286,11 +286,10 @@ jobs:
 
       - name: Configure dev mcpp sandbox to reuse xlings LLVM
         run: |
-          # Discover whatever LLVM version `xlings install llvm` provided —
-          # hardcoding it (20.1.7) broke the job the day xlings bumped its
-          # default to 22.1.8.
-          LLVM_PKG=$(ls -d "$HOME/.xlings/data/xpkgs/xim-x-llvm"/*/ 2>/dev/null | head -1)
-          LLVM_PKG="${LLVM_PKG%/}"
+          # Use EXACTLY the LLVM the install step resolved (env LLVM_ROOT) —
+          # a version-glob pick chose a stale cached 20.1.7 next to the
+          # freshly installed 22.1.8 and silently tested the wrong toolchain.
+          LLVM_PKG="$LLVM_ROOT"
           test -d "$LLVM_PKG"
           LLVM_VER=$(basename "$LLVM_PKG")
           echo "MCPP_LLVM_VER=$LLVM_VER" >> "$GITHUB_ENV"
@@ -339,6 +338,42 @@ jobs:
           "$MCPP" self config --mirror GLOBAL
           "$MCPP" test
 
+      - name: Forensics — test-binary link + load state (on failure)
+        if: failure()
+        run: |
+          BIN=$(find target -path "*/bin/test_manifest" | head -1)
+          echo "binary: $BIN"
+          [ -n "$BIN" ] || exit 0
+          echo "--- otool -L ---";  otool -L "$BIN" || true
+          echo "--- rpaths ---";    otool -l "$BIN" | grep -A2 LC_RPATH || true
+          echo "--- statically embedded libc++? ---"
+          nm "$BIN" 2>/dev/null | grep -cE "T __ZNSt3__1" || echo "0 (good: no libc++ code in binary)"
+          echo "--- direct run ---"
+          set +e
+          "$BIN" > run.out 2>&1
+          echo "exit=$?"
+          head -20 run.out
+          sleep 5
+          echo "--- newest crash report (termination) ---"
+          CR=$(ls -t "$HOME/Library/Logs/DiagnosticReports"/*.ips 2>/dev/null | head -1)
+          if [ -n "$CR" ]; then
+            python3 - "$CR" <<'PY'
+          import json, sys
+          lines = open(sys.argv[1]).read().splitlines()
+          meta = json.loads(lines[0]); body = json.loads("\n".join(lines[1:]))
+          print("proc:", meta.get("app_name"), "| exc:", body.get("exception", {}))
+          print("termination:", body.get("termination", {}))
+          t = [th for th in body.get("threads", []) if th.get("triggered")]
+          for fr in (t[0].get("frames", [])[:12] if t else []):
+              print("  ", fr.get("imageIndex"), fr.get("symbol", fr.get("imageOffset")))
+          imgs = body.get("usedImages", [])
+          for i, im in enumerate(imgs[:12]):
+              print("img", i, im.get("path"))
+          PY
+          else
+            echo "none"
+          fi
+
       - name: E2E suite
         # See ci-linux.yml — fail-fast on hung tests instead of burning the
         # whole job budget. Per-test 600s timeout lives in run_all.sh.

mcpp.toml

@@ -18,7 +18,7 @@ include_dirs = ["src/libs/json"]
 
 [toolchain]
 default = "gcc@16.1.0"
-macos   = "llvm@20.1.7"
+macos   = "llvm@22.1.8"
 windows = "llvm@20.1.7"
 
 # Per-target overrides: `mcpp build --target x86_64-linux-musl` (or the

src/build/flags.cppm

@@ -35,10 +35,11 @@ struct CompileFlags {
     std::string linkage;  // "static" or ""
     // macOS per-unit C++ stdlib link (appended via unit_ldflags):
     // distributable targets get the static LLVM libc++ (portable across
-    // macOS versions), TestBinary targets get the system -lc++ — they
-    // only ever run on the build host, and statically linked libc++
-    // SIGABRTs during static destruction unless the entry point guards
-    // with _Exit (mcpp/xlings do; gtest main does not). Empty on other
+    // macOS versions); TestBinary targets get the toolchain's own libc++
+    // DYNAMICALLY (-L + -lc++ + rpath into the toolchain) — host-only
+    // binaries, so the rpath is fine, and it keeps headers and dylib the
+    // same version (the system -lc++ they used before was a version split
+    // that broke on libc++ 22's out-of-line __hash_memory). Empty on other
     // platforms (stdlib handled by their existing paths).
     std::string ldStdlibDefault;
     std::string ldStdlibTest;
@@ -385,6 +386,30 @@ CompileFlags compute_flags(const BuildPlan& plan) {
                   + " -Wl,-load_hidden," + escape_path(libcxxAbiA);
             }
         }
+        // TestBinary: SAME static -load_hidden libc++ as distributables.
+        // Tests previously took the SYSTEM -lc++ while compiling against the
+        // toolchain's libc++ HEADERS — a header/dylib version split that
+        // detonated when libc++ 22 moved string hashing out of line
+        // (undefined __hash_memory, 2026-07-08). The dynamic alternative
+        // (toolchain libc++.dylib + rpath) is a dead end with this
+        // distribution: its abi/unwind dylibs upward-link /usr/lib/libc++,
+        // so the SYSTEM libc++ still loads next to the toolchain's and
+        // gtest's initializers freed across the two copies
+        // (BUG_IN_CLIENT_OF_LIBMALLOC, CI crash forensics rounds 5-6).
+        // Static hidden archives keep exactly ONE libc++, inside the
+        // binary — the same already-proven shape mcpp/xlings ship with.
+        // Design: .agents/docs/2026-07-08-root-cause-remediation-design.md A1.
+        if (!llvmRootForStdlib.empty()) {
+            auto libDir     = llvmRootForStdlib / "lib";
+            auto libcxxA    = libDir / "libc++.a";
+            auto libcxxAbiA = libDir / "libc++abi.a";
+            if (std::filesystem::exists(libcxxA)
+                && std::filesystem::exists(libcxxAbiA)) {
+                f.ldStdlibTest = " -nostdlib++"
+                    " -Wl,-load_hidden," + escape_path(libcxxA)
+                  + " -Wl,-load_hidden," + escape_path(libcxxAbiA);
+            }
+        }
         std::string version_min;
         if (!macosDeploymentTarget.empty()) {
             version_min = " -mmacosx-version-min=" + macosDeploymentTarget;

tests/e2e/94_macos_test_stdlib.sh

@@ -0,0 +1,49 @@
+#!/usr/bin/env bash
+# requires: macos
+# 94_macos_test_stdlib.sh — A1 regression: test binaries must link the
+# TOOLCHAIN's libc++, not the system one. The old system -lc++ exception
+# split headers (toolchain libc++) from the dylib (Apple's older libc++)
+# and broke on libc++ 22's out-of-line __hash_memory — exercised here via
+# std::unordered_map<std::string,...> string hashing.
+# Design: .agents/docs/2026-07-08-root-cause-remediation-design.md A1.
+set -e
+
+TMP=$(mktemp -d)
+trap "rm -rf $TMP" EXIT
+cd "$TMP"
+
+"$MCPP" new hashapp > /dev/null
+cd hashapp
+
+mkdir -p tests
+cat > tests/hash_test.cpp <<'EOF'
+// Forces the string-hash path (__hash_memory-class symbols on libc++ >= 22).
+import std;
+
+int main() {
+    std::unordered_map<std::string, int> m;
+    for (int i = 0; i < 100; ++i) m["key" + std::to_string(i)] = i;
+    return (m.size() == 100 && m.at("key42") == 42) ? 0 : 1;
+}
+EOF
+
+"$MCPP" test > test.log 2>&1 || { cat test.log; echo "FAIL: mcpp test failed"; exit 1; }
+grep -Eq "[0-9]+ passed; 0 failed" test.log   # template ships its own test_smoke too
+
+# The link assertion only applies to clang/llvm toolchains (gcc on macOS is
+# not a supported config; if resolution picked something else, the behavior
+# test above already covered the regression).
+TEST_BIN=$(find target -path "*/bin/hash_test" | head -1)
+test -x "$TEST_BIN"
+if grep -q "Resolved llvm@" test.log || otool -L "$TEST_BIN" | grep -q "libc++"; then
+    echo "otool -L:"
+    otool -L "$TEST_BIN"
+    if otool -L "$TEST_BIN" | grep -q "/usr/lib/libc++"; then
+        echo "FAIL: test binary links the SYSTEM libc++ (header/dylib split)"
+        exit 1
+    fi
+    # Either the toolchain's dylib (rpath link) or no libc++ dylib at all
+    # (static fallback) is acceptable — never Apple's.
+fi
+
+echo "PASS 94_macos_test_stdlib"

tests/e2e/run_all.sh

@@ -58,7 +58,7 @@ case "$OS" in
         fi
         ;;
     Darwin)
-        CAPS+=(unix-shell fresh-sandbox)
+        CAPS+=(unix-shell fresh-sandbox macos)
         # macOS g++ is Apple Clang, not real GCC — don't add gcc capability.
         # Tests requiring gcc need actual GNU GCC (modules, gcm.cache, etc.)
         ;;