|
| 1 | +# 08 — Toolchain Internals |
| 2 | + |
| 3 | +> How mcpp's toolchain machinery works under the hood, and how to extend it |
| 4 | +> with new toolchains, new architectures, and (eventually) embedded targets. |
| 5 | +> Companion to [03 — Toolchain Management](03-toolchains.md), which covers the |
| 6 | +> user-facing CLI. This document is for contributors and maintainers. |
| 7 | +
|
| 8 | +## 1. The model in one picture |
| 9 | + |
| 10 | +``` |
| 11 | +mcpp.toml [toolchain] / global default / `mcpp toolchain install` |
| 12 | + │ (three entry paths — ONE shared pipeline) |
| 13 | + ▼ |
| 14 | +resolve payload (xim:gcc / xim:llvm / xim:musl-gcc xpkg under the sandbox) |
| 15 | + ▼ |
| 16 | +ensure_post_install_fixup() ← idempotent convergence (marker-gated) |
| 17 | + ▼ |
| 18 | +detect / probe ← triple, sysroot, payload paths (glibc, linux-headers) |
| 19 | + ▼ |
| 20 | +ToolchainLinkModel (single resolver for the C-library axis) |
| 21 | + ├──► flags.cppm (main build compile/link flags) |
| 22 | + ├──► stdmod.cppm (`import std;` BMI precompile) |
| 23 | + ├──► build_program (build.mcpp host compiles) |
| 24 | + └──► cfg regeneration (the human-facing clang++.cfg) |
| 25 | + ▼ |
| 26 | +hermetic link check (`-###` dry-run) ← asserts CRT/loader resolve inside the sandbox |
| 27 | +``` |
| 28 | + |
| 29 | +Two principles run through everything: |
| 30 | + |
| 31 | +1. **Sandbox toolchains are self-contained.** A produced binary's CRT startup |
| 32 | + objects, libc, and dynamic linker come from sandbox payloads — never |
| 33 | + silently from the host. On a machine with no compiler and no |
| 34 | + `/usr/lib/**/Scrt1.o` (fresh WSL2, minimal containers), everything still |
| 35 | + works; on a machine *with* a host toolchain, nothing leaks in. |
| 36 | +2. **Path knowledge has one owner per layer.** What used to be four divergent |
| 37 | + copies of "how to link against the payload glibc" is now one resolver |
| 38 | + (`linkmodel`); what used to be per-entry-path fixup behavior is now one |
| 39 | + pipeline. Divergence between copies is where an entire class of bugs came |
| 40 | + from (issue #195). |
| 41 | + |
| 42 | +## 2. Toolchain resolution |
| 43 | + |
| 44 | +A toolchain spec (`gcc@16.1.0`, `llvm@22.1.8`, `gcc@15.1.0-musl`) maps to an |
| 45 | +xim package (`src/toolchain/registry.cppm`: `parse_toolchain_spec` → |
| 46 | +`to_xim_package`, producing an `XimToolchainPackage` with the xim name, |
| 47 | +version, and frontend candidates). The payload is resolved/auto-installed via |
| 48 | +the xlings backend into the sandbox |
| 49 | +(`$MCPP_HOME/registry/data/xpkgs/xim-x-<name>/<version>/`). |
| 50 | + |
| 51 | +`detect`/`probe` (`src/toolchain/detect.cppm`, `probe.cppm`) then derive: |
| 52 | + |
| 53 | +| Field | How | |
| 54 | +|---|---| |
| 55 | +| `targetTriple` | `<compiler> -dumpmachine` | |
| 56 | +| `sysroot` | `-print-sysroot` (validated: must actually carry libc headers), with a remap fallback for xlings-built GCC whose baked build-time path doesn't exist locally | |
| 57 | +| `payloadPaths` | sibling xpkg discovery: glibc payload (`include/` + `lib64|lib/`) and linux-headers payload — the *payload-first* fine-grained sysroot | |
| 58 | +| runtime dirs | toolchain-private lib dirs for produced binaries' `-L`/`-rpath` | |
| 59 | + |
| 60 | +Note the probe deliberately does **not** mine the clang cfg for `--sysroot` |
| 61 | +anymore: the cfg is an output of this machinery, not an input (§5). |
| 62 | + |
| 63 | +## 3. The link model (`src/toolchain/linkmodel.cppm`) |
| 64 | + |
| 65 | +`ToolchainLinkModel` answers exactly one question — *how do we compile and |
| 66 | +link against this toolchain's C library* — and every consumer derives its |
| 67 | +flags from it: |
| 68 | + |
| 69 | +``` |
| 70 | +CLibMode::PayloadFirst glibc/linux-headers xpkgs found (the normal bundled-LLVM |
| 71 | + and no-usable-sysroot GCC case) |
| 72 | + compile: -isystem (clang) / -idirafter (gcc) payload headers |
| 73 | + link: -B <glibcLib> ← CRT discovery (Scrt1.o/crti.o/crtn.o; |
| 74 | + the driver never consults -L for these) |
| 75 | + -L <glibcLib> [+ -rpath + --dynamic-linker for clang] |
| 76 | +CLibMode::Sysroot a usable --sysroot (GCC include-fixed world, self-contained |
| 77 | + musl sysroots, the macOS SDK) |
| 78 | +CLibMode::None nothing usable — host defaults apply and the hermetic |
| 79 | + check (§6) reports whatever leaks in |
| 80 | +``` |
| 81 | + |
| 82 | +`ClangDriverModel` is the companion for bundled LLVM: mcpp always passes |
| 83 | +`--no-default-config` (bypassing the install-time cfg for reproducibility) |
| 84 | +and re-provides libc++ headers/libs plus |
| 85 | +`-fuse-ld=lld --rtlib=compiler-rt --unwindlib=libunwind` explicitly. |
| 86 | + |
| 87 | +**Loader resolution** is data-driven, never hardcoded: a per-arch triple map |
| 88 | +(x86_64 / aarch64 / riscv64 / loongarch64 / i686, glibc and musl spellings), |
| 89 | +then a `ld-*.so*` glob of the payload as the fallback for arches the map |
| 90 | +doesn't know. A third source — declared metadata persisted by the installer |
| 91 | +(`.xpkg-exports.json`) — was implemented, evaluated, and **removed**: its |
| 92 | +only consumer would have been this resolver, the two sources above already |
| 93 | +cover every real payload (the entire 0.0.83 verification matrix ran green |
| 94 | +without the file ever existing), and a general-purpose package manager |
| 95 | +shouldn't carry a mechanism whose sole reader is one downstream tool. If an |
| 96 | +installed-state metadata DB ever appears, it must be designed with xlings |
| 97 | +itself as its first consumer; mcpp can then re-add a reader. |
| 98 | + |
| 99 | +## 4. The unified post-install fixup pipeline (`src/toolchain/post_install.cppm`) |
| 100 | + |
| 101 | +Sandbox payloads are prebuilt ELF trees. Two kinds of paths baked into them |
| 102 | +are unknowable at packaging time and must be aligned to the *local* sandbox: |
| 103 | +`PT_INTERP`/`RUNPATH` inside binaries, and the loader/rpath lines inside GCC |
| 104 | +specs. `ensure_post_install_fixup(cfg, payloadRoot, pkg)` is the **single |
| 105 | +entry** for that alignment, called from all three entry paths (explicit |
| 106 | +install, default auto-install, manifest auto-install). |
| 107 | + |
| 108 | +> Historical note: before 0.0.83 each path remembered — or forgot — its own |
| 109 | +> subset. The manifest path ran *nothing*, which is how a freshly |
| 110 | +> auto-installed llvm kept a stale, environment-dependent cfg (issue #195), |
| 111 | +> and how gcc once shipped a sandbox that couldn't find `stdlib.h`. "Which |
| 112 | +> command you installed with" must never decide "whether the toolchain |
| 113 | +> works". |
| 114 | +
|
| 115 | +**Trigger semantics — ask every build, act once:** |
| 116 | + |
| 117 | +``` |
| 118 | +every build → ensure() → read <payload>/.mcpp-fixup.json |
| 119 | + marker == {schema, kind, rev, glibcLib}? → return (ms-level) |
| 120 | + mismatch → run the fixup for this kind, write marker |
| 121 | +``` |
| 122 | + |
| 123 | +The marker is a *content-fingerprinted cache*, not an event flag: it encodes |
| 124 | +the fixup revision and the glibc payload it was aligned against. The |
| 125 | +"act" branch therefore fires exactly once per |
| 126 | +`(payload × fixup-rev × glibc-fingerprint)` — first use, plus the two |
| 127 | +re-convergence events that genuinely require rewriting (a fixup-logic |
| 128 | +upgrade via `kFixupRev`, or the glibc payload changing underneath). mcpp |
| 129 | +asks on every build because the events that invalidate a payload (xlings |
| 130 | +swapping glibc, a payload inherited from another home) happen outside |
| 131 | +mcpp's sight — trust-but-verify is the only reliable semantic. |
| 132 | + |
| 133 | +**Per-kind actions:** |
| 134 | + |
| 135 | +| kind | actions | |
| 136 | +|---|---| |
| 137 | +| `gcc` (glibc) | patchelf walk over the gcc payload **and the shared binutils payload** (PT_INTERP → sandbox loader, RUNPATH → glibc+gcc lib dirs); specs rewrite (baked loader/rpath → payload glibc, specs-grammar-aware — `%{...}` conditionals must never be corrupted) | |
| 138 | +| `llvm` | patchelf walk over `lib/` only (runtime `.so` RUNPATH; `bin/` is left alone to preserve xlings-set RUNPATHs); deterministic cfg regeneration (§5) | |
| 139 | +| `musl-gcc` | nothing — self-contained sysroot, static world | |
| 140 | + |
| 141 | +**Safety invariants** (each earned by a real incident): |
| 142 | + |
| 143 | +- **Never patch in place.** patchelf operates on a copy which is then |
| 144 | + atomically `rename()`d in: the payload can contain libraries the *current |
| 145 | + process* (a self-hosted, dynamically linked mcpp) or a concurrent build |
| 146 | + has mmapped, and rewriting a live mapping's backing file corrupts the |
| 147 | + running process (observed: exit-time SIGSEGV in `_dl_fini`). `rename` gives |
| 148 | + new content a fresh inode; live processes keep the old one. |
| 149 | +- **Ownership guard.** Payloads that resolve outside this home's registry |
| 150 | + (symlink-inherited from another `MCPP_HOME`) are never patched — their |
| 151 | + owner already converged them, and patching through the symlink would brick |
| 152 | + the owner's toolchain. |
| 153 | +- Specs rewriting is content-aware (already-aligned specs are skipped). |
| 154 | + Extending the same check to the patchelf walk (compare |
| 155 | + `--print-interpreter`/`--print-rpath` before writing, so an already-aligned |
| 156 | + payload converges with **zero writes**) is a known follow-up. |
| 157 | +- The long-term direction is for the *installer* (xlings) to own all |
| 158 | + writes — at install time and when a payload enters a new home — leaving |
| 159 | + mcpp read-only + verification. The pipeline here is the compatibility |
| 160 | + layer until then, and the self-healing mechanism for drift either way. |
| 161 | + |
| 162 | +## 5. The clang cfg: for humans only |
| 163 | + |
| 164 | +`bin/clang++.cfg` exists so a human running the bundled `clang++` directly |
| 165 | +gets a working, hermetic compiler. mcpp's own builds never read it |
| 166 | +(`--no-default-config` always). The fixup pipeline **regenerates** it |
| 167 | +deterministically from the link model — same payload ⇒ byte-identical cfg on |
| 168 | +every machine and install path — rather than line-patching whatever an |
| 169 | +install produced. On Linux that means CRT discovery (`-B`), payload loader + |
| 170 | +rpath, lld/compiler-rt/libunwind, and bundled libc++ for the C++ drivers; on |
| 171 | +macOS it keeps the historical shape (`--sysroot=<SDK>` + payload libc++ |
| 172 | +headers — the C++ *runtime link* stays with the platform's |
| 173 | +`needs_explicit_libcxx` handling in the main build). |
| 174 | + |
| 175 | +## 6. The hermetic link check (`src/build/hermetic.cppm`) |
| 176 | + |
| 177 | +Before running a build with a sandbox toolchain on Linux, mcpp dry-runs the |
| 178 | +driver with the exact link flags (`-### -x c++ /dev/null`) and asserts every |
| 179 | +CRT object and the *effective* dynamic linker (last occurrence wins) resolve |
| 180 | +under allowed sandbox prefixes. This turns both silent failure modes into |
| 181 | +one actionable diagnostic: bare CRT names that lld can't open (the #195 |
| 182 | +symptom on clean machines) and quiet host-CRT contamination (which made |
| 183 | +green CI a false signal on machines with a host toolchain). The verdict is |
| 184 | +cached per flag-set (`.mcpp-hermetic-ok`); escape hatches: |
| 185 | +`[build] allow_host_libs = true` or `MCPP_ALLOW_HOST_LIBS=1`. System/PATH |
| 186 | +compilers are exempt — using the host world explicitly is the user's choice. |
| 187 | + |
| 188 | +CI keeps this honest with a job that has **no host toolchain at all** |
| 189 | +(`debian:stable-slim`, no gcc, no host `Scrt1.o`) — the only environment |
| 190 | +class that faithfully reproduces the clean-machine failure mode, plus e2e |
| 191 | +`86_llvm_hermetic_link.sh` which re-checks the `-###` resolution on every |
| 192 | +machine. |
| 193 | + |
| 194 | +## 7. Extending the machinery |
| 195 | + |
| 196 | +### 7.1 Adding a new toolchain (new compiler family or distribution) |
| 197 | + |
| 198 | +1. **Index side** (xim-pkgindex): a package with the payload assets and — |
| 199 | + critically — `deps` on whatever C library payload it needs (`xim:glibc`, |
| 200 | + `xim:linux-headers`). Follow the llvm/gcc packaging SOP including the |
| 201 | + admission gate (`verify-toolchain.sh`): completeness + hermetic CRT |
| 202 | + resolution + a real compile/link/run before an asset ships. |
| 203 | +2. **Registry** (`src/toolchain/registry.cppm`): teach |
| 204 | + `parse_toolchain_spec`/`to_xim_package` the spec spelling, xim package |
| 205 | + name, and `frontendCandidates` (which binary is the C++ driver). |
| 206 | +3. **Capabilities** (`src/toolchain/provider.cppm`): stdlib identity, BMI |
| 207 | + traits, and feature switches consumed by `flags.cppm`. |
| 208 | +4. **Fixup kind** (`post_install.cppm`): decide what post-install alignment |
| 209 | + the payload needs — gcc-like (patchelf + specs), llvm-like (lib patchelf + |
| 210 | + cfg), or none (self-contained). Wire it into |
| 211 | + `ensure_post_install_fixup`'s dispatch. |
| 212 | +5. **e2e**: a hermetic-link test in the spirit of |
| 213 | + `86_llvm_hermetic_link.sh`, and coverage in the no-host-toolchain CI job. |
| 214 | + |
| 215 | +### 7.2 Adding a new CPU architecture (Linux) |
| 216 | + |
| 217 | +The machinery is already arch-parameterized; the work is data: |
| 218 | + |
| 219 | +1. add the glibc/musl loader names to the triple map in |
| 220 | + `linkmodel.cppm::loader_filename` (the glob fallback covers you until |
| 221 | + then); |
| 222 | +2. ship payload assets for the arch (glibc, linux-headers, the toolchain |
| 223 | + itself) — the aarch64-linux-musl cross target is the working precedent |
| 224 | + (`[target.aarch64-linux-musl]`, cross frontend resolution via the spec's |
| 225 | + `targetTriple`); |
| 226 | +3. nothing else: `-B`/`-L`/loader emission, the fixup pipeline, and the |
| 227 | + hermetic check are all name-agnostic. |
| 228 | + |
| 229 | +### 7.3 Embedded / bare-metal toolchains (outlook) |
| 230 | + |
| 231 | +The model extends naturally to `arm-none-eabi`-class toolchains because the |
| 232 | +hard parts of the hosted world *disappear* rather than multiply: |
| 233 | + |
| 234 | +- **No dynamic linker**: `loader` stays empty — already legal everywhere |
| 235 | + (renderers omit `--dynamic-linker`; the pack/deploy story is flashing, not |
| 236 | + ELF interp). |
| 237 | +- **No glibc payload**: newlib/picolibc live inside the toolchain's own |
| 238 | + sysroot ⇒ `CLibMode::Sysroot`, the exact mode self-contained musl uses |
| 239 | + today. `is_musl_target`-style self-containment detection generalizes to a |
| 240 | + capability flag ("ships own C library"). |
| 241 | +- **Fixup kind = none or gcc-like** depending on how the payload is built |
| 242 | + (a cross gcc payload still wants PT_INTERP/RUNPATH alignment for the |
| 243 | + *host-run* compiler binaries — that part is identical to today's gcc kind; |
| 244 | + the *target* side needs nothing). |
| 245 | +- **Hermetic check** generalizes: assert crt0/semihosting stubs resolve |
| 246 | + inside the toolchain payload instead of Scrt1.o/loader. |
| 247 | +- What genuinely needs new design: per-target `[target.'cfg(...)']` specs |
| 248 | + for MCU flags (`-mcpu`, `--specs=nosys.specs`), linker-script handling, |
| 249 | + and a run/flash story — build-graph concerns above this document's layer. |
| 250 | + |
| 251 | +### 7.4 Non-ELF platforms |
| 252 | + |
| 253 | +macOS (Mach-O) and Windows (PE) intentionally bypass most of this document: |
| 254 | +macOS resolves its C world from the SDK (`CLibMode::Sysroot`) with its own |
| 255 | +libc++ linkage handling; Windows has no rpath — mcpp deploys runtime DLLs |
| 256 | +next to the produced exe, which is the platform's native equivalent of |
| 257 | +everything §3–§4 does for ELF. |
| 258 | + |
| 259 | +## 8. Source map |
| 260 | + |
| 261 | +| Concern | File | |
| 262 | +|---|---| |
| 263 | +| spec → xim package, frontends | `src/toolchain/registry.cppm` | |
| 264 | +| detect/probe (triple, sysroot, payloads) | `src/toolchain/detect.cppm`, `probe.cppm` | |
| 265 | +| link model + loader resolution | `src/toolchain/linkmodel.cppm` | |
| 266 | +| unified fixup pipeline (patchelf/specs/cfg, marker) | `src/toolchain/post_install.cppm` | |
| 267 | +| install/lifecycle entry | `src/toolchain/lifecycle.cppm`; auto-install entries in `src/build/prepare.cppm` | |
| 268 | +| flag assembly (main build) | `src/build/flags.cppm` | |
| 269 | +| `import std;` precompile | `src/toolchain/stdmod.cppm` | |
| 270 | +| build.mcpp host flags | `src/build/build_program.cppm` | |
| 271 | +| hermetic link check | `src/build/hermetic.cppm` | |
| 272 | +| regression fences | `tests/e2e/86_llvm_hermetic_link.sh`, unit `test_linkmodel.cpp`, `test_post_install.cpp`; the no-host-toolchain CI job in `ci-linux-e2e.yml` | |
| 273 | + |
| 274 | +Design history: `.agents/docs/2026-07-07-hermetic-toolchain-link-model-design.md`. |
0 commit comments