0.0.85: descriptor grammar v2, xpkg parse (strict), scan_overrides + ddi audit, index version floor (#200)

* feat(manifest): Lua long-bracket strings + block comments in xpkg descriptor reader

Descriptor grammar v2 (design: .agents/docs/2026-07-08-index-version-semantics-
and-descriptor-grammar-design.md D1):

- LuaCursor gains long_bracket_level_at / read_long_bracket / at_string_start;
  read_string accepts [[...]] / [=[...]=] with Lua first-newline stripping.
- skip_ws_and_comments / skip_table / read_table_body understand --[[ ]]
  block comments and long strings (content may contain braces).
- strip_lua_comments_and_strings blanks long-string content and block
  comments so the mcpp-segment locator's depth count stays correct.
- Unknown top-level mcpp-segment keys are now RECORDED (Manifest.
  xpkgUnknownKeys) instead of silently dropped — surfaced by the upcoming
  `mcpp xpkg parse`; `schema` becomes a known (informational) key.

Verified: 4 new unit tests (embedded ]], braces in content, decoy
`mcpp = {` inside block comment, CRLF/first-newline, unknown-key capture);
full `mcpp test` green; integration: mcpp-index fmtlib.fmt descriptor
rewritten with [==[ ]==] parses and its member test passes.

* feat(cli): mcpp xpkg parse — single-source-of-truth descriptor validation

`mcpp xpkg parse <file.lua> [--json] [--allow-unknown]` parses an xpkg
descriptor with EXACTLY the resolver's grammar (synthesize_from_xpkg_lua +
canonical identity), so index lint and user builds can never disagree
(design D2). Strict by default: unknown mcpp-segment keys exit 1 — they
would be silently ignored at build time; --allow-unknown downgrades to
warnings. --json emits identity, versions, and verbatim generated_files
contents (the lua5.4 parity oracle for descriptor migrations).

Verified: e2e 93_xpkg_parse (long brackets + block-comment decoy, JSON
round-trip byte-identical, unknown-key strictness both ways, malformed
segment fails); parity vs lua5.4 extraction on fmtlib.fmt = identical.

* refactor(manifest): split into :types / :toml / :xpkg partitions

manifest.cppm mixed three concerns in 2560 lines: the shared data model,
mcpp.toml parsing, and the xpkg .lua mcpp-segment reader. Now a directory
module with one data model and two independent surface grammars:

  src/manifest/manifest.cppm  primary — export import :types/:toml/:xpkg
  src/manifest/types.cppm     Manifest/Target/BuildConfig/errors + shared
                              helpers (module linkage, not exported)
  src/manifest/toml.cppm      parse_string/load/default_template
  src/manifest/xpkg.cppm      LuaCursor, identity, synthesize_from_xpkg_lua

Module name and exported API unchanged — zero importer churn. Full test
suite + e2e 93 green.

* feat(scanner): scan_overrides — author-asserted scan results bypass the text scan

For sha256-pinned third-party module units whose imports sit behind
preprocessor guards (fmt's official src/fmt.cc: #ifdef FMT_IMPORT_STD
import std;), the import set is a CONSTANT — recomputing it per build with
compiler processes is the wrong cost placement, and the M1 text scan must
reject it. Instead the descriptor asserts the scan result as data:

    scan_overrides = {
        ["*/src/fmt.cc"] = { provides = { "fmt" }, imports = { "std" } },
    }

Matched files skip scan_file(); the declared unit enters the graph with
scanOverridden=true (the flag plan-vs-ddi verification keys on). Both
surfaces parse it: xpkg mcpp segment and mcpp.toml [scan_overrides."glob"].
Every glob must match >=1 collected source (unmatched = loud error).
A unit may declare at most one provided module.

Verified: 3 unit tests (xpkg parse, empty-declaration reject, toml parse);
full suite green; end-to-end: mcpp-index fmtlib.fmt rewritten WITHOUT its
3.4KB generated_files copy — upstream fmt.cc verbatim + -DFMT_IMPORT_STD +
override — builds and its member test passes (import fmt; works).

Design: .agents/docs/2026-07-08-scanner-backend-abstraction-design.md §3-pre.

* feat(dyndep): plan-vs-ddi reconciliation — the compiler audits the planner

The .ddi (compiler's own P1689 scan under real flags) is the ground truth
of what phase 4 saw. ninja_backend now embeds the planner's per-TU
assumption on the dyndep edge (--expect-provides / --expect-imports /
--expect-none); `mcpp dyndep --single` compares and fails the edge on
divergence with both sides named. Mandatory for scan_overrides units (an
assertion needs its auditor); MCPP_VERIFY_MODGRAPH=1 at generation time
extends it to every module unit. Zero extra compiler invocations.

Verified: 3 unit tests; e2e negative — an fmt override deliberately
omitting imports={std} fails at DYNDEP with
  planned : provides [fmt] imports [<none>]
  compiler: provides [fmt] imports [std]
while the correct declaration builds green.

Design: .agents/docs/2026-07-08-scanner-backend-abstraction-design.md §3d.

* feat(pm): index.toml version contract — floor check at the index-open choke point

An index tree may carry index.toml ([index] spec / min_mcpp / latest_mcpp).
The contract travels WITH the tree, so ONE check in
read_identity_verified_xpkg_lua — the choke point every transport
converges on (artifact snapshot, git clone, [indices] path, CI-restored
cache) — covers all of them, offline included. own < min_mcpp → loud
E0006 with the upgrade one-liner, once per index per process, then the
resolve fails with the cause already printed. Missing index.toml → no
constraint (back-compat, third-party indices). Malformed versions never
brick a client. Escape hatch: MCPP_INDEX_FLOOR=ignore.

Deviation from design D3 noted: staged-unpack + atomic swap for the
artifact refresh lives in vendored xlings (separate codebase) — the
open-time check is the mcpp-side enforcement; staging is a follow-up
there.

Verified: 3 unit tests (ordering incl. 0.0.90>0.0.85, malformed/absent
tolerance, TOML round-trip); e2e on mcpp-index: min_mcpp=9.9.9 → E0006 +
resolve stop; =0.0.84 → builds; MCPP_INDEX_FLOOR=ignore bypasses.

Design: .agents/docs/2026-07-08-index-version-semantics-and-descriptor-
grammar-design.md D3.

* release: 0.0.85 — descriptor grammar v2, xpkg parse, scan_overrides, ddi audit, index floor

Version bump (mcpp.toml + MCPP_VERSION) and docs: [scan_overrides] section
in 05-mcpp-toml.md. One-time compatibility note for release notes:
mcpp-index will declare min_mcpp = 0.0.85 (index.toml floor) and start
using long-bracket descriptors + scan_overrides; pre-0.0.85 binaries
reading those descriptors fail to parse them — upgrade via install.sh.
From 0.0.85 on, the floor mechanism turns every future evolution into a
graceful keep-old-snapshot + upgrade hint instead of a parse error.

* docs(roadmap): live progress table for the 0.0.85 train

* fix(xpkg): survive the full mcpp-index corpus — 42/42 descriptors parse strict

Dry-running the new lint over every existing mcpp-index descriptor found
three gaps, all fixed:

- list_xpkg_versions only accepted double-quoted version keys; tensorvia's
  ['0.1.1'] spelling made its versions invisible. Accept both quotes.
- linux/macosx/windows per-platform sub-tables are KNOWN schema keys (the
  current platform's body is merged before the key loop); they no longer
  read as unknown on other platforms.
- Form A descriptors (no mcpp segment; build info from the source's own
  mcpp.toml) are a valid form: `xpkg parse` reports form A and passes
  instead of failing on the missing segment.

Corpus check: all 42 pkgs/*.lua in mcpp-index now pass `xpkg parse`
(strict) with zero warnings.

* fix(manifest): partitions -> separate modules + umbrella (GCC 15 cross link)

GCC 15 (the aarch64-linux-musl cross toolchain) drops implicit template
instantiations of module-attached types from partition object files —
undefined refs to std::map<..., ScanOverride>::map() etc. at final link.
Same split, same importer API, but as three plain modules re-exported by
an umbrella mcpp.manifest, the pattern the rest of the codebase already
links with everywhere. Shared parser helpers become exported (separate-
module visibility requires it).

* docs(roadmap): W6 findings — corpus dry-run catches, gcc15 partition workaround

* fix(manifest): GCC 15 cross-link — anchor member instantiations in the types module

GCC 15 (aarch64-linux-musl cross) does not emit implicit std::map/...
member instantiations of module-attached structs in importer objects; the
old single-file mcpp.manifest provided them by accident (its parser code
constructed every struct). A non-inline exported
force_template_instantiations() in mcpp.manifest.types recreates that
guarantee deliberately. Remove when the cross toolchain floor is GCC 16.

Verified locally: mcpp build --target aarch64-linux-musl with the same
xim gcc 15.1.0 toolchain links (static aarch64 ELF produced); native
build + full unit suite green.

* chore: ignore .cache/ (clangd index)

* ci(macos): discover the xlings LLVM version instead of hardcoding 20.1.7

xlings bumped its default llvm to 22.1.8; the hardcoded sandbox link path
and toolchain pin made the job fail at 'Configure dev mcpp sandbox'
deterministically. Discover the installed version from
~/.xlings/data/xpkgs/xim-x-llvm/*/ and thread it via MCPP_LLVM_VER.
(Install step keeps 'xlings install llvm -y || llvm@20.1.7' as fallback.)

* ci(macos): pin xlings llvm@20.1.7 (llvm 22.1.8 libc++ ABI link breakage)

xlings' default llvm moved to 22.1.8, whose libc++ headers reference
std::__1::__hash_memory — undefined at link on macOS (ld64.lld), failing
any gtest-using e2e (78_test_main_combinations). Pin the known-good
20.1.7 first, keep latest as fallback. Real fix (link against the
toolchain's own libc++) belongs to the hermetic-toolchain-link track
(.agents/docs/2026-07-07-hermetic-toolchain-link-model-design.md).
The version-discovery from the previous commit stays (picks whatever is
installed).

Author: Sunrisepeak

.agents/docs/2026-07-08-descriptor-index-evolution-roadmap.md

@@ -0,0 +1,106 @@
+# Descriptor & index evolution — 0.0.85 release train roadmap (cross-repo)
+
+Master plan executing, in one pass, the full necessity-ordered set from the two
+sister designs:
+
+- `2026-07-08-index-version-semantics-and-descriptor-grammar-design.md`
+  (D1 long brackets, D2 single-source lint, D3 index floor)
+- `2026-07-08-scanner-backend-abstraction-design.md`
+  (scan_overrides, L3 plan-vs-ddi reconciliation; p1689 backend deferred)
+
+Scope decision: user base is small — the one-time compatibility break is
+absorbed NOW, all five items ship in **mcpp 0.0.85**, and mcpp-index adopts
+immediately after. The p1689 per-package backend is explicitly OUT of this
+train (seam documented, deferred until demanded).
+
+---
+
+## 0. PROGRESS (live, updated as the train executes)
+
+| Item | Status | Evidence |
+|---|---|---|
+| W1 long brackets | ✅ done (b6c5793) | 4 unit tests; fmt descriptor in `[==[ ]==]` form parses + member test green |
+| W2 xpkg parse | ✅ done (0b2b9b9) | strict IS default (unknown key → exit 1; `--allow-unknown` downgrades); e2e 93; `--json` parity vs lua5.4 = byte-identical |
+| (extra) manifest split | ✅ done (bc24f8e) | :types/:toml/:xpkg partitions, API unchanged, suite green |
+| W3 scan_overrides | ✅ done (ea7ea43) | fmt WITHOUT generated_files builds + tests green end-to-end |
+| W4 ddi reconciliation | ✅ done (9211fd4) | negative test: omitted imports={std} → DYNDEP fails with planned-vs-compiler delta |
+| W5 index floor | ✅ done (94e4c6b) | E0006 + upgrade hint + once-per-index; ignore hatch; 9.9.9/0.0.84 e2e. Deviation: staged-unpack lives in vendored xlings → follow-up there; open-time check is the mcpp-side enforcement |
+| W6 release 0.0.85 | 🔄 PR #200 open, CI running | version bumped; docs/05 scan_overrides section; key e2e green |
+
+Findings during W6 hardening (both fixed in PR #200):
+- **Corpus dry-run of strict lint over all 42 mcpp-index descriptors** caught:
+  single-quoted version keys invisible to list_xpkg_versions (real bug —
+  tensorvia had NO visible versions); platform sub-tables misread as unknown
+  keys; Form A descriptors failing on the missing segment. 42/42 now pass.
+- **GCC 15 partition bug**: module partitions drop implicit template
+  instantiations of module-attached types at the aarch64 cross link; the
+  manifest split became three separate modules + umbrella (same API).
+
+| P1..P3 prepared | see mcpp-index adoption plan progress table | branches feat/index-floor-0.0.85 (2523dd9), feat/long-bracket-migrations (7f1e624) |
+| P0 merge PR #63 | pending maintainer | tested locally 2026-07-08, green on 0.0.81 |
+| P1 index floor PR | pending (after 0.0.85 release) | |
+| P2 fmt → scan_overrides | ✅ validated locally only (per instruction: user's PR untouched) | override descriptor kept out of tree |
+| P3/P4 migrations + docs | pending (after P1) | |
+
+Decision taken during execution (user-directed): `--strict` semantics are the
+DEFAULT of `xpkg parse`; the escape flag is `--allow-unknown`.
+
+## 1. Work breakdown — repo `mcpp` (release 0.0.85)
+
+| # | Item | Design ref | Depends on |
+|---|------|-----------|------------|
+| W1 | **D1 long brackets**: `LuaCursor::read_long_bracket()` (`[[`/`[=[…]=]`, first-newline strip), `read_string()` branch, `strip_lua_comments_and_strings()` long strings + `--[[ ]]` block comments; lua5.4-parity unit fixtures (level ≥ 1, embedded `]]`, CRLF, comment containing `mcpp = {`) | grammar design D1 | — |
+| W2 | **D2 CLI**: `mcpp xpkg parse <file> [--json] [--strict]` wrapping `synthesize_from_xpkg_lua` + identity gate; `--strict` warns on unknown mcpp-segment keys (parser collects skipped keys instead of dropping them silently); e2e good/malformed fixtures | grammar design D2 | W1 (validates new grammar) |
+| W3 | **scan_overrides**: parse in mcpp segment + project `mcpp.toml`; override-matched files bypass the text scan, declared `(provides, imports)` enter the graph; shape validation (glob must match ≥1 source, module names well-formed) | scanner design §3-pre | — |
+| W4 | **L3 reconciliation**: `dyndep.cppm` diffs planned `(provides, requires)` vs compiler `.ddi` per TU; **mandatory for override units**, `MCPP_VERIFY_MODGRAPH=1`-gated for the rest (default-on in a later release once quiet); E-code with file + delta; watch logical-name sanitization parity (`bmi_basename`) | scanner design §3d | W3 (override units are the must-verify set) |
+| W5 | **D3 index floor**: `index.toml` reader; check at the index-open choke point (`package_fetcher` index-dir layer); refresh becomes staged-unpack → floor check → atomic swap (keep-old + warn on incompatible); `MCPP_INDEX_FLOOR=ignore`; E-code + `mcpp explain`; version compare via `version_req.cppm` | grammar design D3 | — |
+| W6 | **Release**: docs 04 (grammar: long brackets, scan_overrides key, unknown-key strictness) + 05 (`[build]`/override surface, `index.toml`); release notes carry the one-time-break notice for pre-0.0.85 clients | — | W1–W5 |
+
+Parallelism: W1 ∥ W3 ∥ W5 are independent tracks; W2 follows W1; W4 follows W3.
+
+Known implementation caveats:
+- **Two refresh paths**: staged+atomic swap applies to the artifact-download
+  path; the xlings-git path can't stage inside mcpp — there the open-time
+  floor check is the guard (acceptable: git path is maintainer/dev-facing).
+- Reconciliation false-positive risk (ddi logical-name formatting vs planned
+  names) is why non-override units start env-gated.
+
+## 2. Work breakdown — repo `mcpp-index` (after 0.0.85 is released + mirrored)
+
+Ordered PRs; details in mcpp-index
+`.agents/docs/2026-07-08-index-side-adoption-plan.md`:
+
+| # | PR | Content | Gate |
+|---|----|---------|------|
+| P0 | (pre-train) merge PR #63 as-is | fmt via `generated_files`, works on 0.0.81 today; don't block it on the train | tested 2026-07-08 |
+| P1 | **floor PR (atomic)** | `index.toml` (`min_mcpp = "0.0.85"`); CI `MCPP_VERSION` 0.0.81 → 0.0.85; lint job gains pinned-mcpp download + `xpkg parse --strict` loop; `publish_mcpp_index.sh` packs `index.toml` (one `cp`) | 0.0.85 released |
+| P2 | fmt → `scan_overrides` | descriptor drops the 3.4 KB `generated_files` copy; upstream `src/fmt.cc` verbatim + `cxxflags -DFMT_IMPORT_STD` + declared `(provides={fmt}, imports={std})` | P1 merged |
+| P3 | long-bracket migrations | `nlohmann.json`, `compat.eigen` `generated_files` → `[==[…]==]`; mechanical, parity-oracle-checked, one PR each | P1 merged |
+| P4 | docs | `repository-and-schema.md`: replace "不支持 `[[…]]`" with "requires mcpp ≥ 0.0.85 (lint enforces)"; document `scan_overrides` as the pattern for guarded-import module units; add floor explanation + case-index rows | P1–P3 |
+
+Invariant enforced mechanically, not by policy: **no descriptor may use new
+grammar/keys before P1** — while lint pins 0.0.81 the `xpkg parse` step doesn't
+exist and old-lint passes would be meaningless; after P1, lint parses with
+0.0.85, so pre-P1 usage simply cannot merge.
+
+## 3. Verification plan
+
+- mcpp unit: lua5.4-parity grammar fixtures; override shape validation.
+- mcpp e2e: fixture index with `min_mcpp = "9.9.9"` → build fails with upgrade
+  message, `MCPP_INDEX_FLOOR=ignore` passes; staged refresh against an
+  incompatible snapshot keeps the old one; override with a deliberately wrong
+  `imports` → reconciliation error names file + delta; `xpkg parse --strict`
+  flags an unknown key.
+- Integration bed: mcpp-index workspace CI (`mcpp test --workspace`, 3 OS) —
+  P2's fmt-via-override member is the live proof; note it is also the first
+  `import_std = true` package build on macOS/Windows in the matrix.
+
+## 4. Compatibility ledger (explicit)
+
+- Pre-0.0.85 clients + post-P2/P3 index: descriptor parse errors with no
+  guidance — **accepted one-time break**, announced in 0.0.85 release notes;
+  floor protects every client ≥ 0.0.85 from all future breaks (staged refresh
+  keeps last compatible snapshot).
+- `format_version` of artifact/pointer: unchanged. Pointer stays dumb.
+- xim/xlings side keeps executing descriptors as real Lua — long brackets and
+  new keys are valid Lua; no impact.