Total parse/edit: never crash on input, errors as data, equivalence gated (closes #39)

[johnsoncodehk]

parse/edit on the handle API are now total: every input produces a tree plus a cst.errors field ([{ offset, end, message }], empty when clean) — never a throw. API misuse (no changes, out-of-range coordinates, foreign/stale handles) still throws; the module-level strict API is untouched, so the whole conformance/parity gate chain runs unchanged.

const cst = p.parse(anything);     // total
p.edit(cst, changes);              // total — error states included
cst.errors                          // what went wrong, current after every edit

Design

• Strict pass first: valid inputs take today's strict path exclusively — byte-identical trees, full PEG arm exploration (gated). Only a strict reject re-parses with recovery enabled; lexer errors flip the same edit into the recovering pass mid-flight.
• Repetition recovery at spine-shaped loops (statement/member lists): a failing element absorbs tokens into an $error row (its leaves keep the text-tiling invariant) up to the element's FIRST set, the enclosing seq's follower literal, or EOF. Expression-internal hooks were measured to cascade (273 errors from one broken identifier) and are deliberately excluded.
• Bar discipline keeps recovery equivalence-safe and arm-blind: recovery fires only where parsing is stuck at a strict-proven fail point (pos ≤ bar ≤ maxPos ≤ bar+2, stateless so losing longest-match arms cannot consume bars); a failure past the bars aborts the attempt, mints the next bar, and the pass re-runs — adoption keeps re-runs cheap, and the runParse safety net obeys the same gate.
• Diagnostics are derived, not collected: $error rows are found by descending the structurally-propagated rowRM spine of the final tree (per-pass candidate lists double-count under stateless re-adoption); lexer diagnostics persist as structured entries formatted at settle time with current offsets, maintained by the window splice.
• Recovered streams broke two strict-era invariants, both fixed: the relex window must anchor below the earliest lexer diagnostic before the damage (a dangling quote pairs with a later edit — backward coupling; forward coupling was already guarded by resync equality), and '>' splits disable adoption for the rest of the parse (the frozen damage mapping is invalid after a mid-parse index shift).

A latent strict-mode hole, found and closed

The last equivalence divergence traced to a watermark contract violation that predates this PR: a Pratt rule's winning row is built before its failed LED extension arms run, so the row's stored lookahead extent under-records the rule's true probes — the memo watermark was always right, but the memo dies with its generation and adoption reads the row. A later edit inside a failed arm's reads could keep a stale row alive (typing the ) that turns a failed call arm into a successful one). Strict sessions never hit it because the triggering texts reject, and the reject was the firewall — total parsing removed the firewall and exposed it. Fix: the rule-level watermark is written back to the row at memo-store time.

Gates (32/32)

• test/recovery.ts (new): valid corpus ≡ strict byte-identical with empty errors; invalid corpus total + deterministic + spans-in-bounds; a char-by-char typing session through transiently-invalid states — every keystroke ≡ a fresh parse, tree and errors.
• test/incremental-verify.ts reworked to total semantics: every seeded step (syntax-breaking included) compares tree + errors against a fresh recovering parse — 128 steps, 0 mismatches.
• test/multi-doc.ts reworked: 60 interleaved steps across two documents including broken text, plus the 9-point handle contract.
• Strict parity 0 mismatches, lexer streams byte-identical, batch in band (11.2×), agnostic 9/9.

Numbers

Valid-state keystrokes are unchanged (9MB median 0.05ms warm). Broken-state keystrokes are incremental too (third follow-up commit): ~3–7ms on 9MB, ~100× over the interim full-re-parse design — see the recovering-adoption section below. Break/fix transitions still cost ~0.4s (the absorbed error region re-parses; recorded follow-up).

Closes #39.

Missing-token synthesis (follow-up commit)

Required token matchers in recovering mode synthesize a zero-width $missing leaf instead of failing, tsc-style: const x = f(1, 2; keeps its Call structure and reports expected ')'; function g() { return 1; closes the body and reports expected '}'. Synthesis is budget-free and position-pure — it fires iff a recovery bar lies in [pos, pos+2], never under probes (not()/optional/separator), never in free-fire — so it is a pure function of (text, bars) and cannot desync the two engines.

Two structural consequences, both gated:

• Zero-width matches now exist (only via synthesis — a strict zero-width element would never terminate its loop). Every loop discards them: plain reps break on pos === before alone, hooked reps discard + resync, leftRec continuations and Pratt LEDs refuse zero-width wraps. Same-position re-entry of a rule through a synthesized leading token is an unbounded recursion no grammar shape rules out — recovering runs keep a (rule, pos) in-progress set and fail the re-entry (PEG cycle semantics, zero strict-path cost). The sentinel also dissolved the bar +1 ladders the recursion crashes were minting: broken-doc recovery in the incremental gate dropped 10.7s → 1.2s.
• The bar protocol's input must be adoption-invariant (bars = strict-fail maxPos; the next bar must be a function of (text, bars), not of what got adopted). Three fixes: frameMax, a frame-local advance watermark, makes recorded probe reaches (rowExt/memo watermarks) exact instead of contaminated by earlier-sibling probes — closing the recorded "exact per-frame extents" item at one extra compare on frontier breaches; recovery runs are adoption-free (attempt loop and the lex-recovered first run — a recovering row's reach is bar-history-dependent, so replaying it poisons the next bar); surgery refuses recovery-made trees (a strict splice into kept $error/$missing siblings was a fake strict success that froze the old text's recovery shape, shifted).

New gate: test/incremental-grammars.ts — all 7 grammars × generative inputs × seeded edit sessions, every step edit ≡ fresh (tree + errors), self-consistency walk, totality: 672/672. test/recovery.ts gains exact-diagnostic synthesis pins; test/incremental-verify.ts gains the five protocol-pin GLUE pairs. Suite 33/33.

Missing-nonterminal synthesis (second follow-up commit)

$missing generalizes from tokens to RULES — the tsc "Expression expected" analog: const a = ; → expected Expr (one diagnostic, right offset), same for operator rhs (a + ;, -;), mixfix arms (x ? y : ;), and list elements after a real separator (a, ;, f(1, ;). Hooks at parseRuleEntry's fail exit + the three Pratt rhs sites that bypass rule entries.

Placement follows commitment semantics: an optional group / repetition element fails freely while uncommitted (its absence is not a diagnostic), but once it consumes a real token, missing pieces synthesize — opt('=', Expr) commits at the real =; rep(seq(',', Expr)) cannot mint a phantom , but synthesizes the Expr after a real one. not() and separator probes stay absolutely suppressed; FIRST-token call-site guards open under recovering so the rule-entry hook is reachable.

The new shapes surfaced two latent bugs, both fixed: the previous commit's frameMax conversion had been double-applied at the 12 token-advance sites by a patch-script composition hole (token consumes never raised maxPos; both engines ran the same wrong bar protocol, so equivalence gates stayed green — a reminder that edit≡fresh cannot see consistently-wrong protocol inputs), and the memo-jump coordinate refresh read toff(tokN) past the token columns for zero-width rows minted at EOF (stale slots under handle reuse → an expected Expr at offset 8 in a 5-char document).

Synthesis quality pins: 3 → 9 exact-match cases in recovery.ts. Suite 33/33; 9MB: fresh 438ms, valid keystroke ~0.6–5ms warm, breaking 649ms, while-broken 438ms, fixing 368ms.

Recovering adoption under bar purity (third follow-up commit)

Typing in a broken 9MB document: ~440ms → ~3–7ms per keystroke. Recovery runs adopt rows from the previous tree again, made sound by two steps:

• Position-pure decisions: recoverArmed now takes the failing element's own frame-local probe reach (staged frameMax) instead of reading the global maxPos — a global frontier parked on a far bar can no longer arm an unrelated loop. With that, every recovery decision (hook arming, token/nonterminal synthesis, the cycle sentinel) is a function of the row's window.
• The bars-window predicate: a row adopts in a recovering run iff the bars inside its window [start, reach+2] are identical (shifted) to the bars the build run saw there (lastBars rides the document register set; strict trees carry [], free-fire trees null). Window text + window bars then determine the frame's behavior completely — including losing-arm fires and synthesis — so $error-containing rows adopt too (the error region is exactly what stays stable across far edits).

The changed fire pattern exposed a latent bug in committed code, now fixed: collectErrRows decoded a $error row's first kid as a token leaf unconditionally, but the runParse leftover net builds a wrapper $error whose kids are nodes — the message quoted text from a garbage offset, and the two text layers resolved it differently (equal trees, different messages — exactly the kind of impurity only the equivalence gate's error-field comparison can catch).

Gates: 672/672 cross-grammar, incremental gate 9.9× vs fresh on mixed sessions, suite 33/33.

The transition cliff, killed (fourth–sixth commits)

The recorded ~0.4s break/fix transitions decomposed into an attempt-re-parse cost and a lexer relex-to-EOF, each closed by a proven mechanism:

• Cross-attempt memo survival: recovery attempts parse the same stream under a monotonically growing bar list, so a memo entry whose window is bar-free behaved strictly (no synthesis, no skip arming) and is a pure function of window text — valid in every later attempt. The exception is the cycle sentinel, which can fire without synthesis and depends on the ancestor stack: recRunning now carries entry serials, and a refusal leaning on a frame entered before the current one taints its memo entry (own-generation-only, taint propagating on reuse). This is precisely the hole that sank the first survival attempt.
• Conditional lexer resync: the windowed relex previously resynced only on exact stack-depth equality, so a paren-balance-changing edit shifted the suffix's depth column and regrew the window to EOF (~130ms on 9MB). Two proven conditions replace it — content-equal stacks (equal depth, neither lex dipped below it since the damage start) in O(1), or a depth-shifted adoption when the old suffix never pops an entry open at the candidate (lazy suffix-min over the depth records), with the adopted depth column re-based by the shift. Closes four latent unsoundness classes of the old equality path along the way (flag-bearing candidates, template brace counters).
• Recovering surgery: trySurgery now accepts recovery-made trees when the damage and the re-parsed span sit clear of every bar window — the splice then provably commutes with every recovery decision (attempt bar lists are prefixes of the final list, so one check covers all attempts). The suffix bars shift with the splice instead of being cleared.

9MB transitions: breaking 335ms → ~6ms, fixing 230ms → ~3ms; valid keystroke ~1ms; while-broken typing at valid-path parity. perf-bench worst 472ms (was 802ms).

Diagnostics: viable sets + paired openers (seventh commit)

• expected ',' or ']' — companion literals are provably still accepted at the failure position (repetitions before the failing literal are always re-enterable; one-shot optionals are crossed but contribute nothing, since they may already have consumed). A static FIRST union would name impossible continuations (after [1, 2 an expression is not viable); tsc under-reports the same position as ')' expected.
• to match this '(' — per-closer structural openers derived by intersecting preceding-literal sets across all seq occurrences ()←(, ]←[, while←do fall out; no bracket list is hardcoded), attached as a related: { offset, end, message } field pointing at the opener leaf.

Measured against tsc and tree-sitter (eighth commit)

test/head-to-head.ts, one 9MB document, identical single-character edit scripts (README has the full table): Monogram beats tsc on every phase (valid keystroke 0.37ms vs 37ms; while-broken 0.21ms vs 13.6ms) and beats or matches tree-sitter everywhere except the two break/fix transition edits — profiling attributes those almost entirely to lexer-layer suffix bookkeeping on the bench's first-touch 4.5MB cursor jump (one-time suffix-min allocation + EOF-relative re-basing); the strict-fail pass measures 0.35ms, the attempts 0.6ms, and repeated transitions at one cursor position settle to ~2ms. test/recovery-conformance.ts scores our diagnostics against tsc's parser on every conformance file it rejects: recall 59.1%, precision 82.4%.

Two syntactic over-accepts, fixed (ninth commit)

Unterminated templates (`tpl ${x;) parsed clean — a template head now commits to its middle/tail chain in both engines; and a bare reserved case slid through the identifier-expression fallback, letting switch (x) { case 1 y(); } parse as three statements. A full accept/reject flip scan over the conformance corpus shows exactly one flip: an intentionally-invalid error test now correctly rejects.

The formal spine + a complete gate (tenth commit)

TOTAL-PARSING.md writes up the contract, the bar-determinism theorem, the three structural theorems, the window-replay theorem with its three corollaries, the lexer-resync soundness conditions, and the one known open caveat (row-level taint). test/exhaustive-edits.ts (CI, suite 34/34) checks edit ≡ fresh completely within a bound: every document ≤ 4 chars over a small bracket grammar × every single-character edit (~330k steps; the 3.2M-step depth-5 run is also clean). It caught a real one-case regression in the surgery length update on its first run.

Ceiling round (eleventh–fourteenth commits)

• Row-level taint closed: rowRM is now bitwise — bit 1 keeps the structural error containment the diagnostics walk descends, bit 2 marks a context-tainted result (a frame whose parse leaned on the cycle sentinel finding an ancestor). Tainted rows refuse recovering adoption and run extension; the one open caveat in TOTAL-PARSING.md is gone.
• Depth-0 shifted resync is O(1): the lexer keeps an ascending list of )-pops that found an empty paren stack (almost always empty); the dominant statement-boundary resync case then needs one end-of-list check instead of an O(suffix) minimum build. Steady break/fix transitions settle at ~2 ms (strict-fail pass 0.23 ms, attempts 0.46 ms; the raw 7-column suffix memmove measures 0.07 ms — no storage floor). The head-to-head's 12 ms transition rows measure a one-time 4.5 MB cursor jump (the EOF-relative bias boundary moves with the cursor — exactly the design that makes valid keystrokes 0.37 ms); the docs now attribute this correctly.
• Statement keywords blocked as expressions, for-in takes comma objects: bare if/for/return/… parsed as identifier expressions, letting namespace if {} fall apart into accepted identifier statements. Blocking for exposed a real masked gap — for (a in b, c) had been parsing as a call of the identifier for; the for-in object now takes a full comma Expression while for-of correctly keeps one AssignmentExpression. Every accept/reject flip across the conformance corpus was individually adjudicated against tsc: 7 flips, all toward tsc, zero away ('var' and 'extends' were caught by this scan leaning on tsc-accepted shapes and stay out). Error-report recall vs tsc's parser: 55.7% → 61.2%; the remaining 108 divergent files are enumerated in the ROADMAP (31 = the [Await]/[Yield] context class, 77 = named per-shape strictness items).