Skip to content

feat(qualification): freeze the independent qualification contract (#655) - #681

Draft
mohanagy wants to merge 5 commits into
nextfrom
roadmap/655-qualification-contract
Draft

feat(qualification): freeze the independent qualification contract (#655)#681
mohanagy wants to merge 5 commits into
nextfrom
roadmap/655-qualification-contract

Conversation

@mohanagy

@mohanagy mohanagy commented Aug 12, 2026

Copy link
Copy Markdown
Owner

Closes #655

Read this first — what this contract gives you today

This contract has never been executed. It currently produces no measured evidence of any kind. Every target is a real external repository pinned at an immutable commit and every task has independent truth, so the corpus can produce evidence about natural code — but nothing has been run against Madar yet. Executing the Tier 1 subset is #661.

Three further limits apply the moment it is executed, and running it fixes none of them:

  1. Regression only, never generalization. The sealed holdout slot is unsatisfied because Madar has one author, so every result must carry sealed holdout unsatisfied; results measure regression only.
  2. Thresholds are pre-registered, not calibrated. Nobody knows how many Tier 1 cells currently pass. That is the correct order — a threshold fitted to observed output would describe current behaviour instead of testing it — but it means the first run is a measurement, not a pass/fail gate.
  3. Tier 1 needs network access to clone the pinned targets. It stays deterministic: the commit SHA and the patch fix the content exactly, and a warm clone cache or local mirror satisfies it without changing any result.

The same callout is the first section of docs/qualification/README.md, and a test asserts it stays there.

Note on an earlier version of this branch

The first commit (6da28ac4) built Tier 1 on two self-authored fixture workspaces and left two natural repositories in Tier 2 at pinned_no_truth. That combination produced no measurable evidence on natural code at all. Commit d49b86cb replaced it: Tier 1 is now natural repositories with independent truth, and there are no fixture proxies anywhere in the corpus.

The qualification gate was verified on one lane out of six

Read this before the CRLF fix below — it reframes the historical evidence, not just the current failure.

Validate qualification contract was gated to if: matrix.os == 'ubuntu-latest' && matrix.node-version == '22'. On every other lane it reported conclusion: skipped. The freeze — SHA-256 over raw bytes, the entire point of the artifact — was therefore verified on one lane out of six. On 765802fe the Windows tests failed on digests that the validator checking those same digests never ran against.

The failure mode is the one worth naming: the step advertised a guarantee broader than the one it enforced, and it would have kept passing indefinitely, because passing was exactly what it was configured to do. Any earlier statement that "the qualification gate passed" meant something narrower than it appeared to.

It is now ungated and runs on all six lanes. Confirmed on run 31580083584: Validate qualification contract = success on both windows-latest lanes, where it previously read skipped.

Cross-platform portability

765802fe was red on both Windows lanes (run 31569725619) while ubuntu and macOS passed. Three failures, all in tests/unit/qualification-contract.test.ts, all caused by the Windows checkout converting LF to CRLF: two prose assertions used LF-specific multiline substrings, and the freeze digest is SHA-256 over raw bytes, which CRLF changes.

A second and more serious defect surfaced while reading those logs. The Validate qualification contract step was gated to ubuntu-latest && Node 22, so on Windows it shows conclusion: skipped — the byte-exact freeze was only ever verified on one of six lanes, and the Windows tests were failing on digests that the validator never checked there. The gate was narrower than the guarantee it advertised. It now runs on every lane; the default mode is pure local file I/O with no network and no spend.

The fix pins the bytes rather than loosening the check. freeze.json is not regenerated and no frozen content changedgit add --renormalize . produces no diff, and freeze.json does not appear in 88c4d526's file list.

  • .gitattributes pins docs/qualification/** to text eol=lf, following the precedent already in that file for the hashed isolation CLAUDE.md, and marks patches/*.patch as -text so seeded-defect patches reach git apply byte-for-byte.
  • Prose assertions read through a normalizing helper, so they test semantic content rather than checkout representation. The freeze digest deliberately does not use it — it reads a raw Buffer, because that guarantee is about bytes. The two paths are separated by name and comment so they are not merged later.
  • The validator normalizes patch text before structural parsing (a stray \r could otherwise be captured into a path), rejects a CRLF patch outright, and on a digest mismatch now reports that the file contains CRLF, that this is a checkout problem, and that regenerating freeze.json is not the fix.
  • Two new tests: .gitattributes must keep the pin, and no frozen file may contain CRLF whatever the checkout did.

Verified locally by simulating the Windows checkout with git clone -c core.autocrlf=true:

Check Result
Simulation is genuinely active README.md checks out CRLF
docs/qualification/** under that clone zero CRLF files
qualify:validate in the simulated checkout pass
38 qualification tests in the simulated checkout pass
Negative test — force CRLF onto a frozen file still fails the digest check and the new CRLF check, diagnostic fires

The negative test is the one that matters: it shows the guarantee was preserved rather than made vacuous, which is the real risk in a fix whose goal is to turn a platform failure green.

Protected CI on the exact head

Run 31580083584 on 88c4d526all six jobs pass:

Lane Result Validate qualification contract
ubuntu-latest, Node 20 pass success
ubuntu-latest, Node 22 pass success
macos-latest, Node 20 pass success
macos-latest, Node 22 pass success
windows-latest, Node 20 pass success (was skipped)
windows-latest, Node 22 pass success (was skipped)

Previous run 31569725619 on 765802fe: both Windows lanes failed, ubuntu and macOS passed.

Audit: which other protected steps are single-lane gated

Requested during review, after the skipped-validator finding. Reporting only — nothing outside docs/qualification/ was changed. Classification is based on reading each script, not on inference from its name.

Gated step Gate Guarantee Assessment
registry:validate ubuntu / 22 Ajv validation of a JSON manifest Platform-independent. Gating is correct.
release:verify ubuntu / 22 Version, changelog, README link checks — single-line includes() on text, no digests, no multiline substrings Platform-independent. Gating is correct.
npm audit --audit-level=high ubuntu / 22 Registry advisory data Platform-independent. Gating is correct.
npm pack --dry-run ubuntu / 20 Package file list Largely platform-independent; npm's files globs do not vary by OS. Low sensitivity.
verify:pack-parity ubuntu / 22 Semantic parity between checkout build and unpacked tarball via the MCP retrieval surface Platform-sensitive and gated. The script explicitly handles Windows paths — archivePath.replaceAll('\\', '/'), sep, a normalizedResponse helper — but it only ever executes on Linux, so none of that Windows-specific handling is exercised anywhere. Retrieval responses carry file paths, which is precisely where separator drift would appear.
Eval regression (generate + threshold enforcement) ubuntu / 22 Runs the real CLI over examples/demo-repo and enforces retrieval thresholds Platform-sensitive and gated. Graph generation walks the filesystem and emits paths — the most separator-sensitive behaviour in the repository — and is never exercised on Windows or macOS.

The two bolded rows are new issue candidates, not changes in this PR: they are outside the qualification namespace and each needs its own evidence and rollback story.

Problem and evidence

At the audited baseline 06b373a447acfce895412ac10eb4e5228c5df0b7 (v0.32.1) there is no contract that lets a roadmap change be judged independently. An evaluation corpus needs two properties, and the existing one has neither.

Independence from Madar output is missing. docs/benchmarks/suite/runtime-proof.json is the current source of per-task expectations. It contains exact expected symbols and paths per repository (sendDocument(), server-only/document/send-document.ts, …), authored alongside the product. methodology.md asserts this file "is not passed into retrieval" and that its obligation checklist "is not written into the answering agent's prompt" — but that isolation is prose. No test, lint rule, or CI check enforces it.

Naturalness is partly missing. Measured against docs/benchmarks/suite/repos.json at the baseline: of eleven rows, five are in-repo proxies keyed by pathts-small (examples/sample-workspace), nestjs-mid and ts-monorepo-large (tests/fixtures/pack-quality/**/workspace), python-service and go-service (fixture directories under the suite). The other six are git-backed and do pin a URL plus an immutable SHA (documenso, formbricks, dub, twenty, cal-diy, novu). The five proxy rows are the ones backing the checked-in deterministic fixture bundles.

And there is no valid outcome evidence at all. All six July 15 2026 public rows are not_measured; four recorded no attributable Madar call. The June 10 rows are real measurements, but the prompts carried proof checklists and retrieval could load expected files from runtime-proof.json — controlled, profile-assisted, not untuned.

Exact scope

A new self-contained namespace docs/qualification/ at contract version 1.0.0, plus one validator, one npm script, one CI step, one vitest file.

Deliverable (issue #655) Artifact
Versioned corpus manifest docs/qualification/corpus.json
Versioned task definitions docs/qualification/tasks.json
Independent truth/rubric files docs/qualification/truth/*.json, docs/qualification/rubrics.json
Environment and run receipt schema docs/qualification/receipt-schema.json + two example receipts
Validity/invalidation rules docs/qualification/validity-rules.md
Hidden-holdout handling policy docs/qualification/holdout-policy.md
Stop/rollback/publication rule docs/qualification/stop-rule.md
Clear evidence-category documentation docs/qualification/evidence-categories.md
Small deterministic Tier 1 subset docs/qualification/tier1.json
Planned Tier 2 repeated-run matrix docs/qualification/tier2-matrix.json

Corpus — natural repositories only

Every target is a real, externally authored, MIT-licensed TypeScript project pinned at an immutable commit. corpus.json#/proxy_targets is empty and documents the conditions under which an entry would be permitted.

Target Repository Commit Supplies
hono honojs/hono 26de73133b8552f56ba72e025ecd82b08900d796 execution-flow, impact-analysis
unstorage unjs/unstorage e6be6135832f350ca16f9a77432e1d4f0aa85ed7 architecture-understanding, implementation-planning
hono-seeded-compose hono + patches/hono-compose-reentrancy-guard.patch same bug/root-cause
hono-seeded-error-disclosure hono + patches/hono-error-message-disclosure.patch same review/security
sealed-holdout-a undisclosed unsatisfied

Seeded defects are injected into real code as patches against the pinned commit, per the issue's "seeded defects" truth source — not recreated inside a synthetic workspace built around the answer. Both patches are one-line changes to genuine framework internals: the middleware re-entrancy guard in src/compose.ts, and the default error handler in src/hono-base.ts.

Each target records a cited_blobs map of path to git blob SHA. That lets the validator check every truth citation offline, and lets --verify-corpus prove the pins against the live repositories.

This supersedes the first pass

The first commit on this branch authored two fixture workspaces (ledger-service, plugin-host). They are deleted. They satisfied independence from Madar output but forfeited naturalness, reproducing the defect this issue exists to cure. Both properties are required and neither substitutes for the other — a point now written into holdout-policy.md.

Explicit non-goals honoured

  • No production retrieval, ranking, context, graph, or reporting code is modified. The diff touches only docs/qualification/, .github/scripts/, .github/workflows/ci.yml, package.json (one script), and one new test.
  • The 480+ public superiority experiment was not run.
  • No ranking was tuned against any target. Madar was never executed against any target at all.
  • No superiority claim is published; evidence-categories.md records that zero product-outcome evidence is held.
  • String-presence scoring is not accepted as truth. Tier 2 uses a different method per category (blinded_rubric, ordered_path_rubric, affected_set_precision_recall, single_root_cause_adjudication, seeded_defect_detection); Tier 1 measures evidence sufficiency, never answer text.

Disclosed coupling — Madar ships a Hono adapter

While wiring the independence guard I found that src/pipeline/spi/framework-hono.ts exists, and src/runtime/retrieve.ts carries Hono-aware query classification (an explicit-Hono token check at line 4449, hono_route and hono_middleware roles at 4854–4857).

This is declared framework support within the TypeScript/Node corridor, not a repository-specific special case, so it is not a #660 contamination finding. But it means a hono result partly measures that adapter. Rather than hide it or drop the target:

  • corpus.json records production_coupling: declared_framework_adapter on hono, with the consequence that results there are not evidence about frameworks with no adapter;
  • every frozen hono prompt deliberately avoids naming the framework (says "this framework", "the application's entry point") so the query classifier is not handed the framework identity — enforced by a test;
  • unstorage records production_coupling: none_found and acts as the uncoupled contrast.

The independence guard was adjusted accordingly: bare target ids are no longer forbidden literals, because a target id may legitimately equal a supported framework name. Forbidden literals are now repository URLs, commit SHAs, task ids, full prompt strings, and a curated per-target symbol list in corpus.json#/forbidden_target_symbols.

Files and symbols changed

  • .github/scripts/validate-qualification-contract.mjs — Ajv draft-07 validation plus target-pinning, patch, blob-manifest, provenance, independence, and freeze-digest checks. --write regenerates freeze.json; --verify-corpus performs the network verification.
  • package.json — adds qualify:validate.
  • .github/workflows/ci.yml — one Validate qualification contract step on the ubuntu/Node 22 leg, beside the existing registry and release-hygiene checks.
  • tests/unit/qualification-contract.test.ts — 34 assertions over the contract's substance.

Tests

tests/unit/qualification-contract.test.ts asserts, among others:

  • every target is natural: true, pins a 40-char SHA and an https URL, and records a license and prepare steps; proxy_targets is empty;
  • every seeded target shares its base target's commit, and its patch is a unified diff touching only recorded paths;
  • the Hono adapter coupling is disclosed and unstorage is none_found;
  • no hono prompt contains the word "hono";
  • every prompt matches its recorded SHA-256, in tasks.json and in the probes;
  • every truth citation resolves to a path in that target's blob manifest;
  • every truth file records an author, a real derivation source, and madar_derived_sources_used: [];
  • an invalid receipt marked aggregatable: true is rejected by the schema;
  • an unmeasured score carrying a value is rejected by the schema;
  • patch_application_failure is an available invalidation reason;
  • costs has exactly indexing, context_build, agent;
  • adoption and broad fallback exploration are non-gating;
  • the stop rule states S1.1–S1.8 and the 0.05 margin;
  • every frozen file matches its digest.

Differential behaviour

None at runtime. No production module is imported, changed, or re-exported. npm pack --dry-run shows zero qualification entries. The only behavioural change is a new CI step that can fail a pull request.

Validation commands and results

Run in the worktree after npm ci:

Command Result
npm ci pass — 88 packages
npm run typecheck pass — no output
npm run build pass
npm run qualify:validate pass — 4 pinned natural targets, 0 proxy targets, 6 tasks, 6 Tier 1 cells, 2 negative-trust probes, 21 frozen files
npm run qualify:validate -- --verify-corpus pass — all four targets verified against the live repositories: commits resolve, all 25 cited blob digests match, both patches apply cleanly
npm run registry:validate pass
npm run release:verify pass
npx vitest run tests/unit/qualification-contract.test.ts pass — 38/38
npx vitest run tests/unit/{package-metadata,release-hygiene,benchmark-docs,benchmark-suite-docs,benchmark-suite,mcp-registry-metadata}.test.ts pass — 67/67
npm pack --dry-run pass — 0 qualification entries

The validator caught two real problems during authoring: a truth file citing a path the plan proposes creating (key renamed to new_path), and the Hono adapter collision described above.

A third was caught in review. corpus.json#/forbidden_target_symbols carries a _note key whose value is a string, so Object.values(...).flat() folded the whole 277-character note into FORBIDDEN_LITERALS as literal #1 of 8. Harmless as written — no production file contains that string — but it would have become a live trap the moment anyone shortened the note to something short or common, at which point the guard starts failing production files over a documentation string. Fixed in 765802fe: _-prefixed keys are skipped, anything that is not an array of identifiers now fails validation loudly, and each remaining key must name a real corpus target. The guard was re-verified end to end by appending SmartRouter to src/runtime/graph-summary.ts and confirming qualify:validate fails with production file src/runtime/graph-summary.ts contains qualification literal "SmartRouter" — so the filter narrowed the scan to the 7 real symbols without disabling it.

Incomplete or failed commands

  • The complete local Vitest suite was not run. The test-gate teammate owns all resource-sensitive full-suite work under [P0] Stabilize the complete Vitest suite and protected CI merge gate #654 and a competing run would corrupt their evidence. Focused tests only.
  • npm run verify:pack-parity was not run. It exercises the packed MCP retrieval surface, which this change does not touch, and it is resource-heavy.
  • The Tier 1 subset has never been executed against Madar. This is a stated design decision, not an omission. Building the executor here would have required running Madar against the targets to check the thresholds, and inspecting that output before freezing would destroy the independence the contract exists to create — the truth files would silently become a description of current behaviour. Executing it is [P0] Add independent Tier 1 graph, retrieval, Pack, and negative-trust evaluation #661.

Compatibility impact

None. No public API, CLI, MCP, artifact format, or package content changes. docs/ is not in the files array. Existing benchmark manifests, receipts, and runtime-proof.json are untouched; evidence-categories.md classifies them (E3/E4/E6) rather than editing them.

Independence, stated plainly

All six truth files were authored on 2026-08-12 by reading the pinned repository sources directly. Madar was never run against any target, and no Madar retrieval output, context pack, implementationGuidance, Madar-selected file list, or Madar-generated validation command was consulted before freezing.

Two consequences, recorded in the artifacts rather than hidden:

  1. Thresholds are pre-registered, not calibrated. Nobody knows how many Tier 1 cells currently pass. This is the correct order, not a shortfall — a threshold fitted to observed output would describe current behaviour instead of testing it. tier1.json#/calibration_status states that the first execution is a measurement and that a failure there is a product finding, not a reason to edit the contract.
  2. The author is not independent of the production-rule author. Madar has one author. Every task carries independent_of_production_rule_author: false, rubrics.json#/blinding/current_status is unsatisfied, and sealed-holdout-a is status: "unsatisfied" with the human action spelled out in holdout-policy.md. Naturalness fixes "was the target shaped around its own answer"; it does not fix "were the rules shaped around a known target". Until a second person seals a holdout, this corpus measures regression, not generalization.

The hidden acceptance test for the bounded-implementation task is likewise status: "unavailable" with its blocking reason, rather than a self-authored stand-in that would look complete and prove nothing.

Risks

  • Tier 1 now requires network access to clone the pinned targets. A warm clone cache or local mirror satisfies this without changing any result, because the commit and patch fix the content exactly — but the PR gate gains a network dependency it did not have when targets were in-repo. This is the cost of naturalness and is stated in tier1.json#/properties/requires_network_reason.
  • Upstream repositories could be deleted or force-pushed. Commits are immutable, but availability is not guaranteed. Mitigation today is --verify-corpus; a durable mirror is the real fix and is not in scope here.
  • The forbidden-literal guard could false-positive if a curated target symbol (SmartRouter, createStorage, …) ever legitimately appears in src/. It passes today; the list is data in corpus.json, so narrowing it is a contract edit rather than a code change.
  • The freeze file is friction by design. Any edit under docs/qualification/ fails CI until npm run qualify:validate -- --write is run and explained.
  • S1.4's adoption target is deliberately unset for Phase 0, using a relative 10-point decrease instead.

Rollback

Delete docs/qualification/ and .github/scripts/validate-qualification-contract.mjs, remove the qualify:validate script and the CI step, and delete tests/unit/qualification-contract.test.ts. Nothing else depends on any of it. No source regeneration, no artifact migration, no cache invalidation.

Verdicts

Ready for review: yes. Every deliverable and acceptance criterion is addressed; the ones that cannot be satisfied credibly in a single-author context are marked unsatisfied in the artifacts themselves rather than filled with placeholders.

Ready for merge: no. Three gates outstanding:

  1. [P0] Stabilize the complete Vitest suite and protected CI merge gate #654 is confirmed open by evidence, not just by the epic rule. test-gate established a partial root cause only: it fixed one real stdio-server polling flake, but the Failed to start forks worker symptom is unresolved, zero clean full-suite runs were achieved, and its PR test: fix flaky stdio auto-refresh polling timeout (partial evidence for #654) #682 says "Refs [P0] Stabilize the complete Vitest suite and protected CI merge gate #654", not "Closes". No roadmap PR merges until the complete suite is reliable, and this PR has deliberately not run it.
  2. Exact-head protected CI must pass, including the new qualify:validate step and the complete suite.
  3. A reviewer should confirm the acceptance-criteria assessment below, in particular the two criteria reported as partially satisfied.

Acceptance criteria assessment

Criterion Status
Repositories, tasks, prompts, package/commit identity, and environments are pinned Met — real repositories at immutable SHAs with licenses, prepare steps, patches, and blob digests, all verified against the live repositories.
Every task has an independent ground-truth owner and scoring rubric Partially met. Named owner, recorded provenance, a rubric per task, and no truth derived from Madar output. The owner is not independent of the production-rule author, recorded as false rather than asserted.
Hidden holdouts are separated from production-rule development Not met, declared. Policy, mechanism, and slot exist; the slot is unsatisfied and the human action is named.
Adoption failure is measured separately from context quality Met — adoption is its own receipt block; both behaviour dimensions are non-gating and may never enter a quality score.
Invalid runs remain not_measured and cannot enter aggregate wins Met and enforced by the schema; tested in both directions.
Indexing, context-building, and agent costs are separate fields Met in the schema. Flagged: v0.32.1 emits neither of the first two — coverage table in validity-rules.md.
Correctness/completeness gates precede token/latency claims Met — validity-rules.md#gate-ordering, tier2-matrix.json#/ordering_rule, stop rule S1.8.
The stop/rollback rule is objective enough to block a PR or release Met — S1.1–S1.8 with explicit thresholds; S1.3 and S1.6 are machine-checkable today.
Raw transcript and receipt retention is defined Met — 24 months, per-artifact path and SHA-256, missing artifacts invalidate the receipt.
The corpus can be executed reproducibly from a clean checkout Partially met. npm ci && npm run qualify:validate verifies the whole contract offline; --verify-corpus reproduces every pin from a clean checkout over the network. Executing Tier 1 against Madar is #661 by design — building that executor here would have meant running Madar against the targets and destroying the independence this contract exists to create.

New issue candidates for the lead

  1. Unenforced retrieval/grader boundary. docs/benchmarks/suite/runtime-proof.json holds per-repository expected symbols and paths. docs/benchmarks/suite/methodology.md claims it is grader-only, never passed into retrieval and never spliced into the answering prompt. Nothing enforces that claim — no test, no lint rule, no CI check — and nothing fails if a future change violates it. The contrast is the sharpest form of the argument: this PR's own qualify:validate mechanically fails CI when a qualification literal reaches src/. Same class of boundary claim, one enforced and one asserted in prose. Bears directly on [P0] Remove qualification-repository tuning from production retrieval and claims #660.
  2. Receipt emitter gap. NativeAgentCompareReport emits no invalidation reason codes, no aggregatable flag, no adoption classification, and no separate indexing or context-build cost accounts.
  3. Sealed holdout owner. Needs a real second person and a schedule, not a code change.
  4. Durable corpus mirror. Tier 1 now depends on honojs/hono and unjs/unstorage remaining reachable. Commits are immutable but availability is not guaranteed, and a deleted or force-pushed upstream would break the PR gate. --verify-corpus detects it; a vendored archive or mirror would remove the dependency. (The natural-code-truth follow-up previously anticipated here is closed by commit d49b86cb rather than deferred.)

🤖 Generated with Claude Code


Integration target

Integration target: next. This PR now targets the prerelease integration branch, not main.

Stable promotion to main is outside the scope of this PR and happens later through a separate reviewed nextmain promotion PR.

#654 remains open and blocks merge of this PR.

Retargeting note: next was synchronized to main at 3371ada8 before the base change. The merge base is unchanged at 06b373a4, and the diff is byte-for-byte the same as when this PR targeted main — no changes were added or removed by the retarget.

)

Add docs/qualification/, a frozen evaluation contract separate from the
product benchmark suite in docs/benchmarks/suite/, so roadmap changes can
be judged without grading Madar against expectations Madar produced.

Contents:
- corpus.json pins two new TypeScript fixture targets by content digest,
  two public repositories by commit SHA, and one sealed-holdout slot that
  is explicitly unsatisfied.
- tasks.json freezes six prompts with SHA-256 hashes, one per required
  task category, each recording who authored its truth and whether Madar
  output was inspected before freezing.
- truth/ holds per-task independent truth: critical facts, ordered
  execution paths, affected/unaffected sets, seeded-defect root causes,
  required uncertainty, and unsupported-claim traps.
- rubrics.json separates the seven scoring dimensions, uses a different
  method per category, and keeps adoption and broad fallback exploration
  out of the quality score.
- receipt-schema.json requires the full experimental identity, splits
  indexing/context-build/agent cost into separate accounts, and makes an
  invalid run structurally unaggregatable.
- validity-rules.md, holdout-policy.md, stop-rule.md, and
  evidence-categories.md freeze the invalidation, holdout, stop/rollback,
  and evidence-labelling rules.
- tier1.json is the deterministic PR-runnable subset; tier2-matrix.json
  freezes the planned repeated-run matrix while it stays unexecuted.

npm run qualify:validate checks the contract from a clean checkout and
fails if any qualification literal reaches src/, if a frozen file changes
without an explicit refreeze, or if a truth file cites a path that does
not exist.

No production retrieval, ranking, context, or reporting code is touched.
@coderabbitai

coderabbitai Bot commented Aug 12, 2026

Copy link
Copy Markdown

Important

Review skipped

Draft detected.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro Plus

Run ID: cc8efbbe-6f87-49c1-8268-bcc1703335f3

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review

Comment @coderabbitai help to get the list of available commands.

…ories (#655)

The first pass built two self-authored TypeScript fixture workspaces as
qualification targets. That satisfied independence from Madar output but
forfeited naturalness, reproducing in a new namespace the defect #655
exists to cure: a corpus of self-made proxies cannot detect production
behaviour drifting toward benchmark-shaped repositories, because the
proxies are shaped by the same hands as the production rules.

Replace both fixtures with real, externally authored, MIT-licensed
TypeScript projects pinned at immutable commits:

- honojs/hono @ 26de73133b8552f56ba72e025ecd82b08900d796
- unjs/unstorage @ e6be6135832f350ca16f9a77432e1d4f0aa85ed7

Seeded defects are now patches applied to the pinned commit rather than
synthetic workspaces authored around their own answer:

- patches/hono-compose-reentrancy-guard.patch weakens the middleware
  re-entrancy guard for the root-cause task.
- patches/hono-error-message-disclosure.patch makes the default error
  handler return the thrown error's stack for the review task.

All six truth files are rewritten from the pinned sources, read directly
and without ever running Madar. Each target records a cited_blobs map of
path to git blob SHA so the validator can check every truth citation
offline; `--verify-corpus` confirms the commits, all 25 blob digests, and
both patches against the live repositories.

corpus.json also discloses that Madar ships a generic Hono adapter
(src/pipeline/spi/framework-hono.ts plus Hono-aware classification in
src/runtime/retrieve.ts). That is declared framework support, not a
repository-specific special case, but it means a hono result partly
measures that adapter. The hono prompts therefore never name the
framework, and unstorage carries production_coupling: none_found as the
uncoupled contrast.

evidence-categories.md separates target naturalness from evidence class
and records two findings: the existing suite corpus is five in-repo
proxies plus six SHA-pinned git rows, and the retrieval/grader isolation
claimed for runtime-proof.json is asserted in prose with no test or CI
check enforcing it.

No production retrieval, ranking, context, or reporting code is touched.
Add a `## Read this first` callout above the corpus description in
docs/qualification/README.md stating plainly that the contract has never
been executed and currently produces no measured evidence of any kind,
followed by the three limits that executing it will not fix: regression
only and never generalization while the sealed holdout is unsatisfied,
thresholds pre-registered rather than calibrated, and Tier 1's network
dependency on cloning the pinned targets.

A reviewer should not have to infer this by cross-referencing tier, kind,
and status across target entries. A test asserts both the wording and
that the callout precedes the corpus description.

Also fix three references left stale by the corpus migration: two dead
task ids in holdout-policy.md and a `pinned_no_truth` status mention in
validity-rules.md that no longer exists in corpus.json.
`forbidden_target_symbols` carries a `_note` key whose value is a string.
`Object.values(...).flat()` folded that 277-character note into
FORBIDDEN_LITERALS as literal #1 of 8. Harmless today because no
production file contains that exact string, but it becomes a live trap
the moment the note is shortened to something short or common, at which
point the independence guard starts failing production files over a
documentation string.

Skip `_`-prefixed keys, and reject anything that is not an array of
identifiers so a future malformed entry fails loudly instead of silently
widening or narrowing the scan. Each remaining key must also name a real
corpus target.

Verified the guard still fires: appending `SmartRouter` to
src/runtime/graph-summary.ts makes `qualify:validate` fail with
`production file src/runtime/graph-summary.ts contains qualification
literal "SmartRouter"`, and the scan covers 7 symbols rather than 8
entries. A test asserts the map's shape so the defect cannot return.

@mohanagy mohanagy left a comment

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review verdict: changes are required. The latest exact-head CI is red for a deterministic cross-platform defect introduced by this PR, not merely because #654 remains open.

Both Windows Node 20 and Node 22 fail in tests/unit/qualification-contract.test.ts for two related reasons:

  1. freeze.json records SHA-256 over raw LF bytes, while the Windows checkout presents CRLF bytes, so the freeze digest assertion mismatches.
  2. Policy-document assertions use LF-specific multiline substrings, while readFileSync() returns CRLF content on the Windows checkout.

Please make the contract's line-ending policy explicit and portable. The strongest approach is:

  • add a .gitattributes rule enforcing LF for docs/qualification/** (and other frozen contract text files as needed), preserving the stated raw-byte freeze semantics;
  • normalize line endings in prose-oriented test assertions so they test content rather than platform checkout representation;
  • regenerate freeze.json only when the intentionally frozen bytes actually change, and document that change;
  • re-run the entire six-job CI matrix and require both Windows lanes to pass.

Ubuntu and macOS successes do not override this compatibility failure. The qualification contract is meant to be a protected, reproducible gate, so cross-platform determinism is part of the feature, not an optional cleanup.

The _note/forbidden-symbol correction at 765802fe is a good class-level fix, and the natural-target/provenance disclosures are directionally strong. Resolve the Windows portability defect before considering this ready for merge.

Both Windows lanes were red on 765802f while ubuntu and macOS passed.
Three assertions in tests/unit/qualification-contract.test.ts failed
because the Windows checkout converts LF to CRLF: two prose assertions
used LF-specific multiline substrings, and the freeze digest is SHA-256
over raw bytes, which CRLF changes.

The freeze is meaningful precisely because it is byte-exact, so the fix
pins the bytes rather than loosening the check. freeze.json is NOT
regenerated and no frozen content changed; `git add --renormalize .`
produces no diff.

- .gitattributes pins `docs/qualification/**` to `text eol=lf`, following
  the existing precedent for the hashed isolation CLAUDE.md, and marks
  `patches/*.patch` as `-text` so seeded-defect patches reach `git apply`
  byte-for-byte in either direction.
- Prose assertions read through a normalizing helper so they test
  semantic content rather than checkout representation. The freeze digest
  deliberately does not use it: it reads a raw Buffer, because that
  guarantee is about bytes.
- The validator normalizes patch text before structural parsing so a
  stray carriage return cannot be captured into a path, rejects a CRLF
  patch outright, and now explains on a digest mismatch that a CRLF file
  is a checkout problem and that regenerating freeze.json is not the fix.
- Two new tests: `.gitattributes` must keep the pin, and no frozen file
  may contain CRLF whatever the checkout did.

Also widen the CI step. `Validate qualification contract` was gated to
ubuntu/Node 22, so it was skipped on Windows entirely and the byte-exact
freeze was only ever verified on one of six lanes — the tests failed
there while the validator that checks the same digests never ran. It now
runs on every lane; the default mode is pure local file I/O with no
network and no spend. A reproducible gate that only reproduces on Linux
is not yet the thing it claims to be.

Verified by simulating the Windows checkout locally: cloning with
core.autocrlf=true converts README.md to CRLF while leaving every file
under docs/qualification as LF, and both the validator and the 38
qualification tests pass in that clone. Forcing CRLF onto a frozen file
there still fails both the digest check and the new CRLF check, so the
guarantee was preserved rather than made vacuous.
@mohanagy
mohanagy changed the base branch from main to next August 12, 2026 16:08
mohanagy added a commit that referenced this pull request Aug 12, 2026
…pin actions, remove workflow_dispatch

Remediates four findings from review of #688 / #687:

- Finding A: removed the workflow_dispatch trigger and its release_tag input.
  publish-next.yml lives only on `next`, but the repository default branch is
  `main`, so a workflow_dispatch trigger here was a dead control surface that
  could never be invoked from the Actions UI. The workflow now triggers only
  on push of an approved prerelease tag; the tag is derived solely from
  github.ref_name. verify-next-release-state.mjs's assertPublishEventAllowed
  no longer accepts workflow_dispatch either.

- Finding B: split the single privileged `publish` job into three:
  validate (contents: read, no environment) builds, tests, and packs the one
  tarball that will ever be published, records its SHA-256/size/identity/
  source commit/tag/lockfile hash/toolchain versions in a receipt, and
  uploads it as one artifact.
  publish (needs: validate, contents: read + id-token: write, environment:
  npm-next) has no checkout, no npm ci, no tests, no build, no repository
  scripts, and no dependency cache -- it downloads and independently
  re-verifies the exact artifact validate produced, then npm publish is its
  last step with nothing after it.
  post_publish (needs: [validate, publish], contents: write, no id-token)
  verifies latest was preserved, runs the clean-install smoke test, and
  creates/edits the GitHub prerelease. It is idempotent so a failure here can
  be rerun without ever republishing an immutable npm version.

- Finding C: pinned every action in publish-next.yml (checkout, setup-node,
  upload-artifact, download-artifact) to full 40-character commit SHAs with
  a readable version comment. ci.yml and release.yml are untouched.

- Finding D: publish ends at the publish command with nothing after it, and
  post_publish's create-or-edit release step makes reruns safe.

qualify:validate stays a presence-detection gate (hard fail when the script
exists, notice when it doesn't) -- this is already deterministic since it is
driven by the checked-out commit's package.json content, not a flag; once
#681 lands the script on `next`, the next tag push automatically takes the
hard-fail branch.

tests/unit/release-pipeline.test.ts: 27 -> 44 tests. Every prior test is
retained (relocated to the job it now covers); new tests assert the trigger
change, the three-job graph and its needs/permissions/environment placement,
that publish has no checkout/ci/tests/build/repo-scripts, that exactly one
live npm publish exists and is publish's final step, exact-tarball
publication, artifact SHA-256 recording/verification, latest preservation,
and that every action uses a 40-character SHA (mutable refs rejected).

Nothing published, tagged, or released. PR #688 stays a draft.

Refs #687, #688.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
mohanagy added a commit that referenced this pull request Aug 12, 2026
…present

Lead review found the qualify:validate gate was only half-deterministic:
running and hard-failing when the script exists covers one direction, but
if the script is ever removed after landing (bad refactor, lost merge, a
dependency bump rewriting package.json) the old presence-only check
silently fell back to the notice path and would publish anyway -- the exact
"permanently optional gate" the requirement forbids.

Add a second, independent signal via new .github/scripts/check-qualification-gate.mjs:
compares whether docs/qualification/ (the qualification contract, landing
with #681 in the same merge as the script) exists against whether
package.json still defines qualify:validate (the contract's validator).

  script present                    -> run: execute for real, hard-fail on
                                        non-zero exit
  script absent, contract present   -> missing: hard fail with a message
                                        naming exactly what's wrong
  script absent, contract absent    -> notice: today's ordinary state

publish-next.yml's qualification step now delegates to this script instead
of inlining the presence check. Both fail-closed directions are unit
tested directly against the script (27 -> 48 tests total in this
remediation round).

Refs #687, #688.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[P0] Freeze the natural qualification corpus, independent rubrics, and stop rule

1 participant