feat(qualification): freeze the independent qualification contract (#655) - #681
feat(qualification): freeze the independent qualification contract (#655)#681mohanagy wants to merge 5 commits into
Conversation
) Add docs/qualification/, a frozen evaluation contract separate from the product benchmark suite in docs/benchmarks/suite/, so roadmap changes can be judged without grading Madar against expectations Madar produced. Contents: - corpus.json pins two new TypeScript fixture targets by content digest, two public repositories by commit SHA, and one sealed-holdout slot that is explicitly unsatisfied. - tasks.json freezes six prompts with SHA-256 hashes, one per required task category, each recording who authored its truth and whether Madar output was inspected before freezing. - truth/ holds per-task independent truth: critical facts, ordered execution paths, affected/unaffected sets, seeded-defect root causes, required uncertainty, and unsupported-claim traps. - rubrics.json separates the seven scoring dimensions, uses a different method per category, and keeps adoption and broad fallback exploration out of the quality score. - receipt-schema.json requires the full experimental identity, splits indexing/context-build/agent cost into separate accounts, and makes an invalid run structurally unaggregatable. - validity-rules.md, holdout-policy.md, stop-rule.md, and evidence-categories.md freeze the invalidation, holdout, stop/rollback, and evidence-labelling rules. - tier1.json is the deterministic PR-runnable subset; tier2-matrix.json freezes the planned repeated-run matrix while it stays unexecuted. npm run qualify:validate checks the contract from a clean checkout and fails if any qualification literal reaches src/, if a frozen file changes without an explicit refreeze, or if a truth file cites a path that does not exist. No production retrieval, ranking, context, or reporting code is touched.
|
Important Review skippedDraft detected. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Pro Plus Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
Comment |
…ories (#655) The first pass built two self-authored TypeScript fixture workspaces as qualification targets. That satisfied independence from Madar output but forfeited naturalness, reproducing in a new namespace the defect #655 exists to cure: a corpus of self-made proxies cannot detect production behaviour drifting toward benchmark-shaped repositories, because the proxies are shaped by the same hands as the production rules. Replace both fixtures with real, externally authored, MIT-licensed TypeScript projects pinned at immutable commits: - honojs/hono @ 26de73133b8552f56ba72e025ecd82b08900d796 - unjs/unstorage @ e6be6135832f350ca16f9a77432e1d4f0aa85ed7 Seeded defects are now patches applied to the pinned commit rather than synthetic workspaces authored around their own answer: - patches/hono-compose-reentrancy-guard.patch weakens the middleware re-entrancy guard for the root-cause task. - patches/hono-error-message-disclosure.patch makes the default error handler return the thrown error's stack for the review task. All six truth files are rewritten from the pinned sources, read directly and without ever running Madar. Each target records a cited_blobs map of path to git blob SHA so the validator can check every truth citation offline; `--verify-corpus` confirms the commits, all 25 blob digests, and both patches against the live repositories. corpus.json also discloses that Madar ships a generic Hono adapter (src/pipeline/spi/framework-hono.ts plus Hono-aware classification in src/runtime/retrieve.ts). That is declared framework support, not a repository-specific special case, but it means a hono result partly measures that adapter. The hono prompts therefore never name the framework, and unstorage carries production_coupling: none_found as the uncoupled contrast. evidence-categories.md separates target naturalness from evidence class and records two findings: the existing suite corpus is five in-repo proxies plus six SHA-pinned git rows, and the retrieval/grader isolation claimed for runtime-proof.json is asserted in prose with no test or CI check enforcing it. No production retrieval, ranking, context, or reporting code is touched.
Add a `## Read this first` callout above the corpus description in docs/qualification/README.md stating plainly that the contract has never been executed and currently produces no measured evidence of any kind, followed by the three limits that executing it will not fix: regression only and never generalization while the sealed holdout is unsatisfied, thresholds pre-registered rather than calibrated, and Tier 1's network dependency on cloning the pinned targets. A reviewer should not have to infer this by cross-referencing tier, kind, and status across target entries. A test asserts both the wording and that the callout precedes the corpus description. Also fix three references left stale by the corpus migration: two dead task ids in holdout-policy.md and a `pinned_no_truth` status mention in validity-rules.md that no longer exists in corpus.json.
`forbidden_target_symbols` carries a `_note` key whose value is a string. `Object.values(...).flat()` folded that 277-character note into FORBIDDEN_LITERALS as literal #1 of 8. Harmless today because no production file contains that exact string, but it becomes a live trap the moment the note is shortened to something short or common, at which point the independence guard starts failing production files over a documentation string. Skip `_`-prefixed keys, and reject anything that is not an array of identifiers so a future malformed entry fails loudly instead of silently widening or narrowing the scan. Each remaining key must also name a real corpus target. Verified the guard still fires: appending `SmartRouter` to src/runtime/graph-summary.ts makes `qualify:validate` fail with `production file src/runtime/graph-summary.ts contains qualification literal "SmartRouter"`, and the scan covers 7 symbols rather than 8 entries. A test asserts the map's shape so the defect cannot return.
mohanagy
left a comment
There was a problem hiding this comment.
Review verdict: changes are required. The latest exact-head CI is red for a deterministic cross-platform defect introduced by this PR, not merely because #654 remains open.
Both Windows Node 20 and Node 22 fail in tests/unit/qualification-contract.test.ts for two related reasons:
freeze.jsonrecords SHA-256 over raw LF bytes, while the Windows checkout presents CRLF bytes, so the freeze digest assertion mismatches.- Policy-document assertions use LF-specific multiline substrings, while
readFileSync()returns CRLF content on the Windows checkout.
Please make the contract's line-ending policy explicit and portable. The strongest approach is:
- add a
.gitattributesrule enforcing LF fordocs/qualification/**(and other frozen contract text files as needed), preserving the stated raw-byte freeze semantics; - normalize line endings in prose-oriented test assertions so they test content rather than platform checkout representation;
- regenerate
freeze.jsononly when the intentionally frozen bytes actually change, and document that change; - re-run the entire six-job CI matrix and require both Windows lanes to pass.
Ubuntu and macOS successes do not override this compatibility failure. The qualification contract is meant to be a protected, reproducible gate, so cross-platform determinism is part of the feature, not an optional cleanup.
The _note/forbidden-symbol correction at 765802fe is a good class-level fix, and the natural-target/provenance disclosures are directionally strong. Resolve the Windows portability defect before considering this ready for merge.
Both Windows lanes were red on 765802f while ubuntu and macOS passed. Three assertions in tests/unit/qualification-contract.test.ts failed because the Windows checkout converts LF to CRLF: two prose assertions used LF-specific multiline substrings, and the freeze digest is SHA-256 over raw bytes, which CRLF changes. The freeze is meaningful precisely because it is byte-exact, so the fix pins the bytes rather than loosening the check. freeze.json is NOT regenerated and no frozen content changed; `git add --renormalize .` produces no diff. - .gitattributes pins `docs/qualification/**` to `text eol=lf`, following the existing precedent for the hashed isolation CLAUDE.md, and marks `patches/*.patch` as `-text` so seeded-defect patches reach `git apply` byte-for-byte in either direction. - Prose assertions read through a normalizing helper so they test semantic content rather than checkout representation. The freeze digest deliberately does not use it: it reads a raw Buffer, because that guarantee is about bytes. - The validator normalizes patch text before structural parsing so a stray carriage return cannot be captured into a path, rejects a CRLF patch outright, and now explains on a digest mismatch that a CRLF file is a checkout problem and that regenerating freeze.json is not the fix. - Two new tests: `.gitattributes` must keep the pin, and no frozen file may contain CRLF whatever the checkout did. Also widen the CI step. `Validate qualification contract` was gated to ubuntu/Node 22, so it was skipped on Windows entirely and the byte-exact freeze was only ever verified on one of six lanes — the tests failed there while the validator that checks the same digests never ran. It now runs on every lane; the default mode is pure local file I/O with no network and no spend. A reproducible gate that only reproduces on Linux is not yet the thing it claims to be. Verified by simulating the Windows checkout locally: cloning with core.autocrlf=true converts README.md to CRLF while leaving every file under docs/qualification as LF, and both the validator and the 38 qualification tests pass in that clone. Forcing CRLF onto a frozen file there still fails both the digest check and the new CRLF check, so the guarantee was preserved rather than made vacuous.
…pin actions, remove workflow_dispatch Remediates four findings from review of #688 / #687: - Finding A: removed the workflow_dispatch trigger and its release_tag input. publish-next.yml lives only on `next`, but the repository default branch is `main`, so a workflow_dispatch trigger here was a dead control surface that could never be invoked from the Actions UI. The workflow now triggers only on push of an approved prerelease tag; the tag is derived solely from github.ref_name. verify-next-release-state.mjs's assertPublishEventAllowed no longer accepts workflow_dispatch either. - Finding B: split the single privileged `publish` job into three: validate (contents: read, no environment) builds, tests, and packs the one tarball that will ever be published, records its SHA-256/size/identity/ source commit/tag/lockfile hash/toolchain versions in a receipt, and uploads it as one artifact. publish (needs: validate, contents: read + id-token: write, environment: npm-next) has no checkout, no npm ci, no tests, no build, no repository scripts, and no dependency cache -- it downloads and independently re-verifies the exact artifact validate produced, then npm publish is its last step with nothing after it. post_publish (needs: [validate, publish], contents: write, no id-token) verifies latest was preserved, runs the clean-install smoke test, and creates/edits the GitHub prerelease. It is idempotent so a failure here can be rerun without ever republishing an immutable npm version. - Finding C: pinned every action in publish-next.yml (checkout, setup-node, upload-artifact, download-artifact) to full 40-character commit SHAs with a readable version comment. ci.yml and release.yml are untouched. - Finding D: publish ends at the publish command with nothing after it, and post_publish's create-or-edit release step makes reruns safe. qualify:validate stays a presence-detection gate (hard fail when the script exists, notice when it doesn't) -- this is already deterministic since it is driven by the checked-out commit's package.json content, not a flag; once #681 lands the script on `next`, the next tag push automatically takes the hard-fail branch. tests/unit/release-pipeline.test.ts: 27 -> 44 tests. Every prior test is retained (relocated to the job it now covers); new tests assert the trigger change, the three-job graph and its needs/permissions/environment placement, that publish has no checkout/ci/tests/build/repo-scripts, that exactly one live npm publish exists and is publish's final step, exact-tarball publication, artifact SHA-256 recording/verification, latest preservation, and that every action uses a 40-character SHA (mutable refs rejected). Nothing published, tagged, or released. PR #688 stays a draft. Refs #687, #688. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…present Lead review found the qualify:validate gate was only half-deterministic: running and hard-failing when the script exists covers one direction, but if the script is ever removed after landing (bad refactor, lost merge, a dependency bump rewriting package.json) the old presence-only check silently fell back to the notice path and would publish anyway -- the exact "permanently optional gate" the requirement forbids. Add a second, independent signal via new .github/scripts/check-qualification-gate.mjs: compares whether docs/qualification/ (the qualification contract, landing with #681 in the same merge as the script) exists against whether package.json still defines qualify:validate (the contract's validator). script present -> run: execute for real, hard-fail on non-zero exit script absent, contract present -> missing: hard fail with a message naming exactly what's wrong script absent, contract absent -> notice: today's ordinary state publish-next.yml's qualification step now delegates to this script instead of inlining the presence check. Both fail-closed directions are unit tested directly against the script (27 -> 48 tests total in this remediation round). Refs #687, #688. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Closes #655
Read this first — what this contract gives you today
Three further limits apply the moment it is executed, and running it fixes none of them:
unsatisfiedbecause Madar has one author, so every result must carrysealed holdout unsatisfied; results measure regression only.The same callout is the first section of
docs/qualification/README.md, and a test asserts it stays there.Note on an earlier version of this branch
The first commit (
6da28ac4) built Tier 1 on two self-authored fixture workspaces and left two natural repositories in Tier 2 atpinned_no_truth. That combination produced no measurable evidence on natural code at all. Commitd49b86cbreplaced it: Tier 1 is now natural repositories with independent truth, and there are no fixture proxies anywhere in the corpus.The qualification gate was verified on one lane out of six
Read this before the CRLF fix below — it reframes the historical evidence, not just the current failure.
Validate qualification contractwas gated toif: matrix.os == 'ubuntu-latest' && matrix.node-version == '22'. On every other lane it reportedconclusion: skipped. The freeze — SHA-256 over raw bytes, the entire point of the artifact — was therefore verified on one lane out of six. On765802fethe Windows tests failed on digests that the validator checking those same digests never ran against.The failure mode is the one worth naming: the step advertised a guarantee broader than the one it enforced, and it would have kept passing indefinitely, because passing was exactly what it was configured to do. Any earlier statement that "the qualification gate passed" meant something narrower than it appeared to.
It is now ungated and runs on all six lanes. Confirmed on run
31580083584:Validate qualification contract=successon bothwindows-latestlanes, where it previously readskipped.Cross-platform portability
765802fewas red on both Windows lanes (run31569725619) while ubuntu and macOS passed. Three failures, all intests/unit/qualification-contract.test.ts, all caused by the Windows checkout converting LF to CRLF: two prose assertions used LF-specific multiline substrings, and the freeze digest is SHA-256 over raw bytes, which CRLF changes.A second and more serious defect surfaced while reading those logs. The
Validate qualification contractstep was gated toubuntu-latest && Node 22, so on Windows it showsconclusion: skipped— the byte-exact freeze was only ever verified on one of six lanes, and the Windows tests were failing on digests that the validator never checked there. The gate was narrower than the guarantee it advertised. It now runs on every lane; the default mode is pure local file I/O with no network and no spend.The fix pins the bytes rather than loosening the check.
freeze.jsonis not regenerated and no frozen content changed —git add --renormalize .produces no diff, andfreeze.jsondoes not appear in88c4d526's file list..gitattributespinsdocs/qualification/**totext eol=lf, following the precedent already in that file for the hashed isolationCLAUDE.md, and markspatches/*.patchas-textso seeded-defect patches reachgit applybyte-for-byte.Buffer, because that guarantee is about bytes. The two paths are separated by name and comment so they are not merged later.\rcould otherwise be captured into a path), rejects a CRLF patch outright, and on a digest mismatch now reports that the file contains CRLF, that this is a checkout problem, and that regeneratingfreeze.jsonis not the fix..gitattributesmust keep the pin, and no frozen file may contain CRLF whatever the checkout did.Verified locally by simulating the Windows checkout with
git clone -c core.autocrlf=true:README.mdchecks out CRLFdocs/qualification/**under that clonequalify:validatein the simulated checkoutThe negative test is the one that matters: it shows the guarantee was preserved rather than made vacuous, which is the real risk in a fix whose goal is to turn a platform failure green.
Protected CI on the exact head
Run
31580083584on88c4d526— all six jobs pass:Validate qualification contractskipped)skipped)Previous run
31569725619on765802fe: both Windows lanes failed, ubuntu and macOS passed.Audit: which other protected steps are single-lane gated
Requested during review, after the skipped-validator finding. Reporting only — nothing outside
docs/qualification/was changed. Classification is based on reading each script, not on inference from its name.registry:validaterelease:verifyincludes()on text, no digests, no multiline substringsnpm audit --audit-level=highnpm pack --dry-runfilesglobs do not vary by OS. Low sensitivity.verify:pack-parityarchivePath.replaceAll('\\', '/'),sep, anormalizedResponsehelper — but it only ever executes on Linux, so none of that Windows-specific handling is exercised anywhere. Retrieval responses carry file paths, which is precisely where separator drift would appear.generate+ threshold enforcement)examples/demo-repoand enforces retrieval thresholdsThe two bolded rows are new issue candidates, not changes in this PR: they are outside the qualification namespace and each needs its own evidence and rollback story.
Problem and evidence
At the audited baseline
06b373a447acfce895412ac10eb4e5228c5df0b7(v0.32.1) there is no contract that lets a roadmap change be judged independently. An evaluation corpus needs two properties, and the existing one has neither.Independence from Madar output is missing.
docs/benchmarks/suite/runtime-proof.jsonis the current source of per-task expectations. It contains exact expected symbols and paths per repository (sendDocument(),server-only/document/send-document.ts, …), authored alongside the product.methodology.mdasserts this file "is not passed into retrieval" and that its obligation checklist "is not written into the answering agent's prompt" — but that isolation is prose. No test, lint rule, or CI check enforces it.Naturalness is partly missing. Measured against
docs/benchmarks/suite/repos.jsonat the baseline: of eleven rows, five are in-repo proxies keyed bypath—ts-small(examples/sample-workspace),nestjs-midandts-monorepo-large(tests/fixtures/pack-quality/**/workspace),python-serviceandgo-service(fixture directories under the suite). The other six are git-backed and do pin a URL plus an immutable SHA (documenso,formbricks,dub,twenty,cal-diy,novu). The five proxy rows are the ones backing the checked-in deterministic fixture bundles.And there is no valid outcome evidence at all. All six July 15 2026 public rows are
not_measured; four recorded no attributable Madar call. The June 10 rows are real measurements, but the prompts carried proof checklists and retrieval could load expected files fromruntime-proof.json— controlled, profile-assisted, not untuned.Exact scope
A new self-contained namespace
docs/qualification/at contract version1.0.0, plus one validator, one npm script, one CI step, one vitest file.docs/qualification/corpus.jsondocs/qualification/tasks.jsondocs/qualification/truth/*.json,docs/qualification/rubrics.jsondocs/qualification/receipt-schema.json+ two example receiptsdocs/qualification/validity-rules.mddocs/qualification/holdout-policy.mddocs/qualification/stop-rule.mddocs/qualification/evidence-categories.mddocs/qualification/tier1.jsondocs/qualification/tier2-matrix.jsonCorpus — natural repositories only
Every target is a real, externally authored, MIT-licensed TypeScript project pinned at an immutable commit.
corpus.json#/proxy_targetsis empty and documents the conditions under which an entry would be permitted.hono26de73133b8552f56ba72e025ecd82b08900d796unstoragee6be6135832f350ca16f9a77432e1d4f0aa85ed7hono-seeded-composepatches/hono-compose-reentrancy-guard.patchhono-seeded-error-disclosurepatches/hono-error-message-disclosure.patchsealed-holdout-aSeeded defects are injected into real code as patches against the pinned commit, per the issue's "seeded defects" truth source — not recreated inside a synthetic workspace built around the answer. Both patches are one-line changes to genuine framework internals: the middleware re-entrancy guard in
src/compose.ts, and the default error handler insrc/hono-base.ts.Each target records a
cited_blobsmap of path to git blob SHA. That lets the validator check every truth citation offline, and lets--verify-corpusprove the pins against the live repositories.This supersedes the first pass
The first commit on this branch authored two fixture workspaces (
ledger-service,plugin-host). They are deleted. They satisfied independence from Madar output but forfeited naturalness, reproducing the defect this issue exists to cure. Both properties are required and neither substitutes for the other — a point now written intoholdout-policy.md.Explicit non-goals honoured
docs/qualification/,.github/scripts/,.github/workflows/ci.yml,package.json(one script), and one new test.evidence-categories.mdrecords that zero product-outcome evidence is held.blinded_rubric,ordered_path_rubric,affected_set_precision_recall,single_root_cause_adjudication,seeded_defect_detection); Tier 1 measures evidence sufficiency, never answer text.Disclosed coupling — Madar ships a Hono adapter
While wiring the independence guard I found that
src/pipeline/spi/framework-hono.tsexists, andsrc/runtime/retrieve.tscarries Hono-aware query classification (an explicit-Hono token check at line 4449,hono_routeandhono_middlewareroles at 4854–4857).This is declared framework support within the TypeScript/Node corridor, not a repository-specific special case, so it is not a #660 contamination finding. But it means a
honoresult partly measures that adapter. Rather than hide it or drop the target:corpus.jsonrecordsproduction_coupling: declared_framework_adapteronhono, with the consequence that results there are not evidence about frameworks with no adapter;honoprompt deliberately avoids naming the framework (says "this framework", "the application's entry point") so the query classifier is not handed the framework identity — enforced by a test;unstoragerecordsproduction_coupling: none_foundand acts as the uncoupled contrast.The independence guard was adjusted accordingly: bare target ids are no longer forbidden literals, because a target id may legitimately equal a supported framework name. Forbidden literals are now repository URLs, commit SHAs, task ids, full prompt strings, and a curated per-target symbol list in
corpus.json#/forbidden_target_symbols.Files and symbols changed
.github/scripts/validate-qualification-contract.mjs— Ajv draft-07 validation plus target-pinning, patch, blob-manifest, provenance, independence, and freeze-digest checks.--writeregeneratesfreeze.json;--verify-corpusperforms the network verification.package.json— addsqualify:validate..github/workflows/ci.yml— oneValidate qualification contractstep on the ubuntu/Node 22 leg, beside the existing registry and release-hygiene checks.tests/unit/qualification-contract.test.ts— 34 assertions over the contract's substance.Tests
tests/unit/qualification-contract.test.tsasserts, among others:natural: true, pins a 40-char SHA and an https URL, and records a license and prepare steps;proxy_targetsis empty;unstorageisnone_found;honoprompt contains the word "hono";tasks.jsonand in the probes;madar_derived_sources_used: [];aggregatable: trueis rejected by the schema;patch_application_failureis an available invalidation reason;costshas exactlyindexing,context_build,agent;0.05margin;Differential behaviour
None at runtime. No production module is imported, changed, or re-exported.
npm pack --dry-runshows zeroqualificationentries. The only behavioural change is a new CI step that can fail a pull request.Validation commands and results
Run in the worktree after
npm ci:npm cinpm run typechecknpm run buildnpm run qualify:validate4 pinned natural targets, 0 proxy targets, 6 tasks, 6 Tier 1 cells, 2 negative-trust probes, 21 frozen filesnpm run qualify:validate -- --verify-corpusnpm run registry:validatenpm run release:verifynpx vitest run tests/unit/qualification-contract.test.tsnpx vitest run tests/unit/{package-metadata,release-hygiene,benchmark-docs,benchmark-suite-docs,benchmark-suite,mcp-registry-metadata}.test.tsnpm pack --dry-runqualificationentriesThe validator caught two real problems during authoring: a truth file citing a path the plan proposes creating (key renamed to
new_path), and the Hono adapter collision described above.A third was caught in review.
corpus.json#/forbidden_target_symbolscarries a_notekey whose value is a string, soObject.values(...).flat()folded the whole 277-character note intoFORBIDDEN_LITERALSas literal #1 of 8. Harmless as written — no production file contains that string — but it would have become a live trap the moment anyone shortened the note to something short or common, at which point the guard starts failing production files over a documentation string. Fixed in765802fe:_-prefixed keys are skipped, anything that is not an array of identifiers now fails validation loudly, and each remaining key must name a real corpus target. The guard was re-verified end to end by appendingSmartRoutertosrc/runtime/graph-summary.tsand confirmingqualify:validatefails withproduction file src/runtime/graph-summary.ts contains qualification literal "SmartRouter"— so the filter narrowed the scan to the 7 real symbols without disabling it.Incomplete or failed commands
test-gateteammate owns all resource-sensitive full-suite work under [P0] Stabilize the complete Vitest suite and protected CI merge gate #654 and a competing run would corrupt their evidence. Focused tests only.npm run verify:pack-paritywas not run. It exercises the packed MCP retrieval surface, which this change does not touch, and it is resource-heavy.Compatibility impact
None. No public API, CLI, MCP, artifact format, or package content changes.
docs/is not in thefilesarray. Existing benchmark manifests, receipts, andruntime-proof.jsonare untouched;evidence-categories.mdclassifies them (E3/E4/E6) rather than editing them.Independence, stated plainly
All six truth files were authored on 2026-08-12 by reading the pinned repository sources directly. Madar was never run against any target, and no Madar retrieval output, context pack,
implementationGuidance, Madar-selected file list, or Madar-generated validation command was consulted before freezing.Two consequences, recorded in the artifacts rather than hidden:
tier1.json#/calibration_statusstates that the first execution is a measurement and that a failure there is a product finding, not a reason to edit the contract.independent_of_production_rule_author: false,rubrics.json#/blinding/current_statusisunsatisfied, andsealed-holdout-aisstatus: "unsatisfied"with the human action spelled out inholdout-policy.md. Naturalness fixes "was the target shaped around its own answer"; it does not fix "were the rules shaped around a known target". Until a second person seals a holdout, this corpus measures regression, not generalization.The hidden acceptance test for the bounded-implementation task is likewise
status: "unavailable"with its blocking reason, rather than a self-authored stand-in that would look complete and prove nothing.Risks
tier1.json#/properties/requires_network_reason.--verify-corpus; a durable mirror is the real fix and is not in scope here.SmartRouter,createStorage, …) ever legitimately appears insrc/. It passes today; the list is data incorpus.json, so narrowing it is a contract edit rather than a code change.docs/qualification/fails CI untilnpm run qualify:validate -- --writeis run and explained.Rollback
Delete
docs/qualification/and.github/scripts/validate-qualification-contract.mjs, remove thequalify:validatescript and the CI step, and deletetests/unit/qualification-contract.test.ts. Nothing else depends on any of it. No source regeneration, no artifact migration, no cache invalidation.Verdicts
Ready for review: yes. Every deliverable and acceptance criterion is addressed; the ones that cannot be satisfied credibly in a single-author context are marked unsatisfied in the artifacts themselves rather than filled with placeholders.
Ready for merge: no. Three gates outstanding:
test-gateestablished a partial root cause only: it fixed one real stdio-server polling flake, but theFailed to start forks workersymptom is unresolved, zero clean full-suite runs were achieved, and its PR test: fix flaky stdio auto-refresh polling timeout (partial evidence for #654) #682 says "Refs [P0] Stabilize the complete Vitest suite and protected CI merge gate #654", not "Closes". No roadmap PR merges until the complete suite is reliable, and this PR has deliberately not run it.qualify:validatestep and the complete suite.Acceptance criteria assessment
falserather than asserted.unsatisfiedand the human action is named.adoptionis its own receipt block; both behaviour dimensions are non-gating and may never enter a quality score.not_measuredand cannot enter aggregate winsv0.32.1emits neither of the first two — coverage table invalidity-rules.md.validity-rules.md#gate-ordering,tier2-matrix.json#/ordering_rule, stop rule S1.8.npm ci && npm run qualify:validateverifies the whole contract offline;--verify-corpusreproduces every pin from a clean checkout over the network. Executing Tier 1 against Madar is #661 by design — building that executor here would have meant running Madar against the targets and destroying the independence this contract exists to create.New issue candidates for the lead
docs/benchmarks/suite/runtime-proof.jsonholds per-repository expected symbols and paths.docs/benchmarks/suite/methodology.mdclaims it is grader-only, never passed into retrieval and never spliced into the answering prompt. Nothing enforces that claim — no test, no lint rule, no CI check — and nothing fails if a future change violates it. The contrast is the sharpest form of the argument: this PR's ownqualify:validatemechanically fails CI when a qualification literal reachessrc/. Same class of boundary claim, one enforced and one asserted in prose. Bears directly on [P0] Remove qualification-repository tuning from production retrieval and claims #660.NativeAgentCompareReportemits no invalidation reason codes, noaggregatableflag, no adoption classification, and no separate indexing or context-build cost accounts.honojs/honoandunjs/unstorageremaining reachable. Commits are immutable but availability is not guaranteed, and a deleted or force-pushed upstream would break the PR gate.--verify-corpusdetects it; a vendored archive or mirror would remove the dependency. (The natural-code-truth follow-up previously anticipated here is closed by commitd49b86cbrather than deferred.)🤖 Generated with Claude Code
Integration target
Integration target:
next. This PR now targets the prerelease integration branch, notmain.Stable promotion to
mainis outside the scope of this PR and happens later through a separate reviewednext→mainpromotion PR.#654 remains open and blocks merge of this PR.
Retargeting note:
nextwas synchronized tomainat3371ada8before the base change. The merge base is unchanged at06b373a4, and the diff is byte-for-byte the same as when this PR targetedmain— no changes were added or removed by the retarget.