From e2ea3312f83e00c0ce4c2120aaa2f86d7df2b6e4 Mon Sep 17 00:00:00 2001 From: Wibias <37517432+Wibias@users.noreply.github.com> Date: Sat, 8 Aug 2026 14:49:10 +0200 Subject: [PATCH 01/20] docs(lab): define compatibility evidence contracts Freeze the CL-00 evidence, verdict, scenario, identity, privacy, and incident contracts so conformance implementation can proceed without duplicating routing authority. --- .../000_master_plan.md | 303 ++++++++++ .../001_pr_stack_status.md | 72 +++ .../010_architecture_and_evidence_contract.md | 549 ++++++++++++++++++ .../020_scenario_contract_and_catalogue.md | 394 +++++++++++++ .../021_protocol_v1_manifest_authority.md | 342 +++++++++++ .../022_protocol_v1_cases.json | 456 +++++++++++++++ .../030_incident_corpus.md | 457 +++++++++++++++ .../040_security_and_privacy.md | 239 ++++++++ .../050_cl00_acceptance_review.md | 147 +++++ 9 files changed, 2959 insertions(+) create mode 100644 devlog/_plan/260807_compatibility_lab/000_master_plan.md create mode 100644 devlog/_plan/260807_compatibility_lab/001_pr_stack_status.md create mode 100644 devlog/_plan/260807_compatibility_lab/010_architecture_and_evidence_contract.md create mode 100644 devlog/_plan/260807_compatibility_lab/020_scenario_contract_and_catalogue.md create mode 100644 devlog/_plan/260807_compatibility_lab/021_protocol_v1_manifest_authority.md create mode 100644 devlog/_plan/260807_compatibility_lab/022_protocol_v1_cases.json create mode 100644 devlog/_plan/260807_compatibility_lab/030_incident_corpus.md create mode 100644 devlog/_plan/260807_compatibility_lab/040_security_and_privacy.md create mode 100644 devlog/_plan/260807_compatibility_lab/050_cl00_acceptance_review.md diff --git a/devlog/_plan/260807_compatibility_lab/000_master_plan.md b/devlog/_plan/260807_compatibility_lab/000_master_plan.md new file mode 100644 index 000000000..fa41e24b4 --- /dev/null +++ b/devlog/_plan/260807_compatibility_lab/000_master_plan.md @@ -0,0 +1,303 @@ +# OpenCodex Compatibility Lab / EvalGrid + +Status: CL-00 architecture authority +Authority baseline: `upstream/dev` at `3ad5bb6bd3f76f6879d84b78ea39edd3e01ec296` +Package/runtime at baseline: OpenCodex `2.10.2`, Bun `1.3.14` + +## Purpose + +Compatibility Lab turns compatibility claims into bounded, reproducible +evidence. It tests OpenCodex protocol behavior, exact configured routes, and +later execution-grounded task outcomes without becoming a provider registry, a +user-policy system, or a production router. + +This directory is the programme authority. The contracts frozen by CL-00 are: + +- [Architecture and evidence](./010_architecture_and_evidence_contract.md) +- [Scenario model and initial catalogue](./020_scenario_contract_and_catalogue.md) +- [Protocol V1 manifest authority](./021_protocol_v1_manifest_authority.md) +- [Protocol V1 canonical cases](./022_protocol_v1_cases.json) +- [Historical incident corpus](./030_incident_corpus.md) +- [Security and privacy](./040_security_and_privacy.md) +- [CL-00 independent acceptance review](./050_cl00_acceptance_review.md) +- [PR stack status](./001_pr_stack_status.md) + +Later phases may add implementation detail, but must amend these contracts +explicitly rather than silently changing their meaning. + +## Live repository truth + +CL-00 audited the live `dev` tree before defining new authority. + +### Shipped and authoritative + +- Provider declarations and model metadata: + `src/providers/registry.ts`, `src/providers/derive.ts`, `src/types.ts`; + route-time claimed capability assembly in `src/routing/capability.ts` also + consumes provider config, cached Codex catalog rows and native metadata. + Generated fallback metadata lives in `src/generated/model-metadata.ts` and is + sourced by `scripts/model-metadata.source.json`. +- Routing Profile public/config types: `OcxRoutingProfileConfig` in + `src/types.ts`; validation, normalization, revision hashing, persistence and + resolution in `src/routing/profile.ts`. +- Deterministic profile evaluation and route traces: + `src/routing/evaluator.ts`, `src/routing/trace.ts`, and + `src/router.ts`. +- Profile management CRUD and dry-run: + `src/server/management/routing-profile-routes.ts`. +- Dashboard profile editor, dry-run and routing analytics: + `gui/src/pages/RoutingProfiles.tsx`, mounted under Models -> Routing. +- Immutable request/usage evidence and rebuildable history projection: + `src/usage/log.ts`, `src/routing/history/indexer.ts`, and + `src/routing/analytics.ts`. +- Why-this-route evidence: + `RouteDecisionTraceV1`, request-history explain endpoints in + `src/server/management/request-history-routes.ts`, and CLI explain support. + The GUI Logs modal renders only a compact route summary through + `gui/src/pages/log-route-decision.ts`; there is no GUI request-history browser + or full trace + attempts + outcome view on this baseline. +- Existing diagnostics are narrower than Compatibility Lab: + `ocx doctor` is observe-only environment/OAuth/runtime diagnosis, while + `POST /api/providers/test` performs a bounded live `/models` connectivity + check only when applicable; forward providers return configured status and + static catalogues return not-applicable without network access. +- Protocol behavior is already covered by many focused tests under `tests/`, + but those tests are not a versioned scenario catalogue or evidence ledger. + +### Not shipped + +- No Compatibility Lab runner, scenario registry, evidence ledger, SQLite + projection, CLI, management API, or UI exists. +- Generated Cursor agent protobufs include task/grind/subagent message types, + but OpenCodex has no native Agent Fabric task persistence, harness handoff, + portable task-state model, or management API. Router Intelligence's own + master plan explicitly excluded Agent Fabric. +- No Routing Profile compatibility fields exist. + +Current routing nuances that later phases must preserve rather than +over-describe: + +- selection traces and execution attempts are separate; the explain API merges + trace + `attempts[]` + final outcome at read time; +- `optimize.latency` is currently a declaration-priority share, while observed + latency contributes through health evidence rather than an independent + top-level score; +- cost evidence is commonly unknown on the live pre-dispatch path because + request usage is not yet available; +- profile dry-run is evaluation-only and never dispatches upstream; +- an unknown canonical `policy/` currently falls through to ordinary model + routing rather than failing closed. + +Consequently, CL-00 defines future contracts and integration seams only. It +does not rename existing Router Intelligence concepts or describe speculative +Agent Fabric endpoints as current behavior. + +## Architectural invariant + +```text +Provider Registry + ↓ +Compatibility Lab + ↓ +Compatibility Graph / Verified Evidence + ↓ +Routing Profiles + ↓ +Router Intelligence + ↓ +Selected Model / Route + ↓ +Agent Fabric / Real Execution + └──────────────→ execution-grounded outcomes back to Lab +``` + +The arrows are data dependencies, not ownership transfers. + +### Provider Registry + +The Provider Registry declares what a provider/model is believed to support and +supplies defaults used to construct an effective route. The shipped +`candidateCapabilityEvidence()` also combines explicit provider config, cached +catalog rows, adapter-level inference and native-model metadata. These local +declarations are claims. They may seed `CLAIMED`; they cannot by themselves +produce `PROBED`, `VERIFIED`, `DEGRADED`, or `UNSUPPORTED`. + +The Lab may snapshot a registry claim with its source revision for +reproducibility. It must not create a parallel provider catalogue or write +provider declarations back into the registry. + +Registry-owned runtime defaults such as model wire selection, discovery policy, +upstream streaming, reasoning replay, service-tier support, and item-ID repair +are intentionally not all persisted to `config.json`. Claim snapshots capture +the effective sources; they do not freeze runtime defaults into user config. + +### Compatibility Lab + +The Lab owns versioned scenarios, immutable compatibility evidence, failure +attribution, freshness, derived verdicts, and regression history. It may +project evidence into a compatibility graph keyed by exact route subject and +evidence layer. + +The Lab never chooses a production candidate, mutates a Routing Profile, +changes provider metadata, or turns a probe result directly into a route. + +### Routing Profiles + +Routing Profiles remain the only user-policy layer. Future compatibility +requirements extend `OcxRoutingProfileConfig`, its normalizer/revision, the +existing evaluator, the existing management CRUD/dry-run endpoints, and the +Models dashboard editor. There will be no compatibility-specific profile +store, evaluator, or editor. + +### Router Intelligence + +Router Intelligence combines the selected profile with current capability, +compatibility, health, quota, cost, and latency evidence and makes the +deterministic route decision. Its existing `RouteDecisionTraceV1` remains the +authority for explaining that decision. Compatibility inputs will later add +bounded evidence to that trace rather than introduce a second explanation +record. + +### Agent Fabric + +Agent Fabric is a future producer of task-effectiveness observations. It owns +real task execution and its sandbox. The Lab accepts only structured outcome +data and sanitized content-addressed artifact references; it does not copy task +repositories, prompts, worktrees, or hidden reasoning. + +Because a native Agent Fabric is not present on the CL-00 baseline, this +programme freezes the consumer semantics, not a fictitious production API. A +later producer contract must identify its schema version, task class, exact +route subject, deterministic verifier results, timing, resource limits, +outcome, and sanitized artifact references. Existing request-grounded evidence +may be linked through `RouteDecisionTraceV1`, `PersistedUsageAttempt`, and the +final request outcome; prompt-bearing `responses-state.json` and generated +Cursor task protobufs are not Lab feeds. + +## Evidence-layer invariant + +Every scenario and observation has exactly one layer: + +1. `protocol_conformance`: whether OpenCodex translates and preserves a + protocol contract correctly. +2. `live_route_compatibility`: whether an exact + provider/model/adapter/configuration route works now. +3. `task_effectiveness`: whether that route produces verifier-confirmed + outcomes for a versioned class of coding work. + +Verdicts are projected per `(subject, layer, suite)`. Evidence from one layer +may be shown as a prerequisite or correlated signal, but cannot promote or +degrade another layer's verdict. There is no universal compatibility score. + +## Persistence authority + +Future implementation uses the existing OpenCodex config root returned by +`getConfigDir()` (`OPENCODEX_HOME`, default `~/.opencodex`) and owns: + +```text +~/.opencodex/lab/ + compatibility.jsonl + compatibility.sqlite + artifacts/ +``` + +- `compatibility.jsonl` is the canonical append-only evidence/event ledger. +- `compatibility.sqlite` is a disposable query projection rebuilt from JSONL. +- `artifacts/` contains bounded, sanitized, content-addressed artifacts. +- Scenario/suite manifests and synthetic fixture/source anchors are + content-addressed contract artifacts retained with the observations that + reference them. +- Verdicts are derived projections, never mutable canonical booleans. +- Corrections append invalidation/supersession events; prior bytes are not + edited. +- The Lab does not copy `usage.jsonl` or routing-history rows. When useful, an + observation references an existing request ID or route decision ID. +- Agent Fabric supplies structured outcome data/references, never repositories + or prompt transcripts. + +This location follows current repository state-root conventions. No filename +or location change from the proposed architecture was justified by the audit. + +## Routing Profiles boundary for CL-06 + +CL-06 must add optional compatibility controls alongside existing capability, +health, quota, cost, and latency policy: + +- required compatibility suites; +- minimum compatibility status; +- maximum evidence age; +- unknown-evidence behavior; +- degraded-evidence behavior. + +`minimum compatibility status` is not a total ordering across all verdicts. +Only `PROBED` and `VERIFIED` are positive thresholds. `DEGRADED` is governed by +its explicit behavior, `UNKNOWN`/`CLAIMED`/`BLOCKED` by unknown-evidence +behavior, and `UNSUPPORTED` fails a required suite. + +The exact future flow is: + +```text +Routing Profile + ↓ +Configured candidates + ↓ +Hard capability gates + ↓ +Compatibility requirements / penalties + ↓ +Eligible candidates + ↓ +Health / quota / cost / latency scoring + ↓ +Deterministic winner +``` + +All compatibility fields are optional. Profiles that omit them retain their +current validation, revision, eligibility and scoring behavior. A profile +evaluation reads an existing projection only. No compatibility probe, network +request, task, or projection rebuild may run synchronously on the production +request path. + +## Programme phases + +Only CL-00 is authorized by this document at present. + +| Phase | Purpose | Authorization | +|---|---|---| +| CL-00 | Architecture authority, contracts, scenario catalogue, incident corpus | This PR | +| CL-01 | Deterministic protocol-conformance runner and fixtures | Not started; requires accepted CL-00 | +| CL-02 | Immutable JSONL ledger, artifacts and SQLite projection | Not started | +| CL-03 | Bounded live-route probes | Not started | +| CL-04 | Lab CLI and management read surfaces | Not started | +| CL-05 | Compatibility Matrix UI | Not started | +| CL-06 | Existing Routing Profile compatibility controls and Router Intelligence consumption | Not started | +| CL-07 | Agent Fabric task-effectiveness ingestion | Not started | +| CL-08 | Shadow/automatic/public evidence workflows | Not started | + +Phase numbering after CL-01 is programme planning, not implementation +authorization. A later accepted plan may split a phase while preserving these +ownership boundaries. + +## CL-00 acceptance criteria + +CL-00 is accepted only when: + +1. all three evidence layers have separate subjects, scenarios and verdicts; +2. every canonical verdict is reproducible from immutable inputs; +3. environmental blockers cannot poison compatibility conclusions; +4. exact route identity prevents evidence reuse across behavior changes; +5. scenario semantics and initial IDs are implementable without an LLM judge; +6. representative historical incidents map to abstract regression scenarios; +7. future compatibility policy extends existing Routing Profiles; +8. probes and task execution are excluded from production request routing; +9. privacy and sandbox ceilings are explicit; +10. an independent review finds no unresolved Critical, High, or Medium issue. + +## CL-00 hard stop + +This phase does not implement a runner, mock upstream, persistence code, live +probe, CLI, management endpoint, UI, profile field, routing behavior, shadow +route, Fabric ingestion, automatic routing, or public publisher. + +Acceptance of CL-00 authorizes discussion and planning of CL-01; it does not +start CL-01 automatically. diff --git a/devlog/_plan/260807_compatibility_lab/001_pr_stack_status.md b/devlog/_plan/260807_compatibility_lab/001_pr_stack_status.md new file mode 100644 index 000000000..e084fa66a --- /dev/null +++ b/devlog/_plan/260807_compatibility_lab/001_pr_stack_status.md @@ -0,0 +1,72 @@ +# Compatibility Lab PR stack status + +Updated throughout the programme. Every phase records its branch, exact base +and implementation head, PR, verification, independent review, blockers, and +whether the next phase is authorized. + +## Programme facts + +- Repository: `lidge-jun/opencodex` +- Integration target: `dev` +- CL-00 starting `upstream/dev`: + `3ad5bb6bd3f76f6879d84b78ea39edd3e01ec296` +- Package/runtime at start: OpenCodex `2.10.2`, Bun `1.3.14` +- CL-00 branch: `feat/cl-00-compatibility-contracts` +- CL-00 scope: documentation/contracts/incident corpus only +- PR target: `lidge-jun/opencodex:dev` + +## Stack + +| Phase | Branch | Base SHA | Implementation head | PR | State | +|---|---|---|---|---|---| +| CL-00 | `feat/cl-00-compatibility-contracts` | `3ad5bb6bd3f76f6879d84b78ea39edd3e01ec296` | pending commit | pending | ACCEPTED | +| CL-01 | not created | CL-00 must be accepted first | not started | none | NOT AUTHORIZED | + +## CL-00 acceptance log + +- Live-tree audit: complete against the exact base above. Audited provider + registry/derivation, Routing Profile types/normalization/evaluator/API/UI/ + dry-run, route traces and Why-this-route, usage/request-history/analytics, + doctor/provider connectivity validation, protocol regression tests, and + relevant open/closed devlog incidents. +- Live-tree correction: generated Cursor task/grind protobuf messages exist, + but no native Agent Fabric task persistence or management contract and no + Compatibility Lab implementation exists. CL-00 freezes consumer semantics + without claiming production endpoints. Future compatibility policy must + extend the shipped Routing Profiles system. +- Documents: + - `000_master_plan.md` + - `010_architecture_and_evidence_contract.md` + - `020_scenario_contract_and_catalogue.md` + - `021_protocol_v1_manifest_authority.md` + - `022_protocol_v1_cases.json` + - `030_incident_corpus.md` + - `040_security_and_privacy.md` + - `050_cl00_acceptance_review.md` (recorded after independent review) +- Baseline verification on clean CL-00 worktree: + - `bun x tsc --noEmit`: passed, 0 errors. + - `bun run privacy:scan`: passed. + - `bun run test`: **not green** on this Windows/Bun 1.3.14 host. The full + run exited 3 after a cache-invalidation failure, an empty Windows + effective-account lookup, and a Bun + `panic(main thread): index out of bounds: index 0, len 0`. + - Serial isolation: + `tests/codex-models-cache-invalidate.test.ts` passed 6/6; + `tests/codex-native-residue.test.ts` passed 63 with 2 platform skips. + - Focused protocol/compatibility suite excluding privileged-symlink state + cases: 395 passed, 0 failed across 24 files. + - Focused continuation state semantics: 2 passed, 95 filtered, 0 failed. + - A broader focused run including all `responses-state.test.ts` cases had + 488 pass and 4 fail; all four failures were Windows `EPERM` creating + symlinks on this host. + - `tests/repo-hygiene.test.ts`: 11 passed, 0 failed. +- Documentation verification: complete. JSON authority parses; all 35 cases, + 46 fixture records, eight suites and fixture digests validate; all local + CL-00 links resolve; `git diff --check` passed. +- Independent acceptance review: accepted after correction; all ten required + challenges pass and no Critical/High/Medium findings remain. +- Blockers: none for CL-00. Full-suite green remains unavailable on this host + for the Windows/Bun reasons above. +- CL-00 ending implementation SHA: pending finalization. +- Draft PR: pending. +- CL-01 authorized: **NO**. Acceptance of CL-00 does not start CL-01. diff --git a/devlog/_plan/260807_compatibility_lab/010_architecture_and_evidence_contract.md b/devlog/_plan/260807_compatibility_lab/010_architecture_and_evidence_contract.md new file mode 100644 index 000000000..11c7ba60c --- /dev/null +++ b/devlog/_plan/260807_compatibility_lab/010_architecture_and_evidence_contract.md @@ -0,0 +1,549 @@ +# CL-00 architecture and evidence contract + +This document freezes the semantic model consumed by later Compatibility Lab +phases. Names shown in code blocks are contract names, not claims that +production TypeScript types already exist. + +## 1. Evidence layers + +### `protocol_conformance` + +Question: does this OpenCodex build preserve the declared inbound-to-upstream +and upstream-to-client protocol contract? + +Inputs are deterministic fixture requests and deterministic mock-upstream +responses. The subject includes the OpenCodex compatibility version, adapter, +inbound protocol, upstream protocol, surface, and relevant behavior +fingerprint. A real provider account is neither required nor permitted. + +A pass proves only the exercised OpenCodex translation. It says nothing about a +provider's current availability or a model's coding quality. + +### `live_route_compatibility` + +Question: does this exact configured route satisfy this versioned scenario now? + +The subject is an exact provider/model/effective-adapter/configuration route. +The run may contact only that route's configured upstream under the Lab +sandbox. A pass cannot be reused for a different route fingerprint. + +A pass proves only the exercised route behavior at the observation time. It +does not prove task effectiveness. + +### `task_effectiveness` + +Question: did this exact route produce a successful, deterministically verified +outcome for a versioned task class? + +Agent Fabric, not the Lab, owns real execution. The Lab receives a structured +outcome containing deterministic verifier results and sanitized artifact +references. Human ratings or LLM-judge output may be stored as advisory +annotations in a later phase, but cannot produce a canonical verdict. + +### Non-collapse rule + +The canonical projection key is: + +```text +(subjectId, evidenceLayer, suiteId, suiteVersion, suiteManifestDigest, + projectionSpecVersion) +``` + +There is no projection across all layers and no weighted universal score. +Callers may present multiple layer verdicts next to each other. A prerequisite +failure in one layer may make a later-layer run inapplicable, but it does not +rewrite evidence in the other layer. + +## 2. Immutable ledger contract + +The canonical JSONL ledger is a sequence of versioned events. The minimum +event kinds are: + +```text +observation +claim_snapshot +invalidation +``` + +An `observation` records one scenario attempt. A `claim_snapshot` captures the +local declared-capability inputs and source revisions needed to reproduce +`CLAIMED`: registry/config, cached catalog or native metadata, including the +adapter inference currently assembled by `src/routing/capability.ts`. An +`invalidation` identifies prior evidence that a later-discovered harness, +fixture, redaction, or integrity defect makes unusable. Invalidations append; +they never delete or edit prior lines. + +Each event has: + +```text +schemaVersion +eventId +eventKind +recordedAt +producer +producerVersion +``` + +An observation additionally has: + +```text +evidenceLayer +scenarioId +scenarioVersion +scenarioManifestDigest +suiteId +suiteVersion +suiteManifestDigest +fixtureDigests[] +subject +subjectId +startedAt +completedAt +executionMode fixture | live | fabric +attempt +limits +outcome pass | fail | blocked | inconclusive +assertions[] +failure? { class, code, retryable, attribution } +expectedFailure? +environment +artifactRefs[] +sourceRefs? +``` + +Rules: + +- Canonical JSON is RFC 8785 JSON Canonicalization Scheme (JCS), encoded as + UTF-8 with no BOM. +- `eventId` is lowercase + `sha256("ocx-lab:event:v1\0" || JCS(event without eventId))`. +- `subjectId` is lowercase + `sha256("ocx-lab:subject:v1\0" || JCS(subject))`. +- Scenario and suite manifests use the same JCS construction with domains + `ocx-lab:scenario-manifest:v1` and `ocx-lab:suite-manifest:v1`. Fixture + digests are lowercase + `sha256("ocx-lab:fixture:v1\0" || exact fixture bytes)`. A manifest's digest + field and storage path are excluded from its preimage. Domain text is UTF-8 + and terminated by one NUL byte. +- Every observation carries the exact scenario, suite and fixture digests it + executed. Version strings without matching digests are invalid evidence. +- Timestamps are UTC epoch milliseconds; duration alone is not sufficient. +- Assertions record expected value/shape, observed normalized value/shape, and + pass/fail. They never require raw prompts or unbounded bodies. +- `failure.attribution` records whether the failure is attributable to + OpenCodex, the exact route, the environment, or the harness. +- Artifact references contain digest, media type, byte count, redaction + policy, and local relative path; never an arbitrary filesystem path. +- `sourceRefs` may contain existing request IDs, route-decision IDs, or future + Fabric outcome IDs. It must not inline the referenced request or task. +- The shipped request-history explain surface already composes selection trace, + `PersistedUsageAttempt[]`, and final outcome. A later Lab consumer should link + that normalized composition instead of reading prompt-bearing + `responses-state.json` or treating generated Cursor task protobufs as durable + OpenCodex state. +- A structurally invalid or partially written line contributes no evidence and + is reported as ledger corruption. SQLite must be rebuildable from all valid + complete lines. + +The canonical scenario manifest, suite manifest and synthetic fixture bytes +referenced by an observation are retained as content-addressed +`scenario_manifest`, `suite_manifest`, and `fixture` artifacts. These contract +artifacts have indefinite retention while any non-invalidated observation +references them. A missing or digest-mismatched contract artifact makes that +observation unusable and yields `harness_failure`; projection code must never +substitute the current manifest for the historical one. + +The SQLite projection may cache derived verdict rows. Such rows must include +their `asOf`, projection spec, scenario/suite/fixture manifest digests, and +contributing event IDs. Deleting SQLite and replaying JSONL plus the referenced +content-addressed contract artifacts must reproduce them. + +## 3. Canonical verdict contract + +The closed verdict set is: + +```text +UNKNOWN +CLAIMED +PROBED +VERIFIED +DEGRADED +BLOCKED +UNSUPPORTED +``` + +Verdicts are projections, not mutable evidence fields. + +### `UNKNOWN` + +Produced when no current registry support claim and no current, valid, +attributable observation can classify the projection key. + +It can also result when all prior evidence became stale or was invalidated and +there is no current claim or blocker. Absence of a test is not +`UNSUPPORTED`. + +### `CLAIMED` + +Produced only by a current snapshotted positive local capability declaration +for the exact capability/subject, when no current executable evidence yields a +stronger state. The snapshot records whether the declaration came from explicit +provider config, Provider Registry, cached catalog, native metadata, or current +adapter inference rather than pretending every claim is registry-authored. + +A claim cannot produce `PROBED` or `VERIFIED`. A registry negative declaration +is shown as claim metadata but does not by itself prove `UNSUPPORTED`. + +### `PROBED` + +Produced when at least one required executable scenario completed with an +attributable pass, but the suite's versioned verification rule is not yet +satisfied. Examples are partial required-scenario coverage or a scenario whose +assertions establish reachability/shape but not full suite verification. + +Connectivity-only `/models` checks, registry discovery, doctor output, health +samples, and blocked attempts cannot produce `PROBED`. + +### `VERIFIED` + +Produced only when every requirement in the suite manifest's verification rule +is met by current, valid, attributable observations for the exact projection +key, and no newer current contradictory attributable failure remains +unresolved. + +The projection must expose: + +- exact contributing event IDs; +- suite/scenario versions and manifest digest; +- subject ID and full local subject; +- projection algorithm version and `asOf`; +- freshness calculation; +- any invalidations and contradictory events considered. + +An LLM judge, user assertion, registry declaration, successful model listing, +or mutable `verified=true` flag cannot produce `VERIFIED`. + +### `DEGRADED` + +Produced when executable evidence proves that the capability works only +partially, loses required semantics, violates a required assertion while a +usable subset remains, or requires a suite-declared workaround. + +Examples include malformed tool-call/result correlation, dropped reasoning +replay required by the suite, or incomplete stream semantics with otherwise +usable output. Environmental blockers cannot produce `DEGRADED`. + +### `BLOCKED` + +Produced when a current attempt cannot reach a compatibility assertion because +of an environmental or administrative precondition and no current +compatibility-attributable verdict should take precedence. + +Authentication, quota, region policy, local network failure, provider +transients, harness failure, or exhausted Lab budget can yield `BLOCKED`. The +projection retains those classes separately. + +A blocked retry does not erase a still-current `VERIFIED`, `PROBED`, +`DEGRADED`, or `UNSUPPORTED` verdict. Once that prior verdict is stale, +`BLOCKED` may become the current state until a conclusive run succeeds. + +### `UNSUPPORTED` + +Produced only by executable, suite-declared evidence that the exact capability +is unavailable by contract for this subject. The scenario must define an +unambiguous unsupported signal or negative-control assertion; a generic 4xx, +timeout, empty output, registry omission, or failed authentication is +insufficient. + +Expected rejection of a deliberately unsupported feature can prove +`UNSUPPORTED` when the rejection itself matches the deterministic contract. It +is not a failed harness run. + +### Projection precedence + +For current valid evidence at the same projection key: + +1. Satisfied full verification rule yields `VERIFIED`. +2. An unresolved attributable required-assertion failure yields `DEGRADED` or + `UNSUPPORTED` according to the scenario's failure rule. +3. Partial positive coverage yields `PROBED`. +4. A blocker yields `BLOCKED` only when 1-3 have no current result. +5. A current positive registry claim yields `CLAIMED`. +6. Otherwise the result is `UNKNOWN`. + +This is precedence, not a quality scale. In particular, `DEGRADED`, +`BLOCKED`, and `UNSUPPORTED` are not numeric values below `PROBED`. + +## 4. Transitions, contradiction and freshness + +Because verdicts are recomputed, a "transition" means that new events, time, or +version inputs change a projection. + +Allowed transitions: + +| From | May move to | Cause | +|---|---|---| +| `UNKNOWN` | any state | claim, observation, or blocker | +| `CLAIMED` | `UNKNOWN`, `PROBED`, `VERIFIED`, `DEGRADED`, `BLOCKED`, `UNSUPPORTED` | claim removal/staleness or executable evidence | +| `PROBED` | `UNKNOWN`, `CLAIMED`, `VERIFIED`, `DEGRADED`, `BLOCKED`, `UNSUPPORTED` | coverage, contradiction, staleness, invalidation | +| `VERIFIED` | `UNKNOWN`, `CLAIMED`, `PROBED`, `DEGRADED`, `BLOCKED`, `UNSUPPORTED` | partial remaining coverage after invalidation, contradiction, staleness, or changed inputs | +| `DEGRADED` | `UNKNOWN`, `CLAIMED`, `PROBED`, `VERIFIED`, `BLOCKED`, `UNSUPPORTED` | repair evidence, staleness, invalidation, or reclassification | +| `BLOCKED` | any state | blocker clears, prior evidence becomes current/stale, or claim changes | +| `UNSUPPORTED` | `UNKNOWN`, `CLAIMED`, `PROBED`, `VERIFIED`, `DEGRADED`, `BLOCKED` | route/version/config change, invalidation, or new evidence | + +Direct transitions not listed are forbidden; implementations must not invent a +state outside this set. + +Contradictory attributable evidence is never overwritten. The projection: + +1. filters by exact subject/layer/suite/scenario versions; +2. applies invalidation events; +3. applies freshness; +4. orders observations by completion time and deterministic event-ID tie-break; +5. applies the suite's contradiction rule; +6. emits the contributing and contradicting event IDs. + +The initial contradiction rule is conservative: a newer required-scenario +failure prevents `VERIFIED` until a newer pass of that scenario and all other +required coverage exists. A newer pass can restore `VERIFIED`; history remains. + +### Freshness + +Each scenario manifest declares its maximum evidence age. The suite manifest +may declare a stricter maximum, and a future Routing Profile may tighten it +again. Effective maximum age is the minimum of all finite scenario, suite and +profile values; `null` means no bound at that layer: + +- deterministic protocol evidence has no wall-clock expiry by default, but is + exact-match bound to scenario, suite, compatibility version, adapter and + behavior fingerprint; +- initial live-route manifests default to seven days; +- initial task-effectiveness manifests default to thirty days; +- a profile's maximum evidence age is an additional upper bound, never an + extension. + +Stale observations remain queryable but cannot support current +`PROBED`/`VERIFIED`/`DEGRADED`/`UNSUPPORTED`. The projection may display a +`lastKnownVerdict` separately. Its current verdict falls to `CLAIMED`, +`BLOCKED`, or `UNKNOWN` according to current inputs. + +### Version and configuration changes + +- Evidence matches an exact scenario and suite version in contract v1. No + implicit semver range reuse is allowed. +- A changed scenario assertion, fixture, requirement, or classification rule + requires a new scenario version and invalidates old evidence for the new + projection key. +- `opencodexCompatibilityVersion` is lowercase + `sha256("ocx-lab:compatibility-version:v1\0" || JCS(manifest))`. + The exact manifest object is: + + ```text + { + "schemaVersion": 1, + "assertionDslVersion": "1.0.0", + "evidenceSchemaVersion": "1.0.0", + "bunRuntimeVersion": , + "files": [ + { + "path": , + "sha256": + }, + ... + ] + } + ``` + + `files` contains every Git-index-tracked regular file under `src/`, plus + `package.json`, `bun.lock`, and `scripts/model-metadata.source.json`, sorted + by UTF-8 path bytes. Generation reads current working-tree bytes so a dirty + behavior change cannot reuse clean-tree evidence. A missing file, a tracked + symlink, a non-regular file, duplicate normalized path, invalid UTF-8 path, + or unreadable file makes the run `harness_failure`; untracked files are not + loaded by the compatibility harness. Release/package builds embed this + generated manifest so an installed runtime does not require Git. This + conservative whole-runtime input set may invalidate unrelated evidence, but + cannot falsely reuse evidence after a behavior change. The package marketing + version remains provenance only. +- A compatibility-version change starts a new subject projection. +- Any behavior-relevant configuration fingerprint change starts a new subject. +- Credential rotation alone does not change the subject. + +## 5. Failure-attribution contract + +Every non-pass observation uses exactly one primary class. Stable secondary +codes may add detail without changing these semantics. + +| Class | Meaning | Affects verdict? | Default action | +|---|---|---:|---| +| `protocol_failure` | OpenCodex or the exact route emitted, accepted, ordered, translated, or terminated protocol data incorrectly | Yes, in the observation's layer | Conclusive; reverify after code/config/version change | +| `capability_failure` | The exact route cannot satisfy a capability assertion that it was expected to support | Yes | Conclusive when the scenario rules out an unsupported contract; otherwise retry once then `inconclusive` | +| `behavioral_failure` | A task-effectiveness deterministic verifier failed although protocol/capability prerequisites completed | Yes, task layer only | Conclusive for that task scenario | +| `authentication_blocked` | Missing, expired, rejected, or insufficient credentials prevented the assertion | No | `BLOCKED`; reauthenticate and retry | +| `quota_blocked` | Rate, credit, token, concurrency, or account quota prevented the assertion | No | `BLOCKED`; retry after reset/backoff | +| `region_blocked` | Region/tenant policy prevented execution | No | `BLOCKED`; retry only when route context changes | +| `network_failure` | DNS, TLS establishment, connect, local proxy, or transport reachability failed without provider response evidence | No | `BLOCKED`; repair environment and retry | +| `provider_transient` | Upstream returned a recognized transient/overload failure or interrupted a previously valid service path | No by default | `BLOCKED`; bounded retry/reverification | +| `timeout` | A versioned scenario deadline expired; secondary code distinguishes connect, first-byte, inactivity, or total budget | No by default | `BLOCKED`; bounded retry; reclassify only with deterministic protocol evidence | +| `harness_failure` | Runner, fixture, mock, sandbox, assertion engine, or artifact writer failed | No | Invalidate affected evidence and fix harness | +| `budget_exhausted` | Lab request/token/tool/byte/time budget ended the run before its assertion | No | `BLOCKED`; revise scenario limits/version or retry | +| `inconclusive` | Observations conflict or lack enough information for another class | No | No promotion/degradation; investigate/reverify | + +Safety rules: + +- Expired credentials never imply broken tool support. +- Quota exhaustion never implies model incompatibility. +- Local DNS/TLS/connect failure never degrades provider capability. +- A generic timeout never proves missing terminal semantics. A deterministic + mock stream that closes without its required terminal event is + `protocol_failure`; a live body that simply stalls is `timeout`. +- Malformed tool-call semantics, broken tool-result correlation, or lost + required event ordering may legitimately produce `protocol_failure` and + `DEGRADED`. +- `provider_transient` may be promoted to a compatibility-affecting class only + by a scenario-specific deterministic rule and a new observation; projection + code must not infer promotion from retry count. +- An expected failure is first-class scenario data. When the observed + rejection exactly matches a declared unsupported assertion, it can produce + `UNSUPPORTED`. Any other expected failure remains a pass/fail of the + assertion, not a blanket suppression. + +## 6. Canonical route subject + +Evidence is never keyed by model name alone. `RouteSubjectV1` contains: + +```text +subjectSchemaVersion +providerId +providerInstanceFingerprint +clientModelId +upstreamModelId +effectiveAdapter +inboundProtocol +upstreamProtocol +surface +opencodexCompatibilityVersion +behaviorFingerprint +endpointFingerprint +dependencies[] +``` + +Semantics: + +- `providerId` is the built-in registry ID or `custom`; it is not a display + label. +- `providerInstanceFingerprint` is a locally salted HMAC over the configured + provider identity, allowing two instances of one preset to differ without + leaking a user-selected name. All local opaque fingerprints use lowercase + HMAC-SHA-256 over + `UTF8("ocx-lab:local-fingerprint:v1\0" + fieldName + "\0") || JCS(value)` + with the installation salt as key. +- `clientModelId` is the selected canonical route model; `upstreamModelId` is + the effective wire model after namespace, virtual-model, combo and suffix + resolution. +- `effectiveAdapter` reflects model-specific wire defaults/overrides and wire + pins, not merely the provider-wide configured adapter. +- Protocol values distinguish OpenAI Responses, OpenAI Chat Completions, + Anthropic Messages, and provider-specific wires. +- `surface` distinguishes behaviorally different ingress/transport paths such + as Responses HTTP, Responses WebSocket, Chat HTTP/SSE, and Anthropic + Messages HTTP/SSE. +- The package/build version remains observation provenance in + `producerVersion`; only `opencodexCompatibilityVersion` participates in the + subject so an unrelated release does not discard valid conformance evidence. +- `endpointFingerprint` is a locally salted HMAC of the normalized destination + scheme/host/port/base path. Raw URLs, userinfo, query strings, and fragments + are not evidence fields. +- `dependencies` is an ordered list of flat `RouteDependencyV1` records for + behaviorally invoked sidecars. Each record contains role, provider ID, + provider-instance fingerprint, client/upstream model IDs, effective adapter, + upstream protocol, endpoint fingerprint, and behavior fingerprint. It cannot + nest. Records sort by role, provider ID, upstream model ID, then endpoint + fingerprint. An empty list is canonical when no sidecar is invoked. + +### Behavior fingerprint allowlist + +The fingerprint is SHA-256 over canonical JSON containing only effective, +behavior-changing values applicable to the selected model/surface: + +- adapter/wire resolution and `responsesPath`; +- auth mode and auth transport, but no credential/account identity; +- stateful/stateless Responses behavior, upstream streaming mode, service-tier + support, snapshot and item-ID repair; +- context/input/output limits and input modalities; +- reasoning capability, effort/default/mapping/wire/summary/replay/split/ + toggle/budget behavior; +- tool-choice restrictions, parallel-tool support, hosted-tool preference, + freeform/custom-tool handling, built-in-name escaping; +- prompt-cache forwarding, Anthropic EOF policy, model suffix handling; +- Google mode and opaque project/location fingerprints where applicable; +- OpenRouter routing preferences where applicable; +- actual Bun runtime version and platform/architecture whenever the selected + stream path or adapter has platform-sensitive behavior; +- effective vision and web-search sidecar enablement, backend, model, + reasoning, per-turn limits, timeout/stall limits, and the matching flat + dependency subject when that sidecar can execute; +- MCP schema/result/tool count bounds, with all Lab execution facilities forced + to the sandbox settings in the security contract; +- effective global stream mode, fast/service-tier behavior, and effort caps + when they can alter the scenario; +- a digest of non-credential custom header behavior. + +Values that are inapplicable to the selected model/surface are omitted rather +than copied wholesale. Canonical JSON sorts keys, normalizes absent/default +values to their effective value, and sorts set-like arrays while preserving +order-sensitive arrays. + +The following never participate: + +- API keys, OAuth/access/refresh tokens, cookies, authorization headers; +- account IDs, emails, labels, aliases, quota balances or plan names; +- raw custom/private header names or values; +- prompts, messages, tool results, repository paths or contents; +- timestamps, transient health, latency, cost, quota or retry state. + +Credential headers are excluded. Non-credential custom headers contribute only +through a locally salted HMAC over normalized names/values, so a behavior +change alters identity without disclosing the header. Project/location and +custom endpoint values use the same local opaque treatment. + +Public export replaces all local fingerprints with export-scoped opaque IDs +and redacts custom model IDs unless the export policy explicitly classifies +them as public. + +## 7. Task-effectiveness ingress + +A future Fabric observation must provide, at minimum: + +```text +producerSchemaVersion +outcomeId +taskClassId +taskClassVersion +subject +startedAt +completedAt +resourceLimits +result success | failure | blocked | inconclusive +verifiers[] { id, version, result, normalizedMetrics? } +artifactRefs[] +``` + +The Lab rejects an outcome if the subject cannot be reconstructed, a verifier +is nondeterministic for a canonical assertion, or an artifact violates the +security contract. Free-form narrative may be retained only as bounded, +sanitized advisory metadata and never determines the verdict. + +## 8. Consumer boundaries + +- Provider Registry supplies claims; Lab does not rewrite them. +- Request history supplies route/outcome references; Lab does not copy its + ledger. +- Selection and execution stay separate: `RouteDecisionTraceV1` records the + pre-dispatch choice, `attempts[]` records physical execution/fallback, and + final outcome is joined at read time. +- Routing Profiles express user requirements; Lab does not evaluate policy. +- Router Intelligence reads projections; Lab does not rank candidates. +- Route decision traces explain compatibility exclusions/penalties in the + existing trace; Lab does not create a parallel route explanation. +- Agent Fabric executes tasks; Lab does not run arbitrary repository work. diff --git a/devlog/_plan/260807_compatibility_lab/020_scenario_contract_and_catalogue.md b/devlog/_plan/260807_compatibility_lab/020_scenario_contract_and_catalogue.md new file mode 100644 index 000000000..fa0e610ff --- /dev/null +++ b/devlog/_plan/260807_compatibility_lab/020_scenario_contract_and_catalogue.md @@ -0,0 +1,394 @@ +# CL-00 scenario contract and initial catalogue + +This document freezes the scenario schema and the initial IDs. CL-01 implements +the `*.protocol.*` scenarios only. Entries marked live or Fabric-reserved define +future semantics and are not authorization to execute them. + +The normative V1 selector/operator semantics, immutable fixture anchors, +expanded defaults and complete protocol scenario/suite manifest records are in +[the protocol V1 manifest authority](./021_protocol_v1_manifest_authority.md); +its canonical fixture vectors and literal expectations are in +[`022_protocol_v1_cases.json`](./022_protocol_v1_cases.json). + +## 1. Versioned scenario model + +A `CompatibilityScenarioV1` has: + +```text +schemaVersion 1 +id stable lowercase dotted ID +version exact semver +suite { id, version } +evidenceLayer protocol_conformance | + live_route_compatibility | + task_effectiveness +capability stable capability ID +verificationRole required | supplemental | negative_control +requirements +fixtures +executionLimits +assertions[] +failureRules[] +artifactPolicy +freshness +``` + +### Identity and versioning + +- `id` names semantics and does not contain a provider, model, or version. +- `version` is exact-match in contract v1. +- Any assertion, fixture, limit that affects expected behavior, failure rule, + artifact exposure, or requirement change increments the scenario version. +- Editorial description changes do not require a version change. +- A suite manifest has its own exact version and lists scenario IDs, versions, + roles, and its verification rule. +- Scenario and suite manifests are RFC 8785 canonical JSON with the + domain-separated digest defined by the evidence contract. + +### Requirements + +Requirements are declarative and may include: + +```text +inboundProtocols[] +upstreamProtocols[] +surfaces[] +requiredClaims[] +requiredHarnessFeatures[] +platforms[] +routePreconditions[] +``` + +An unmet deterministic fixture requirement is `harness_failure`. An unmet live +route precondition is either inapplicable or a typed blocker; it is never +silently counted as a capability failure. + +### Fixtures + +Fixture references include ID, content digest, media type, generator version +where generated, and role (`client_request`, `upstream_response`, +`adapter_vector`, `synthetic_tool`, `synthetic_image`, or `task_fixture`). A +fixture never embeds credentials, user data, or an external mutable URL. + +Protocol fixtures run against a deterministic mock upstream. Live scenarios use +only Lab-owned synthetic requests and inert tools. Fabric scenarios refer to a +versioned synthetic task class; they do not place a repository in the Lab +ledger. + +### Execution limits + +Every scenario states: + +```text +totalTimeoutMs +connectTimeoutMs? +firstByteTimeoutMs? +inactivityTimeoutMs? +maxRequests +maxInputBytes +maxOutputBytes +maxOutputTokens? +maxToolCalls +maxArtifactBytes +``` + +Absent limits are invalid. Limits may be stricter than Lab-wide ceilings but +not wider without a scenario version change and security review. Expiry of a +limit classifies as `timeout` or `budget_exhausted` according to the failed +limit; it does not imply incompatibility. + +### Deterministic assertion DSL + +Canonical assertions use a closed set of observable operators: + +```text +http_status_equals +header_present +header_absent +header_value_equals +json_schema_matches +json_path_equals +json_path_present +json_path_absent +sse_field_equals +sse_event_sequence +sse_event_count +terminal_signal_equals +id_matches +id_stable_across_events +id_correlates +tool_call_equals +tool_result_correlates +fixture_request_matches +normalized_text_equals +byte_limit_observed +process_exit_equals +verifier_result_equals +``` + +Each assertion has an ID, operator, selector, expected value, required flag, +and redaction-safe observed summary. Implementations must exhaustively reject +unknown operators. Protocol V1 permits no arbitrary regular expressions; ID +grammars and all operator type/missing-value behavior are closed in the +manifest authority. + +Core verdicts cannot use an LLM judge, free-form human interpretation, or a +snapshot that contains unstable timestamps/IDs without normalization. + +### Failure rules + +Rules are ordered and explicit: + +```text +match assertion IDs, normalized status/error/event/timeout +classification canonical failure class +secondaryCode +verdictEffect none | degraded | unsupported +retry never | bounded | after_precondition_change +expected boolean +``` + +The first exact rule wins. If no rule establishes a compatibility-attributable +class, the attempt is `inconclusive`. Generic HTTP 4xx/5xx rules may classify a +blocker or transient but cannot prove `UNSUPPORTED`. + +### Artifact policy + +The policy is deny-by-default and names allowed normalized artifacts: + +```text +assertion_report +sanitized_request_shape +sanitized_response_shape +normalized_event_trace +sanitized_error +verifier_summary +``` + +It states per-artifact and aggregate byte limits, retention class, local/public +visibility, and redaction profile. Raw credentials, prompts, hidden reasoning, +full task repositories, arbitrary headers, and arbitrary response bodies are +not valid artifact kinds. + +Each scenario declares `freshness.maxAgeMs`; its suite may declare a stricter +bound, and a future profile may tighten it again. Effective maximum age is the +minimum finite bound, with `null` meaning unbounded at that layer. + +## 2. Suite projection rules + +For each exact suite manifest: + +- `VERIFIED`: all `required` scenarios applicable to the subject have current + passes and every negative control observed its required rejection. +- `PROBED`: at least one required scenario passed, with no current + compatibility-attributable required-scenario failure, but coverage is + incomplete. +- `DEGRADED`: a failure rule on an applicable required scenario yields + `degraded`. +- `UNSUPPORTED`: a required scenario's deterministic unsupported rule or + negative control proves the capability unavailable. +- `BLOCKED`: only blockers exist and no current attributable verdict takes + precedence. +- `CLAIMED`/`UNKNOWN`: follow the evidence contract. + +Supplemental scenarios never block `VERIFIED` unless a new suite version makes +them required. + +## 3. Initial suite catalogue + +All initial scenario versions and suite versions are `1.0.0`. + +For protocol V1, the literal fixtures/assertions in `022` are the complete +verification boundary. Descriptions below summarize those exact vectors; they +do not silently incorporate every historical incident mapped to the same +scenario ID. An incident absent from `022` is candidate coverage for a reviewed +scenario/suite version amendment and cannot be claimed by a V1 `VERIFIED` +verdict. + +### `responses-core` + +Purpose: preserve the OpenAI Responses request, output-item lifecycle, stream +framing, IDs, terminal state, and JSON/SSE equivalence. + +Capability: `protocol.responses.core`. + +| Scenario ID | Layer/applicability | Required observable assertions | +|---|---|---| +| `responses-core.protocol.request-shape` | Deterministic; CL-01 | Mock receives the exact model, first user text and zero temperature | +| `responses-core.protocol.sse-framing` | Deterministic; CL-01 | Spaced and unspaced data fields, a non-record `null` frame, data-only event inference, exact text and completion terminal | +| `responses-core.protocol.item-lifecycle` | Deterministic; CL-01 | One added/done/completed lifecycle with stable valid message ID | +| `responses-core.protocol.terminal-state` | Deterministic; CL-01 | One explicit failed terminal is preserved exactly | +| `responses-core.protocol.json-sse-equivalence` | Deterministic; CL-01 | One normalized JSON/SSE pair has equal text and completion terminal | +| `responses-core.live.basic-turn` | Live-reserved | 2xx, bounded output, valid lifecycle and terminal state from exact route | + +Unsupported means a route deterministically rejects the Responses surface with +a suite-recognized unsupported signal. Semantic loss, invalid IDs, malformed +event order, or missing terminal state is degraded. Auth/quota/region/network, +transient upstream errors, and body stalls are blocked. + +### `chat-core` + +Purpose: preserve OpenAI Chat Completions request/response semantics for JSON +and streaming routes. + +Capability: `protocol.chat.core`. + +| Scenario ID | Layer/applicability | Required observable assertions | +|---|---|---| +| `chat-core.protocol.request-mapping` | Deterministic; CL-01 | System/developer/user order and JSON-object response format match the fixture | +| `chat-core.protocol.nonstream-envelope` | Deterministic; CL-01 | One valid choice/message/finish/usage envelope yields exact text and terminal | +| `chat-core.protocol.stream-assembly` | Deterministic; CL-01 | Fragmented/interleaved deltas assemble in order and emit one finish | +| `chat-core.protocol.stream-terminal` | Deterministic; CL-01 | One stop finish plus `[DONE]` yields exact text and one terminal | +| `chat-core.live.basic-turn` | Live-reserved | Exact route returns bounded text and a valid finish contract | + +Unsupported is a deterministic surface rejection. Incorrect role mapping, +malformed choices, lost stream fragments, or invalid finish semantics is +degraded. Environmental and transient failures are blocked. + +### `anthropic-core` + +Purpose: preserve Anthropic Messages roles/content blocks, tool/thinking block +ordering, stop reasons, usage, and SSE lifecycle. + +Capability: `protocol.anthropic.messages.core`. + +| Scenario ID | Layer/applicability | Required observable assertions | +|---|---|---| +| `anthropic-core.protocol.request-mapping` | Deterministic; CL-01 | Model, system instruction and first user text map exactly | +| `anthropic-core.protocol.content-sequence` | Deterministic; CL-01 | `message_start`, monotonic content blocks/deltas/stops, `message_delta`, `message_stop` | +| `anthropic-core.protocol.tool-round-trip` | Deterministic; CL-01 | One `tool_use`/`tool_result` pair preserves its ID correlation and result text | +| `anthropic-core.protocol.terminal-errors` | Deterministic; CL-01 | One explicit Responses failure maps to the Anthropic error/failed terminal | +| `anthropic-core.live.basic-turn` | Live-reserved | Exact route returns a valid bounded Messages lifecycle and terminal | + +Unsupported is a recognized Messages-surface rejection. Wrong block ordering, +lost tool correlation, invalid stop reason, or clean EOF accepted without the +suite's terminal contract is degraded. Authentication, quota, region, network, +transient failure, and silence timeout are blocked. + +### `tools-core` + +Purpose: prove deterministic function/custom tool declaration, call assembly, +parallel correlation, and result continuation. + +Capability: `tools.round_trip`. + +| Scenario ID | Layer/applicability | Required observable assertions | +|---|---|---| +| `tools-core.protocol.function-round-trip` | Deterministic; CL-01 | One function call preserves ID/name/parsed arguments and its continuation result correlates | +| `tools-core.protocol.custom-freeform-round-trip` | Deterministic; CL-01 | One `apply_patch` call preserves exact freeform input and its continuation result correlates | +| `tools-core.protocol.parallel-correlation` | Deterministic; CL-01 | Two interleaved calls assemble once in first-seen order without overlap | +| `tools-core.protocol.result-content` | Deterministic; CL-01 | One result preserves exact text and data-image parts | +| `tools-core.protocol.choice-and-allowed-set` | Deterministic; CL-01 | One required single-tool allowed set narrows exactly without widening | +| `tools-core.live.function-round-trip` | Live-reserved | Inert deterministic function is called once with schema-valid args and static result is continued | +| `tools-core.live.custom-freeform-round-trip` | Live-reserved | Route emits exact custom/freeform call and accepts static result continuation | + +An explicit route contract that rejects a tool kind can prove unsupported. +Malformed arguments, dangling IDs, widened choice, dropped calls/results, or +incorrect parallel assembly is degraded. A model choosing not to call an +`auto` tool is inconclusive; required tool choice is used for conclusive live +coverage. Environmental failures are blocked. + +### `codex-core` + +Purpose: establish the minimum end-to-end semantics required to advertise a +route as usable by Codex. A basic Responses text request is insufficient. + +Capability: `client.codex.core`. + +| Scenario ID | Layer/applicability | Required observable assertions | +|---|---|---| +| `codex-core.protocol.streaming-turn` | Deterministic; CL-01 | One Chat-backed stream yields exact text, final-answer phase and one completed terminal | +| `codex-core.protocol.apply-patch-turn` | Deterministic; CL-01 | One custom `apply_patch` call preserves exact patch text and its result ID correlates | +| `codex-core.protocol.tool-continuation` | Deterministic; CL-01 | One function result follows and correlates with its prior call | +| `codex-core.protocol.previous-response-replay` | Deterministic; CL-01 | One local expansion preserves stored input/output/new-input order and strips `previous_response_id` | +| `codex-core.protocol.structured-output` | Deterministic; CL-01 | One JSON-schema request maps to the exact Chat `response_format` | +| `codex-core.protocol.compaction-and-special-items` | Deterministic; CL-01 | Compaction, local shell, tool search and hosted-tool items are normalized without leaking opaque raw data | +| `codex-core.live.tool-turn` | Live-reserved | Valid stream plus required inert tool call/result continuation and terminal | +| `codex-core.live.custom-tool-turn` | Live-reserved | Required custom/freeform call/result continuation and valid terminal | + +The `codex-core` manifest requires all six protocol scenarios for conformance +verification. Future live verification requires both live scenarios plus +current `responses-core.live.basic-turn`. Text-only success is at most partial +coverage, never `codex-core: VERIFIED`. + +An exact route may be unsupported when it deterministically lacks a mandatory +Codex surface or tool kind. Lossy lifecycle, call/result correlation, +continuation, or special-item behavior is degraded. Environmental failures are +blocked. + +### `vision-core` + +Purpose: preserve declared image input and tool-result image behavior and prove +exact-route image understanding without user media. + +Capability: `modalities.image.input`. + +| Scenario ID | Layer/applicability | Required observable assertions | +|---|---|---| +| `vision-core.protocol.input-image` | Deterministic; CL-01 | One data-URL image preserves detail and text/image ordering | +| `vision-core.protocol.tool-result-image` | Deterministic; CL-01 | Image content in function/tool result remains structured and correlated | +| `vision-core.protocol.modality-gate` | Deterministic negative control; CL-01 | A text-only/no-sidecar synthetic vector produces the typed unsupported path without silent image drop | +| `vision-core.live.synthetic-ocr` | Live-reserved | Lab-generated image nonce is returned in an exact JSON schema | + +A deterministic declared no-image contract may prove unsupported. Dropping, +textifying without a declared sidecar, corrupting, or misordering image content +is degraded. Failure of the optional sidecar route is attributed to that exact +subject. Auth/quota/network/transient failures are blocked. + +### `reasoning-core` + +Purpose: preserve supported reasoning controls, summaries, signatures and +replay while preventing provider-private reasoning material from crossing an +incompatible boundary. + +Capability: `reasoning.round_trip`. + +| Scenario ID | Layer/applicability | Required observable assertions | +|---|---|---| +| `reasoning-core.protocol.effort-mapping` | Deterministic; CL-01 | Effective effort maps to the declared wire form and unsupported parameters are omitted | +| `reasoning-core.protocol.summary-stream` | Deterministic; CL-01 | One summary-part/delta/completed sequence preserves ordering and reasoning ID | +| `reasoning-core.protocol.replay` | Deterministic; CL-01 | One synthetic plaintext/signature replay reaches the second turn exactly | +| `reasoning-core.protocol.private-content-isolation` | Deterministic; CL-01 | One provider-private value is absent from an incompatible upstream and client response | +| `reasoning-core.live.replay` | Live-reserved | Synthetic two-turn route accepts its declared replay form and completes | + +An explicit no-reasoning route can prove unsupported. Rejected or corrupted +declared replay, lost required signatures, or private content sent across an +incompatible provider boundary is degraded (and the latter is also a security +finding). Environmental failures are blocked. + +### `mcp-core` + +Purpose: preserve MCP namespace, schema, tool/resource invocation and result +contracts through supported adapters without touching user MCP servers. + +Capability: `tools.mcp.core`. + +| Scenario ID | Layer/applicability | Required observable assertions | +|---|---|---| +| `mcp-core.protocol.namespace-mapping` | Deterministic; CL-01 | One namespace/name pair flattens and reverses exactly | +| `mcp-core.protocol.schema-and-bounds` | Deterministic; CL-01 | Tool schemas encode correctly; exact configured bounds admit and one-byte-over rejects atomically | +| `mcp-core.protocol.call-result` | Deterministic; CL-01 | Lab stub receives one exact call and returns one successful text result | +| `mcp-core.protocol.resource-round-trip` | Deterministic; CL-01 | One list/read resource success shape preserves URI, name and text | +| `mcp-core.live.synthetic-tool` | Live-reserved | Lab-owned loopback pure-function MCP tool is advertised, called and correlated | + +Only a Lab-owned in-memory or loopback fixture is allowed. A route that +deterministically cannot expose MCP may be unsupported. Namespace loss, schema +corruption, partial bound commits, or result miscorrelation is degraded. +User-server unavailability is never tested; environmental failures are blocked. + +## 4. CL-01 implementation boundary + +CL-01 may implement the scenario registry, deterministic mock-upstream harness, +closed assertion DSL, and only the `protocol_conformance` manifests frozen in +`021_protocol_v1_manifest_authority.md`. It must not: + +- contact a real provider; +- write the planned production evidence ledger or SQLite projection; +- add profile/routing controls; +- execute a user tool, shell, filesystem, repository, MCP server, or external + network action; +- implement live/Fabric scenarios merely because their IDs are reserved here. + +If CL-01 discovers that an observable assertion cannot be implemented without +new semantics, it must amend this contract in a reviewed change rather than +quietly inventing behavior. diff --git a/devlog/_plan/260807_compatibility_lab/021_protocol_v1_manifest_authority.md b/devlog/_plan/260807_compatibility_lab/021_protocol_v1_manifest_authority.md new file mode 100644 index 000000000..5c5035176 --- /dev/null +++ b/devlog/_plan/260807_compatibility_lab/021_protocol_v1_manifest_authority.md @@ -0,0 +1,342 @@ +# CL-00 protocol V1 manifest authority + +This document closes the executable semantics for the initial +`protocol_conformance` scenarios. It is normative for CL-01 and does not +implement a runner. + +The machine-readable source of truth is +[`022_protocol_v1_cases.json`](./022_protocol_v1_cases.json). It contains 35 +provider-independent canonical fixture vectors, literal expected values, +row-specific requirements, exact roles/media types, execution limits, artifact +policy, failure rules, and domain-separated fixture digests. Historical tests +in the [incident corpus](./030_incident_corpus.md) are provenance and coverage +guidance only; they are not executable manifest semantics. + +## 1. Exact manifest expansion + +For each entry in `cases`, CL-01 constructs `CompatibilityScenarioV1` in this +field order before RFC 8785 canonicalization: + +```text +schemaVersion source.schemaVersion +id case.id +version manifestDefaults.version +suite { id: case.suite, + version: manifestDefaults.suiteVersion } +evidenceLayer manifestDefaults.evidenceLayer +capability case.capability +verificationRole case.verificationRole when present, + otherwise manifestDefaults.verificationRole +requirements case.requirements +fixtures when case.initiatingRequest is present: + [fixtureRef(case.initiatingRequest), + fixtureRef(case.fixture)] + otherwise: [fixtureRef(case.fixture)] +executionLimits manifestDefaults.executionLimits +assertions case.assertions +failureRules failureRuleSets[ + manifestDefaults.failureRuleSet] +artifactPolicy manifestDefaults.artifactPolicy +freshness manifestDefaults.freshness +``` + +`fixtureRef(x)` is exactly: + +```text +{ + id: x.id, + role: x.role, + mediaType: x.mediaType, + digest: x.digest, + byteLength: UTF8(x.bytesUtf8).byteLength +} +``` + +JSON object field order has no digest effect, but the field set above is +closed. Unknown fields reject registration. Arrays preserve source order. +Empty arrays remain present. No default may be read from runtime code. + +The scenario digest is: + +```text +sha256( + UTF8("ocx-lab:scenario-manifest:v1\0") || + UTF8(JCS(expanded scenario)) +) +``` + +The exact fixture bytes are UTF-8 encoding of each `bytesUtf8` field; every +published `fixture` and `initiatingRequest` digest is: + +```text +sha256(UTF8("ocx-lab:fixture:v1\0") || fixture bytes) +``` + +Registration recomputes all digests and rejects mismatch. Every `bytesUtf8` is +retained as a content-addressed fixture artifact, not duplicated into the +expanded manifest. An `upstream_response` case without an initiating +`client_request` rejects registration. + +## 2. Exact suite manifests + +All suite versions are `1.0.0`. Every listed scenario has role `required` +except `vision-core.protocol.modality-gate`, whose role is `negative_control`. +No supplemental protocol V1 scenario exists. Order is the order in the case +authority: + +| Suite | Capability | Required member suffixes | +|---|---|---| +| `responses-core` | `protocol.responses.core` | `request-shape`, `sse-framing`, `item-lifecycle`, `terminal-state`, `json-sse-equivalence` | +| `chat-core` | `protocol.chat.core` | `request-mapping`, `nonstream-envelope`, `stream-assembly`, `stream-terminal` | +| `anthropic-core` | `protocol.anthropic.messages.core` | `request-mapping`, `content-sequence`, `tool-round-trip`, `terminal-errors` | +| `tools-core` | `tools.round_trip` | `function-round-trip`, `custom-freeform-round-trip`, `parallel-correlation`, `result-content`, `choice-and-allowed-set` | +| `codex-core` | `client.codex.core` | `streaming-turn`, `apply-patch-turn`, `tool-continuation`, `previous-response-replay`, `structured-output`, `compaction-and-special-items` | +| `vision-core` | `modalities.image.input` | `input-image`, `tool-result-image`, `modality-gate` | +| `reasoning-core` | `reasoning.round_trip` | `effort-mapping`, `summary-stream`, `replay`, `private-content-isolation` | +| `mcp-core` | `tools.mcp.core` | `namespace-mapping`, `schema-and-bounds`, `call-result`, `resource-round-trip` | + +For each row, the expanded suite manifest is: + +```text +schemaVersion 1 +id table suite +version 1.0.0 +evidenceLayer protocol_conformance +capability table capability +assertionDslVersion 1.0.0 +evidenceSchemaVersion 1.0.0 +freshness { maxAgeMs: null } +contradictionRule newest-required-observation-v1 +scenarios [{ + id: full case ID, + version: 1.0.0, + role: expanded scenario verificationRole, + manifestDigest: expanded scenario digest + }, ...] +verificationRule all-applicable-required-pass-v1 +``` + +Unknown fields reject registration. The suite digest uses +`ocx-lab:suite-manifest:v1` plus JCS exactly as defined by the evidence +contract. `VERIFIED` requires a current pass for every applicable member and +the exact suite/scenario/fixture digests above. + +## 3. Observation selector model + +Assertions use absolute RFC 6901 JSON Pointers into this closed normalized +observation: + +```text +{ + "client": { + "request": { "status": 0, "headers": {}, "json": null, "rawBytes": 0 }, + "response": { + "status": 0, + "headers": {}, + "json": null, + "events": [], + "terminal": null, + "normalizedText": "" + } + }, + "upstream": { + "requests": [ + { "status": 0, "headers": {}, "json": null, "rawBytes": 0 } + ], + "responses": [] + }, + "process": { "exitCode": null }, + "verifiers": {} +} +``` + +Header names are lowercase. JSON pointers use `~0` and `~1` escaping. Array +indexes are decimal with no leading zero except `0`; `-` is forbidden. +Wildcard, filter, recursive descent, script expression, URI, and filesystem +selectors do not exist in V1. + +Unless the operator is `json_path_absent`, a missing selector fails with +`selector_missing`. Unless the operator checks presence/absence, a wrong JSON +type fails with `selector_type_mismatch`. Both are required-assertion failures, +not harness failures. + +Objects compare by JCS bytes. Arrays are order-sensitive. Strings compare +without trimming or Unicode normalization. Numbers use JCS representation. +`null`, missing, empty string, empty array, and empty object are distinct. + +## 4. Fixture roles and execution + +Closed V1 fixture roles are: + +- `client_request`: inject exact bytes at the named inbound protocol surface; +- `upstream_response`: return exact bytes from the loopback mock; +- `adapter_vector`: decode the fixture JSON and feed its documented fields to + the selected adapter boundary without network access; +- `synthetic_tool`: decode the fixture JSON into the in-memory inert tool/MCP + stub. It never executes model arguments. + +Closed media types are `application/json`, `text/event-stream`, +`application/vnd.opencodex.adapter-vector+json`, and +`application/vnd.opencodex.mcp-stub+json`. + +Each case's exact `requirements` selects the adapter/surface and harness +features. `adapter_vector` keys are scenario-specific closed input fields +defined by the literal vector and scenario assertions; unknown keys reject the +fixture. CL-01 must encode those fields as a discriminated union keyed by the +scenario ID, not a generic callback or dynamic module. + +The MCP cases use only `in_memory_mcp_stub`. No case authorizes stdio, a child +process, user MCP configuration, filesystem access, or a user tool. + +## 5. SSE normalization + +The harness retains exact fixture bytes and normalizes only for assertions: + +1. UTF-8 must decode without replacement. A BOM is allowed only at byte zero + and is removed. +2. CRLF and CR become LF. +3. An empty line terminates a frame. Comment lines beginning `:` are ignored. +4. The first `:` separates field and value. No colon means an empty value. + Exactly one optional leading U+0020 after `:` is removed; no other + whitespace is trimmed. +5. Repeated `data` fields join with LF. The last `event` field wins. +6. `[DONE]` is a sentinel only for Chat surfaces. +7. When `event` is absent and parsed `data` is an object with string `type`, + Responses/Anthropic normalization infers that `type`. Explicit event wins. + Parsed `null`, scalar, array, or empty data is padding and emits no event. + Syntactically malformed nonempty JSON is terminal. +8. Arrival order is preserved; events are never sorted or deduplicated. + +Each normalized event is: + +```text +{ "event": string, "data": JSON value, "ordinal": integer } +``` + +## 6. Assertion operators + +- `http_status_equals`: selected integer equals expected integer. +- `header_present` / `header_absent`: selected lowercase header key exists/does + not exist. +- `header_value_equals`: selected normalized header string equals expected. +- `json_schema_matches`: selected value validates against embedded JSON Schema + draft 2020-12. Only local `$defs`/`$ref` are allowed; coercion, defaults, + custom formats, and network resolution are forbidden. +- `json_path_equals`: selected value equals literal expected under JCS rules. +- `json_path_present` / `json_path_absent`: pointer succeeds/fails; a present + `null` is present. Expected must be literal `true`. +- `sse_field_equals`: selected normalized field equals expected string. +- `sse_event_sequence`: exact event-name array; no subsequence or extras. +- `sse_event_count`: expected is `{event,count}` and exact count is required. +- `terminal_signal_equals`: expected is `completed`, `failed`, `incomplete`, + `done`, `message_stop`, `eof_tolerated`, or `none`. Exactly one terminal is + required unless expected is `none`. +- `id_matches`: expected is a closed grammar: + - `responses_message`: `msg_` plus 1..128 ASCII alphanumeric/underscore/dash; + - `responses_reasoning`: `rs_` plus 1..128 of that set; + - `responses_call`: `call_` plus 1..128 of that set; + - `nonempty_128`: 1..128 printable non-whitespace ASCII characters. + Arbitrary regular expressions are forbidden. +- `id_stable_across_events`: expected is an ordered pointer list. Every + resolved string is byte-equal. +- `id_correlates`: expected is exactly two pointers resolving to byte-equal + strings. +- `tool_call_equals`: selected normalized call equals + `{id,name,arguments,kind,ordinal}`. Function arguments are parsed JSON; + custom/freeform arguments are exact strings. +- `tool_result_correlates`: expected is `{call,result}` pointers. IDs match, + result follows call, and no intervening call reuses the ID. +- `fixture_request_matches`: method, normalized path, allowlisted headers and + JSON body equal the literal fixture expectation. +- `normalized_text_equals`: selected string equals expected exactly. +- `byte_limit_observed`: selected nonnegative integer is `<=` expected. +- `process_exit_equals`: selected integer equals expected. +- `verifier_result_equals`: selected value `pass|fail|blocked|inconclusive` + equals expected. + +Unknown operators, selectors, expected shapes, fixture roles, media types, or +requirements reject registration. + +## 7. Closed verifier derivations + +`/verifiers` is populated only by these pure V1 functions. They may read the +current case's decoded synthetic fixture and normalized observation, but no +clock, random source, network, filesystem, environment, runtime callback, or +model output outside that observation. + +- `json_sse_equivalence`: build the JSON projection + `{text,terminal}` where `text` concatenates, in order, every + `output[].content[]` `output_text.text`, and `terminal` is top-level + `status`. Build the SSE projection where `text` concatenates every + `response.output_text.delta.data.delta`, and `terminal` is the normalized + terminal. Return `pass` iff the two JCS objects are equal, else `fail`. +- `nonoverlap_order`: read normalized `tool_call` events in ordinal order. + Return their IDs only when every ID occurs once, ordinals are contiguous + from zero, and each event contains its complete arguments. Otherwise return + an empty array. +- `call_result_order`: over the normalized two-turn input, return `pass` iff a + `function_call` occurs in turn 1, exactly one `function_call_output` with the + same `call_id` occurs in turn 2, and no result precedes its call; otherwise + `fail`. +- `compaction_replayed`: return `true` iff the one + `context_compaction.encrypted_content` value is accepted into the parser's + normalized compaction slot and is absent from user-visible output; otherwise + `false`. The synthetic value is never decrypted or executed. +- `local_shell_correlated`: return `true` iff the `local_shell_call.call_id` + equals the following `function_call_output.call_id` and neither item invokes + a process; otherwise `false`. +- `tool_search_error`: for the one failed `tool_search_output`, return its exact + `error` string; missing, duplicate, or non-failed items return `null`. +- `modality_path`: return `native` when `requestHasImage` is true and + `modelInputModalities` contains `image`; otherwise return `sidecar` when an + enabled authorized vision sidecar exists; otherwise return `unsupported`. +- `silent_image_drop`: return `true` only when an image-bearing input is + omitted from adapter output without either a `native`/`sidecar` path or the + typed unsupported rejection; otherwise `false`. +- `exact_bound`: UTF-8 encode `exactSchema`; return `pass` iff its byte length + equals `limitBytes`, JSON parsing succeeds, and the complete staged catalogue + commits; otherwise `fail`. +- `one_over_rejected`: UTF-8 encode `overSchema`; return `pass` iff its byte + length equals `limitBytes + 1` and admission rejects it before commit; + otherwise `fail`. +- `partial_commit`: return `true` iff any tool from the rejected one-byte-over + staging transaction is visible in the committed catalogue; otherwise + `false`. +- `stub_received`: the in-memory MCP stub records exactly + `{namespace,name,arguments}` from the one decoded invocation. Duplicate or + missing invocations produce `null`. + +Verifier outputs use only the literal types above. A missing or type-invalid +input returns `fail`, `false`, `[]`, or `null` as specified and therefore fails +the corresponding required assertion; it is not silently repaired. + +## 8. Failure rules and freshness + +The exact ordered `protocol-v1-default` records are in the case authority. +Fixture/manifest integrity and harness failures do not affect compatibility. +Time/resource limits are environmental blockers. Exact unsupported controls +produce `UNSUPPORTED`; the exact expected rejection of a `negative_control` +satisfies that control without producing `UNSUPPORTED`; other required +deterministic mismatches are `protocol_failure`/`DEGRADED`. + +Protocol V1 scenario and suite freshness are both unbounded (`null`) because +their exact scenario, suite, fixture, compatibility-version, adapter, and +behavior digests invalidate behavior changes. A future profile may still set a +stricter maximum age. + +## 9. CL-01 boundary + +CL-01 may materialize and execute only the protocol manifests in the case +authority. It must: + +1. parse the authority as JSON and reject unknown fields; +2. recompute all fixture, scenario, and suite digests; +3. retain the exact case fixture bytes and expanded manifests + content-addressably; +4. execute only loopback mocks, closed adapter vectors, and in-memory inert MCP + stubs; +5. fail registration rather than invent semantics. + +This document and the JSON authority authorize no runner, mock server, ledger, +SQLite projection, fixture extraction from tests, live probe, or Fabric +ingestion implementation in CL-00. diff --git a/devlog/_plan/260807_compatibility_lab/022_protocol_v1_cases.json b/devlog/_plan/260807_compatibility_lab/022_protocol_v1_cases.json new file mode 100644 index 000000000..087f25634 --- /dev/null +++ b/devlog/_plan/260807_compatibility_lab/022_protocol_v1_cases.json @@ -0,0 +1,456 @@ +{ + "schemaVersion": 1, + "authority": "CL-00 design contract; not a runtime registry", + "sourceCommit": "3ad5bb6bd3f76f6879d84b78ea39edd3e01ec296", + "assertionDslVersion": "1.0.0", + "evidenceSchemaVersion": "1.0.0", + "failureRuleSets": { + "protocol-v1-default": [ + { "id": "contract-integrity", "match": ["fixture_digest_mismatch", "manifest_digest_mismatch", "fixture_decode_failure", "harness_failure", "sanitizer_failure"], "classification": "harness_failure", "secondaryCode": "contract_integrity", "verdictEffect": "none", "retry": "never", "expected": false }, + { "id": "time-limit", "match": ["connect_timeout", "first_byte_timeout", "inactivity_timeout", "total_timeout"], "classification": "timeout", "secondaryCode": "scenario_time_limit", "verdictEffect": "none", "retry": "never", "expected": false }, + { "id": "resource-limit", "match": ["request_limit", "input_byte_limit", "output_byte_limit", "output_token_limit", "tool_call_limit", "artifact_byte_limit"], "classification": "budget_exhausted", "secondaryCode": "scenario_resource_limit", "verdictEffect": "none", "retry": "never", "expected": false }, + { "id": "negative-control-exact-rejection", "match": ["negative_control_exact_rejection"], "classification": "capability_failure", "secondaryCode": "expected_negative_control", "verdictEffect": "none", "retry": "never", "expected": true }, + { "id": "exact-unsupported", "match": ["unsupported_control_exact_rejection"], "classification": "capability_failure", "secondaryCode": "deterministic_unsupported", "verdictEffect": "unsupported", "retry": "never", "expected": true }, + { "id": "required-assertion", "match": ["required_assertion_failed"], "classification": "protocol_failure", "secondaryCode": "deterministic_assertion", "verdictEffect": "degraded", "retry": "never", "expected": false }, + { "id": "fallback", "match": ["no_prior_rule"], "classification": "inconclusive", "secondaryCode": "unclassified", "verdictEffect": "none", "retry": "never", "expected": false } + ] + }, + "manifestDefaults": { + "version": "1.0.0", + "suiteVersion": "1.0.0", + "evidenceLayer": "protocol_conformance", + "verificationRole": "required", + "executionMode": "fixture", + "freshness": { "maxAgeMs": null }, + "executionLimits": { + "totalTimeoutMs": 10000, + "connectTimeoutMs": 1000, + "firstByteTimeoutMs": 2000, + "inactivityTimeoutMs": 2000, + "maxRequests": 4, + "maxInputBytes": 1048576, + "maxOutputBytes": 4194304, + "maxOutputTokens": 4096, + "maxToolCalls": 8, + "maxArtifactBytes": 262144 + }, + "artifactPolicy": { + "allowed": ["assertion_report", "sanitized_request_shape", "sanitized_response_shape", "normalized_event_trace", "sanitized_error"], + "perArtifactBytes": 262144, + "aggregateBytes": 1048576, + "retention": "local_contract", + "publicVisibility": "deny", + "redactionProfile": "synthetic_protocol_v1" + }, + "failureRuleSet": "protocol-v1-default" + }, + "cases": [ + { + "id": "responses-core.protocol.request-shape", + "suite": "responses-core", + "capability": "protocol.responses.core", + "requirements": { "inboundProtocols": ["openai-responses"], "upstreamProtocols": ["openai-chat"], "surfaces": ["responses-http"], "requiredClaims": [], "requiredHarnessFeatures": ["adapter_vector"], "platforms": [], "routePreconditions": [] }, + "fixture": { "id": "rsp-request-shape", "role": "adapter_vector", "mediaType": "application/vnd.opencodex.adapter-vector+json", "bytesUtf8": "{\"modelId\":\"fixture-model\",\"context\":{\"messages\":[{\"role\":\"user\",\"content\":\"PING\",\"timestamp\":0}]},\"stream\":false,\"options\":{\"temperature\":0}}", "digest": "ccc7549e8bcfe4e28d0d4a87c14e622ecfb75973600b5eef830d83620c5bd0f8" }, + "assertions": [ + { "id": "method", "operator": "json_path_equals", "selector": "/upstream/requests/0/json/model", "expected": "fixture-model", "required": true }, + { "id": "message", "operator": "json_path_equals", "selector": "/upstream/requests/0/json/messages/0/content", "expected": "PING", "required": true }, + { "id": "temperature", "operator": "json_path_equals", "selector": "/upstream/requests/0/json/temperature", "expected": 0, "required": true } + ] + }, + { + "id": "responses-core.protocol.sse-framing", + "suite": "responses-core", + "capability": "protocol.responses.core", + "requirements": { "inboundProtocols": ["openai-responses"], "upstreamProtocols": ["openai-responses"], "surfaces": ["responses-sse"], "requiredClaims": [], "requiredHarnessFeatures": ["raw_sse_capture"], "platforms": [], "routePreconditions": [] }, + "initiatingRequest": { "id": "rsp-sse-framing-request", "role": "client_request", "mediaType": "application/json", "bytesUtf8": "{\"model\":\"fixture-model\",\"input\":\"PING\",\"stream\":true}", "digest": "c2c39a78b939e6c5182d3206f86aa86d6fd706959de9c37072db759aa510a1f6" }, + "fixture": { "id": "rsp-sse-framing", "role": "upstream_response", "mediaType": "text/event-stream", "bytesUtf8": "data:{\"type\":\"response.output_text.delta\",\"delta\":\"A\"}\n\ndata: null\n\ndata: {\"type\":\"response.output_text.delta\",\"delta\":\"B\"}\n\ndata:{\"type\":\"response.completed\",\"response\":{\"id\":\"resp_fixture\",\"status\":\"completed\"}}\n\n", "digest": "1c384ef32886054d8f15c14cbcbcc9af4a3bed845d6f820691368614d61515e7" }, + "assertions": [ + { "id": "events", "operator": "sse_event_sequence", "selector": "/client/response/events", "expected": ["response.output_text.delta", "response.output_text.delta", "response.completed"], "required": true }, + { "id": "text", "operator": "normalized_text_equals", "selector": "/client/response/normalizedText", "expected": "AB", "required": true }, + { "id": "terminal", "operator": "terminal_signal_equals", "selector": "/client/response/terminal", "expected": "completed", "required": true } + ] + }, + { + "id": "responses-core.protocol.item-lifecycle", + "suite": "responses-core", + "capability": "protocol.responses.core", + "requirements": { "inboundProtocols": ["openai-responses"], "upstreamProtocols": ["openai-responses"], "surfaces": ["responses-sse"], "requiredClaims": [], "requiredHarnessFeatures": ["raw_sse_capture"], "platforms": [], "routePreconditions": [] }, + "initiatingRequest": { "id": "rsp-item-lifecycle-request", "role": "client_request", "mediaType": "application/json", "bytesUtf8": "{\"model\":\"fixture-model\",\"input\":\"PING\",\"stream\":true}", "digest": "c2c39a78b939e6c5182d3206f86aa86d6fd706959de9c37072db759aa510a1f6" }, + "fixture": { "id": "rsp-item-lifecycle", "role": "upstream_response", "mediaType": "text/event-stream", "bytesUtf8": "event: response.output_item.added\ndata: {\"type\":\"response.output_item.added\",\"output_index\":0,\"item\":{\"id\":\"msg_fixture\",\"type\":\"message\",\"status\":\"in_progress\",\"role\":\"assistant\",\"content\":[]}}\n\nevent: response.output_item.done\ndata: {\"type\":\"response.output_item.done\",\"output_index\":0,\"item\":{\"id\":\"msg_fixture\",\"type\":\"message\",\"status\":\"completed\",\"role\":\"assistant\",\"content\":[]}}\n\nevent: response.completed\ndata: {\"type\":\"response.completed\",\"response\":{\"id\":\"resp_fixture\",\"status\":\"completed\",\"output\":[]}}\n\n", "digest": "ef271e8aaa1d63d51d4e7e0d47facadf39603c1ffa2e871865ace3684feead08" }, + "assertions": [ + { "id": "events", "operator": "sse_event_sequence", "selector": "/client/response/events", "expected": ["response.output_item.added", "response.output_item.done", "response.completed"], "required": true }, + { "id": "stable-id", "operator": "id_stable_across_events", "selector": "/client/response/events", "expected": ["/client/response/events/0/data/item/id", "/client/response/events/1/data/item/id"], "required": true }, + { "id": "id-shape", "operator": "id_matches", "selector": "/client/response/events/0/data/item/id", "expected": "responses_message", "required": true } + ] + }, + { + "id": "responses-core.protocol.terminal-state", + "suite": "responses-core", + "capability": "protocol.responses.core", + "requirements": { "inboundProtocols": ["openai-responses"], "upstreamProtocols": ["openai-responses"], "surfaces": ["responses-sse"], "requiredClaims": [], "requiredHarnessFeatures": ["raw_sse_capture"], "platforms": [], "routePreconditions": [] }, + "initiatingRequest": { "id": "rsp-terminal-state-request", "role": "client_request", "mediaType": "application/json", "bytesUtf8": "{\"model\":\"fixture-model\",\"input\":\"PING\",\"stream\":true}", "digest": "c2c39a78b939e6c5182d3206f86aa86d6fd706959de9c37072db759aa510a1f6" }, + "fixture": { "id": "rsp-terminal-state", "role": "upstream_response", "mediaType": "text/event-stream", "bytesUtf8": "event: response.failed\ndata: {\"type\":\"response.failed\",\"response\":{\"id\":\"resp_fixture\",\"status\":\"failed\",\"error\":{\"type\":\"server_error\",\"code\":\"fixture_failure\"}}}\n\n", "digest": "472735364ce0ee28e68192d478ccb658ec8d6a149dba6fe914e5ab35cc1a41d7" }, + "assertions": [ + { "id": "events", "operator": "sse_event_sequence", "selector": "/client/response/events", "expected": ["response.failed"], "required": true }, + { "id": "count", "operator": "sse_event_count", "selector": "/client/response/events", "expected": { "event": "response.failed", "count": 1 }, "required": true }, + { "id": "terminal", "operator": "terminal_signal_equals", "selector": "/client/response/terminal", "expected": "failed", "required": true } + ] + }, + { + "id": "responses-core.protocol.json-sse-equivalence", + "suite": "responses-core", + "capability": "protocol.responses.core", + "requirements": { "inboundProtocols": ["openai-responses"], "upstreamProtocols": ["openai-responses"], "surfaces": ["responses-http", "responses-sse"], "requiredClaims": [], "requiredHarnessFeatures": ["adapter_vector", "raw_sse_capture"], "platforms": [], "routePreconditions": [] }, + "fixture": { "id": "rsp-json-sse-equivalence", "role": "adapter_vector", "mediaType": "application/vnd.opencodex.adapter-vector+json", "bytesUtf8": "{\"json\":{\"id\":\"resp_fixture\",\"status\":\"completed\",\"output\":[{\"id\":\"msg_fixture\",\"type\":\"message\",\"role\":\"assistant\",\"status\":\"completed\",\"content\":[{\"type\":\"output_text\",\"text\":\"OK\"}]}]},\"sse\":\"event: response.output_text.delta\\ndata: {\\\"type\\\":\\\"response.output_text.delta\\\",\\\"delta\\\":\\\"OK\\\"}\\n\\nevent: response.completed\\ndata: {\\\"type\\\":\\\"response.completed\\\",\\\"response\\\":{\\\"id\\\":\\\"resp_fixture\\\",\\\"status\\\":\\\"completed\\\"}}\\n\\n\"}", "digest": "b7288170258b91361530d1dd5a0a818859ff9b6176793554ec8b0ae1177d87cf" }, + "assertions": [ + { "id": "text", "operator": "normalized_text_equals", "selector": "/client/response/normalizedText", "expected": "OK", "required": true }, + { "id": "terminal", "operator": "terminal_signal_equals", "selector": "/client/response/terminal", "expected": "completed", "required": true }, + { "id": "equivalent", "operator": "verifier_result_equals", "selector": "/verifiers/json_sse_equivalence", "expected": "pass", "required": true } + ] + }, + { + "id": "chat-core.protocol.request-mapping", + "suite": "chat-core", + "capability": "protocol.chat.core", + "requirements": { "inboundProtocols": ["openai-responses"], "upstreamProtocols": ["openai-chat"], "surfaces": ["responses-http"], "requiredClaims": [], "requiredHarnessFeatures": ["adapter_vector"], "platforms": [], "routePreconditions": [] }, + "fixture": { "id": "chat-request-mapping", "role": "adapter_vector", "mediaType": "application/vnd.opencodex.adapter-vector+json", "bytesUtf8": "{\"context\":{\"systemPrompt\":[\"SYS\"],\"messages\":[{\"role\":\"developer\",\"content\":\"DEV\",\"timestamp\":0},{\"role\":\"user\",\"content\":\"PING\",\"timestamp\":1}]},\"options\":{\"textFormat\":{\"type\":\"json_object\"}}}", "digest": "0a9c319b3a6dadbf581d0d2185f57527cd28aa57127e0eac91a421735b4c2ad9" }, + "assertions": [ + { "id": "roles", "operator": "json_path_equals", "selector": "/upstream/requests/0/json/messages", "expected": [{"role":"system","content":"SYS"},{"role":"developer","content":"DEV"},{"role":"user","content":"PING"}], "required": true }, + { "id": "format", "operator": "json_path_equals", "selector": "/upstream/requests/0/json/response_format", "expected": {"type":"json_object"}, "required": true } + ] + }, + { + "id": "chat-core.protocol.nonstream-envelope", + "suite": "chat-core", + "capability": "protocol.chat.core", + "requirements": { "inboundProtocols": ["openai-responses"], "upstreamProtocols": ["openai-chat"], "surfaces": ["responses-http"], "requiredClaims": [], "requiredHarnessFeatures": ["adapter_vector"], "platforms": [], "routePreconditions": [] }, + "initiatingRequest": { "id": "chat-nonstream-envelope-request", "role": "client_request", "mediaType": "application/json", "bytesUtf8": "{\"model\":\"fixture-model\",\"input\":\"PING\",\"stream\":false}", "digest": "4f6e495840e4fc80f833aa8cc09c09ee765ae9f8134db70565f3445989892db7" }, + "fixture": { "id": "chat-nonstream-envelope", "role": "upstream_response", "mediaType": "application/json", "bytesUtf8": "{\"id\":\"chatcmpl_fixture\",\"choices\":[{\"index\":0,\"message\":{\"role\":\"assistant\",\"content\":\"OK\"},\"finish_reason\":\"stop\"}],\"usage\":{\"prompt_tokens\":1,\"completion_tokens\":1,\"total_tokens\":2}}", "digest": "4a9a0352daa284e73850ce613b1cc939a534d6a930c7b68c8eb28e3fcca5b248" }, + "assertions": [ + { "id": "status", "operator": "http_status_equals", "selector": "/client/response/status", "expected": 200, "required": true }, + { "id": "text", "operator": "normalized_text_equals", "selector": "/client/response/normalizedText", "expected": "OK", "required": true }, + { "id": "terminal", "operator": "terminal_signal_equals", "selector": "/client/response/terminal", "expected": "done", "required": true } + ] + }, + { + "id": "chat-core.protocol.stream-assembly", + "suite": "chat-core", + "capability": "protocol.chat.core", + "requirements": { "inboundProtocols": ["openai-responses"], "upstreamProtocols": ["openai-chat"], "surfaces": ["responses-sse"], "requiredClaims": [], "requiredHarnessFeatures": ["raw_sse_capture"], "platforms": [], "routePreconditions": [] }, + "initiatingRequest": { "id": "chat-stream-assembly-request", "role": "client_request", "mediaType": "application/json", "bytesUtf8": "{\"model\":\"fixture-model\",\"input\":\"PING\",\"stream\":true}", "digest": "c2c39a78b939e6c5182d3206f86aa86d6fd706959de9c37072db759aa510a1f6" }, + "fixture": { "id": "chat-stream-assembly", "role": "upstream_response", "mediaType": "text/event-stream", "bytesUtf8": "data: {\"choices\":[{\"index\":0,\"delta\":{\"tool_calls\":[{\"index\":0,\"id\":\"call_a\",\"function\":{\"name\":\"alpha\",\"arguments\":\"{\\\"x\\\":\"}},{\"index\":1,\"id\":\"call_b\",\"function\":{\"name\":\"beta\",\"arguments\":\"{\\\"y\\\":\"}}]}}]}\n\ndata: {\"choices\":[{\"index\":0,\"delta\":{\"tool_calls\":[{\"index\":1,\"function\":{\"arguments\":\"2}\"}},{\"index\":0,\"function\":{\"arguments\":\"1}\"}}]},\"finish_reason\":\"tool_calls\"}]}\n\ndata: [DONE]\n\n", "digest": "0085f298a8690aefb74bb09ea2e0cb77703aaf6cfd822c6ce0d4d334ad4b9b3f" }, + "assertions": [ + { "id": "alpha", "operator": "tool_call_equals", "selector": "/client/response/events/0", "expected": {"id":"call_a","name":"alpha","arguments":{"x":1},"kind":"function","ordinal":0}, "required": true }, + { "id": "beta", "operator": "tool_call_equals", "selector": "/client/response/events/1", "expected": {"id":"call_b","name":"beta","arguments":{"y":2},"kind":"function","ordinal":1}, "required": true }, + { "id": "terminal", "operator": "terminal_signal_equals", "selector": "/client/response/terminal", "expected": "done", "required": true } + ] + }, + { + "id": "chat-core.protocol.stream-terminal", + "suite": "chat-core", + "capability": "protocol.chat.core", + "requirements": { "inboundProtocols": ["openai-responses"], "upstreamProtocols": ["openai-chat"], "surfaces": ["responses-sse"], "requiredClaims": [], "requiredHarnessFeatures": ["raw_sse_capture"], "platforms": [], "routePreconditions": [] }, + "initiatingRequest": { "id": "chat-stream-terminal-request", "role": "client_request", "mediaType": "application/json", "bytesUtf8": "{\"model\":\"fixture-model\",\"input\":\"PING\",\"stream\":true}", "digest": "c2c39a78b939e6c5182d3206f86aa86d6fd706959de9c37072db759aa510a1f6" }, + "fixture": { "id": "chat-stream-terminal", "role": "upstream_response", "mediaType": "text/event-stream", "bytesUtf8": "data: {\"choices\":[{\"index\":0,\"delta\":{\"content\":\"OK\"},\"finish_reason\":\"stop\"}]}\n\ndata: [DONE]\n\n", "digest": "6e0e4e8d32d8575db6a09e89c222b16338e1499e940e038599f7a6b5332e59e6" }, + "assertions": [ + { "id": "text", "operator": "normalized_text_equals", "selector": "/client/response/normalizedText", "expected": "OK", "required": true }, + { "id": "terminal", "operator": "terminal_signal_equals", "selector": "/client/response/terminal", "expected": "done", "required": true } + ] + }, + { + "id": "anthropic-core.protocol.request-mapping", + "suite": "anthropic-core", + "capability": "protocol.anthropic.messages.core", + "requirements": { "inboundProtocols": ["anthropic-messages"], "upstreamProtocols": ["openai-responses"], "surfaces": ["anthropic-http"], "requiredClaims": [], "requiredHarnessFeatures": ["adapter_vector"], "platforms": [], "routePreconditions": [] }, + "fixture": { "id": "anthropic-request-mapping", "role": "client_request", "mediaType": "application/json", "bytesUtf8": "{\"model\":\"fixture-model\",\"system\":\"SYS\",\"messages\":[{\"role\":\"user\",\"content\":\"PING\"}],\"max_tokens\":32,\"stream\":false}", "digest": "deeca799f660f413d0cb85263aa332bdc05aa995ccf8c7322f6af43e9bf6a627" }, + "assertions": [ + { "id": "model", "operator": "json_path_equals", "selector": "/upstream/requests/0/json/model", "expected": "fixture-model", "required": true }, + { "id": "system", "operator": "json_path_equals", "selector": "/upstream/requests/0/json/instructions", "expected": "SYS", "required": true }, + { "id": "input", "operator": "json_path_equals", "selector": "/upstream/requests/0/json/input/0/content/0/text", "expected": "PING", "required": true } + ] + }, + { + "id": "anthropic-core.protocol.content-sequence", + "suite": "anthropic-core", + "capability": "protocol.anthropic.messages.core", + "requirements": { "inboundProtocols": ["anthropic-messages"], "upstreamProtocols": ["openai-responses"], "surfaces": ["anthropic-sse"], "requiredClaims": [], "requiredHarnessFeatures": ["raw_sse_capture"], "platforms": [], "routePreconditions": [] }, + "initiatingRequest": { "id": "anthropic-content-sequence-request", "role": "client_request", "mediaType": "application/json", "bytesUtf8": "{\"model\":\"fixture-model\",\"messages\":[{\"role\":\"user\",\"content\":\"PING\"}],\"max_tokens\":32,\"stream\":true}", "digest": "96e15d2044ccaacca81d32bda4157e4baf82ef98c7640f27034e10285f5de8f3" }, + "fixture": { "id": "anthropic-content-sequence", "role": "upstream_response", "mediaType": "text/event-stream", "bytesUtf8": "data:{\"type\":\"response.output_text.delta\",\"delta\":\"OK\"}\n\ndata:{\"type\":\"response.completed\",\"response\":{\"id\":\"resp_fixture\",\"status\":\"completed\",\"usage\":{\"input_tokens\":1,\"output_tokens\":1}}}\n\n", "digest": "1f8148d142038f42fadf4b3e938b45f4313986cbbd6338feac3b8db8f355299a" }, + "assertions": [ + { "id": "events", "operator": "sse_event_sequence", "selector": "/client/response/events", "expected": ["message_start","content_block_start","content_block_delta","content_block_stop","message_delta","message_stop"], "required": true }, + { "id": "terminal", "operator": "terminal_signal_equals", "selector": "/client/response/terminal", "expected": "message_stop", "required": true } + ] + }, + { + "id": "anthropic-core.protocol.tool-round-trip", + "suite": "anthropic-core", + "capability": "protocol.anthropic.messages.core", + "requirements": { "inboundProtocols": ["anthropic-messages"], "upstreamProtocols": ["openai-responses"], "surfaces": ["anthropic-http"], "requiredClaims": ["tools"], "requiredHarnessFeatures": ["adapter_vector"], "platforms": [], "routePreconditions": [] }, + "fixture": { "id": "anthropic-tool-roundtrip", "role": "client_request", "mediaType": "application/json", "bytesUtf8": "{\"model\":\"fixture-model\",\"messages\":[{\"role\":\"assistant\",\"content\":[{\"type\":\"tool_use\",\"id\":\"call_fixture\",\"name\":\"lookup\",\"input\":{\"q\":\"x\"}}]},{\"role\":\"user\",\"content\":[{\"type\":\"tool_result\",\"tool_use_id\":\"call_fixture\",\"content\":\"RESULT\"}]}],\"tools\":[{\"name\":\"lookup\",\"input_schema\":{\"type\":\"object\",\"properties\":{\"q\":{\"type\":\"string\"}},\"required\":[\"q\"]}}],\"max_tokens\":32}", "digest": "f8dfefb427ce81fb4570f83d8d24c91e79350a7a256ecde7e636e0d70fbdff64" }, + "assertions": [ + { "id": "call-id", "operator": "id_correlates", "selector": "/upstream/requests", "expected": ["/upstream/requests/0/json/input/0/call_id","/upstream/requests/0/json/input/1/call_id"], "required": true }, + { "id": "result", "operator": "json_path_equals", "selector": "/upstream/requests/0/json/input/1/output", "expected": "RESULT", "required": true } + ] + }, + { + "id": "anthropic-core.protocol.terminal-errors", + "suite": "anthropic-core", + "capability": "protocol.anthropic.messages.core", + "requirements": { "inboundProtocols": ["anthropic-messages"], "upstreamProtocols": ["openai-responses"], "surfaces": ["anthropic-sse"], "requiredClaims": [], "requiredHarnessFeatures": ["raw_sse_capture"], "platforms": [], "routePreconditions": [] }, + "initiatingRequest": { "id": "anthropic-terminal-error-request", "role": "client_request", "mediaType": "application/json", "bytesUtf8": "{\"model\":\"fixture-model\",\"messages\":[{\"role\":\"user\",\"content\":\"PING\"}],\"max_tokens\":32,\"stream\":true}", "digest": "96e15d2044ccaacca81d32bda4157e4baf82ef98c7640f27034e10285f5de8f3" }, + "fixture": { "id": "anthropic-terminal-error", "role": "upstream_response", "mediaType": "text/event-stream", "bytesUtf8": "event: response.failed\ndata: {\"type\":\"response.failed\",\"response\":{\"status\":\"failed\",\"error\":{\"type\":\"server_error\",\"code\":\"overloaded\",\"message\":\"fixture\"}}}\n\n", "digest": "fad0d0edca35d066e89de5488635a2912930d5dedc79d047759fb6ecc6567718" }, + "assertions": [ + { "id": "events", "operator": "sse_event_sequence", "selector": "/client/response/events", "expected": ["error"], "required": true }, + { "id": "terminal", "operator": "terminal_signal_equals", "selector": "/client/response/terminal", "expected": "failed", "required": true } + ] + }, + { + "id": "tools-core.protocol.function-round-trip", + "suite": "tools-core", + "capability": "tools.round_trip", + "requirements": { "inboundProtocols": ["openai-responses"], "upstreamProtocols": ["openai-chat"], "surfaces": ["responses-http"], "requiredClaims": ["tools"], "requiredHarnessFeatures": ["adapter_vector"], "platforms": [], "routePreconditions": [] }, + "fixture": { "id": "tools-function", "role": "adapter_vector", "mediaType": "application/vnd.opencodex.adapter-vector+json", "bytesUtf8": "{\"tools\":[{\"name\":\"lookup\",\"parameters\":{\"type\":\"object\",\"properties\":{\"q\":{\"type\":\"string\"}},\"required\":[\"q\"]}}],\"upstreamToolCall\":{\"id\":\"call_fixture\",\"name\":\"lookup\",\"arguments\":\"{\\\"q\\\":\\\"x\\\"}\"},\"toolResult\":{\"toolCallId\":\"call_fixture\",\"content\":\"RESULT\"}}", "digest": "9107f4dfdd7da8340c866c9fb6f42854437cebb98592d0510969c810c1eeb0ad" }, + "assertions": [ + { "id": "call", "operator": "tool_call_equals", "selector": "/client/response/events/0", "expected": {"id":"call_fixture","name":"lookup","arguments":{"q":"x"},"kind":"function","ordinal":0}, "required": true }, + { "id": "result", "operator": "tool_result_correlates", "selector": "/upstream/requests", "expected": {"call":"/client/response/events/0/id","result":"/upstream/requests/1/json/input/0/call_id"}, "required": true } + ] + }, + { + "id": "tools-core.protocol.custom-freeform-round-trip", + "suite": "tools-core", + "capability": "tools.round_trip", + "requirements": { "inboundProtocols": ["openai-responses"], "upstreamProtocols": ["openai-responses"], "surfaces": ["responses-http"], "requiredClaims": ["custom_tools"], "requiredHarnessFeatures": ["adapter_vector"], "platforms": [], "routePreconditions": [] }, + "fixture": { "id": "tools-custom", "role": "adapter_vector", "mediaType": "application/vnd.opencodex.adapter-vector+json", "bytesUtf8": "{\"tool\":{\"type\":\"custom\",\"name\":\"apply_patch\",\"format\":{\"type\":\"grammar\",\"syntax\":\"lark\",\"definition\":\"start: /[\\\\s\\\\S]+/\"}},\"call\":{\"id\":\"call_patch\",\"name\":\"apply_patch\",\"input\":\"*** Begin Patch\\n*** End Patch\\n\"},\"output\":{\"call_id\":\"call_patch\",\"output\":\"Done\"}}", "digest": "752750104e99602d9160feaa591bcbfcfd0c8c53fc9feda4a48c3b6813b74d44" }, + "assertions": [ + { "id": "call", "operator": "tool_call_equals", "selector": "/client/response/events/0", "expected": {"id":"call_patch","name":"apply_patch","arguments":"*** Begin Patch\n*** End Patch\n","kind":"custom","ordinal":0}, "required": true }, + { "id": "result", "operator": "tool_result_correlates", "selector": "/upstream/requests", "expected": {"call":"/client/response/events/0/id","result":"/upstream/requests/1/json/input/0/call_id"}, "required": true } + ] + }, + { + "id": "tools-core.protocol.parallel-correlation", + "suite": "tools-core", + "capability": "tools.round_trip", + "requirements": { "inboundProtocols": ["openai-responses"], "upstreamProtocols": ["openai-chat"], "surfaces": ["responses-sse"], "requiredClaims": ["parallel_tools"], "requiredHarnessFeatures": ["raw_sse_capture"], "platforms": [], "routePreconditions": [] }, + "initiatingRequest": { "id": "tools-parallel-request", "role": "client_request", "mediaType": "application/json", "bytesUtf8": "{\"model\":\"fixture-model\",\"input\":\"PING\",\"stream\":true}", "digest": "c2c39a78b939e6c5182d3206f86aa86d6fd706959de9c37072db759aa510a1f6" }, + "fixture": { "id": "tools-parallel", "role": "upstream_response", "mediaType": "text/event-stream", "bytesUtf8": "data:{\"choices\":[{\"delta\":{\"tool_calls\":[{\"index\":0,\"id\":\"call_a\",\"function\":{\"name\":\"a\",\"arguments\":\"{\"}},{\"index\":1,\"id\":\"call_b\",\"function\":{\"name\":\"b\",\"arguments\":\"{\"}}]}}]}\n\ndata:{\"choices\":[{\"delta\":{\"tool_calls\":[{\"index\":1,\"function\":{\"arguments\":\"}\"}},{\"index\":0,\"function\":{\"arguments\":\"}\"}}]},\"finish_reason\":\"tool_calls\"}]}\n\ndata:[DONE]\n\n", "digest": "7a954d390bdf48d0dec3ed2515a5bbbedd4165656fcb7f0643ce743d17bb39f0" }, + "assertions": [ + { "id": "count", "operator": "sse_event_count", "selector": "/client/response/events", "expected": {"event":"tool_call","count":2}, "required": true }, + { "id": "order", "operator": "json_path_equals", "selector": "/verifiers/nonoverlap_order", "expected": ["call_a","call_b"], "required": true } + ] + }, + { + "id": "tools-core.protocol.result-content", + "suite": "tools-core", + "capability": "tools.round_trip", + "requirements": { "inboundProtocols": ["openai-responses"], "upstreamProtocols": ["openai-chat"], "surfaces": ["responses-http"], "requiredClaims": ["tools","image"], "requiredHarnessFeatures": ["adapter_vector"], "platforms": [], "routePreconditions": [] }, + "fixture": { "id": "tools-result-content", "role": "adapter_vector", "mediaType": "application/vnd.opencodex.adapter-vector+json", "bytesUtf8": "{\"callId\":\"call_fixture\",\"content\":[{\"type\":\"input_text\",\"text\":\"RESULT\"},{\"type\":\"input_image\",\"image_url\":\"data:image/png;base64,iVBORw0KGgo=\",\"detail\":\"high\"}],\"isError\":false}", "digest": "ec81d47d3d6a67254afcc21b55f458269d8dd34ab3b3d52a6c12fec9bec814ab" }, + "assertions": [ + { "id": "text", "operator": "json_path_equals", "selector": "/upstream/requests/0/json/messages/0/content", "expected": "RESULT", "required": true }, + { "id": "image", "operator": "json_path_equals", "selector": "/upstream/requests/0/json/messages/1/content/0/image_url/url", "expected": "data:image/png;base64,iVBORw0KGgo=", "required": true } + ] + }, + { + "id": "tools-core.protocol.choice-and-allowed-set", + "suite": "tools-core", + "capability": "tools.round_trip", + "requirements": { "inboundProtocols": ["openai-responses"], "upstreamProtocols": ["openai-chat"], "surfaces": ["responses-http"], "requiredClaims": ["tools"], "requiredHarnessFeatures": ["adapter_vector"], "platforms": [], "routePreconditions": [] }, + "fixture": { "id": "tools-choice", "role": "client_request", "mediaType": "application/json", "bytesUtf8": "{\"model\":\"fixture-model\",\"input\":\"PING\",\"tools\":[{\"type\":\"function\",\"name\":\"alpha\",\"parameters\":{\"type\":\"object\"}},{\"type\":\"function\",\"name\":\"beta\",\"parameters\":{\"type\":\"object\"}}],\"tool_choice\":{\"type\":\"allowed_tools\",\"mode\":\"required\",\"tools\":[{\"type\":\"function\",\"name\":\"beta\"}]}}", "digest": "fe8b6dde44f88cb9e9a7c6b2bb290e2ee57e7ed425ca8249fb4b7804feff148a" }, + "assertions": [ + { "id": "choice", "operator": "json_path_equals", "selector": "/upstream/requests/0/json/tool_choice", "expected": {"type":"function","function":{"name":"beta"}}, "required": true }, + { "id": "set", "operator": "json_path_equals", "selector": "/upstream/requests/0/json/tools", "expected": [{"type":"function","function":{"name":"beta","parameters":{"type":"object"}}}], "required": true } + ] + }, + { + "id": "codex-core.protocol.streaming-turn", + "suite": "codex-core", + "capability": "client.codex.core", + "requirements": { "inboundProtocols": ["openai-responses"], "upstreamProtocols": ["openai-chat"], "surfaces": ["responses-sse"], "requiredClaims": [], "requiredHarnessFeatures": ["raw_sse_capture"], "platforms": [], "routePreconditions": [] }, + "initiatingRequest": { "id": "codex-streaming-request", "role": "client_request", "mediaType": "application/json", "bytesUtf8": "{\"model\":\"fixture-model\",\"input\":\"PING\",\"stream\":true}", "digest": "c2c39a78b939e6c5182d3206f86aa86d6fd706959de9c37072db759aa510a1f6" }, + "fixture": { "id": "codex-streaming", "role": "upstream_response", "mediaType": "text/event-stream", "bytesUtf8": "data:{\"choices\":[{\"delta\":{\"content\":\"OK\"},\"finish_reason\":\"stop\"}],\"usage\":{\"prompt_tokens\":1,\"completion_tokens\":1}}\n\ndata:[DONE]\n\n", "digest": "f109d35734ecca8e71226ff739b6a0783aca283a9b0d0238a7974b2d7fd9af53" }, + "assertions": [ + { "id": "text", "operator": "normalized_text_equals", "selector": "/client/response/normalizedText", "expected": "OK", "required": true }, + { "id": "terminal", "operator": "terminal_signal_equals", "selector": "/client/response/terminal", "expected": "completed", "required": true }, + { "id": "phase", "operator": "json_path_equals", "selector": "/client/response/events/0/data/phase", "expected": "final_answer", "required": true } + ] + }, + { + "id": "codex-core.protocol.apply-patch-turn", + "suite": "codex-core", + "capability": "client.codex.core", + "requirements": { "inboundProtocols": ["openai-responses"], "upstreamProtocols": ["openai-chat"], "surfaces": ["responses-http"], "requiredClaims": ["custom_tools"], "requiredHarnessFeatures": ["adapter_vector"], "platforms": [], "routePreconditions": [] }, + "fixture": { "id": "codex-patch", "role": "adapter_vector", "mediaType": "application/vnd.opencodex.adapter-vector+json", "bytesUtf8": "{\"name\":\"apply_patch\",\"input\":\"*** Begin Patch\\n*** Add File: x\\n+x\\n*** End Patch\\n\",\"callId\":\"call_patch\",\"result\":\"Done\"}", "digest": "668baa1fbea1d7a6556f717467fc3b90a47b2edfaa2ccf0c7950fd30dfe27a81" }, + "assertions": [ + { "id": "call", "operator": "tool_call_equals", "selector": "/client/response/events/0", "expected": {"id":"call_patch","name":"apply_patch","arguments":"*** Begin Patch\n*** Add File: x\n+x\n*** End Patch\n","kind":"custom","ordinal":0}, "required": true }, + { "id": "result", "operator": "tool_result_correlates", "selector": "/upstream/requests", "expected": {"call":"/client/response/events/0/id","result":"/upstream/requests/1/json/input/0/call_id"}, "required": true } + ] + }, + { + "id": "codex-core.protocol.tool-continuation", + "suite": "codex-core", + "capability": "client.codex.core", + "requirements": { "inboundProtocols": ["openai-responses"], "upstreamProtocols": ["openai-responses"], "surfaces": ["responses-http"], "requiredClaims": ["tools"], "requiredHarnessFeatures": ["adapter_vector"], "platforms": [], "routePreconditions": [] }, + "fixture": { "id": "codex-tool-continuation", "role": "adapter_vector", "mediaType": "application/vnd.opencodex.adapter-vector+json", "bytesUtf8": "{\"turn1\":{\"output\":[{\"type\":\"function_call\",\"id\":\"fc_fixture\",\"call_id\":\"call_fixture\",\"name\":\"lookup\",\"arguments\":\"{}\"}]},\"turn2\":{\"input\":[{\"type\":\"function_call_output\",\"call_id\":\"call_fixture\",\"output\":\"RESULT\"}]}}", "digest": "0b1e955829282c51e056e0bd1d6eb88d62fbae1accd52bdda579c3fce9eac205" }, + "assertions": [ + { "id": "correlation", "operator": "id_correlates", "selector": "/upstream/requests", "expected": ["/upstream/requests/0/json/input/0/call_id","/upstream/requests/0/json/input/1/call_id"], "required": true }, + { "id": "order", "operator": "json_path_equals", "selector": "/verifiers/call_result_order", "expected": "pass", "required": true } + ] + }, + { + "id": "codex-core.protocol.previous-response-replay", + "suite": "codex-core", + "capability": "client.codex.core", + "requirements": { "inboundProtocols": ["openai-responses"], "upstreamProtocols": ["openai-responses"], "surfaces": ["responses-http"], "requiredClaims": [], "requiredHarnessFeatures": ["adapter_vector"], "platforms": [], "routePreconditions": [] }, + "fixture": { "id": "codex-replay", "role": "adapter_vector", "mediaType": "application/vnd.opencodex.adapter-vector+json", "bytesUtf8": "{\"stored\":{\"id\":\"resp_prev\",\"input\":[{\"role\":\"user\",\"content\":\"ONE\"}],\"output\":[{\"role\":\"assistant\",\"content\":\"TWO\"}]},\"next\":{\"previous_response_id\":\"resp_prev\",\"input\":[{\"role\":\"user\",\"content\":\"THREE\"}]}}", "digest": "e849a72d9772616a5ca8853bef48fd2f0884fd006b9ac747bd442513ad05e0f4" }, + "assertions": [ + { "id": "expanded", "operator": "json_path_equals", "selector": "/upstream/requests/0/json/input", "expected": [{"role":"user","content":"ONE"},{"role":"assistant","content":"TWO"},{"role":"user","content":"THREE"}], "required": true }, + { "id": "private-id", "operator": "json_path_absent", "selector": "/upstream/requests/0/json/previous_response_id", "expected": true, "required": true } + ] + }, + { + "id": "codex-core.protocol.structured-output", + "suite": "codex-core", + "capability": "client.codex.core", + "requirements": { "inboundProtocols": ["openai-responses"], "upstreamProtocols": ["openai-chat"], "surfaces": ["responses-http"], "requiredClaims": ["structured_output"], "requiredHarnessFeatures": ["adapter_vector"], "platforms": [], "routePreconditions": [] }, + "fixture": { "id": "codex-structured", "role": "client_request", "mediaType": "application/json", "bytesUtf8": "{\"model\":\"fixture-model\",\"input\":\"PING\",\"text\":{\"format\":{\"type\":\"json_schema\",\"name\":\"answer\",\"schema\":{\"type\":\"object\",\"properties\":{\"ok\":{\"type\":\"boolean\"}},\"required\":[\"ok\"],\"additionalProperties\":false},\"strict\":true}}}", "digest": "e6278954535f4d482a9bb1f6c0189ef7aed00294a7bde695747b7898886cf937" }, + "assertions": [ + { "id": "format", "operator": "json_path_equals", "selector": "/upstream/requests/0/json/response_format", "expected": {"type":"json_schema","json_schema":{"name":"answer","schema":{"type":"object","properties":{"ok":{"type":"boolean"}},"required":["ok"],"additionalProperties":false},"strict":true}}, "required": true } + ] + }, + { + "id": "codex-core.protocol.compaction-and-special-items", + "suite": "codex-core", + "capability": "client.codex.core", + "requirements": { "inboundProtocols": ["openai-responses"], "upstreamProtocols": ["openai-chat"], "surfaces": ["responses-http"], "requiredClaims": [], "requiredHarnessFeatures": ["adapter_vector"], "platforms": [], "routePreconditions": [] }, + "fixture": { "id": "codex-special-items", "role": "client_request", "mediaType": "application/json", "bytesUtf8": "{\"model\":\"fixture-model\",\"input\":[{\"type\":\"context_compaction\",\"encrypted_content\":\"ocx1:fixture\"},{\"type\":\"local_shell_call\",\"id\":\"shell_fixture\",\"call_id\":\"call_shell\",\"status\":\"completed\",\"action\":{\"type\":\"exec\",\"command\":[\"echo\",\"ok\"]}},{\"type\":\"function_call_output\",\"call_id\":\"call_shell\",\"output\":\"ok\"},{\"type\":\"tool_search_output\",\"status\":\"failed\",\"error\":\"fixture\"}]}", "digest": "bf61cb0783f288a4dd6b0b8c0f9a2ddb60f02d0f8ea1d2875ea7e3fe1740b043" }, + "assertions": [ + { "id": "compaction", "operator": "json_path_equals", "selector": "/verifiers/compaction_replayed", "expected": true, "required": true }, + { "id": "shell", "operator": "json_path_equals", "selector": "/verifiers/local_shell_correlated", "expected": true, "required": true }, + { "id": "search", "operator": "json_path_equals", "selector": "/verifiers/tool_search_error", "expected": "fixture", "required": true } + ] + }, + { + "id": "vision-core.protocol.input-image", + "suite": "vision-core", + "capability": "modalities.image.input", + "requirements": { "inboundProtocols": ["openai-responses"], "upstreamProtocols": ["openai-chat"], "surfaces": ["responses-http"], "requiredClaims": ["image"], "requiredHarnessFeatures": ["adapter_vector"], "platforms": [], "routePreconditions": [] }, + "fixture": { "id": "vision-input", "role": "client_request", "mediaType": "application/json", "bytesUtf8": "{\"model\":\"fixture-model\",\"input\":[{\"role\":\"user\",\"content\":[{\"type\":\"input_text\",\"text\":\"READ\"},{\"type\":\"input_image\",\"image_url\":\"data:image/png;base64,iVBORw0KGgo=\",\"detail\":\"high\"}]}]}", "digest": "a26ba5209858c3658d698c1dcb6c92845b2e6aae6bab70a7b9ad1cba1d8aa6a5" }, + "assertions": [ + { "id": "text", "operator": "json_path_equals", "selector": "/upstream/requests/0/json/messages/0/content/0/text", "expected": "READ", "required": true }, + { "id": "image", "operator": "json_path_equals", "selector": "/upstream/requests/0/json/messages/0/content/1/image_url", "expected": {"url":"data:image/png;base64,iVBORw0KGgo=","detail":"high"}, "required": true } + ] + }, + { + "id": "vision-core.protocol.tool-result-image", + "suite": "vision-core", + "capability": "modalities.image.input", + "requirements": { "inboundProtocols": ["openai-responses"], "upstreamProtocols": ["openai-chat"], "surfaces": ["responses-http"], "requiredClaims": ["tools","image"], "requiredHarnessFeatures": ["adapter_vector"], "platforms": [], "routePreconditions": [] }, + "fixture": { "id": "vision-tool-result", "role": "adapter_vector", "mediaType": "application/vnd.opencodex.adapter-vector+json", "bytesUtf8": "{\"callId\":\"call_fixture\",\"result\":[{\"type\":\"input_text\",\"text\":\"RESULT\"},{\"type\":\"input_image\",\"image_url\":\"data:image/png;base64,iVBORw0KGgo=\"}]}", "digest": "02c724259bb3c98002842cafad6d890d3dab7db287f1803fd9ce97ec79630a6d" }, + "assertions": [ + { "id": "tool-text", "operator": "json_path_equals", "selector": "/upstream/requests/0/json/messages/0", "expected": {"role":"tool","tool_call_id":"call_fixture","content":"RESULT"}, "required": true }, + { "id": "image-carrier", "operator": "json_path_equals", "selector": "/upstream/requests/0/json/messages/1/content/0/image_url/url", "expected": "data:image/png;base64,iVBORw0KGgo=", "required": true } + ] + }, + { + "id": "vision-core.protocol.modality-gate", + "suite": "vision-core", + "capability": "modalities.image.input", + "verificationRole": "negative_control", + "requirements": { "inboundProtocols": ["openai-responses"], "upstreamProtocols": ["openai-chat"], "surfaces": ["responses-http"], "requiredClaims": [], "requiredHarnessFeatures": ["adapter_vector"], "platforms": [], "routePreconditions": [] }, + "fixture": { "id": "vision-gate", "role": "adapter_vector", "mediaType": "application/vnd.opencodex.adapter-vector+json", "bytesUtf8": "{\"model\":\"text-only\",\"modelInputModalities\":[\"text\"],\"visionSidecar\":{\"enabled\":false},\"requestHasImage\":true}", "digest": "d5b438fb3fad873b0a1bb1b6c91539862e4f3aa8690a963eb6121c5a3229818a" }, + "assertions": [ + { "id": "path", "operator": "json_path_equals", "selector": "/verifiers/modality_path", "expected": "unsupported", "required": true }, + { "id": "no-drop", "operator": "json_path_equals", "selector": "/verifiers/silent_image_drop", "expected": false, "required": true } + ] + }, + { + "id": "reasoning-core.protocol.effort-mapping", + "suite": "reasoning-core", + "capability": "reasoning.round_trip", + "requirements": { "inboundProtocols": ["openai-responses"], "upstreamProtocols": ["openai-chat"], "surfaces": ["responses-http"], "requiredClaims": ["reasoning"], "requiredHarnessFeatures": ["adapter_vector"], "platforms": [], "routePreconditions": [] }, + "fixture": { "id": "reasoning-effort", "role": "adapter_vector", "mediaType": "application/vnd.opencodex.adapter-vector+json", "bytesUtf8": "{\"requested\":\"high\",\"reasoningEffortMap\":{\"high\":\"adaptive\"},\"reasoningWireFormat\":\"gateway-object\"}", "digest": "d9d5cce104809764d5edbc833088a0a9bb3b4d678a4f135353cc5fecf62e8b57" }, + "assertions": [ + { "id": "wire", "operator": "json_path_equals", "selector": "/upstream/requests/0/json/reasoning", "expected": {"effort":"adaptive"}, "required": true }, + { "id": "legacy-absent", "operator": "json_path_absent", "selector": "/upstream/requests/0/json/reasoning_effort", "expected": true, "required": true } + ] + }, + { + "id": "reasoning-core.protocol.summary-stream", + "suite": "reasoning-core", + "capability": "reasoning.round_trip", + "requirements": { "inboundProtocols": ["openai-responses"], "upstreamProtocols": ["openai-responses"], "surfaces": ["responses-sse"], "requiredClaims": ["reasoning"], "requiredHarnessFeatures": ["raw_sse_capture"], "platforms": [], "routePreconditions": [] }, + "initiatingRequest": { "id": "reasoning-summary-request", "role": "client_request", "mediaType": "application/json", "bytesUtf8": "{\"model\":\"fixture-model\",\"input\":\"PING\",\"stream\":true}", "digest": "c2c39a78b939e6c5182d3206f86aa86d6fd706959de9c37072db759aa510a1f6" }, + "fixture": { "id": "reasoning-summary", "role": "upstream_response", "mediaType": "text/event-stream", "bytesUtf8": "event: response.reasoning_summary_part.added\ndata: {\"type\":\"response.reasoning_summary_part.added\",\"item_id\":\"rs_fixture\",\"summary_index\":0,\"part\":{\"type\":\"summary_text\",\"text\":\"\"}}\n\nevent: response.reasoning_summary_text.delta\ndata: {\"type\":\"response.reasoning_summary_text.delta\",\"item_id\":\"rs_fixture\",\"summary_index\":0,\"delta\":\"WHY\"}\n\nevent: response.completed\ndata: {\"type\":\"response.completed\",\"response\":{\"status\":\"completed\"}}\n\n", "digest": "d9c4fa73b67f7a92d9ec55af7ff12b16ddc9870059e5e530ee04006b171367e7" }, + "assertions": [ + { "id": "events", "operator": "sse_event_sequence", "selector": "/client/response/events", "expected": ["response.reasoning_summary_part.added","response.reasoning_summary_text.delta","response.completed"], "required": true }, + { "id": "id", "operator": "id_matches", "selector": "/client/response/events/0/data/item_id", "expected": "responses_reasoning", "required": true } + ] + }, + { + "id": "reasoning-core.protocol.replay", + "suite": "reasoning-core", + "capability": "reasoning.round_trip", + "requirements": { "inboundProtocols": ["openai-responses"], "upstreamProtocols": ["openai-responses"], "surfaces": ["responses-http"], "requiredClaims": ["reasoning"], "requiredHarnessFeatures": ["adapter_vector"], "platforms": [], "routePreconditions": [] }, + "fixture": { "id": "reasoning-replay", "role": "adapter_vector", "mediaType": "application/vnd.opencodex.adapter-vector+json", "bytesUtf8": "{\"turn1\":{\"reasoning\":{\"id\":\"rs_fixture\",\"text\":\"PLAN\",\"signature\":\"sig_fixture\"},\"toolCall\":{\"callId\":\"call_fixture\"}},\"turn2\":{\"toolResult\":{\"callId\":\"call_fixture\",\"output\":\"RESULT\"}}}", "digest": "6e137e06f52c32e9f7d394b92343a8b849103328e958ab7c9b2825a799ea60c3" }, + "assertions": [ + { "id": "text", "operator": "json_path_equals", "selector": "/upstream/requests/1/json/input/0/content/0/text", "expected": "PLAN", "required": true }, + { "id": "signature", "operator": "json_path_equals", "selector": "/upstream/requests/1/json/input/0/signature", "expected": "sig_fixture", "required": true } + ] + }, + { + "id": "reasoning-core.protocol.private-content-isolation", + "suite": "reasoning-core", + "capability": "reasoning.round_trip", + "requirements": { "inboundProtocols": ["openai-responses"], "upstreamProtocols": ["openai-chat"], "surfaces": ["responses-http"], "requiredClaims": ["reasoning"], "requiredHarnessFeatures": ["adapter_vector"], "platforms": [], "routePreconditions": [] }, + "fixture": { "id": "reasoning-private", "role": "adapter_vector", "mediaType": "application/vnd.opencodex.adapter-vector+json", "bytesUtf8": "{\"origin\":{\"provider\":\"alpha\",\"encrypted\":\"opaque_fixture\"},\"destination\":{\"provider\":\"beta\",\"adapter\":\"openai-chat\"}}", "digest": "3519bd299fe2cd5b8069e0cd1c3b65b61d5ce9588c66e54b49d42af5ccf0c81e" }, + "assertions": [ + { "id": "upstream-absent", "operator": "json_path_absent", "selector": "/upstream/requests/0/json/encrypted_content", "expected": true, "required": true }, + { "id": "client-absent", "operator": "json_path_absent", "selector": "/client/response/json/hidden_reasoning", "expected": true, "required": true } + ] + }, + { + "id": "mcp-core.protocol.namespace-mapping", + "suite": "mcp-core", + "capability": "tools.mcp.core", + "requirements": { "inboundProtocols": ["openai-responses"], "upstreamProtocols": ["cursor-protobuf"], "surfaces": ["responses-http"], "requiredClaims": ["mcp"], "requiredHarnessFeatures": ["adapter_vector","in_memory_mcp_stub"], "platforms": [], "routePreconditions": [] }, + "fixture": { "id": "mcp-namespace", "role": "synthetic_tool", "mediaType": "application/vnd.opencodex.mcp-stub+json", "bytesUtf8": "{\"namespace\":\"mcp__fixture\",\"name\":\"lookup\",\"description\":\"fixture\",\"inputSchema\":{\"type\":\"object\",\"properties\":{\"q\":{\"type\":\"string\"}}}}", "digest": "91a53f8c580d461d0f5e0d7209e5d4b95249bdfa8bd3fd4f298e18bdeadb0693" }, + "assertions": [ + { "id": "wire-name", "operator": "json_path_equals", "selector": "/upstream/requests/0/json/tools/0/name", "expected": "mcp__fixture__lookup", "required": true }, + { "id": "reverse", "operator": "json_path_equals", "selector": "/client/response/events/0", "expected": {"namespace":"mcp__fixture","name":"lookup"}, "required": true } + ] + }, + { + "id": "mcp-core.protocol.schema-and-bounds", + "suite": "mcp-core", + "capability": "tools.mcp.core", + "requirements": { "inboundProtocols": ["openai-responses"], "upstreamProtocols": ["cursor-protobuf"], "surfaces": ["responses-http"], "requiredClaims": ["mcp"], "requiredHarnessFeatures": ["adapter_vector","in_memory_mcp_stub"], "platforms": [], "routePreconditions": [] }, + "fixture": { "id": "mcp-bounds", "role": "synthetic_tool", "mediaType": "application/vnd.opencodex.mcp-stub+json", "bytesUtf8": "{\"limitBytes\":64,\"exactSchema\":\"{\\\"type\\\":\\\"object\\\",\\\"properties\\\":{\\\"x\\\":{\\\"type\\\":\\\"string\\\"}},\\\"a\\\":\\\"xxx\\\"}\",\"overSchema\":\"{\\\"type\\\":\\\"object\\\",\\\"properties\\\":{\\\"x\\\":{\\\"type\\\":\\\"string\\\"}},\\\"a\\\":\\\"xxxx\\\"}\"}", "digest": "34ff4414dc8e196d460390557f4fd74c32418ea00710167baff2a0dc1f3b643c" }, + "assertions": [ + { "id": "exact", "operator": "verifier_result_equals", "selector": "/verifiers/exact_bound", "expected": "pass", "required": true }, + { "id": "over", "operator": "verifier_result_equals", "selector": "/verifiers/one_over_rejected", "expected": "pass", "required": true }, + { "id": "atomic", "operator": "json_path_equals", "selector": "/verifiers/partial_commit", "expected": false, "required": true } + ] + }, + { + "id": "mcp-core.protocol.call-result", + "suite": "mcp-core", + "capability": "tools.mcp.core", + "requirements": { "inboundProtocols": ["openai-responses"], "upstreamProtocols": ["cursor-protobuf"], "surfaces": ["responses-http"], "requiredClaims": ["mcp"], "requiredHarnessFeatures": ["adapter_vector","in_memory_mcp_stub"], "platforms": [], "routePreconditions": [] }, + "fixture": { "id": "mcp-call", "role": "synthetic_tool", "mediaType": "application/vnd.opencodex.mcp-stub+json", "bytesUtf8": "{\"namespace\":\"mcp__fixture\",\"name\":\"lookup\",\"arguments\":{\"q\":\"x\"},\"result\":{\"content\":[{\"type\":\"text\",\"text\":\"RESULT\"}],\"isError\":false}}", "digest": "986ef5017fbdb46eb18b30daaffe72aecc93868b7d89b11c3e244dc084f46496" }, + "assertions": [ + { "id": "call", "operator": "json_path_equals", "selector": "/verifiers/stub_received", "expected": {"namespace":"mcp__fixture","name":"lookup","arguments":{"q":"x"}}, "required": true }, + { "id": "result", "operator": "json_path_equals", "selector": "/client/response/json", "expected": {"content":[{"type":"text","text":"RESULT"}],"isError":false}, "required": true } + ] + }, + { + "id": "mcp-core.protocol.resource-round-trip", + "suite": "mcp-core", + "capability": "tools.mcp.core", + "requirements": { "inboundProtocols": ["openai-responses"], "upstreamProtocols": ["cursor-protobuf"], "surfaces": ["responses-http"], "requiredClaims": ["mcp"], "requiredHarnessFeatures": ["adapter_vector","in_memory_mcp_stub"], "platforms": [], "routePreconditions": [] }, + "fixture": { "id": "mcp-resource", "role": "synthetic_tool", "mediaType": "application/vnd.opencodex.mcp-stub+json", "bytesUtf8": "{\"resources\":[{\"uri\":\"fixture://one\",\"name\":\"one\"}],\"read\":{\"uri\":\"fixture://one\",\"contents\":[{\"uri\":\"fixture://one\",\"text\":\"RESOURCE\"}]}}", "digest": "a3f6317374ce92da0155dd14bbf0d5822e8687cbe8ef7968221f23acf8b16aa5" }, + "assertions": [ + { "id": "list", "operator": "json_path_equals", "selector": "/client/response/json/resources", "expected": [{"uri":"fixture://one","name":"one"}], "required": true }, + { "id": "read", "operator": "json_path_equals", "selector": "/client/response/json/contents", "expected": [{"uri":"fixture://one","text":"RESOURCE"}], "required": true } + ] + } + ] +} diff --git a/devlog/_plan/260807_compatibility_lab/030_incident_corpus.md b/devlog/_plan/260807_compatibility_lab/030_incident_corpus.md new file mode 100644 index 000000000..2e17f89e3 --- /dev/null +++ b/devlog/_plan/260807_compatibility_lab/030_incident_corpus.md @@ -0,0 +1,457 @@ +# CL-00 compatibility incident corpus + +These are abstract regression specifications distilled from shipped tests, +public issues, and devlog records. Provider names identify historical evidence, +not special cases to encode in the scenario model. + +Each future fixture must reproduce the observable wire condition with a mock +upstream. CL-00 does not reopen or fix the production incidents. + +`Future mapping` identifies the suite/scenario family that should own the +regression. It does not claim protocol V1 already covers the incident. Only a +literal vector in `022_protocol_v1_cases.json` gates a V1 verdict. +Unrepresented incidents below are reviewed inputs to a later scenario/suite +version amendment; prose and source-test references cannot be inferred into +V1. + +## IC-001 - legal SSE field spacing + +- Incident class: valid SSE framing rejected. +- Historical source: + [#1170](https://github.com/lidge-jun/opencodex/issues/1170), + `devlog/_plan/260807_untouched_bug_stack/010_sse_unspaced_data_fields.md`, + `tests/sse-unspaced-data-fields.test.ts`. +- Observable failure: a parser accepts `data: {...}` but rejects legal + `data:{...}`/`event:name`, trims payload whitespace, or treats a bare + `data:` as malformed. +- Expected behavior: accept both legal forms, strip at most one optional space, + preserve payload whitespace, and handle empty field values consistently on + Responses, Chat, Anthropic and sidecar paths. +- Future mapping: `responses-core.protocol.sse-framing`, + `chat-core.protocol.stream-terminal`, + `anthropic-core.protocol.content-sequence`. +- Classification: `protocol_failure` -> `DEGRADED`. +- Deterministic fixture: yes; paired frames differing only in legal spacing. + +## IC-002 - null and empty SSE data frames + +- Incident class: ignorable frame mishandled as payload or terminal error. +- Historical source: + `devlog/_plan/260808_bug_campaign/020_wp2_sse_frame_contract.md`, + `tests/sse-null-data-frame.test.ts`. +- Observable failure: `data: null`, a bare `data:` field, or a comment frame + crashes decoding, creates a synthetic event, or hides a later valid event. +- Expected behavior: apply each surface's explicit ignorable-frame contract; + continue parsing without fabricating output, while malformed non-null JSON + still fails closed. +- Future mapping: `responses-core.protocol.sse-framing`, + `chat-core.protocol.stream-terminal`, + `anthropic-core.protocol.content-sequence`. +- Classification: `protocol_failure` -> `DEGRADED`. +- Deterministic fixture: yes; null/empty/comment/malformed controls. + +## IC-003 - missing or incorrect terminal stream signal + +- Incident class: clean EOF, `[DONE]`, completion event, and terminal state + confused. +- Historical source: `tests/openai-chat-eof.test.ts`, + `tests/sse-failed-tail.test.ts`, `tests/claude-outbound.test.ts`, + `tests/responses-stream-tool-events.test.ts`, + [#658](https://github.com/lidge-jun/opencodex/issues/658), + [#735](https://github.com/lidge-jun/opencodex/issues/735). +- Observable failure: a stream ends without the protocol-required terminal, + emits more than one terminal, accepts `[DONE]` as a Responses completion + without a terminal event, or maps failed/incomplete to successful end-turn. +- Expected behavior: exactly one surface-correct terminal. Responses remains + strict. A fingerprinted Chat/Anthropic EOF-tolerance contract may complete + only after visible output or a fully assembled tool call and only when no + incomplete call remains; every other deterministic EOF fails closed and + preserves the typed failed/incomplete reason. +- Future mapping: `responses-core.protocol.terminal-state`, + `chat-core.protocol.stream-terminal`, + `anthropic-core.protocol.terminal-errors`, + `codex-core.protocol.streaming-turn`. +- Classification: deterministic close is `protocol_failure` -> `DEGRADED`. +- Deterministic fixture: yes; completed, failed, incomplete, duplicate and + missing-terminal tails. + +## IC-004 - body stall versus protocol truncation + +- Incident class: environmental/provider timeout misreported as incompatibility. +- Historical source: + [#875](https://github.com/lidge-jun/opencodex/issues/875), + [#1065](https://github.com/lidge-jun/opencodex/issues/1065), + `devlog/_plan/260805_bug_stack_campaign/050_issue875_deepseek_flash_stall.md`, + `devlog/_plan/260806_overnight_triage_round2/020_bounded_body_first_byte.md`. +- Observable failure: no first byte or no later body byte arrives before the + deadline; the system labels the model's protocol unsupported, or waits + without a bound. +- Expected behavior: connect, first-byte, inactivity and total deadlines remain + distinguishable. A silent live stall is a blocker, not proof of malformed + protocol. A mock that deliberately closes without terminal data remains + IC-003. +- Future mapping: supplemental timeout controls for + `responses-core.protocol.terminal-state` and every future live suite. +- Classification: `timeout` -> `BLOCKED`; `provider_transient` when an + authoritative transient response exists. +- Deterministic fixture: yes for timeout attribution; no deterministic fixture + can convert an arbitrary live stall into incompatibility evidence. + +## IC-005 - sparse lifecycle snapshots + +- Incident class: incomplete Responses lifecycle snapshots forwarded as valid. +- Historical source: + [#893](https://github.com/lidge-jun/opencodex/issues/893), + `devlog/_plan/260805_bug_stack_campaign/040_issue893_sparse_snapshot_repair.md`, + `tests/responses-snapshot-repair.test.ts`, + `tests/responses-snapshot-repair-server.test.ts`. +- Observable failure: added/done snapshots omit required ID, type, role, status, + output index or closing item, producing a client-invalid lifecycle. +- Expected behavior: preserve a complete canonical lifecycle or apply an + explicitly configured, assertion-visible repair; never claim a sparse stream + is valid without proving the repaired output. +- Future mapping: `responses-core.protocol.item-lifecycle`, + `codex-core.protocol.streaming-turn`. +- Classification: `protocol_failure` -> `DEGRADED`. +- Deterministic fixture: yes; sparse permutations plus no-repair control. + +## IC-006 - invalid, reused, or missing item IDs + +- Incident class: client-facing Responses item identity violates its grammar or + event correlation. +- Historical source: + [#938](https://github.com/lidge-jun/opencodex/issues/938), + `devlog/_plan/260805_bug_stack_campaign/060_issue938_uuid_item_ids.md`, + `tests/responses-item-id-repair.test.ts`, + `tests/deepseek-responses-item-id-repair.test.ts`. +- Observable failure: UUID/placeholder/missing IDs reach a client contract that + requires typed IDs, or added/done events use inconsistent IDs. +- Expected behavior: valid stable IDs on the client surface; any configured + repair is deterministic, type-scoped, and never rewrites function call IDs + or breaks `call_id` correlation. +- Future mapping: `responses-core.protocol.item-lifecycle`, + `codex-core.protocol.streaming-turn`. +- Classification: `protocol_failure` -> `DEGRADED`. +- Deterministic fixture: yes; valid, invalid, reused, missing-terminal and + correlation controls. + +## IC-007 - function schema root normalization + +- Incident class: valid tool rejected because its schema root is missing or + non-object. +- Historical source: + [PR #745](https://github.com/lidge-jun/opencodex/pull/745), + `tests/responses-parser.test.ts`. +- Observable failure: tool definition reaches an object-schema-only upstream + with absent/invalid root shape, or normalization corrupts an already valid + schema. +- Expected behavior: produce the required object root without changing valid + properties/required/additionalProperties semantics. +- Future mapping: `tools-core.protocol.function-round-trip`, + `mcp-core.protocol.schema-and-bounds`. +- Classification: `protocol_failure` -> `DEGRADED`. +- Deterministic fixture: yes; absent, malformed and valid schema controls. + +## IC-008 - custom/freeform tool envelope mismatch + +- Incident class: function-only route or token preset rejects a valid + custom/freeform tool. +- Historical source: + `devlog/_plan/260807_untouched_bug_stack/070_mimo_token_plan_preset.md`, + `tests/responses-parser.test.ts` (exact `apply_patch` envelope), + `tests/responses-tool-groups.test.ts`. +- Observable failure: a custom tool is serialized as a function, its freeform + input/output is JSON-wrapped or dropped, or the route rejects the tool without + an honest unsupported result. +- Expected behavior: preserve the exact custom tool declaration, call and + output grammar, or deterministically classify the exact route unsupported for + custom tools. +- Future mapping: `tools-core.protocol.custom-freeform-round-trip`, + `codex-core.protocol.apply-patch-turn`. +- Classification: malformed translation is `protocol_failure` -> `DEGRADED`; + a proven route contract is `capability_failure` -> `UNSUPPORTED`. +- Deterministic fixture: yes for translation; a future live negative control + proves route support. + +## IC-009 - dangling tool calls and result correlation + +- Incident class: tool call/result pair becomes orphaned or misassociated. +- Historical source: + `devlog/_fin/260718_dangling_toolcall_hardening/010_record.md`, + `tests/openai-chat-dangling-toolcalls.test.ts`, + `tests/issue-702-expired-replay-state.test.ts`, + [#334](https://github.com/lidge-jun/opencodex/issues/334), + [#620](https://github.com/lidge-jun/opencodex/issues/620). +- Observable failure: an assistant tool call is forwarded without a matching + result, a result is attached to the wrong ID, or expired continuation state + resurrects an unrelated call. +- Expected behavior: preserve exact call/result identity and order; repair only + the narrowly declared orphan case; otherwise fail closed without fabricating + a successful tool result. +- Future mapping: `tools-core.protocol.function-round-trip`, + `codex-core.protocol.tool-continuation`, + `codex-core.protocol.previous-response-replay`. +- Classification: `protocol_failure` -> `DEGRADED`. +- Deterministic fixture: yes; missing, duplicate, out-of-order, expired and + mismatched IDs. + +## IC-010 - parallel tool fragment assembly + +- Incident class: interleaved calls merged, lost, reordered, or correlated to + the wrong result. +- Historical source: + `devlog/_fin/260709_parallel_tool_calls/000_plan.md`, + `tests/openai-chat-parallel-stream.test.ts`, + `tests/parallel-tool-calls-optin.test.ts`, + [#361](https://github.com/lidge-jun/opencodex/issues/361). +- Observable failure: fragmented deltas from two calls produce one argument + buffer, unstable ordering, duplicate completion, or incorrect result IDs. +- Expected behavior: assemble each indexed call independently, never duplicate + argument fragments, preserve stable order/identity, and advertise parallel + capability only when the effective adapter contract supports it. Provider + interleaving does not require overlapping canonical adapter events; atomic + sequential emission is a valid compatibility-preserving bridge contract. +- Future mapping: `tools-core.protocol.parallel-correlation`. +- Classification: `protocol_failure` -> `DEGRADED`; explicit no-parallel + contract -> `UNSUPPORTED` for that capability only. +- Deterministic fixture: yes; interleaved, fragmented, out-of-order and + single-call controls. + +## IC-011 - wrong upstream wire for a model + +- Incident class: Responses-capable and Chat-only models behind one gateway use + the provider-wide wire indiscriminately. +- Historical source: `src/types.ts` and `src/providers/registry.ts` model-wire + contract for [#404](https://github.com/lidge-jun/opencodex/issues/404), + `tests/adapter-resolve.test.ts`, `tests/deepseek-inbound-wire.test.ts`, + `tests/chat-completions-endpoint.test.ts`. +- Observable failure: an exact model is sent to the wrong endpoint/request + shape, producing rejection or silent semantic loss. +- Expected behavior: resolve the effective model-specific adapter before + subject identity and send the declared wire shape. Evidence for one wire is + never reused for the other. +- Future mapping: `responses-core.protocol.request-shape`, + `chat-core.protocol.request-mapping`; future live route variants. +- Classification: deterministic resolver/translation error is + `protocol_failure` -> `DEGRADED`; a correctly selected but unsupported route + is `capability_failure` -> `UNSUPPORTED`. +- Deterministic fixture: yes; mixed gateway with endpoint-specific fixtures. + +## IC-012 - reasoning replay form mismatch + +- Incident class: plaintext reasoning, signature, redacted block, or thought + signature is dropped or replayed in the wrong form. +- Historical source: `tests/deepseek-reasoning-replay.test.ts`, + `tests/deepseek-reasoning-replay-gaps.test.ts`, + `tests/google-antigravity-replay.test.ts`, + `tests/anthropic-thinking-signature.test.ts`, + `tests/kiro-reasoning-roundtrip.test.ts`. +- Observable failure: a second turn is rejected, reasoning text leaks into + visible output, required signature data is lost, or incompatible replay data + is forwarded. +- Expected behavior: use the exact selected adapter's replay contract, preserve + opaque data only on its compatible route, and omit/normalize it safely + elsewhere. +- Future mapping: `reasoning-core.protocol.replay`, + `reasoning-core.protocol.summary-stream`. +- Classification: `protocol_failure` -> `DEGRADED`. +- Deterministic fixture: yes; two-turn fixtures for each abstract replay form. + +## IC-013 - provider-private content crosses a route boundary + +- Incident class: encrypted/task/reasoning content from one provider is sent to + an incompatible provider or exposed as ordinary text. +- Historical source: + [#92](https://github.com/lidge-jun/opencodex/issues/92), + `devlog/_fin/260706_previous-response-id-400/000_plan.md`, + `tests/responses-parser.test.ts` encrypted-content case, + `tests/bridge-raw-reasoning-hidden.test.ts`, + `tests/v2-agent-message-failfast.test.ts`. +- Observable failure: opaque encrypted content is forwarded where it cannot be + decrypted, causes a 400, or becomes user-visible/private evidence. +- Expected behavior: provider-private envelopes remain origin-scoped; cross + route replay fails closed or uses a bounded opaque marker expressly allowed + by the protocol, never raw private data. +- Future mapping: `reasoning-core.protocol.private-content-isolation`, + `codex-core.protocol.previous-response-replay`; a later encrypted-task + capability scenario when its upstream contract is implementable. +- Classification: unsafe translation is `protocol_failure` -> `DEGRADED`; a + route proven unable to consume the encrypted task capability is + `UNSUPPORTED`; the current explicit fail-fast is safe `UNSUPPORTED` evidence + only when the scenario's exact route and encrypted-task preconditions match. + Raw disclosure is also a security Critical independent of compatibility + verdict. +- Deterministic fixture: yes for local origin isolation and fail-fast + mitigation; partial for true cross-provider encrypted task execution. + +## IC-014 - image modality or tool-result image mismatch + +- Incident class: structured image content is dropped, stringified, sent to a + text-only route, or advertised inaccurately. +- Historical source: + [#888](https://github.com/lidge-jun/opencodex/issues/888), + `tests/openai-chat-tool-result-images.test.ts`, + `tests/responses-parser.test.ts`, `tests/vision-anthropic.test.ts`, + `tests/vision-fail-closed.test.ts`, `tests/request-evidence.test.ts`. +- Observable failure: image order/detail/MIME is lost, a tool-result image + becomes raw JSON/text, or capability gating disagrees with the effective + native/sidecar path. +- Expected behavior: preserve structured image parts and honestly choose + native, declared sidecar, or unsupported behavior without silent loss. +- Future mapping: `vision-core.protocol.input-image`, + `vision-core.protocol.tool-result-image`, + `vision-core.protocol.modality-gate`. +- Classification: `protocol_failure` -> `DEGRADED`; proven no-image route -> + `UNSUPPORTED`; sidecar/network unavailability -> `BLOCKED`. +- Deterministic fixture: yes; synthetic data image and text-only controls. + +## IC-015 - malformed continuation and previous-response state + +- Incident class: stateful continuation is forwarded to a stateless/incompatible + route or local replay is incomplete. +- Historical source: + [#702](https://github.com/lidge-jun/opencodex/issues/702), + `devlog/_fin/260706_previous-response-id-400/000_plan.md`, + `tests/responses-state.test.ts`, + `tests/issue-702-expired-replay-state.test.ts`, + `tests/grok-orphan-adoption.test.ts`. +- Observable failure: upstream 400, duplicate history, missing prior tool call, + orphaned result, or continuation state reused after expiry/route change. +- Expected behavior: use valid provider-private continuation only on its exact + compatible subject; otherwise perform bounded ordered local expansion or fail + closed. +- Future mapping: `codex-core.protocol.previous-response-replay`, + `codex-core.protocol.tool-continuation`. +- Classification: `protocol_failure` -> `DEGRADED`. +- Deterministic fixture: yes; stateful, stateless, expired and route-change + matrices. + +## IC-016 - Anthropic terminal/error taxonomy corruption + +- Incident class: failed/incomplete/upstream-overload response appears as + successful `end_turn` or wrong Anthropic error type. +- Historical source: `tests/claude-outbound.test.ts`, + `tests/anthropic-eof-tolerance.test.ts`, + `tests/anthropic-compatible-stream.test.ts`. +- Observable failure: missing `message_stop` is accepted outside a declared + tolerance, transient 502 becomes a normal message, or content-filter/max-token + stop reason is mapped incorrectly. +- Expected behavior: preserve exact content-block and message terminal + sequence; map failure classes and stop reasons deterministically; apply any + EOF tolerance only to its exact fingerprinted route. +- Future mapping: `anthropic-core.protocol.terminal-errors`, + `anthropic-core.protocol.content-sequence`. +- Classification: `protocol_failure` -> `DEGRADED`; live transient -> + `provider_transient` -> `BLOCKED`. +- Deterministic fixture: yes; strict/tolerant, failed, incomplete and transient + controls. + +## IC-017 - MCP namespace, bound, and result atomicity + +- Incident class: namespace collision, oversized schema/result partial commit, + or result type loss. +- Historical source: `tests/cursor-mcp-manager.test.ts`, + `tests/cursor-mcp-stdio.test.ts`. +- Observable failure: flattened names cannot map back, one-byte-over input + leaves a partial catalogue, image/error result changes type, or unknown tool + becomes an untyped exception. +- Expected behavior: collision-safe namespace mapping, exact atomic bounds, and + typed result/error/resource behavior through a Lab-owned stub. +- Future mapping: all `mcp-core.protocol.*` scenarios. +- Classification: `protocol_failure` -> `DEGRADED`; declared no-MCP route -> + `UNSUPPORTED`. +- Deterministic fixture: yes; in-memory/loopback stub only. + +## IC-018 - DNS/connect failure poisons account or capability evidence + +- Incident class: pre-connection transport failure attributed to credentials, + account, model, or capability. +- Historical source: + [#914](https://github.com/lidge-jun/opencodex/issues/914), + `devlog/_plan/260805_bug_stack_campaign/030_issue914_dns_transport_attribution.md`, + `devlog/_plan/260803_transport_attribution/000_plan.md`, + `tests/upstream-connect-error.test.ts`. +- Observable failure: DNS/connect/TLS setup rotates account state, marks a + route capability degraded, or becomes authentication evidence. +- Expected behavior: classify pre-response transport evidence as environment/ + network, leave compatibility and credential capability unchanged, and permit + retry after environment repair. +- Future mapping: blocker controls shared by every future live suite. +- Classification: `network_failure` -> `BLOCKED`. +- Deterministic fixture: yes for attribution using an injected connect failure; + it never contributes a compatibility failure. + +## IC-019 - malformed error or empty success envelope + +- Incident class: upstream error/empty payload accepted as a successful model + response. +- Historical source: `tests/openai-chat-hardening.test.ts`, + `tests/error-fidelity.test.ts`, `tests/upstream-http-error.test.ts`. +- Observable failure: falsey error payload, empty choices, null choice, missing + message, or malformed SSE data is emitted as success or hidden by a terminal. +- Expected behavior: fail closed with a typed normalized error while preserving + any safe usage/status evidence. +- Future mapping: `chat-core.protocol.nonstream-envelope`, + `chat-core.protocol.stream-terminal`. +- Classification: `protocol_failure` -> `DEGRADED` for deterministic malformed + protocol; recognized live transient remains `provider_transient`. +- Deterministic fixture: yes. + +## IC-020 - structured-output wire mismatch + +- Incident class: Responses `text.format` is lost, malformed, or sent to an + upstream in the wrong shape. +- Historical source: `tests/responses-parser.test.ts`, + `tests/openai-chat-hardening.test.ts`, + `tests/deepseek-inbound-wire.test.ts`. +- Observable failure: JSON schema/object request widens to plain text, schema + nesting changes, or a strict unsupported route receives an invalid parameter. +- Expected behavior: preserve the known equivalent wire form, or return a + deterministic unsupported result without pretending structured output was + honored. +- Future mapping: `codex-core.protocol.structured-output`, + `responses-core.protocol.request-shape`, + `chat-core.protocol.request-mapping`. +- Classification: translation error is `protocol_failure` -> `DEGRADED`; + proven route limitation is `capability_failure` -> `UNSUPPORTED`. +- Deterministic fixture: yes for wire translation; future live negative control + for route support. + +## IC-021 - data-only Responses SSE + +- Incident class: valid Responses events rejected because the producer omits + the redundant `event:` field. +- Historical source: + [#700](https://github.com/lidge-jun/opencodex/issues/700), + `tests/claude-outbound.test.ts`. +- Observable failure: a payload with a valid typed Responses JSON record in + `data:` is ignored or treated as a truncated stream when no `event:` line is + present. +- Expected behavior: infer the event name from the payload's canonical `type` + when the surface permits data-only events, permit explicit and inferred + frames to interleave, and keep untyped data-only records ignored/fail-closed + according to the scenario. +- Future mapping: `responses-core.protocol.sse-framing`, + `anthropic-core.protocol.content-sequence`. +- Classification: `protocol_failure` -> `DEGRADED`. +- Deterministic fixture: yes; explicit-only, data-only, mixed and untyped + controls. + +## Corpus maintenance rule + +New incidents enter this corpus only when they add a reusable wire condition, +assertion, or attribution boundary. A provider-specific workaround is not a +scenario. The abstraction must state: + +```text +incident class +historical source/reference +observable failure +expected correct behavior +future scenario/suite mapping +expected failure classification +deterministic fixture feasibility +``` + +When a future fix changes the expected contract, bump the mapped scenario +version and preserve this historical record. diff --git a/devlog/_plan/260807_compatibility_lab/040_security_and_privacy.md b/devlog/_plan/260807_compatibility_lab/040_security_and_privacy.md new file mode 100644 index 000000000..08f236ae1 --- /dev/null +++ b/devlog/_plan/260807_compatibility_lab/040_security_and_privacy.md @@ -0,0 +1,239 @@ +# CL-00 security, privacy, and probe sandbox contract + +Compatibility evidence is useful only if collecting it does not turn the Lab +into a data-exfiltration or arbitrary-execution surface. These requirements are +release blockers for later implementation. + +## 1. Data prohibition + +The Lab must not read, accept, persist, or export: + +- user prompts or conversation history; +- real user repositories, worktrees, patches, source files, or file paths; +- user MCP server definitions, resources, results, or credentials by default; +- arbitrary shell commands or process output; +- arbitrary filesystem contents; +- arbitrary external-network tool requests or responses; +- API keys, OAuth/access/refresh tokens, cookies, authorization material, or + raw credential errors; +- account IDs, account emails, aliases, plan labels, tenant IDs, or other PII; +- raw private/custom headers; +- hidden reasoning, chain of thought, encrypted reasoning payloads, provider + thought signatures, or decrypted private task content. + +Scenarios contain Lab-authored synthetic prompts, fixtures, tool definitions +and results only. They must be recognizable as synthetic and contain no copied +customer material. + +## 2. Future live-probe sandbox + +A live probe is an explicit background/management/CLI action. It is never +started by the production request path, profile evaluator, Router Intelligence, +request-history read, dashboard render, or provider discovery. + +The future runner must enforce a capability-deny sandbox: + +### Network + +- The immutable scenario manifest may authorize only fixed dependency roles + and protocol classes, never a route-local URL. The only remote destinations + are the exact primary endpoint and flat sidecar dependency endpoints named + in the composite route subject, after existing provider destination-policy + validation. +- DNS resolution and every redirect are revalidated. Redirects cannot widen + scheme, host, port, or private-network access. +- Private/loopback endpoints require the route's existing explicit private + network opt-in and an explicit Lab-run confirmation. Metadata endpoints + remain blocked. +- No scenario-supplied URL, model output, tool argument, or redirect may add a + destination. +- A sidecar dependency is allowed only when the scenario explicitly authorizes + its role/protocol class, the composite subject names the exact dependency + fingerprint and endpoint, the operator approves the composite live probe, + and its credential is destination-bound independently. Unmanifested roles or + subject-external/dynamically widened endpoints make the run + `harness_failure`. +- Tools have no network capability. A model-requested web search, image + generation, URL fetch, computer use, or hosted external tool is disabled or + classified inapplicable unless a future separately reviewed scenario owns a + fixed synthetic sidecar. +- Deterministic protocol tests may contact only a Lab-owned loopback mock. + +### Credentials + +- A credential broker resolves the existing route credential immediately + before the request and binds it to the validated destination. +- Credentials exist in memory for the request only and are never included in a + subject, assertion, error, artifact, log, event ID input, or SQLite row. +- Probe code receives no entire auth store and no unrelated provider/account + credential. +- Credential absence/rejection produces `authentication_blocked`, never a + compatibility failure. + +### Process and system access + +- The scenario DSL cannot express a shell command, executable, arbitrary + module, callback, script, filesystem path, or dynamic import. +- The runner receives no general shell/process API and no inherited stdin. +- Filesystem access is restricted to a fresh Lab scratch directory, read-only + packaged synthetic fixtures, and the bounded artifact writer. +- Environment inheritance is an allowlist. Secrets and proxy variables are + supplied only through reviewed destination/credential plumbing, not copied + wholesale. +- The run has enforced wall-clock, inactivity, byte, request, token, tool-call, + memory, process and artifact limits. If the platform cannot enforce a + required boundary, the run fails as `harness_failure`. +- Scratch data is deleted after artifact sanitization. Cleanup failure is + visible and retried by bounded maintenance; it does not silently retain user + data because none was admitted. + +### Tools and MCP + +- Function/custom tool scenarios expose inert Lab-authored definitions. The + harness returns static or pure-function results and never executes model + arguments. +- `apply_patch`, shell, file, browser, web-search, image-generation, computer + use and similar names are protocol tokens only. They do not invoke the real + facility. +- MCP scenarios use an in-memory or Lab-owned loopback stub with fixed schemas, + resources and pure results. User MCP configuration is not loaded. +- Cursor `nativeLocalExec`, `unsafeAllowNativeLocalExec`, desktop executors and + configured `mcpServers` are forced off for the Lab subject. Their disabled + state participates in the behavior fingerprint. + +### Agent Fabric + +- Real task execution remains in Agent Fabric's separately reviewed sandbox. +- The Lab accepts a structured outcome and sanitized content-addressed + references only. +- Outcome ingestion cannot dereference an arbitrary path or URL. Artifact + transfer uses an allowlisted broker and re-runs Lab validation. +- No task repository, prompt transcript, worktree, patch body, terminal log, or + hidden reasoning is copied into `~/.opencodex/lab/`. + +## 3. Artifact contract + +Artifacts are deny-by-default, normalized, sanitized, bounded, and +content-addressed after redaction. + +Initial hard ceilings: + +```text +maximum artifacts per run 16 +maximum bytes per artifact 256 KiB +maximum aggregate artifact data 1 MiB +maximum normalized events 4,096 +maximum sanitized string field 4 KiB +``` + +Scenario limits may be lower. Raising a hard ceiling requires a reviewed +security-contract change; a scenario manifest alone cannot raise it. + +Allowed artifact classes: + +- canonical scenario manifest; +- canonical suite manifest; +- canonical synthetic fixture; +- assertion report containing normalized expected/observed summaries; +- sanitized request shape with content replaced by type/length/digest markers; +- sanitized response shape with visible synthetic fixture output only; +- normalized bounded event trace; +- sanitized error taxonomy/status; +- deterministic verifier summary. + +Artifact paths are derived from the SHA-256 digest and fixed extension under +`~/.opencodex/lab/artifacts/`. Manifests reject traversal, symlinks, +device/special files, alternate data streams, and digest/size mismatch. The +ledger stores relative content-addressed references, never arbitrary paths. + +Scenario/suite manifests and synthetic fixtures use the domain-separated +digests in the evidence contract and remain retained while referenced by any +non-invalidated observation. Their content is still subject to the same +synthetic-data and size rules. + +Redaction occurs before hashing and writing. A redaction failure discards the +artifact and marks the run `harness_failure`; "write now, redact later" is +forbidden. + +## 4. Diagnostic sanitization + +Provider diagnostics retain only: + +- normalized HTTP status; +- allowlisted non-sensitive error type/code; +- coarse phase (`dns`, `connect`, `tls`, `first_byte`, `stream`, `terminal`); +- bounded latency/duration; +- redacted, bounded message selected by an explicit provider sanitizer. + +They remove URLs, query strings, authorization values, header dumps, request/ +response bodies, account identifiers, project/tenant names, local paths, IPs +where identifying, and token-like strings. Unknown provider diagnostics are +reduced to taxonomy and phase rather than persisted verbatim. + +Sanitizers are tested with seeded canary secrets and common credential forms. +`bun run privacy:scan` remains required but is defense in depth, not the +redaction mechanism. + +## 5. Subject privacy + +The local route subject distinguishes exact behavior without raw secrets: + +- configured instance, endpoint, custom headers, project and location use a + per-installation keyed HMAC; +- credential/account identity does not participate; +- raw base URLs and private/custom headers are absent; +- model IDs are retained locally because they are required route identity, but + custom model IDs are private-by-default for export; +- rotating the local subject salt invalidates local correlation and requires + re-projection/reverification, never reverse lookup. + +The salt is stored with secret-file permissions outside the JSONL/artifact +tree. It is not exported. + +## 6. Local evidence versus public export + +Local evidence is already sanitized. Public export is stricter and uses a new, +allowlist-only schema: + +- include suite/scenario versions, evidence layer, verdict, observation time + bucket, public registry provider/model where permitted, assertion summaries, + and public incident/scenario references; +- replace local subject/event/artifact IDs with export-scoped opaque IDs; +- omit endpoint and provider-instance fingerprints, local request/decision/ + Fabric references, precise local paths, custom headers, project/location, + custom provider/model names, account context, raw latency traces, and local + errors; +- include artifact content only when its policy explicitly says + `public_export`; local visibility does not imply export permission; +- run export-specific secret/PII scanning and fail closed on an unknown field. + +Public publishing is not authorized in CL-00 and remains a later phase. + +## 7. Retention and deletion + +- JSONL is the immutable local authority, but a user can delete the entire Lab + directory. Immutability describes in-ledger correction semantics, not a + promise to resist user deletion. +- Artifact retention classes are versioned and bounded by storage policy. + Deleting an expired artifact leaves its digest/reference and a typed + unavailable marker; it does not alter the observation. +- SQLite is disposable and contains no data absent from valid ledger events and + artifact metadata. +- Invalid or sensitive evidence is neutralized by an appended invalidation and + secure artifact deletion. A security incident may require deleting the local + ledger; append-only semantics never override the duty to remove leaked + secrets. + +## 8. Security acceptance tests required later + +Before any live runner ships, tests must prove: + +1. prompt/repository/MCP/user-tool inputs are unreachable from the scenario DSL; +2. redirects and model-supplied URLs cannot widen network access; +3. credential, account, custom header and endpoint canaries never enter + evidence, errors, SQLite or artifacts; +4. tool arguments cannot execute; +5. artifact traversal/symlink/oversize/digest attacks fail closed; +6. timeout, quota, auth, DNS and harness failures remain blockers; +7. public export rejects unknown/private fields; +8. no probe runs from the production routing path. diff --git a/devlog/_plan/260807_compatibility_lab/050_cl00_acceptance_review.md b/devlog/_plan/260807_compatibility_lab/050_cl00_acceptance_review.md new file mode 100644 index 000000000..8e70b2896 --- /dev/null +++ b/devlog/_plan/260807_compatibility_lab/050_cl00_acceptance_review.md @@ -0,0 +1,147 @@ +# CL-00 independent acceptance review + +Date: 2026-08-08 + +Scope: the complete CL-00 contract set on +`feat/cl-00-compatibility-contracts`, based on +`3ad5bb6bd3f76f6879d84b78ea39edd3e01ec296`. + +The review was read-only and separate from the authoring pass. Every validated +Critical, High, and Medium finding was corrected and re-reviewed before this +record was finalized. + +## Findings and corrections + +### Critical + +None. + +### High + +1. Initial scenario prose did not define executable selector/operator semantics + or canonical per-case manifests. + - Correction: added the closed assertion/selector/SSE contract in `021` and + the 35-case machine-readable authority in `022`, with literal fixtures, + expected values, row-specific requirements, roles, media types, limits, + artifact policy, and failure rules. +2. Immutable observations lacked enough manifest/fixture provenance to + reproduce `VERIFIED`. + - Correction: observations now carry scenario, suite, and fixture digests; + domain-separated digest preimages are exact; referenced manifests and + fixtures are retained content-addressably and cannot be replaced by the + current version during replay. +3. Route identity did not close compatibility-version and sidecar-dependent + behavior. + - Correction: froze the compatibility-version manifest/preimage, included + effective runtime and sidecar settings, added flat dependency identities, + and kept route-local endpoints in the composite subject rather than the + provider-independent scenario manifest. +4. Response-only protocol vectors did not identify an initiating client + request. + - Correction: added 11 explicit initiating request fixtures. All + `upstream_response` cases now have one request that fixes model, input, + stream mode, and inbound surface. +5. Named verifier values had no deterministic derivation. + - Correction: defined every V1 verifier as a closed pure function over the + current synthetic fixture and normalized observation. +6. Catalogue prose and incident mappings initially implied coverage beyond the + literal V1 assertions. + - Correction: narrowed every protocol V1 row to its exact `022` evidence and + made incident mappings explicit future scenario/version inputs when no + literal V1 vector exists. + +### Medium + +1. `VERIFIED -> PROBED` was missing after partial invalidation. + - Correction: added the transition for remaining partial coverage. +2. Scenario and suite freshness authorities conflicted. + - Correction: effective age is the minimum finite scenario, suite, and + profile bound. +3. Compatibility-version file hashing and dirty/missing/symlink behavior were + underspecified. + - Correction: froze the canonical object, file set, raw-byte hashes, sort + order, current-working-tree behavior, and fail-closed cases. +4. Sidecar network wording incorrectly put route-local endpoints in scenario + manifests. + - Correction: manifests authorize only dependency roles/protocol classes; + the composite subject owns exact destination fingerprints. +5. The MCP exact-bound vector was not actually at its stated boundary. + - Correction: replaced it with exact 64-byte and 65-byte UTF-8 JSON schema + payloads and a recomputed fixture digest. +6. The vision modality control could have made a compatible suite + `UNSUPPORTED`. + - Correction: made it a `negative_control`; its exact rejection satisfies + the suite without projecting route-level `UNSUPPORTED`. +7. The compaction assertion tested presence rather than truth. + - Correction: changed it to exact equality with `true`. +8. One result-content description claimed call correlation absent from its + assertions. + - Correction: removed the claim. + +### Low + +- Corrected the provider-test description: forward/static providers do not + always perform a live `/models` request. +- Corrected historical reference `#745` from issue to pull request. +- Added `021`/`022` to the stack ledger and created this review record, closing + all local document links. + +## Mechanical review evidence + +- `022_protocol_v1_cases.json` parses as JSON. +- 35 unique cases cover all required members of the eight initial suites. +- 46 fixture artifacts are present: 35 primary vectors and 11 initiating + requests. +- Every fixture digest matches + `sha256("ocx-lab:fixture:v1\0" || UTF8(bytesUtf8))`. +- Every response fixture has one initiating request. +- The MCP bound vector is exactly 64/65 UTF-8 bytes. +- All named verifier selectors have one closed deterministic definition. +- `vision-core.protocol.modality-gate` is the sole V1 negative control and is + represented as such in case and suite expansion. + +## Repository verification + +- `bun run typecheck`: passed. +- `bun run privacy:scan`: passed. +- `bun test tests/repo-hygiene.test.ts`: 11 passed, 0 failed. +- Focused protocol/compatibility suite excluding Windows privileged-symlink + state cases: 395 passed, 0 failed across 24 files. +- Focused continuation-state semantics: 2 passed, 95 filtered, 0 failed. +- Serial isolation of failures observed in the full run: + - `tests/codex-models-cache-invalidate.test.ts`: 6 passed, 0 failed. + - `tests/codex-native-residue.test.ts`: 63 passed, 2 platform skips, + 0 failed. +- Local link validation, canonical case/digest validation, and + `git diff --check`: passed. + +The full `bun run test` result is **not green**. On Windows with Bun 1.3.14 it +exited 3 after a cache-invalidation failure, an empty effective-account lookup, +and a Bun `index out of bounds` panic. A broader focused run separately found +four `responses-state.test.ts` failures, all Windows `EPERM` errors creating +symlinks (488 passed, 4 failed). The isolated cache/native tests and the +non-privileged protocol suite pass, but this review does not claim the full +suite passed. + +## Required challenge results + +1. Protocol conformance, live compatibility, and task effectiveness are + separated: **PASS**. +2. Environmental failures cannot poison compatibility verdicts: **PASS**. +3. `VERIFIED` is reproducible from immutable evidence: **PASS**. +4. Exact route identity prevents false evidence reuse: **PASS**. +5. Routing Profiles remain the sole user-policy layer: **PASS**. +6. The Lab cannot become a second router: **PASS**. +7. The Lab cannot become a second provider registry: **PASS**. +8. Probes cannot access user data or arbitrary tools: **PASS**. +9. Historical incidents are representable as deterministic versioned + scenarios: **PASS**. +10. CL-01 is implementable without semantic invention: **PASS**. + +## Verdict + +No Critical, High, or Medium findings remain. + +**CL-00: ACCEPTED** + +CL-01 remains not started and is not authorized by this review. From e0dcc2e13e4f553116fa76f6f586e7f68908819a Mon Sep 17 00:00:00 2001 From: Wibias <37517432+Wibias@users.noreply.github.com> Date: Sat, 8 Aug 2026 14:51:19 +0200 Subject: [PATCH 02/20] docs(lab): record CL-00 acceptance delivery Record the accepted implementation head and draft pull request so the programme ledger is complete without authorizing CL-01. --- .../_plan/260807_compatibility_lab/001_pr_stack_status.md | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/devlog/_plan/260807_compatibility_lab/001_pr_stack_status.md b/devlog/_plan/260807_compatibility_lab/001_pr_stack_status.md index e084fa66a..b72481e59 100644 --- a/devlog/_plan/260807_compatibility_lab/001_pr_stack_status.md +++ b/devlog/_plan/260807_compatibility_lab/001_pr_stack_status.md @@ -19,7 +19,7 @@ whether the next phase is authorized. | Phase | Branch | Base SHA | Implementation head | PR | State | |---|---|---|---|---|---| -| CL-00 | `feat/cl-00-compatibility-contracts` | `3ad5bb6bd3f76f6879d84b78ea39edd3e01ec296` | pending commit | pending | ACCEPTED | +| CL-00 | `feat/cl-00-compatibility-contracts` | `3ad5bb6bd3f76f6879d84b78ea39edd3e01ec296` | `e2ea3312f83e00c0ce4c2120aaa2f86d7df2b6e4` | [draft #1286](https://github.com/lidge-jun/opencodex/pull/1286) | ACCEPTED | | CL-01 | not created | CL-00 must be accepted first | not started | none | NOT AUTHORIZED | ## CL-00 acceptance log @@ -67,6 +67,7 @@ whether the next phase is authorized. challenges pass and no Critical/High/Medium findings remain. - Blockers: none for CL-00. Full-suite green remains unavailable on this host for the Windows/Bun reasons above. -- CL-00 ending implementation SHA: pending finalization. -- Draft PR: pending. +- CL-00 ending implementation SHA: + `e2ea3312f83e00c0ce4c2120aaa2f86d7df2b6e4`. +- Draft PR: [#1286](https://github.com/lidge-jun/opencodex/pull/1286). - CL-01 authorized: **NO**. Acceptance of CL-00 does not start CL-01. From f2f2837c46a3ed20ca5d787dde7342af6d7c0fc5 Mon Sep 17 00:00:00 2001 From: Wibias <37517432+Wibias@users.noreply.github.com> Date: Sat, 8 Aug 2026 14:52:58 +0200 Subject: [PATCH 03/20] docs(lab): fix master plan whitespace Remove Markdown hard-break spaces so the final branch diff passes repository whitespace checks. --- devlog/_plan/260807_compatibility_lab/000_master_plan.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/devlog/_plan/260807_compatibility_lab/000_master_plan.md b/devlog/_plan/260807_compatibility_lab/000_master_plan.md index fa41e24b4..b8483839d 100644 --- a/devlog/_plan/260807_compatibility_lab/000_master_plan.md +++ b/devlog/_plan/260807_compatibility_lab/000_master_plan.md @@ -1,7 +1,7 @@ # OpenCodex Compatibility Lab / EvalGrid -Status: CL-00 architecture authority -Authority baseline: `upstream/dev` at `3ad5bb6bd3f76f6879d84b78ea39edd3e01ec296` +Status: CL-00 architecture authority +Authority baseline: `upstream/dev` at `3ad5bb6bd3f76f6879d84b78ea39edd3e01ec296` Package/runtime at baseline: OpenCodex `2.10.2`, Bun `1.3.14` ## Purpose From 12e50a3502fb4af25283538cc717ead2291edd8b Mon Sep 17 00:00:00 2001 From: Wibias <37517432+Wibias@users.noreply.github.com> Date: Sat, 8 Aug 2026 22:32:50 +0200 Subject: [PATCH 04/20] docs(lab): close delayed CL-00 acceptance findings Freeze layer-specific subjects, claim supersession, closed behavior keys, scenario-local expectedFailure expansion, and canonical toolCall projections so CL-01 needs no semantic invention. --- .../000_master_plan.md | 21 +- .../001_pr_stack_status.md | 6 +- .../010_architecture_and_evidence_contract.md | 295 +++++++++++++++--- .../020_scenario_contract_and_catalogue.md | 77 ++++- .../021_protocol_v1_manifest_authority.md | 97 +++++- .../022_protocol_v1_cases.json | 37 ++- .../040_security_and_privacy.md | 27 +- .../050_cl00_acceptance_review.md | 40 ++- 8 files changed, 507 insertions(+), 93 deletions(-) diff --git a/devlog/_plan/260807_compatibility_lab/000_master_plan.md b/devlog/_plan/260807_compatibility_lab/000_master_plan.md index b8483839d..fae01e95d 100644 --- a/devlog/_plan/260807_compatibility_lab/000_master_plan.md +++ b/devlog/_plan/260807_compatibility_lab/000_master_plan.md @@ -47,7 +47,8 @@ CL-00 audited the live `dev` tree before defining new authority. `src/server/management/routing-profile-routes.ts`. - Dashboard profile editor, dry-run and routing analytics: `gui/src/pages/RoutingProfiles.tsx`, mounted under Models -> Routing. -- Immutable request/usage evidence and rebuildable history projection: +- Canonical append-only request/usage evidence and rebuildable history + projection: `src/usage/log.ts`, `src/routing/history/indexer.ts`, and `src/routing/analytics.ts`. - Why-this-route evidence: @@ -57,7 +58,8 @@ CL-00 audited the live `dev` tree before defining new authority. `gui/src/pages/log-route-decision.ts`; there is no GUI request-history browser or full trace + attempts + outcome view on this baseline. - Existing diagnostics are narrower than Compatibility Lab: - `ocx doctor` is observe-only environment/OAuth/runtime diagnosis, while + the default `ocx doctor` path is observe-only environment/OAuth/runtime + diagnosis; explicit `--fix-codex-runtime` may persist a repair. In contrast, `POST /api/providers/test` performs a bounded live `/models` connectivity check only when applicable; forward providers return configured status and static catalogues return not-applicable without network access. @@ -143,11 +145,12 @@ changes provider metadata, or turns a probe result directly into a route. ### Routing Profiles -Routing Profiles remain the only user-policy layer. Future compatibility -requirements extend `OcxRoutingProfileConfig`, its normalizer/revision, the -existing evaluator, the existing management CRUD/dry-run endpoints, and the -Models dashboard editor. There will be no compatibility-specific profile -store, evaluator, or editor. +Routing Profiles remain the sole compatibility-policy surface. Future +compatibility requirements extend `OcxRoutingProfileConfig`, its +normalizer/revision, the existing evaluator, the existing management +CRUD/dry-run endpoints, and the Models dashboard editor. Existing combo and +account-pool controls retain their separate non-compatibility responsibilities. +There will be no compatibility-specific profile store, evaluator, or editor. ### Router Intelligence @@ -208,8 +211,8 @@ Future implementation uses the existing OpenCodex config root returned by content-addressed contract artifacts retained with the observations that reference them. - Verdicts are derived projections, never mutable canonical booleans. -- Corrections append invalidation/supersession events; prior bytes are not - edited. +- Corrections append invalidation events or a new claim snapshot with explicit + `supersedes[]`; prior bytes are not edited. - The Lab does not copy `usage.jsonl` or routing-history rows. When useful, an observation references an existing request ID or route decision ID. - Agent Fabric supplies structured outcome data/references, never repositories diff --git a/devlog/_plan/260807_compatibility_lab/001_pr_stack_status.md b/devlog/_plan/260807_compatibility_lab/001_pr_stack_status.md index b72481e59..1676521e9 100644 --- a/devlog/_plan/260807_compatibility_lab/001_pr_stack_status.md +++ b/devlog/_plan/260807_compatibility_lab/001_pr_stack_status.md @@ -19,7 +19,7 @@ whether the next phase is authorized. | Phase | Branch | Base SHA | Implementation head | PR | State | |---|---|---|---|---|---| -| CL-00 | `feat/cl-00-compatibility-contracts` | `3ad5bb6bd3f76f6879d84b78ea39edd3e01ec296` | `e2ea3312f83e00c0ce4c2120aaa2f86d7df2b6e4` | [draft #1286](https://github.com/lidge-jun/opencodex/pull/1286) | ACCEPTED | +| CL-00 | `feat/cl-00-compatibility-contracts` | `3ad5bb6bd3f76f6879d84b78ea39edd3e01ec296` | pending commit after delayed-review corrections | [draft #1286](https://github.com/lidge-jun/opencodex/pull/1286) | ACCEPTED | | CL-01 | not created | CL-00 must be accepted first | not started | none | NOT AUTHORIZED | ## CL-00 acceptance log @@ -67,7 +67,7 @@ whether the next phase is authorized. challenges pass and no Critical/High/Medium findings remain. - Blockers: none for CL-00. Full-suite green remains unavailable on this host for the Windows/Bun reasons above. -- CL-00 ending implementation SHA: - `e2ea3312f83e00c0ce4c2120aaa2f86d7df2b6e4`. +- CL-00 ending implementation SHA: pending commit after delayed-review + corrections from the independent acceptance pass. - Draft PR: [#1286](https://github.com/lidge-jun/opencodex/pull/1286). - CL-01 authorized: **NO**. Acceptance of CL-00 does not start CL-01. diff --git a/devlog/_plan/260807_compatibility_lab/010_architecture_and_evidence_contract.md b/devlog/_plan/260807_compatibility_lab/010_architecture_and_evidence_contract.md index 11c7ba60c..820611af9 100644 --- a/devlog/_plan/260807_compatibility_lab/010_architecture_and_evidence_contract.md +++ b/devlog/_plan/260807_compatibility_lab/010_architecture_and_evidence_contract.md @@ -42,6 +42,19 @@ annotations in a later phase, but cannot produce a canonical verdict. ### Non-collapse rule +Each layer has one executable subject kind: + +```text +protocol_conformance -> ProtocolSubjectV1 +live_route_compatibility -> RouteSubjectV1 +task_effectiveness -> TaskSubjectV1 +``` + +`EvidenceSubjectV1` is that closed discriminated union. An observation whose +layer and `subjectKind` do not match is invalid evidence. Suite manifests also +belong to exactly one evidence layer; a manifest cannot mix protocol, live, and +task scenarios even when their human-facing suite stem is the same. + The canonical projection key is: ```text @@ -54,6 +67,48 @@ Callers may present multiple layer verdicts next to each other. A prerequisite failure in one layer may make a later-layer run inapplicable, but it does not rewrite evidence in the other layer. +### `ProtocolSubjectV1` + +Protocol evidence identifies OpenCodex translation behavior, not a provider: + +```text +subjectSchemaVersion 1 +subjectKind protocol +opencodexCompatibilityVersion +effectiveAdapter +inboundProtocol +upstreamProtocol +surface +behaviorFingerprint +``` + +The closed `runtime.bunVersion`, `runtime.platform`, and `runtime.arch` behavior +keys cover platform-sensitive paths; there is no second runtime digest. +Provider, credential, account, endpoint, health, quota, cost, and latency +fields are forbidden. + +### `TaskSubjectV1` + +Task evidence identifies one exact route plus one synthetic task/verifier +contract: + +```text +subjectSchemaVersion 1 +subjectKind task +routeSubject RouteSubjectV1 +taskClassId +taskClassVersion +taskFixtureDigest +verifierManifestDigest +fabricCompatibilityVersion +sandboxProfileDigest +``` + +The nested route subject makes execution behavior reproducible. The task +subject never contains repository contents, prompts, user paths, account +identity, or raw artifacts. A changed route, task fixture, verifier, Fabric +runtime, or sandbox profile starts a new task projection. + ## 2. Immutable ledger contract The canonical JSONL ledger is a sequence of versioned events. The minimum @@ -111,6 +166,41 @@ artifactRefs[] sourceRefs? ``` +A `claim_snapshot` additionally has: + +```text +evidenceLayer live_route_compatibility +subject RouteSubjectV1 +subjectId +capability +polarity supported | not_supported | withdrawn +sourceManifestDigest +sourceEventIds[] +supersedes[] +effectiveAt +``` + +Claim rules: + +- Claims exist only for `live_route_compatibility`; protocol and task verdicts + cannot be `CLAIMED`. +- `sourceManifestDigest` identifies the canonical, content-addressed snapshot + of all registry/config/catalog/native/adapter inputs used by the existing + capability resolver. Raw secrets are forbidden. +- `supersedes` explicitly lists every previously current claim event for the + same `(subjectId, capability)`. The first snapshot uses an empty list. +- A source refresh, removal, or changed polarity appends a new snapshot; it + never edits a prior event. `withdrawn` means the local declaration no longer + exists. `not_supported` suppresses `CLAIMED` but cannot produce + `UNSUPPORTED`. +- Projection first applies invalidations, then removes every event named by a + valid later snapshot's `supersedes`. Exactly one unsuperseded claim may + remain current for a claim key. Multiple unsuperseded claims, a missing + referenced predecessor, a cross-key supersession, or a cycle makes claim + state `UNKNOWN` and reports ledger corruption. +- Claim ordering is `effectiveAt`, then `recordedAt`, then event ID, but + ordering never substitutes for the explicit supersession graph. + Rules: - Canonical JSON is RFC 8785 JSON Canonicalization Scheme (JCS), encoded as @@ -176,7 +266,7 @@ Verdicts are projections, not mutable evidence fields. ### `UNKNOWN` -Produced when no current registry support claim and no current, valid, +Produced when no current positive claim snapshot and no current, valid, attributable observation can classify the projection key. It can also result when all prior evidence became stale or was invalidated and @@ -191,6 +281,9 @@ stronger state. The snapshot records whether the declaration came from explicit provider config, Provider Registry, cached catalog, native metadata, or current adapter inference rather than pretending every claim is registry-authored. +`CLAIMED` applies only to `live_route_compatibility`; protocol and task layers +have no declaration source and project `UNKNOWN` without executable evidence. + A claim cannot produce `PROBED` or `VERIFIED`. A registry negative declaration is shown as claim metadata but does not by itself prove `UNSUPPORTED`. @@ -251,9 +344,9 @@ A blocked retry does not erase a still-current `VERIFIED`, `PROBED`, Produced only by executable, suite-declared evidence that the exact capability is unavailable by contract for this subject. The scenario must define an -unambiguous unsupported signal or negative-control assertion; a generic 4xx, -timeout, empty output, registry omission, or failed authentication is -insufficient. +unambiguous `capability_absence_control`; a conformance negative control, +generic 4xx, timeout, empty output, registry omission, or failed authentication +is insufficient. Expected rejection of a deliberately unsupported feature can prove `UNSUPPORTED` when the rejection itself matches the deterministic contract. It @@ -268,7 +361,7 @@ For current valid evidence at the same projection key: `UNSUPPORTED` according to the scenario's failure rule. 3. Partial positive coverage yields `PROBED`. 4. A blocker yields `BLOCKED` only when 1-3 have no current result. -5. A current positive registry claim yields `CLAIMED`. +5. A current positive claim snapshot yields `CLAIMED`. 6. Otherwise the result is `UNKNOWN`. This is precedence, not a quality scale. In particular, `DEGRADED`, @@ -389,6 +482,45 @@ codes may add detail without changing these semantics. | `budget_exhausted` | Lab request/token/tool/byte/time budget ended the run before its assertion | No | `BLOCKED`; revise scenario limits/version or retry | | `inconclusive` | Observations conflict or lack enough information for another class | No | No promotion/degradation; investigate/reverify | +Manifest registration enforces this exhaustive class/effect matrix: + +| Class | Legal `verdictEffect` | Additional constraint | +|---|---|---| +| `protocol_failure` | `degraded`, or `none` for a supplemental assertion | Never `unsupported` | +| `capability_failure` | `degraded`, `unsupported`, or `none` for an exact conformance negative control | `unsupported` requires a `capability_absence_control` | +| `behavioral_failure` | `degraded`, or `none` for a supplemental assertion | `task_effectiveness` only | +| `authentication_blocked` | `none` | Environmental blocker only | +| `quota_blocked` | `none` | Environmental blocker only | +| `region_blocked` | `none` | Environmental blocker only | +| `network_failure` | `none` | Environmental blocker only | +| `provider_transient` | `none` | Environmental blocker only | +| `timeout` | `none` | Environmental blocker only | +| `harness_failure` | `none` | Invalidates/blocks evidence only | +| `budget_exhausted` | `none` | Environmental blocker only | +| `inconclusive` | `none` | No verdict promotion or degradation | + +Any other pair rejects the manifest. A retry count cannot change the pair; +reclassification requires a new observation with independently satisfied +classification rules. + +`expectedFailure` is either absent or: + +```text +controlKind conformance_negative_control | + capability_absence_control +expectedClass +expectedCode +assertionIds[] +onMatch pass | unsupported +onMismatch fail | inconclusive +``` + +For `conformance_negative_control`, `onMatch` must be `pass` and the legal +effect is `none`; the expected rejection helps satisfy `VERIFIED`. For +`capability_absence_control`, `onMatch` must be `unsupported` and the class +must be `capability_failure`. No one observation can both satisfy verification +and project `UNSUPPORTED`. + Safety rules: - Expired credentials never imply broken tool support. @@ -403,17 +535,17 @@ Safety rules: - `provider_transient` may be promoted to a compatibility-affecting class only by a scenario-specific deterministic rule and a new observation; projection code must not infer promotion from retry count. -- An expected failure is first-class scenario data. When the observed - rejection exactly matches a declared unsupported assertion, it can produce - `UNSUPPORTED`. Any other expected failure remains a pass/fail of the - assertion, not a blanket suppression. +- Expected failures follow the closed control contract above. A generic + `negative_control` label never implies capability absence. ## 6. Canonical route subject -Evidence is never keyed by model name alone. `RouteSubjectV1` contains: +Live route evidence is never keyed by model name alone. `RouteSubjectV1` +contains: ```text -subjectSchemaVersion +subjectSchemaVersion 1 +subjectKind route providerId providerInstanceFingerprint clientModelId @@ -463,37 +595,112 @@ Semantics: ### Behavior fingerprint allowlist -The fingerprint is SHA-256 over canonical JSON containing only effective, -behavior-changing values applicable to the selected model/surface: - -- adapter/wire resolution and `responsesPath`; -- auth mode and auth transport, but no credential/account identity; -- stateful/stateless Responses behavior, upstream streaming mode, service-tier - support, snapshot and item-ID repair; -- context/input/output limits and input modalities; -- reasoning capability, effort/default/mapping/wire/summary/replay/split/ - toggle/budget behavior; -- tool-choice restrictions, parallel-tool support, hosted-tool preference, - freeform/custom-tool handling, built-in-name escaping; -- prompt-cache forwarding, Anthropic EOF policy, model suffix handling; -- Google mode and opaque project/location fingerprints where applicable; -- OpenRouter routing preferences where applicable; -- actual Bun runtime version and platform/architecture whenever the selected - stream path or adapter has platform-sensitive behavior; -- effective vision and web-search sidecar enablement, backend, model, - reasoning, per-turn limits, timeout/stall limits, and the matching flat - dependency subject when that sidecar can execute; -- MCP schema/result/tool count bounds, with all Lab execution facilities forced - to the sandbox settings in the security contract; -- effective global stream mode, fast/service-tier behavior, and effort caps - when they can alter the scenario; -- a digest of non-credential custom header behavior. +The fingerprint is SHA-256 over JCS `BehaviorFingerprintV1`: + +```text +{ + "schemaVersion": 1, + "resolverVersion": 1, + "values": { + "": { + "source": "request" | "model_override" | "provider_config" | + "registry_runtime_default" | "generated_model_metadata" | + "global_config" | "adapter_default" | "lab_forced", + "value": + } + } +} +``` + +The production route/model/adapter resolver, not a second Lab merge, emits the +effective value and winning source tag. Thus model-specific wire/config +overrides, registry runtime defaults, generated metadata, global values and +adapter defaults retain the exact live precedence. If precedence is ambiguous, +an effective default cannot be resolved, or scenario execution reads a +behavior-changing input that has no closed key below, subject construction +fails as `harness_failure` with code `unclassified_behavior_input`; no evidence +is emitted. + +Closed V1 keys are: + +```text +wire.adapter +wire.upstreamProtocol +wire.responsesPath +wire.commandCodeVersion +wire.modelSuffixMode +auth.mode +auth.transport +responses.stateful +responses.upstreamStreaming +responses.serviceTier +responses.snapshotRepair +responses.itemIdRepair +limits.contextWindow +limits.maxInputTokens +limits.maxOutputTokens +modalities.input +sampling.omitTemperature +sampling.omitTopP +sampling.omitPenalties +reasoning.supported +reasoning.efforts +reasoning.defaultEffort +reasoning.effortMap +reasoning.wireFormat +reasoning.summaryMode +reasoning.replayMode +reasoning.splitMode +reasoning.toggleMode +reasoning.budgetMode +tools.choiceRestrictions +tools.parallel +tools.hostedPreference +tools.customFreeform +tools.builtinNameEscaping +cache.forwarding +cache.retention +anthropic.eofPolicy +google.mode +google.projectFingerprint +google.locationFingerprint +openrouter.order +openrouter.only +openrouter.allowFallbacks +sidecars.vision +sidecars.webSearch +mcp.maxTools +mcp.maxSchemaBytes +mcp.maxResultBytes +mcp.nativeLocalExec +runtime.bunVersion +runtime.platform +runtime.arch +runtime.streamMode +runtime.fastMode +runtime.effortCap +headers.nonCredentialBehaviorDigest +``` + +`sampling.omitTemperature`, `sampling.omitTopP`, and +`sampling.omitPenalties` are the selected-model effective booleans produced +from `noTemperatureModels`, `noTopPModels`, and `noPenaltyModels`, rather than +hashes of the whole configured arrays. `wire.commandCodeVersion` is the +effective `commandCodeVersion`. `cache.retention` is the effective global +`cacheRetention` value. Sidecar keys contain the effective enablement, backend, +model, reasoning, per-turn limits and timeout/stall limits plus the matching +flat dependency subject ID. Values that are inapplicable to the selected model/surface are omitted rather than copied wholesale. Canonical JSON sorts keys, normalizes absent/default values to their effective value, and sorts set-like arrays while preserving order-sensitive arrays. +Acceptance tests for the future builder must enumerate every +behavior-changing config/default read by the selected adapter and prove that +changing each effective value changes the fingerprint, while changing every +excluded secret/transient value does not. + The following never participate: - API keys, OAuth/access/refresh tokens, cookies, authorization headers; @@ -502,10 +709,11 @@ The following never participate: - prompts, messages, tool results, repository paths or contents; - timestamps, transient health, latency, cost, quota or retry state. -Credential headers are excluded. Non-credential custom headers contribute only -through a locally salted HMAC over normalized names/values, so a behavior -change alters identity without disclosing the header. Project/location and -custom endpoint values use the same local opaque treatment. +Credential headers are excluded. The config-owner fingerprint broker described +by the security contract supplies only +`headers.nonCredentialBehaviorDigest`; Lab code never receives raw header names +or values. Project/location and custom endpoint values use the same local +opaque treatment. Public export replaces all local fingerprints with export-scoped opaque IDs and redacts custom model IDs unless the export policy explicitly classifies @@ -520,12 +728,15 @@ producerSchemaVersion outcomeId taskClassId taskClassVersion -subject +subject TaskSubjectV1 +taskFixtureDigest +verifierManifestDigest startedAt completedAt resourceLimits result success | failure | blocked | inconclusive -verifiers[] { id, version, result, normalizedMetrics? } +verifiers[] { id, version, manifestDigest, result, + normalizedMetrics? } artifactRefs[] ``` diff --git a/devlog/_plan/260807_compatibility_lab/020_scenario_contract_and_catalogue.md b/devlog/_plan/260807_compatibility_lab/020_scenario_contract_and_catalogue.md index fa0e610ff..6a97d4c8f 100644 --- a/devlog/_plan/260807_compatibility_lab/020_scenario_contract_and_catalogue.md +++ b/devlog/_plan/260807_compatibility_lab/020_scenario_contract_and_catalogue.md @@ -18,7 +18,7 @@ A `CompatibilityScenarioV1` has: schemaVersion 1 id stable lowercase dotted ID version exact semver -suite { id, version } +suite { id, version, evidenceLayer } evidenceLayer protocol_conformance | live_route_compatibility | task_effectiveness @@ -42,6 +42,10 @@ freshness - Editorial description changes do not require a version change. - A suite manifest has its own exact version and lists scenario IDs, versions, roles, and its verification rule. +- A suite manifest belongs to exactly one `evidenceLayer`; a scenario and suite + with different layers are invalid. Human-facing suite stems may recur across + layers, but their manifest keys are + `(id, evidenceLayer, version, manifestDigest)`. - Scenario and suite manifests are RFC 8785 canonical JSON with the domain-separated digest defined by the evidence contract. @@ -152,6 +156,18 @@ The first exact rule wins. If no rule establishes a compatibility-attributable class, the attempt is `inconclusive`. Generic HTTP 4xx/5xx rules may classify a blocker or transient but cannot prove `UNSUPPORTED`. +Manifest registration must enforce the exhaustive classification/effect matrix +in `010_architecture_and_evidence_contract.md`. Environmental, timeout, +budget, harness, transient, and inconclusive classes permit only +`verdictEffect: none`; a manifest that maps any of them to `degraded` or +`unsupported` is invalid. + +`expectedFailure`, when present, includes `controlKind`, exact class/code, +assertion IDs, `onMatch`, and `onMismatch`. A +`conformance_negative_control` exact rejection is a verification pass with no +verdict effect. A `capability_absence_control` may produce `UNSUPPORTED`. These +meanings cannot be combined in one scenario observation. + ### Artifact policy The policy is deny-by-default and names allowed normalized artifacts: @@ -179,14 +195,15 @@ minimum finite bound, with `null` meaning unbounded at that layer. For each exact suite manifest: - `VERIFIED`: all `required` scenarios applicable to the subject have current - passes and every negative control observed its required rejection. + passes and every `conformance_negative_control` observed its exact required + rejection. - `PROBED`: at least one required scenario passed, with no current compatibility-attributable required-scenario failure, but coverage is incomplete. - `DEGRADED`: a failure rule on an applicable required scenario yields `degraded`. -- `UNSUPPORTED`: a required scenario's deterministic unsupported rule or - negative control proves the capability unavailable. +- `UNSUPPORTED`: a required scenario's exact + `capability_absence_control` proves the capability unavailable. - `BLOCKED`: only blockers exist and no current attributable verdict takes precedence. - `CLAIMED`/`UNKNOWN`: follow the evidence contract. @@ -282,7 +299,8 @@ Capability: `tools.round_trip`. | `tools-core.live.function-round-trip` | Live-reserved | Inert deterministic function is called once with schema-valid args and static result is continued | | `tools-core.live.custom-freeform-round-trip` | Live-reserved | Route emits exact custom/freeform call and accepts static result continuation | -An explicit route contract that rejects a tool kind can prove unsupported. +An explicit route `capability_absence_control` that rejects a tool kind can +prove unsupported. Malformed arguments, dangling IDs, widened choice, dropped calls/results, or incorrect parallel assembly is degraded. A model choosing not to call an `auto` tool is inconclusive; required tool choice is used for conclusive live @@ -330,9 +348,11 @@ Capability: `modalities.image.input`. | `vision-core.protocol.modality-gate` | Deterministic negative control; CL-01 | A text-only/no-sidecar synthetic vector produces the typed unsupported path without silent image drop | | `vision-core.live.synthetic-ocr` | Live-reserved | Lab-generated image nonce is returned in an exact JSON schema | -A deterministic declared no-image contract may prove unsupported. Dropping, -textifying without a declared sidecar, corrupting, or misordering image content -is degraded. Failure of the optional sidecar route is attributed to that exact +A deterministic declared no-image `capability_absence_control` may prove +unsupported. The protocol V1 modality gate is instead a conformance negative +control whose exact rejection is a verification pass. Dropping, textifying +without a declared sidecar, corrupting, or misordering image content is +degraded. Failure of the optional sidecar route is attributed to that exact subject. Auth/quota/network/transient failures are blocked. ### `reasoning-core` @@ -376,6 +396,47 @@ deterministically cannot expose MCP may be unsupported. Namespace loss, schema corruption, partial bound commits, or result miscorrelation is degraded. User-server unavailability is never tested; environmental failures are blocked. +### `fabric-core` task-effectiveness reservation + +This is a distinct `task_effectiveness` suite manifest, not an extension of a +protocol or live-route manifest: + +```text +suite.id fabric-core +suite.version 1.0.0 +suite.evidenceLayer task_effectiveness +verificationRule all-applicable-required-pass-v1 +freshness.maxAgeMs 2592000000 +``` + +The first reserved scenario is +`fabric-core.task.synthetic-patch@1.0.0`: + +- subject: exact `TaskSubjectV1`; +- verification role: `required`; +- fixture: a content-addressed synthetic scratch tree containing + `src/value.txt` with UTF-8 bytes `before\n`, plus a task-class manifest that + requests the exact final bytes `after\n`; +- execution: Fabric-owned, no user repository/prompt, no network, no user MCP, + no shell, and filesystem access restricted to that synthetic scratch tree; +- limits: one file, 64 KiB aggregate input/output, one patch operation, + 30-second total, 5-second inactivity, and 1 MiB aggregate artifacts; +- verifier manifest: `exact-tree-diff-v1`, whose digest participates in + `TaskSubjectV1`; +- deterministic verifier: sort repository-relative POSIX paths by UTF-8 bytes, + reject symlinks/special files/path traversal, hash exact file bytes, and pass + only when the sole diff changes `src/value.txt` from `before\n` to `after\n` + with no added/deleted/renamed file; +- success assertion: verifier result `pass`; +- failure rules: verifier `fail` is `behavioral_failure -> degraded`; + unavailable Fabric/sandbox is `harness_failure -> none`; exhausted time/bytes + is the corresponding blocker with effect `none`; +- artifact policy: retain only the bounded normalized path/digest diff and + verifier summary, never file bodies. + +This reservation freezes task-subject and verifier semantics for a later Fabric +phase. It does not authorize CL-01 to implement or execute the task. + ## 4. CL-01 implementation boundary CL-01 may implement the scenario registry, deterministic mock-upstream harness, diff --git a/devlog/_plan/260807_compatibility_lab/021_protocol_v1_manifest_authority.md b/devlog/_plan/260807_compatibility_lab/021_protocol_v1_manifest_authority.md index 5c5035176..5f098338b 100644 --- a/devlog/_plan/260807_compatibility_lab/021_protocol_v1_manifest_authority.md +++ b/devlog/_plan/260807_compatibility_lab/021_protocol_v1_manifest_authority.md @@ -22,7 +22,9 @@ schemaVersion source.schemaVersion id case.id version manifestDefaults.version suite { id: case.suite, - version: manifestDefaults.suiteVersion } + version: manifestDefaults.suiteVersion, + evidenceLayer: + manifestDefaults.evidenceLayer } evidenceLayer manifestDefaults.evidenceLayer capability case.capability verificationRole case.verificationRole when present, @@ -34,8 +36,8 @@ fixtures when case.initiatingRequest is present: otherwise: [fixtureRef(case.fixture)] executionLimits manifestDefaults.executionLimits assertions case.assertions -failureRules failureRuleSets[ - manifestDefaults.failureRuleSet] +expectedFailure case.expectedFailure, only when present +failureRules expandFailureRules(case) artifactPolicy manifestDefaults.artifactPolicy freshness manifestDefaults.freshness ``` @@ -52,6 +54,31 @@ freshness manifestDefaults.freshness } ``` +`expandFailureRules(case)` copies +`failureRuleSets[manifestDefaults.failureRuleSet]`. When +`case.expectedFailure` is absent, that copy is the complete rule list and no +control-specific rule exists. When present, registration validates the closed +control matrix in the evidence contract and inserts this materialized record +immediately before `required-assertion`: + +```text +id expectedFailureRuleTemplate.id +match expectedFailureRuleTemplate.match +classification case.expectedFailure.expectedClass +secondaryCode case.expectedFailure.expectedCode +verdictEffect none when onMatch is pass, + unsupported when onMatch is unsupported +retry expectedFailureRuleTemplate.retry +expected expectedFailureRuleTemplate.expected +``` + +`expected_failure_exact_match` exists only when the observed class and code +equal `expectedClass`/`expectedCode` and every listed assertion ID passed. An +`onMismatch: fail` falls through to `required-assertion`; an +`onMismatch: inconclusive` falls through to `fallback`. No V1 case declares a +`capability_absence_control`, so no V1 expanded manifest contains an +`unsupported` effect. + JSON object field order has no digest effect, but the field set above is closed. Unknown fields reject registration. Arrays preserve source order. Empty arrays remain present. No default may be read from runtime code. @@ -135,6 +162,8 @@ observation: "headers": {}, "json": null, "events": [], + "toolCalls": [], + "mcpCalls": [], "terminal": null, "normalizedText": "" } @@ -213,6 +242,48 @@ Each normalized event is: { "event": string, "data": JSON value, "ordinal": integer } ``` +### Semantic call projections + +`/client/response/events` always contains the event records above and is never +treated as an array of bare calls. The normalized observation additionally +contains `/client/response/toolCalls` and `/client/response/mcpCalls`. + +`toolCalls` is built from completed client-visible semantic output items in +response order. For non-streaming Responses, use `output[]`; for Responses SSE, +use each `response.output_item.done.data.item` and reject a completed response +whose added item lacks exactly one done item. Chat fragments are first +translated by the adapter into those client-visible Responses items; the +projection never reads the upstream Chat deltas directly. + +Accepted item shapes and exact projections are: + +```text +function_call: + { id: item.call_id, name: item.name, + arguments: JSON.parse(item.arguments), + kind: "function", ordinal: output order } + +custom_tool_call: + { id: item.call_id, name: item.name, + arguments: item.input, + kind: "custom", ordinal: output order } +``` + +Missing/non-string IDs or names, malformed function JSON, duplicate IDs, an +unknown item type, or an added/done mismatch emits no repaired call and causes +the corresponding required assertion to fail. Ordinals are contiguous from +zero; they are not SSE event ordinals. + +`mcpCalls` is derived from `toolCalls` whose name starts with `mcp__` and +contains a final `__` separator. Split at the final separator: + +```text +{ namespace: bytes before final "__", name: bytes after final "__" } +``` + +Empty components, more than 64 UTF-8 bytes per component, invalid UTF-8, or a +non-MCP name emits no MCP call. Order matches `toolCalls`. + ## 6. Assertion operators - `http_status_equals`: selected integer equals expected integer. @@ -270,10 +341,10 @@ model output outside that observation. `status`. Build the SSE projection where `text` concatenates every `response.output_text.delta.data.delta`, and `terminal` is the normalized terminal. Return `pass` iff the two JCS objects are equal, else `fail`. -- `nonoverlap_order`: read normalized `tool_call` events in ordinal order. - Return their IDs only when every ID occurs once, ordinals are contiguous - from zero, and each event contains its complete arguments. Otherwise return - an empty array. +- `nonoverlap_order`: read `/client/response/toolCalls` in array order. Return + their IDs only when every ID occurs once, each record has a complete + arguments value, and ordinals are contiguous from zero and equal the array + indexes. Otherwise return an empty array. - `call_result_order`: over the normalized two-turn input, return `pass` iff a `function_call` occurs in turn 1, exactly one `function_call_output` with the same `call_id` occurs in turn 2, and no result precedes its call; otherwise @@ -312,11 +383,13 @@ the corresponding required assertion; it is not silently repaired. ## 8. Failure rules and freshness -The exact ordered `protocol-v1-default` records are in the case authority. -Fixture/manifest integrity and harness failures do not affect compatibility. -Time/resource limits are environmental blockers. Exact unsupported controls -produce `UNSUPPORTED`; the exact expected rejection of a `negative_control` -satisfies that control without producing `UNSUPPORTED`; other required +The exact ordered base `protocol-v1-default` records and the one +expected-failure rule template are in the case authority. Control-specific +rules are included only by `expandFailureRules(case)`. Fixture/manifest +integrity and harness failures do not affect compatibility. Time/resource +limits are environmental blockers. The exact expected rejection of the V1 +conformance negative control satisfies that control without producing +`UNSUPPORTED`; no protocol V1 case can produce `UNSUPPORTED`. Other required deterministic mismatches are `protocol_failure`/`DEGRADED`. Protocol V1 scenario and suite freshness are both unbounded (`null`) because diff --git a/devlog/_plan/260807_compatibility_lab/022_protocol_v1_cases.json b/devlog/_plan/260807_compatibility_lab/022_protocol_v1_cases.json index 087f25634..0b3fa6e2c 100644 --- a/devlog/_plan/260807_compatibility_lab/022_protocol_v1_cases.json +++ b/devlog/_plan/260807_compatibility_lab/022_protocol_v1_cases.json @@ -9,12 +9,16 @@ { "id": "contract-integrity", "match": ["fixture_digest_mismatch", "manifest_digest_mismatch", "fixture_decode_failure", "harness_failure", "sanitizer_failure"], "classification": "harness_failure", "secondaryCode": "contract_integrity", "verdictEffect": "none", "retry": "never", "expected": false }, { "id": "time-limit", "match": ["connect_timeout", "first_byte_timeout", "inactivity_timeout", "total_timeout"], "classification": "timeout", "secondaryCode": "scenario_time_limit", "verdictEffect": "none", "retry": "never", "expected": false }, { "id": "resource-limit", "match": ["request_limit", "input_byte_limit", "output_byte_limit", "output_token_limit", "tool_call_limit", "artifact_byte_limit"], "classification": "budget_exhausted", "secondaryCode": "scenario_resource_limit", "verdictEffect": "none", "retry": "never", "expected": false }, - { "id": "negative-control-exact-rejection", "match": ["negative_control_exact_rejection"], "classification": "capability_failure", "secondaryCode": "expected_negative_control", "verdictEffect": "none", "retry": "never", "expected": true }, - { "id": "exact-unsupported", "match": ["unsupported_control_exact_rejection"], "classification": "capability_failure", "secondaryCode": "deterministic_unsupported", "verdictEffect": "unsupported", "retry": "never", "expected": true }, { "id": "required-assertion", "match": ["required_assertion_failed"], "classification": "protocol_failure", "secondaryCode": "deterministic_assertion", "verdictEffect": "degraded", "retry": "never", "expected": false }, { "id": "fallback", "match": ["no_prior_rule"], "classification": "inconclusive", "secondaryCode": "unclassified", "verdictEffect": "none", "retry": "never", "expected": false } ] }, + "expectedFailureRuleTemplate": { + "id": "expected-failure-exact-match", + "match": ["expected_failure_exact_match"], + "retry": "never", + "expected": true + }, "manifestDefaults": { "version": "1.0.0", "suiteVersion": "1.0.0", @@ -129,7 +133,7 @@ "assertions": [ { "id": "status", "operator": "http_status_equals", "selector": "/client/response/status", "expected": 200, "required": true }, { "id": "text", "operator": "normalized_text_equals", "selector": "/client/response/normalizedText", "expected": "OK", "required": true }, - { "id": "terminal", "operator": "terminal_signal_equals", "selector": "/client/response/terminal", "expected": "done", "required": true } + { "id": "terminal", "operator": "terminal_signal_equals", "selector": "/client/response/terminal", "expected": "completed", "required": true } ] }, { @@ -140,9 +144,9 @@ "initiatingRequest": { "id": "chat-stream-assembly-request", "role": "client_request", "mediaType": "application/json", "bytesUtf8": "{\"model\":\"fixture-model\",\"input\":\"PING\",\"stream\":true}", "digest": "c2c39a78b939e6c5182d3206f86aa86d6fd706959de9c37072db759aa510a1f6" }, "fixture": { "id": "chat-stream-assembly", "role": "upstream_response", "mediaType": "text/event-stream", "bytesUtf8": "data: {\"choices\":[{\"index\":0,\"delta\":{\"tool_calls\":[{\"index\":0,\"id\":\"call_a\",\"function\":{\"name\":\"alpha\",\"arguments\":\"{\\\"x\\\":\"}},{\"index\":1,\"id\":\"call_b\",\"function\":{\"name\":\"beta\",\"arguments\":\"{\\\"y\\\":\"}}]}}]}\n\ndata: {\"choices\":[{\"index\":0,\"delta\":{\"tool_calls\":[{\"index\":1,\"function\":{\"arguments\":\"2}\"}},{\"index\":0,\"function\":{\"arguments\":\"1}\"}}]},\"finish_reason\":\"tool_calls\"}]}\n\ndata: [DONE]\n\n", "digest": "0085f298a8690aefb74bb09ea2e0cb77703aaf6cfd822c6ce0d4d334ad4b9b3f" }, "assertions": [ - { "id": "alpha", "operator": "tool_call_equals", "selector": "/client/response/events/0", "expected": {"id":"call_a","name":"alpha","arguments":{"x":1},"kind":"function","ordinal":0}, "required": true }, - { "id": "beta", "operator": "tool_call_equals", "selector": "/client/response/events/1", "expected": {"id":"call_b","name":"beta","arguments":{"y":2},"kind":"function","ordinal":1}, "required": true }, - { "id": "terminal", "operator": "terminal_signal_equals", "selector": "/client/response/terminal", "expected": "done", "required": true } + { "id": "alpha", "operator": "tool_call_equals", "selector": "/client/response/toolCalls/0", "expected": {"id":"call_a","name":"alpha","arguments":{"x":1},"kind":"function","ordinal":0}, "required": true }, + { "id": "beta", "operator": "tool_call_equals", "selector": "/client/response/toolCalls/1", "expected": {"id":"call_b","name":"beta","arguments":{"y":2},"kind":"function","ordinal":1}, "required": true }, + { "id": "terminal", "operator": "terminal_signal_equals", "selector": "/client/response/terminal", "expected": "completed", "required": true } ] }, { @@ -154,7 +158,7 @@ "fixture": { "id": "chat-stream-terminal", "role": "upstream_response", "mediaType": "text/event-stream", "bytesUtf8": "data: {\"choices\":[{\"index\":0,\"delta\":{\"content\":\"OK\"},\"finish_reason\":\"stop\"}]}\n\ndata: [DONE]\n\n", "digest": "6e0e4e8d32d8575db6a09e89c222b16338e1499e940e038599f7a6b5332e59e6" }, "assertions": [ { "id": "text", "operator": "normalized_text_equals", "selector": "/client/response/normalizedText", "expected": "OK", "required": true }, - { "id": "terminal", "operator": "terminal_signal_equals", "selector": "/client/response/terminal", "expected": "done", "required": true } + { "id": "terminal", "operator": "terminal_signal_equals", "selector": "/client/response/terminal", "expected": "completed", "required": true } ] }, { @@ -211,8 +215,8 @@ "requirements": { "inboundProtocols": ["openai-responses"], "upstreamProtocols": ["openai-chat"], "surfaces": ["responses-http"], "requiredClaims": ["tools"], "requiredHarnessFeatures": ["adapter_vector"], "platforms": [], "routePreconditions": [] }, "fixture": { "id": "tools-function", "role": "adapter_vector", "mediaType": "application/vnd.opencodex.adapter-vector+json", "bytesUtf8": "{\"tools\":[{\"name\":\"lookup\",\"parameters\":{\"type\":\"object\",\"properties\":{\"q\":{\"type\":\"string\"}},\"required\":[\"q\"]}}],\"upstreamToolCall\":{\"id\":\"call_fixture\",\"name\":\"lookup\",\"arguments\":\"{\\\"q\\\":\\\"x\\\"}\"},\"toolResult\":{\"toolCallId\":\"call_fixture\",\"content\":\"RESULT\"}}", "digest": "9107f4dfdd7da8340c866c9fb6f42854437cebb98592d0510969c810c1eeb0ad" }, "assertions": [ - { "id": "call", "operator": "tool_call_equals", "selector": "/client/response/events/0", "expected": {"id":"call_fixture","name":"lookup","arguments":{"q":"x"},"kind":"function","ordinal":0}, "required": true }, - { "id": "result", "operator": "tool_result_correlates", "selector": "/upstream/requests", "expected": {"call":"/client/response/events/0/id","result":"/upstream/requests/1/json/input/0/call_id"}, "required": true } + { "id": "call", "operator": "tool_call_equals", "selector": "/client/response/toolCalls/0", "expected": {"id":"call_fixture","name":"lookup","arguments":{"q":"x"},"kind":"function","ordinal":0}, "required": true }, + { "id": "result", "operator": "tool_result_correlates", "selector": "/upstream/requests", "expected": {"call":"/client/response/toolCalls/0/id","result":"/upstream/requests/1/json/input/0/call_id"}, "required": true } ] }, { @@ -222,8 +226,8 @@ "requirements": { "inboundProtocols": ["openai-responses"], "upstreamProtocols": ["openai-responses"], "surfaces": ["responses-http"], "requiredClaims": ["custom_tools"], "requiredHarnessFeatures": ["adapter_vector"], "platforms": [], "routePreconditions": [] }, "fixture": { "id": "tools-custom", "role": "adapter_vector", "mediaType": "application/vnd.opencodex.adapter-vector+json", "bytesUtf8": "{\"tool\":{\"type\":\"custom\",\"name\":\"apply_patch\",\"format\":{\"type\":\"grammar\",\"syntax\":\"lark\",\"definition\":\"start: /[\\\\s\\\\S]+/\"}},\"call\":{\"id\":\"call_patch\",\"name\":\"apply_patch\",\"input\":\"*** Begin Patch\\n*** End Patch\\n\"},\"output\":{\"call_id\":\"call_patch\",\"output\":\"Done\"}}", "digest": "752750104e99602d9160feaa591bcbfcfd0c8c53fc9feda4a48c3b6813b74d44" }, "assertions": [ - { "id": "call", "operator": "tool_call_equals", "selector": "/client/response/events/0", "expected": {"id":"call_patch","name":"apply_patch","arguments":"*** Begin Patch\n*** End Patch\n","kind":"custom","ordinal":0}, "required": true }, - { "id": "result", "operator": "tool_result_correlates", "selector": "/upstream/requests", "expected": {"call":"/client/response/events/0/id","result":"/upstream/requests/1/json/input/0/call_id"}, "required": true } + { "id": "call", "operator": "tool_call_equals", "selector": "/client/response/toolCalls/0", "expected": {"id":"call_patch","name":"apply_patch","arguments":"*** Begin Patch\n*** End Patch\n","kind":"custom","ordinal":0}, "required": true }, + { "id": "result", "operator": "tool_result_correlates", "selector": "/upstream/requests", "expected": {"call":"/client/response/toolCalls/0/id","result":"/upstream/requests/1/json/input/0/call_id"}, "required": true } ] }, { @@ -234,7 +238,7 @@ "initiatingRequest": { "id": "tools-parallel-request", "role": "client_request", "mediaType": "application/json", "bytesUtf8": "{\"model\":\"fixture-model\",\"input\":\"PING\",\"stream\":true}", "digest": "c2c39a78b939e6c5182d3206f86aa86d6fd706959de9c37072db759aa510a1f6" }, "fixture": { "id": "tools-parallel", "role": "upstream_response", "mediaType": "text/event-stream", "bytesUtf8": "data:{\"choices\":[{\"delta\":{\"tool_calls\":[{\"index\":0,\"id\":\"call_a\",\"function\":{\"name\":\"a\",\"arguments\":\"{\"}},{\"index\":1,\"id\":\"call_b\",\"function\":{\"name\":\"b\",\"arguments\":\"{\"}}]}}]}\n\ndata:{\"choices\":[{\"delta\":{\"tool_calls\":[{\"index\":1,\"function\":{\"arguments\":\"}\"}},{\"index\":0,\"function\":{\"arguments\":\"}\"}}]},\"finish_reason\":\"tool_calls\"}]}\n\ndata:[DONE]\n\n", "digest": "7a954d390bdf48d0dec3ed2515a5bbbedd4165656fcb7f0643ce743d17bb39f0" }, "assertions": [ - { "id": "count", "operator": "sse_event_count", "selector": "/client/response/events", "expected": {"event":"tool_call","count":2}, "required": true }, + { "id": "calls", "operator": "json_path_equals", "selector": "/client/response/toolCalls", "expected": [{"id":"call_a","name":"a","arguments":{},"kind":"function","ordinal":0},{"id":"call_b","name":"b","arguments":{},"kind":"function","ordinal":1}], "required": true }, { "id": "order", "operator": "json_path_equals", "selector": "/verifiers/nonoverlap_order", "expected": ["call_a","call_b"], "required": true } ] }, @@ -280,8 +284,8 @@ "requirements": { "inboundProtocols": ["openai-responses"], "upstreamProtocols": ["openai-chat"], "surfaces": ["responses-http"], "requiredClaims": ["custom_tools"], "requiredHarnessFeatures": ["adapter_vector"], "platforms": [], "routePreconditions": [] }, "fixture": { "id": "codex-patch", "role": "adapter_vector", "mediaType": "application/vnd.opencodex.adapter-vector+json", "bytesUtf8": "{\"name\":\"apply_patch\",\"input\":\"*** Begin Patch\\n*** Add File: x\\n+x\\n*** End Patch\\n\",\"callId\":\"call_patch\",\"result\":\"Done\"}", "digest": "668baa1fbea1d7a6556f717467fc3b90a47b2edfaa2ccf0c7950fd30dfe27a81" }, "assertions": [ - { "id": "call", "operator": "tool_call_equals", "selector": "/client/response/events/0", "expected": {"id":"call_patch","name":"apply_patch","arguments":"*** Begin Patch\n*** Add File: x\n+x\n*** End Patch\n","kind":"custom","ordinal":0}, "required": true }, - { "id": "result", "operator": "tool_result_correlates", "selector": "/upstream/requests", "expected": {"call":"/client/response/events/0/id","result":"/upstream/requests/1/json/input/0/call_id"}, "required": true } + { "id": "call", "operator": "tool_call_equals", "selector": "/client/response/toolCalls/0", "expected": {"id":"call_patch","name":"apply_patch","arguments":"*** Begin Patch\n*** Add File: x\n+x\n*** End Patch\n","kind":"custom","ordinal":0}, "required": true }, + { "id": "result", "operator": "tool_result_correlates", "selector": "/upstream/requests", "expected": {"call":"/client/response/toolCalls/0/id","result":"/upstream/requests/1/json/input/0/call_id"}, "required": true } ] }, { @@ -360,7 +364,8 @@ "assertions": [ { "id": "path", "operator": "json_path_equals", "selector": "/verifiers/modality_path", "expected": "unsupported", "required": true }, { "id": "no-drop", "operator": "json_path_equals", "selector": "/verifiers/silent_image_drop", "expected": false, "required": true } - ] + ], + "expectedFailure": { "controlKind": "conformance_negative_control", "expectedClass": "capability_failure", "expectedCode": "image_input_unsupported", "assertionIds": ["path", "no-drop"], "onMatch": "pass", "onMismatch": "fail" } }, { "id": "reasoning-core.protocol.effort-mapping", @@ -415,7 +420,7 @@ "fixture": { "id": "mcp-namespace", "role": "synthetic_tool", "mediaType": "application/vnd.opencodex.mcp-stub+json", "bytesUtf8": "{\"namespace\":\"mcp__fixture\",\"name\":\"lookup\",\"description\":\"fixture\",\"inputSchema\":{\"type\":\"object\",\"properties\":{\"q\":{\"type\":\"string\"}}}}", "digest": "91a53f8c580d461d0f5e0d7209e5d4b95249bdfa8bd3fd4f298e18bdeadb0693" }, "assertions": [ { "id": "wire-name", "operator": "json_path_equals", "selector": "/upstream/requests/0/json/tools/0/name", "expected": "mcp__fixture__lookup", "required": true }, - { "id": "reverse", "operator": "json_path_equals", "selector": "/client/response/events/0", "expected": {"namespace":"mcp__fixture","name":"lookup"}, "required": true } + { "id": "reverse", "operator": "json_path_equals", "selector": "/client/response/mcpCalls/0", "expected": {"namespace":"mcp__fixture","name":"lookup"}, "required": true } ] }, { diff --git a/devlog/_plan/260807_compatibility_lab/040_security_and_privacy.md b/devlog/_plan/260807_compatibility_lab/040_security_and_privacy.md index 08f236ae1..d92ee71be 100644 --- a/devlog/_plan/260807_compatibility_lab/040_security_and_privacy.md +++ b/devlog/_plan/260807_compatibility_lab/040_security_and_privacy.md @@ -178,8 +178,8 @@ redaction mechanism. The local route subject distinguishes exact behavior without raw secrets: -- configured instance, endpoint, custom headers, project and location use a - per-installation keyed HMAC; +- configured instance, endpoint, custom-header behavior, project and location + use a per-installation keyed HMAC; - credential/account identity does not participate; - raw base URLs and private/custom headers are absent; - model IDs are retained locally because they are required route identity, but @@ -190,6 +190,29 @@ The local route subject distinguishes exact behavior without raw secrets: The salt is stored with secret-file permissions outside the JSONL/artifact tree. It is not exported. +### Custom-header fingerprint broker + +Raw custom headers remain owned by the provider/config request builder and are +never passed to Lab code. That owner computes +`headers.nonCredentialBehaviorDigest` through a narrow fingerprint broker: + +1. resolve the effective static custom headers after preset/config merge but + before request-specific or credential injection; +2. remove every credential-bearing header according to the same auth transport + classification used by the request builder; +3. lowercase valid ASCII field names, reject invalid names, preserve duplicate + value order, and preserve exact UTF-8 value bytes without trimming; +4. sort entries by lowercase name while retaining duplicate order and encode + JCS `[{"name": string, "values": string[]}, ...]`; +5. return lowercase HMAC-SHA-256 with installation salt and domain + `ocx-lab:local-fingerprint:v1\0customHeaderBehavior\0`. + +The broker returns only the digest. Its API cannot return normalized names, +values, intermediate bytes, the salt, or the credential classification. +Unknown classification fails subject construction; it never falls back to +hashing or logging the raw header. Canary tests must prove raw names/values do +not enter Lab events, errors, SQLite, or artifacts. + ## 6. Local evidence versus public export Local evidence is already sanitized. Public export is stricter and uses a new, diff --git a/devlog/_plan/260807_compatibility_lab/050_cl00_acceptance_review.md b/devlog/_plan/260807_compatibility_lab/050_cl00_acceptance_review.md index 8e70b2896..629a0ed37 100644 --- a/devlog/_plan/260807_compatibility_lab/050_cl00_acceptance_review.md +++ b/devlog/_plan/260807_compatibility_lab/050_cl00_acceptance_review.md @@ -49,6 +49,31 @@ None. - Correction: narrowed every protocol V1 row to its exact `022` evidence and made incident mappings explicit future scenario/version inputs when no literal V1 vector exists. +7. The three evidence layers lacked separate executable subject identities, + and suites could span layers. + - Correction: added the closed `ProtocolSubjectV1`, `RouteSubjectV1`, and + `TaskSubjectV1` union, exact layer/subject matching, layer-qualified suite + manifests, and one reserved deterministic Fabric task/verifier contract. +8. Behavior identity did not explicitly close effective + `commandCodeVersion`, sampling-parameter omission sets, `cacheRetention`, or + unknown future behavior inputs. + - Correction: froze `BehaviorFingerprintV1`, its closed keys and source + tags, required values from the production resolver, and fail-closed + handling/tests for an unclassified behavior input. +9. A global protocol failure-rule list attached both control effects to every + case, violating the legal control matrix. + - Correction: the base rule set contains no control rule; expansion adds + exactly one materialized rule only for a case with `expectedFailure`. + Protocol V1 has one conformance control and no unsupported effect. +10. Tool/MCP assertions selected bare calls from the normalized SSE event array. + - Correction: froze separate canonical `toolCalls[]` and `mcpCalls[]` + semantic projections and moved all call/correlation selectors to them. +11. Chat-backed Responses cases expected adapter `done` instead of the + client-visible `completed` terminal, and parallel-tool counting still used + nonexistent `tool_call` SSE events. + - Correction: those terminals now expect `completed`; parallel count/order + derive from `/client/response/toolCalls`, and `toolCalls`/`mcpCalls` are + part of the closed observation schema. ### Medium @@ -77,6 +102,14 @@ None. 8. One result-content description claimed call correlation absent from its assertions. - Correction: removed the claim. +9. Environmental failure effects, claim supersession/currentness, and + custom-header fingerprint ownership were not mechanically closed. + - Correction: added the exhaustive class/effect matrix, formal + `claim_snapshot`/`supersedes[]` schema and currentness algorithm, and a + config-owner broker that exposes only a domain-separated header digest. +10. `ProtocolSubjectV1` named a second `runtimeFingerprint` without a schema. + - Correction: removed it; the closed `runtime.*` behavior keys are the sole + platform-sensitive identity inputs. ### Low @@ -85,6 +118,9 @@ None. - Corrected historical reference `#745` from issue to pull request. - Added `021`/`022` to the stack ledger and created this review record, closing all local document links. +- Corrected request-history evidence from “immutable” to canonical + append-only, limited the profile claim to compatibility policy, and noted the + explicit state-mutating `ocx doctor --fix-codex-runtime` mode. ## Mechanical review evidence @@ -99,6 +135,8 @@ None. - All named verifier selectors have one closed deterministic definition. - `vision-core.protocol.modality-gate` is the sole V1 negative control and is represented as such in case and suite expansion. +- Base failure rules contain no control effect; the vision case alone expands + the conformance-control rule. ## Repository verification @@ -130,7 +168,7 @@ suite passed. 2. Environmental failures cannot poison compatibility verdicts: **PASS**. 3. `VERIFIED` is reproducible from immutable evidence: **PASS**. 4. Exact route identity prevents false evidence reuse: **PASS**. -5. Routing Profiles remain the sole user-policy layer: **PASS**. +5. Routing Profiles remain the sole compatibility-policy surface: **PASS**. 6. The Lab cannot become a second router: **PASS**. 7. The Lab cannot become a second provider registry: **PASS**. 8. Probes cannot access user data or arbitrary tools: **PASS**. From d3bb98e0976edfa291bb868a1eb285fc5f6a407c Mon Sep 17 00:00:00 2001 From: Wibias <37517432+Wibias@users.noreply.github.com> Date: Sat, 8 Aug 2026 22:33:08 +0200 Subject: [PATCH 05/20] docs(lab): record CL-00 delayed-review head SHA Record the accepted delayed-review correction commit on the stack ledger and acceptance review. --- .../_plan/260807_compatibility_lab/001_pr_stack_status.md | 6 +++--- .../260807_compatibility_lab/050_cl00_acceptance_review.md | 3 ++- 2 files changed, 5 insertions(+), 4 deletions(-) diff --git a/devlog/_plan/260807_compatibility_lab/001_pr_stack_status.md b/devlog/_plan/260807_compatibility_lab/001_pr_stack_status.md index 1676521e9..e7d83b638 100644 --- a/devlog/_plan/260807_compatibility_lab/001_pr_stack_status.md +++ b/devlog/_plan/260807_compatibility_lab/001_pr_stack_status.md @@ -19,7 +19,7 @@ whether the next phase is authorized. | Phase | Branch | Base SHA | Implementation head | PR | State | |---|---|---|---|---|---| -| CL-00 | `feat/cl-00-compatibility-contracts` | `3ad5bb6bd3f76f6879d84b78ea39edd3e01ec296` | pending commit after delayed-review corrections | [draft #1286](https://github.com/lidge-jun/opencodex/pull/1286) | ACCEPTED | +| CL-00 | `feat/cl-00-compatibility-contracts` | `3ad5bb6bd3f76f6879d84b78ea39edd3e01ec296` | `12e50a3502fb4af25283538cc717ead2291edd8b` | [draft #1286](https://github.com/lidge-jun/opencodex/pull/1286) | ACCEPTED | | CL-01 | not created | CL-00 must be accepted first | not started | none | NOT AUTHORIZED | ## CL-00 acceptance log @@ -67,7 +67,7 @@ whether the next phase is authorized. challenges pass and no Critical/High/Medium findings remain. - Blockers: none for CL-00. Full-suite green remains unavailable on this host for the Windows/Bun reasons above. -- CL-00 ending implementation SHA: pending commit after delayed-review - corrections from the independent acceptance pass. +- CL-00 ending implementation SHA: + `12e50a3502fb4af25283538cc717ead2291edd8b`. - Draft PR: [#1286](https://github.com/lidge-jun/opencodex/pull/1286). - CL-01 authorized: **NO**. Acceptance of CL-00 does not start CL-01. diff --git a/devlog/_plan/260807_compatibility_lab/050_cl00_acceptance_review.md b/devlog/_plan/260807_compatibility_lab/050_cl00_acceptance_review.md index 629a0ed37..ee0a0372a 100644 --- a/devlog/_plan/260807_compatibility_lab/050_cl00_acceptance_review.md +++ b/devlog/_plan/260807_compatibility_lab/050_cl00_acceptance_review.md @@ -4,7 +4,8 @@ Date: 2026-08-08 Scope: the complete CL-00 contract set on `feat/cl-00-compatibility-contracts`, based on -`3ad5bb6bd3f76f6879d84b78ea39edd3e01ec296`. +`3ad5bb6bd3f76f6879d84b78ea39edd3e01ec296`, ending at +`12e50a3502fb4af25283538cc717ead2291edd8b`. The review was read-only and separate from the authoring pass. Every validated Critical, High, and Medium finding was corrected and re-reviewed before this From c2113ca47b8a05c5a5f90679e4eaa640ca2c6a66 Mon Sep 17 00:00:00 2001 From: Wibias <37517432+Wibias@users.noreply.github.com> Date: Sat, 8 Aug 2026 22:33:43 +0200 Subject: [PATCH 06/20] docs(lab): pin CL-00 acceptance to correction commit Keep the stack ledger and acceptance review pointed at the delayed-review contract correction commit rather than chasing tip SHA churn. --- devlog/_plan/260807_compatibility_lab/001_pr_stack_status.md | 3 ++- .../260807_compatibility_lab/050_cl00_acceptance_review.md | 4 ++-- 2 files changed, 4 insertions(+), 3 deletions(-) diff --git a/devlog/_plan/260807_compatibility_lab/001_pr_stack_status.md b/devlog/_plan/260807_compatibility_lab/001_pr_stack_status.md index e7d83b638..7214263bc 100644 --- a/devlog/_plan/260807_compatibility_lab/001_pr_stack_status.md +++ b/devlog/_plan/260807_compatibility_lab/001_pr_stack_status.md @@ -68,6 +68,7 @@ whether the next phase is authorized. - Blockers: none for CL-00. Full-suite green remains unavailable on this host for the Windows/Bun reasons above. - CL-00 ending implementation SHA: - `12e50a3502fb4af25283538cc717ead2291edd8b`. + `12e50a3502fb4af25283538cc717ead2291edd8b` (delayed-review corrections); + subsequent ledger sync commits may follow without changing that contract set. - Draft PR: [#1286](https://github.com/lidge-jun/opencodex/pull/1286). - CL-01 authorized: **NO**. Acceptance of CL-00 does not start CL-01. diff --git a/devlog/_plan/260807_compatibility_lab/050_cl00_acceptance_review.md b/devlog/_plan/260807_compatibility_lab/050_cl00_acceptance_review.md index ee0a0372a..331bd35c5 100644 --- a/devlog/_plan/260807_compatibility_lab/050_cl00_acceptance_review.md +++ b/devlog/_plan/260807_compatibility_lab/050_cl00_acceptance_review.md @@ -4,8 +4,8 @@ Date: 2026-08-08 Scope: the complete CL-00 contract set on `feat/cl-00-compatibility-contracts`, based on -`3ad5bb6bd3f76f6879d84b78ea39edd3e01ec296`, ending at -`12e50a3502fb4af25283538cc717ead2291edd8b`. +`3ad5bb6bd3f76f6879d84b78ea39edd3e01ec296`, with delayed-review corrections +accepted at `12e50a3502fb4af25283538cc717ead2291edd8b`. The review was read-only and separate from the authoring pass. Every validated Critical, High, and Medium finding was corrected and re-reviewed before this From df4f171857866190b7277ec9f17b5941fa7c1c28 Mon Sep 17 00:00:00 2001 From: Wibias <37517432+Wibias@users.noreply.github.com> Date: Sat, 8 Aug 2026 22:37:15 +0200 Subject: [PATCH 07/20] docs(lab): address CodeRabbit CL-00 contract findings Tighten destination/proxy sandbox wording, event bounds, retention and sensitive purge rules, dependency total order, and CL-01 gate phrasing. --- .../000_master_plan.md | 2 +- .../010_architecture_and_evidence_contract.md | 7 +- .../021_protocol_v1_manifest_authority.md | 5 + .../040_security_and_privacy.md | 121 ++++++++++++------ 4 files changed, 96 insertions(+), 39 deletions(-) diff --git a/devlog/_plan/260807_compatibility_lab/000_master_plan.md b/devlog/_plan/260807_compatibility_lab/000_master_plan.md index fae01e95d..f9ee5b833 100644 --- a/devlog/_plan/260807_compatibility_lab/000_master_plan.md +++ b/devlog/_plan/260807_compatibility_lab/000_master_plan.md @@ -268,7 +268,7 @@ Only CL-00 is authorized by this document at present. | Phase | Purpose | Authorization | |---|---|---| | CL-00 | Architecture authority, contracts, scenario catalogue, incident corpus | This PR | -| CL-01 | Deterministic protocol-conformance runner and fixtures | Not started; requires accepted CL-00 | +| CL-01 | Deterministic protocol-conformance runner and fixtures | Not started; requires CL-00 to be accepted | | CL-02 | Immutable JSONL ledger, artifacts and SQLite projection | Not started | | CL-03 | Bounded live-route probes | Not started | | CL-04 | Lab CLI and management read surfaces | Not started | diff --git a/devlog/_plan/260807_compatibility_lab/010_architecture_and_evidence_contract.md b/devlog/_plan/260807_compatibility_lab/010_architecture_and_evidence_contract.md index 820611af9..f108ab17d 100644 --- a/devlog/_plan/260807_compatibility_lab/010_architecture_and_evidence_contract.md +++ b/devlog/_plan/260807_compatibility_lab/010_architecture_and_evidence_contract.md @@ -590,8 +590,11 @@ Semantics: behaviorally invoked sidecars. Each record contains role, provider ID, provider-instance fingerprint, client/upstream model IDs, effective adapter, upstream protocol, endpoint fingerprint, and behavior fingerprint. It cannot - nest. Records sort by role, provider ID, upstream model ID, then endpoint - fingerprint. An empty list is canonical when no sidecar is invoked. + nest. Records sort by this total order of UTF-8 string comparisons: + role, provider ID, upstream model ID, endpoint fingerprint, client model + ID, effective adapter, upstream protocol, then behavior fingerprint. A + duplicate full key makes subject construction `harness_failure`. An empty + list is canonical when no sidecar is invoked. ### Behavior fingerprint allowlist diff --git a/devlog/_plan/260807_compatibility_lab/021_protocol_v1_manifest_authority.md b/devlog/_plan/260807_compatibility_lab/021_protocol_v1_manifest_authority.md index 5f098338b..53f093378 100644 --- a/devlog/_plan/260807_compatibility_lab/021_protocol_v1_manifest_authority.md +++ b/devlog/_plan/260807_compatibility_lab/021_protocol_v1_manifest_authority.md @@ -219,6 +219,11 @@ process, user MCP configuration, filesystem access, or a user tool. ## 5. SSE normalization +This section is the Lab harness normalizer for protocol V1 observations. It +does not rewrite production sidecar parsers such as +`src/vision/anthropic-describe.ts`; those remain out of CL-00/CL-01 scope and +may keep closed product-local semantics until a later shared-normalizer phase. + The harness retains exact fixture bytes and normalizes only for assertions: 1. UTF-8 must decode without replacement. A BOM is allowed only at byte zero diff --git a/devlog/_plan/260807_compatibility_lab/040_security_and_privacy.md b/devlog/_plan/260807_compatibility_lab/040_security_and_privacy.md index d92ee71be..56fc532cd 100644 --- a/devlog/_plan/260807_compatibility_lab/040_security_and_privacy.md +++ b/devlog/_plan/260807_compatibility_lab/040_security_and_privacy.md @@ -10,7 +10,8 @@ The Lab must not read, accept, persist, or export: - user prompts or conversation history; - real user repositories, worktrees, patches, source files, or file paths; -- user MCP server definitions, resources, results, or credentials by default; +- user MCP server definitions, resources, results, or credentials, with no + Lab mode, CLI flag, profile, or override that can load them; - arbitrary shell commands or process output; - arbitrary filesystem contents; - arbitrary external-network tool requests or responses; @@ -36,12 +37,27 @@ The future runner must enforce a capability-deny sandbox: ### Network - The immutable scenario manifest may authorize only fixed dependency roles - and protocol classes, never a route-local URL. The only remote destinations - are the exact primary endpoint and flat sidecar dependency endpoints named - in the composite route subject, after existing provider destination-policy - validation. -- DNS resolution and every redirect are revalidated. Redirects cannot widen - scheme, host, port, or private-network access. + and protocol classes, never a route-local URL. +- Network authorization uses a trusted in-memory `LabDestinationV1` record + owned by the existing provider destination/credential plumbing: scheme, + host, port, base path, resolved IP family/address set after policy checks, + TLS SNI/Host values, and private-network opt-in. That record is never + written to JSONL, SQLite, artifacts, or export. +- The composite route subject stores only the keyed opaque + `endpointFingerprint` derived from the normalized destination. Raw URLs are + never evidence fields. +- The only remote destinations are the exact primary and flat sidecar + destinations named by those in-memory records after existing provider + destination-policy validation. +- DNS is resolved once before the policy check. The HTTP client must connect + to the validated IP set (pin/connect to the approved addresses) while + preserving the intended Host/SNI. A later resolution that differs fails + closed as `harness_failure`. +- Redirects are rejected by default for Lab probes, matching the existing + SSRF fail-closed posture. A future scenario that explicitly opts into + redirects must authorize every hop with the same destination policy, IP + pinning, and Host/SNI preservation; redirects still cannot widen scheme, + host, port, or private-network access. - Private/loopback endpoints require the route's existing explicit private network opt-in and an explicit Lab-run confirmation. Metadata endpoints remain blocked. @@ -49,10 +65,14 @@ The future runner must enforce a capability-deny sandbox: destination. - A sidecar dependency is allowed only when the scenario explicitly authorizes its role/protocol class, the composite subject names the exact dependency - fingerprint and endpoint, the operator approves the composite live probe, - and its credential is destination-bound independently. Unmanifested roles or - subject-external/dynamically widened endpoints make the run - `harness_failure`. + fingerprint, an in-memory destination record exists for that fingerprint, + the operator approves the composite live probe, and its credential is + destination-bound independently. Unmanifested roles or subject-external/ + dynamically widened endpoints make the run `harness_failure`. +- Inherited `HTTP_PROXY`, `HTTPS_PROXY`, `ALL_PROXY`, and `NO_PROXY` values are + rejected for Lab runs. If a future reviewed scenario requires a proxy, that + proxy endpoint is authorized as its own exact destination under the same + SSRF checks and never inherits ambient proxy environment variables. - Tools have no network capability. A model-requested web search, image generation, URL fetch, computer use, or hosted external tool is disabled or classified inapplicable unless a future separately reviewed scenario owns a @@ -77,9 +97,9 @@ The future runner must enforce a capability-deny sandbox: - The runner receives no general shell/process API and no inherited stdin. - Filesystem access is restricted to a fresh Lab scratch directory, read-only packaged synthetic fixtures, and the bounded artifact writer. -- Environment inheritance is an allowlist. Secrets and proxy variables are - supplied only through reviewed destination/credential plumbing, not copied - wholesale. +- Environment inheritance is an allowlist. Secrets are supplied only through + reviewed destination/credential plumbing. Ambient proxy variables are never + inherited; see Network above. - The run has enforced wall-clock, inactivity, byte, request, token, tool-call, memory, process and artifact limits. If the platform cannot enforce a required boundary, the run fails as `harness_failure`. @@ -119,13 +139,22 @@ content-addressed after redaction. Initial hard ceilings: ```text -maximum artifacts per run 16 -maximum bytes per artifact 256 KiB -maximum aggregate artifact data 1 MiB -maximum normalized events 4,096 -maximum sanitized string field 4 KiB +maximum artifacts per run 16 +maximum bytes per artifact 256 KiB +maximum aggregate artifact data 1 MiB +maximum normalized events 4,096 +maximum sanitized string field 4 KiB +maximum serialized bytes per event 64 KiB +maximum aggregate normalized event bytes 1 MiB +maximum event JSON nesting depth 8 +maximum object keys per event object 64 +maximum array elements per event array 256 ``` +These event bounds are enforced while decoding/normalizing, before an event +is buffered into the observation or any artifact. Exceeding a bound fails the +run as `harness_failure` without retaining the oversized fragment. + Scenario limits may be lower. Raising a hard ceiling requires a reviewed security-contract change; a scenario manifest alone cannot raise it. @@ -234,29 +263,49 @@ Public publishing is not authorized in CL-00 and remains a later phase. ## 7. Retention and deletion -- JSONL is the immutable local authority, but a user can delete the entire Lab - directory. Immutability describes in-ledger correction semantics, not a - promise to resist user deletion. -- Artifact retention classes are versioned and bounded by storage policy. - Deleting an expired artifact leaves its digest/reference and a typed - unavailable marker; it does not alter the observation. +- JSONL is the immutable local authority for non-sensitive evidence, but a + user can delete the entire Lab directory. Immutability describes in-ledger + correction semantics, not a promise to resist user deletion. +- Retention ceilings by class: + - scratch/temp run directories: deleted at run end; cleanup retry within 24h; + - export staging: maximum 24h; + - disposable SQLite projection: rebuildable anytime; may be deleted at any + time and must be deleted during a sensitive purge; + - sanitized non-contract artifacts (`assertion_report`, shapes, traces, + errors): default 90 days, hard ceiling 365 days; + - content-addressed scenario/suite/fixture contract artifacts: retained + while any non-invalidated observation references them, because + reproducible `VERIFIED` projection requires the exact historical bytes. + Their content remains synthetic-only and size-bounded. User deletion of + the Lab directory remains absolute. +- Deleting an expired non-contract artifact leaves its digest/reference and a + typed unavailable marker; it does not alter the observation. - SQLite is disposable and contains no data absent from valid ledger events and artifact metadata. -- Invalid or sensitive evidence is neutralized by an appended invalidation and - secure artifact deletion. A security incident may require deleting the local - ledger; append-only semantics never override the duty to remove leaked - secrets. +- Invalid non-sensitive evidence is neutralized by an appended invalidation and + secure artifact deletion. +- Confirmed sensitive evidence is distinct from ordinary invalidation. It + requires a fail-closed purge of every local copy: JSONL lines containing the + leak, SQLite rows, artifacts, scratch/temp files, and generated exports. The + purge record stores only taxonomy, time, affected event/artifact digests, + and action taken — never the leaked value. Append-only semantics never + override that duty. ## 8. Security acceptance tests required later Before any live runner ships, tests must prove: 1. prompt/repository/MCP/user-tool inputs are unreachable from the scenario DSL; -2. redirects and model-supplied URLs cannot widen network access; -3. credential, account, custom header and endpoint canaries never enter +2. redirects and model-supplied URLs cannot widen network access, and Lab + clients pin connections to the validated IP set; +3. inherited proxy environment variables cannot route Lab traffic; +4. credential, account, custom header and endpoint canaries never enter evidence, errors, SQLite or artifacts; -4. tool arguments cannot execute; -5. artifact traversal/symlink/oversize/digest attacks fail closed; -6. timeout, quota, auth, DNS and harness failures remain blockers; -7. public export rejects unknown/private fields; -8. no probe runs from the production routing path. +5. tool arguments cannot execute; +6. artifact traversal/symlink/oversize/digest attacks fail closed; +7. normalized event byte/depth/key/array ceilings fail closed before buffering; +8. timeout, quota, auth, DNS and harness failures remain blockers; +9. confirmed sensitive evidence is purged from JSONL, SQLite, artifacts, temp + files and exports without recording the leaked value; +10. public export rejects unknown/private fields; +11. no probe runs from the production routing path. From 4313fb7dacb51a44b2610093a0e774e6d0db2146 Mon Sep 17 00:00:00 2001 From: Wibias <37517432+Wibias@users.noreply.github.com> Date: Sat, 8 Aug 2026 23:56:52 +0200 Subject: [PATCH 08/20] docs(lab): close CL-00 security contract gaps --- .../040_security_and_privacy.md | 73 +++++++++++++------ 1 file changed, 51 insertions(+), 22 deletions(-) diff --git a/devlog/_plan/260807_compatibility_lab/040_security_and_privacy.md b/devlog/_plan/260807_compatibility_lab/040_security_and_privacy.md index 56fc532cd..e925a401d 100644 --- a/devlog/_plan/260807_compatibility_lab/040_security_and_privacy.md +++ b/devlog/_plan/260807_compatibility_lab/040_security_and_privacy.md @@ -43,6 +43,12 @@ The future runner must enforce a capability-deny sandbox: host, port, base path, resolved IP family/address set after policy checks, TLS SNI/Host values, and private-network opt-in. That record is never written to JSONL, SQLite, artifacts, or export. +- Provider destination/credential plumbing creates one per-run immutable + `LabDestinationV1` snapshot before endpoint fingerprinting. The exact same + snapshot must be used unchanged for endpoint fingerprinting, destination + authorization, credential binding, and connection. Mutation, replacement, + re-resolution to a different address set, or any mismatch between those + stages fails closed as `harness_failure` before credentials are sent. - The composite route subject stores only the keyed opaque `endpointFingerprint` derived from the normalized destination. Raw URLs are never evidence fields. @@ -69,10 +75,11 @@ The future runner must enforce a capability-deny sandbox: the operator approves the composite live probe, and its credential is destination-bound independently. Unmanifested roles or subject-external/ dynamically widened endpoints make the run `harness_failure`. -- Inherited `HTTP_PROXY`, `HTTPS_PROXY`, `ALL_PROXY`, and `NO_PROXY` values are - rejected for Lab runs. If a future reviewed scenario requires a proxy, that - proxy endpoint is authorized as its own exact destination under the same - SSRF checks and never inherits ambient proxy environment variables. +- Inherited `HTTP_PROXY`, `HTTPS_PROXY`, `ALL_PROXY`, `NO_PROXY`, `http_proxy`, + `https_proxy`, `all_proxy`, and `no_proxy` values are rejected for Lab runs. + If a future reviewed scenario requires a proxy, that proxy endpoint is + authorized as its own exact destination under the same SSRF checks and never + inherits ambient proxy environment variables. - Tools have no network capability. A model-requested web search, image generation, URL fetch, computer use, or hosted external tool is disabled or classified inapplicable unless a future separately reviewed scenario owns a @@ -97,9 +104,13 @@ The future runner must enforce a capability-deny sandbox: - The runner receives no general shell/process API and no inherited stdin. - Filesystem access is restricted to a fresh Lab scratch directory, read-only packaged synthetic fixtures, and the bounded artifact writer. -- Environment inheritance is an allowlist. Secrets are supplied only through - reviewed destination/credential plumbing. Ambient proxy variables are never - inherited; see Network above. +- The V1 inherited-environment allowlist is empty: Lab code must not read an + ambient variable to determine behavior, routing, credentials, destinations, + paths, locale, or proxying. The runner may expose only a constructed, + non-inherited environment view with exact constants `TZ=UTC` and + `NO_COLOR=1`; every other name is absent. In particular all uppercase and + lowercase proxy variables are rejected as stated in Network above. Secrets + are supplied only through reviewed destination/credential plumbing. - The run has enforced wall-clock, inactivity, byte, request, token, tool-call, memory, process and artifact limits. If the platform cannot enforce a required boundary, the run fails as `harness_failure`. @@ -229,11 +240,16 @@ never passed to Lab code. That owner computes before request-specific or credential injection; 2. remove every credential-bearing header according to the same auth transport classification used by the request builder; -3. lowercase valid ASCII field names, reject invalid names, preserve duplicate +3. before canonicalization, enforce at most 64 non-credential header entries, + at most 16 duplicate values for one lowercase name, at most 256 ASCII bytes + per field name, at most 8 KiB UTF-8 bytes per value, and at most 64 KiB of + aggregate normalized name/value bytes; exceeding any bound is + `harness_failure` and no digest is emitted; +4. lowercase valid ASCII field names, reject invalid names, preserve duplicate value order, and preserve exact UTF-8 value bytes without trimming; -4. sort entries by lowercase name while retaining duplicate order and encode +5. sort entries by lowercase name while retaining duplicate order and encode JCS `[{"name": string, "values": string[]}, ...]`; -5. return lowercase HMAC-SHA-256 with installation salt and domain +6. return lowercase HMAC-SHA-256 with installation salt and domain `ocx-lab:local-fingerprint:v1\0customHeaderBehavior\0`. The broker returns only the digest. Its API cannot return normalized names, @@ -282,8 +298,11 @@ Public publishing is not authorized in CL-00 and remains a later phase. typed unavailable marker; it does not alter the observation. - SQLite is disposable and contains no data absent from valid ledger events and artifact metadata. -- Invalid non-sensitive evidence is neutralized by an appended invalidation and - secure artifact deletion. +- Invalid non-sensitive evidence is neutralized by an appended invalidation. + Event-private non-contract artifacts may then be securely deleted. A shared + scenario, suite, or fixture contract artifact must remain while any other + non-invalidated observation references its digest, and may be deleted only + after the last such reference is gone. - Confirmed sensitive evidence is distinct from ordinary invalidation. It requires a fail-closed purge of every local copy: JSONL lines containing the leak, SQLite rows, artifacts, scratch/temp files, and generated exports. The @@ -298,14 +317,24 @@ Before any live runner ships, tests must prove: 1. prompt/repository/MCP/user-tool inputs are unreachable from the scenario DSL; 2. redirects and model-supplied URLs cannot widen network access, and Lab clients pin connections to the validated IP set; -3. inherited proxy environment variables cannot route Lab traffic; -4. credential, account, custom header and endpoint canaries never enter +3. the inherited-environment allowlist is empty, all uppercase/lowercase proxy + variables are rejected, and no ambient variable changes Lab behavior; +4. destination-record mutation, replacement, or address-set drift between + authorization, fingerprinting, credential binding, and connect fails closed; +5. credential, account, custom header and endpoint canaries never enter evidence, errors, SQLite or artifacts; -5. tool arguments cannot execute; -6. artifact traversal/symlink/oversize/digest attacks fail closed; -7. normalized event byte/depth/key/array ceilings fail closed before buffering; -8. timeout, quota, auth, DNS and harness failures remain blockers; -9. confirmed sensitive evidence is purged from JSONL, SQLite, artifacts, temp - files and exports without recording the leaked value; -10. public export rejects unknown/private fields; -11. no probe runs from the production routing path. +6. custom-header canonicalization is deterministic; unknown credential + classification and every count/name/value/aggregate bound fail closed; +7. local subject-salt rotation breaks prior correlation and forces + re-projection/reverification without reverse lookup; +8. tool arguments cannot execute; +9. artifact traversal/symlink/oversize/digest attacks fail closed; +10. normalized event byte/depth/key/array ceilings fail closed before buffering; +11. timeout, quota, auth, DNS and harness failures remain blockers; +12. retention expiry emits typed unavailable markers; cleanup retry/failure is + visible and bounded; shared contract artifacts survive invalidation while + any non-invalidated observation still references them; +13. confirmed sensitive evidence is purged from JSONL, SQLite, artifacts, temp + files and exports without recording the leaked value; +14. public export rejects unknown/private fields; +15. no probe runs from the production routing path. From b21ca0363226031273cd7e0c8f3109e8c869db55 Mon Sep 17 00:00:00 2001 From: Wibias <37517432+Wibias@users.noreply.github.com> Date: Sat, 8 Aug 2026 23:58:11 +0200 Subject: [PATCH 09/20] docs(lab): make verdict and fingerprint contracts deterministic --- .../010_architecture_and_evidence_contract.md | 24 +++++++++++++++---- 1 file changed, 20 insertions(+), 4 deletions(-) diff --git a/devlog/_plan/260807_compatibility_lab/010_architecture_and_evidence_contract.md b/devlog/_plan/260807_compatibility_lab/010_architecture_and_evidence_contract.md index f108ab17d..c2f4aa777 100644 --- a/devlog/_plan/260807_compatibility_lab/010_architecture_and_evidence_contract.md +++ b/devlog/_plan/260807_compatibility_lab/010_architecture_and_evidence_contract.md @@ -302,7 +302,8 @@ samples, and blocked attempts cannot produce `PROBED`. Produced only when every requirement in the suite manifest's verification rule is met by current, valid, attributable observations for the exact projection key, and no newer current contradictory attributable failure remains -unresolved. +unresolved. A verification rule with zero applicable required scenarios is not +satisfied and cannot produce `PROBED` or `VERIFIED`. The projection must expose: @@ -695,9 +696,24 @@ model, reasoning, per-turn limits and timeout/stall limits plus the matching flat dependency subject ID. Values that are inapplicable to the selected model/surface are omitted rather -than copied wholesale. Canonical JSON sorts keys, normalizes absent/default -values to their effective value, and sorts set-like arrays while preserving -order-sensitive arrays. +than copied wholesale. Canonical JSON sorts object keys and normalizes +absent/default values to their effective value. Array handling is closed and +part of the V1 schema: + +| Array-valued key | Ordering mode | Element comparison | +|---|---|---| +| `modalities.input` | `set` | UTF-8 JCS bytes | +| `reasoning.efforts` | `set` | UTF-8 JCS bytes | +| `tools.choiceRestrictions` | `set` | UTF-8 JCS bytes | +| `openrouter.order` | `ordered` | source order preserved | +| `openrouter.only` | `set` | UTF-8 JCS bytes | + +For `set`, each element is JCS-canonicalized, duplicate JCS byte sequences are +rejected, and elements sort lexicographically by their UTF-8 JCS bytes. For +`ordered`, elements are JCS-canonicalized but their resolved source order is +preserved. Any other closed key resolving to an array, or any array-valued key +without one of these declared modes, fails subject construction as +`harness_failure` with code `unclassified_behavior_input`. Acceptance tests for the future builder must enumerate every behavior-changing config/default read by the selected adapter and prove that From 40b88da09b81541f433ea143b95a377d353c088e Mon Sep 17 00:00:00 2001 From: Wibias <37517432+Wibias@users.noreply.github.com> Date: Sat, 8 Aug 2026 23:59:02 +0200 Subject: [PATCH 10/20] docs(lab): prevent vacuous verified projections --- .../020_scenario_contract_and_catalogue.md | 26 +++++++++++++------ 1 file changed, 18 insertions(+), 8 deletions(-) diff --git a/devlog/_plan/260807_compatibility_lab/020_scenario_contract_and_catalogue.md b/devlog/_plan/260807_compatibility_lab/020_scenario_contract_and_catalogue.md index 6a97d4c8f..fe53ef6e0 100644 --- a/devlog/_plan/260807_compatibility_lab/020_scenario_contract_and_catalogue.md +++ b/devlog/_plan/260807_compatibility_lab/020_scenario_contract_and_catalogue.md @@ -192,14 +192,16 @@ minimum finite bound, with `null` meaning unbounded at that layer. ## 2. Suite projection rules -For each exact suite manifest: - -- `VERIFIED`: all `required` scenarios applicable to the subject have current - passes and every `conformance_negative_control` observed its exact required - rejection. -- `PROBED`: at least one required scenario passed, with no current - compatibility-attributable required-scenario failure, but coverage is - incomplete. +For each exact suite manifest, first compute the required scenarios whose +manifest requirements are applicable to the exact subject. That applicable +required set must be non-empty for any positive executable verdict. + +- `VERIFIED`: the applicable required set is non-empty; every applicable + `required` scenario has a current pass; and every applicable + `conformance_negative_control` observed its exact required rejection. +- `PROBED`: at least one applicable required scenario passed, with no current + compatibility-attributable required-scenario failure, but the suite's + verification rule is not fully satisfied. - `DEGRADED`: a failure rule on an applicable required scenario yields `degraded`. - `UNSUPPORTED`: a required scenario's exact @@ -208,6 +210,14 @@ For each exact suite manifest: precedence. - `CLAIMED`/`UNKNOWN`: follow the evidence contract. +An inapplicable required scenario contributes neither a pass nor a failure. If +no required scenario is applicable, the suite cannot project `PROBED` or +`VERIFIED`; it falls through to `CLAIMED` or `UNKNOWN` under the evidence +contract. If execution was attempted but an environmental or administrative +precondition prevented reaching assertions, that attempt is a typed blocker +and may project `BLOCKED` under normal precedence; it is not treated as +inapplicability. + Supplemental scenarios never block `VERIFIED` unless a new suite version makes them required. From 3ae5a70bd6b40e929ff5fe45ef08f0b325a63e1f Mon Sep 17 00:00:00 2001 From: Wibias <37517432+Wibias@users.noreply.github.com> Date: Sun, 9 Aug 2026 00:01:04 +0200 Subject: [PATCH 11/20] docs(lab): close protocol sentinel semantics --- .../021_protocol_v1_manifest_authority.md | 18 +++++++++++++----- 1 file changed, 13 insertions(+), 5 deletions(-) diff --git a/devlog/_plan/260807_compatibility_lab/021_protocol_v1_manifest_authority.md b/devlog/_plan/260807_compatibility_lab/021_protocol_v1_manifest_authority.md index 53f093378..4086ee09b 100644 --- a/devlog/_plan/260807_compatibility_lab/021_protocol_v1_manifest_authority.md +++ b/devlog/_plan/260807_compatibility_lab/021_protocol_v1_manifest_authority.md @@ -145,8 +145,9 @@ verificationRule all-applicable-required-pass-v1 Unknown fields reject registration. The suite digest uses `ocx-lab:suite-manifest:v1` plus JCS exactly as defined by the evidence -contract. `VERIFIED` requires a current pass for every applicable member and -the exact suite/scenario/fixture digests above. +contract. `VERIFIED` requires at least one applicable required member, a +current pass for every applicable required member, and the exact +suite/scenario/fixture digests above. ## 3. Observation selector model @@ -234,7 +235,15 @@ The harness retains exact fixture bytes and normalizes only for assertions: Exactly one optional leading U+0020 after `:` is removed; no other whitespace is trimmed. 5. Repeated `data` fields join with LF. The last `event` field wins. -6. `[DONE]` is a sentinel only for Chat surfaces. +6. Sentinel interpretation follows the protocol of the byte stream being + normalized, not the client-facing `requirements.surfaces` label. For an + `upstream_response` fixture, the source is the case's single resolved + `requirements.upstreamProtocols` entry: only `openai-chat` recognizes a + data value exactly equal to `[DONE]` as a sentinel. `openai-responses` and + `anthropic-messages` do not. A raw fixture with zero or multiple resolved + source protocols is invalid. Client-output normalization instead follows + the emitted client protocol; a Responses bridge's transport `[DONE]` + padding is not reclassified as an upstream Chat sentinel. 7. When `event` is absent and parsed `data` is an object with string `type`, Responses/Anthropic normalization infers that `type`. Explicit event wins. Parsed `null`, scalar, array, or empty data is padding and emits no event. @@ -267,7 +276,6 @@ function_call: { id: item.call_id, name: item.name, arguments: JSON.parse(item.arguments), kind: "function", ordinal: output order } - custom_tool_call: { id: item.call_id, name: item.name, arguments: item.input, @@ -417,4 +425,4 @@ authority. It must: This document and the JSON authority authorize no runner, mock server, ledger, SQLite projection, fixture extraction from tests, live probe, or Fabric -ingestion implementation in CL-00. +ingestion implementation in CL-00. \ No newline at end of file From b62e29395d25b2cd9a1dfd852101dbf7a2c91507 Mon Sep 17 00:00:00 2001 From: Wibias <37517432+Wibias@users.noreply.github.com> Date: Sun, 9 Aug 2026 00:05:17 +0200 Subject: [PATCH 12/20] docs(lab): fix Chat tool-result selectors --- .../260807_compatibility_lab/022_protocol_v1_cases.json | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/devlog/_plan/260807_compatibility_lab/022_protocol_v1_cases.json b/devlog/_plan/260807_compatibility_lab/022_protocol_v1_cases.json index 0b3fa6e2c..00f196cd1 100644 --- a/devlog/_plan/260807_compatibility_lab/022_protocol_v1_cases.json +++ b/devlog/_plan/260807_compatibility_lab/022_protocol_v1_cases.json @@ -216,7 +216,7 @@ "fixture": { "id": "tools-function", "role": "adapter_vector", "mediaType": "application/vnd.opencodex.adapter-vector+json", "bytesUtf8": "{\"tools\":[{\"name\":\"lookup\",\"parameters\":{\"type\":\"object\",\"properties\":{\"q\":{\"type\":\"string\"}},\"required\":[\"q\"]}}],\"upstreamToolCall\":{\"id\":\"call_fixture\",\"name\":\"lookup\",\"arguments\":\"{\\\"q\\\":\\\"x\\\"}\"},\"toolResult\":{\"toolCallId\":\"call_fixture\",\"content\":\"RESULT\"}}", "digest": "9107f4dfdd7da8340c866c9fb6f42854437cebb98592d0510969c810c1eeb0ad" }, "assertions": [ { "id": "call", "operator": "tool_call_equals", "selector": "/client/response/toolCalls/0", "expected": {"id":"call_fixture","name":"lookup","arguments":{"q":"x"},"kind":"function","ordinal":0}, "required": true }, - { "id": "result", "operator": "tool_result_correlates", "selector": "/upstream/requests", "expected": {"call":"/client/response/toolCalls/0/id","result":"/upstream/requests/1/json/input/0/call_id"}, "required": true } + { "id": "result", "operator": "tool_result_correlates", "selector": "/upstream/requests", "expected": {"call":"/client/response/toolCalls/0/id","result":"/upstream/requests/1/json/messages/1/tool_call_id"}, "required": true } ] }, { @@ -285,7 +285,7 @@ "fixture": { "id": "codex-patch", "role": "adapter_vector", "mediaType": "application/vnd.opencodex.adapter-vector+json", "bytesUtf8": "{\"name\":\"apply_patch\",\"input\":\"*** Begin Patch\\n*** Add File: x\\n+x\\n*** End Patch\\n\",\"callId\":\"call_patch\",\"result\":\"Done\"}", "digest": "668baa1fbea1d7a6556f717467fc3b90a47b2edfaa2ccf0c7950fd30dfe27a81" }, "assertions": [ { "id": "call", "operator": "tool_call_equals", "selector": "/client/response/toolCalls/0", "expected": {"id":"call_patch","name":"apply_patch","arguments":"*** Begin Patch\n*** Add File: x\n+x\n*** End Patch\n","kind":"custom","ordinal":0}, "required": true }, - { "id": "result", "operator": "tool_result_correlates", "selector": "/upstream/requests", "expected": {"call":"/client/response/toolCalls/0/id","result":"/upstream/requests/1/json/input/0/call_id"}, "required": true } + { "id": "result", "operator": "tool_result_correlates", "selector": "/upstream/requests", "expected": {"call":"/client/response/toolCalls/0/id","result":"/upstream/requests/1/json/messages/1/tool_call_id"}, "required": true } ] }, { @@ -458,4 +458,4 @@ ] } ] -} +} \ No newline at end of file From 163e21c3ca57c6f4a6381d00094e4fc23ae11e89 Mon Sep 17 00:00:00 2001 From: Wibias <37517432+Wibias@users.noreply.github.com> Date: Sun, 9 Aug 2026 00:06:35 +0200 Subject: [PATCH 13/20] docs(lab): refresh CL-00 acceptance after CodeRabbit --- .../050_cl00_acceptance_review.md | 133 +++++++++++++++--- 1 file changed, 115 insertions(+), 18 deletions(-) diff --git a/devlog/_plan/260807_compatibility_lab/050_cl00_acceptance_review.md b/devlog/_plan/260807_compatibility_lab/050_cl00_acceptance_review.md index 331bd35c5..4392be86f 100644 --- a/devlog/_plan/260807_compatibility_lab/050_cl00_acceptance_review.md +++ b/devlog/_plan/260807_compatibility_lab/050_cl00_acceptance_review.md @@ -4,12 +4,17 @@ Date: 2026-08-08 Scope: the complete CL-00 contract set on `feat/cl-00-compatibility-contracts`, based on -`3ad5bb6bd3f76f6879d84b78ea39edd3e01ec296`, with delayed-review corrections -accepted at `12e50a3502fb4af25283538cc717ead2291edd8b`. +`3ad5bb6bd3f76f6879d84b78ea39edd3e01ec296`. The initial delayed-review +acceptance was recorded at `12e50a3502fb4af25283538cc717ead2291edd8b`. +A later CodeRabbit re-review was validated against current production code and +contract semantics; its accepted contract corrections are incorporated through +`b62e29395d25b2cd9a1dfd852101dbf7a2c91507` and this record supersedes the +stale acceptance/status statements from the earlier pass. -The review was read-only and separate from the authoring pass. Every validated -Critical, High, and Medium finding was corrected and re-reviewed before this -record was finalized. +The review was read-only and separate from the original authoring pass. Every +validated Critical, High, and Medium finding was corrected and re-reviewed +before this record was finalized. Valid deterministic/security contract defects +were also corrected even when review severity was lower. ## Findings and corrections @@ -123,6 +128,57 @@ None. append-only, limited the profile claim to compatibility policy, and noted the explicit state-mutating `ocx doctor --fix-codex-runtime` mode. +## CodeRabbit re-review remediation + +All ten unresolved CodeRabbit threads visible on PR #1286 were inspected against +current branch code/contracts before editing. The valid findings were resolved +as follows: + +1. Stack audit metadata used a non-SHA dependency label where an exact CL-01 + base revision is required. The stack ledger is refreshed after this review + with exact base/head fields and the CL-01 correction requirement. +2. `BehaviorFingerprintV1` did not define deterministic ordering for every + array-valued closed key. V1 now classifies each allowed array as `set` or + `ordered`, defines JCS-byte sorting/deduplication for sets, preserves source + order for ordered arrays, and fails closed for any undeclared array input. +3. `all-applicable-required-pass-v1` admitted a vacuous `VERIFIED` result when + zero required scenarios applied. Positive executable verdicts now require a + non-empty applicable required set; zero-applicable falls through to current + claim/unknown semantics, while attempted environmental blockers remain + `BLOCKED`. +4. `[DONE]` handling was selected by the client-facing surface even for a Chat + upstream response fixture. Sentinel interpretation now follows the protocol + of the byte stream being normalized: an `upstream_response` uses its single + resolved upstream protocol, and only OpenAI Chat recognizes exact `[DONE]`. +5. Two Chat-backed tool-result assertions incorrectly selected Responses + `input[].call_id`. Production `openai-chat` emits the continuation as + `messages[1].tool_call_id`; both canonical selectors now assert that actual + Chat wire shape. +6. Live-probe destination authorization could drift between endpoint + fingerprinting, credential binding and connect. The security contract now + requires one immutable per-run `LabDestinationV1` snapshot for every stage + and fails closed before credential transmission on mutation/re-resolution or + mismatch. +7. Ambient environment/proxy handling was not executable enough. The V1 + inherited-environment allowlist is empty; the runner constructs only + `TZ=UTC` and `NO_COLOR=1`, rejects uppercase and lowercase proxy variables, + and cannot derive behavior from ambient variables. +8. Custom-header fingerprinting lacked resource/canonicalization bounds. The + broker now enforces entry, duplicate-value, field-name, per-value and + aggregate byte ceilings before JCS/HMAC, with unknown credential + classification or overflow failing closed. +9. Ordinary invalidation wording allowed deletion of shared contract artifacts. + Only event-private non-contract artifacts may be deleted after invalidation; + shared scenario/suite/fixture artifacts survive until no non-invalidated + observation references them. +10. Security acceptance coverage omitted the new invariants. Required tests now + cover environment/proxy denial, destination snapshot mutation/address drift, + custom-header canonicalization and bounds, subject-salt rotation, and + retention expiry/cleanup/unavailable markers. + +These corrections remain CL-00 contracts only. No CL-02 implementation or +runtime live-probe feature was started. + ## Mechanical review evidence - `022_protocol_v1_cases.json` parses as JSON. @@ -138,9 +194,17 @@ None. represented as such in case and suite expansion. - Base failure rules contain no control effect; the vision case alone expands the conformance-control rule. +- The two corrected Chat continuation selectors target + `/upstream/requests/1/json/messages/1/tool_call_id`, matching the current + `openai-chat` request builder's assistant-call then tool-result message order. +- The selector-only `022` correction did not change any fixture bytes or fixture + digest; it changes expanded scenario/suite manifest digests as expected for + the corrected pre-release V1 authority. ## Repository verification +Initial acceptance verification: + - `bun run typecheck`: passed. - `bun run privacy:scan`: passed. - `bun test tests/repo-hygiene.test.ts`: 11 passed, 0 failed. @@ -154,12 +218,17 @@ None. - Local link validation, canonical case/digest validation, and `git diff --check`: passed. -The full `bun run test` result is **not green**. On Windows with Bun 1.3.14 it -exited 3 after a cache-invalidation failure, an empty effective-account lookup, -and a Bun `index out of bounds` panic. A broader focused run separately found -four `responses-state.test.ts` failures, all Windows `EPERM` errors creating -symlinks (488 passed, 4 failed). The isolated cache/native tests and the -non-privileged protocol suite pass, but this review does not claim the full +The current CodeRabbit remediation is documentation/contract-only. Final branch +CI/status and the refreshed unresolved-thread set are checked after the status +ledger sync; this record must not be read as claiming a new full local Bun test +run from the connector environment. + +The earlier full `bun run test` result was **not green**. On Windows with Bun +1.3.14 it exited 3 after a cache-invalidation failure, an empty effective-account +lookup, and a Bun `index out of bounds` panic. A broader focused run separately +found four `responses-state.test.ts` failures, all Windows `EPERM` errors +creating symlinks (488 passed, 4 failed). The isolated cache/native tests and +the non-privileged protocol suite passed; this review does not claim the full suite passed. ## Required challenge results @@ -167,20 +236,48 @@ suite passed. 1. Protocol conformance, live compatibility, and task effectiveness are separated: **PASS**. 2. Environmental failures cannot poison compatibility verdicts: **PASS**. -3. `VERIFIED` is reproducible from immutable evidence: **PASS**. +3. `VERIFIED` is reproducible from immutable evidence and cannot be vacuous: + **PASS**. 4. Exact route identity prevents false evidence reuse: **PASS**. 5. Routing Profiles remain the sole compatibility-policy surface: **PASS**. 6. The Lab cannot become a second router: **PASS**. 7. The Lab cannot become a second provider registry: **PASS**. -8. Probes cannot access user data or arbitrary tools: **PASS**. +8. Probes cannot access user data or arbitrary tools, and live destinations, + environment and custom-header fingerprints are fail-closed: **PASS**. 9. Historical incidents are representable as deterministic versioned scenarios: **PASS**. -10. CL-01 is implementable without semantic invention: **PASS**. +10. CL-01 remains implementable without semantic invention after synchronizing + the corrected V1 authority: **PASS WITH REQUIRED CL-01 CORRECTION**. -## Verdict +## CL-01 impact + +The independently accepted CL-01 branch exists at +`feat/cl-01-conformance-harness` and was built on an earlier CL-00 contract tip. +Its acceptance record explicitly documents a harness-only projection of +Chat-wire `messages` tool rows into a synthetic Responses-shaped `input[]` to +satisfy the old CL-00 selectors. That workaround is no longer authoritative: +CL-00 now selects the actual Chat wire `messages[].tool_call_id` field. + +Before CL-01 is stacked or merged it must therefore: -No Critical, High, or Medium findings remain. +- rebase onto the refreshed CL-00 accepted contract head; +- synchronize its copied `protocol-v1-cases.json` authority with the two new + Chat selectors; +- remove or narrow the synthetic Chat-to-Responses `input[]` observation + projection so the asserted upstream JSON remains the actual Chat request; +- align the harness SSE-normalizer contract with source-protocol sentinel + selection (current Chat upstream execution already goes through the + production Chat parser, but the helper/API contract must not retain the stale + client-surface rule); and +- rerun the 24 canonical CL-01 scenarios, negative controls, digest/manifest + checks and its acceptance review. + +This CL-00 remediation does not modify CL-01 and does not start CL-02. + +## Verdict -**CL-00: ACCEPTED** +No validated unresolved Critical, High, Medium, deterministic-contract, or +security-contract finding remains in the CL-00 contract set after the +CodeRabbit remediation above, subject to the final GitHub thread/CI re-check. -CL-01 remains not started and is not authorized by this review. +**CL-00: ACCEPTED AFTER CODERABBIT REMEDIATION** From 1e1e12add74cb431727176e2cc1d2f30f13f3568 Mon Sep 17 00:00:00 2001 From: Wibias <37517432+Wibias@users.noreply.github.com> Date: Sun, 9 Aug 2026 00:07:10 +0200 Subject: [PATCH 14/20] docs(lab): sync stack status after CL-00 remediation --- .../001_pr_stack_status.md | 92 ++++++++++++++----- 1 file changed, 71 insertions(+), 21 deletions(-) diff --git a/devlog/_plan/260807_compatibility_lab/001_pr_stack_status.md b/devlog/_plan/260807_compatibility_lab/001_pr_stack_status.md index 7214263bc..0ccf56740 100644 --- a/devlog/_plan/260807_compatibility_lab/001_pr_stack_status.md +++ b/devlog/_plan/260807_compatibility_lab/001_pr_stack_status.md @@ -1,8 +1,8 @@ # Compatibility Lab PR stack status -Updated throughout the programme. Every phase records its branch, exact base -and implementation head, PR, verification, independent review, blockers, and -whether the next phase is authorized. +Updated throughout the programme. Every phase records its branch, exact +starting/base revision, accepted contract/implementation head, PR, verification, +independent review, blockers, and whether a later phase is authorized. ## Programme facts @@ -17,10 +17,14 @@ whether the next phase is authorized. ## Stack -| Phase | Branch | Base SHA | Implementation head | PR | State | +| Phase | Branch | Starting/base SHA | Accepted head | PR | State | |---|---|---|---|---|---| -| CL-00 | `feat/cl-00-compatibility-contracts` | `3ad5bb6bd3f76f6879d84b78ea39edd3e01ec296` | `12e50a3502fb4af25283538cc717ead2291edd8b` | [draft #1286](https://github.com/lidge-jun/opencodex/pull/1286) | ACCEPTED | -| CL-01 | not created | CL-00 must be accepted first | not started | none | NOT AUTHORIZED | +| CL-00 | `feat/cl-00-compatibility-contracts` | `3ad5bb6bd3f76f6879d84b78ea39edd3e01ec296` | `163e21c3ca57c6f4a6381d00094e4fc23ae11e89` | [draft #1286](https://github.com/lidge-jun/opencodex/pull/1286) | ACCEPTED AFTER CODERABBIT REMEDIATION | +| CL-01 | `feat/cl-01-conformance-harness` | `c2113ca47b8a05c5a5f90679e4eaa640ca2c6a66` | `cc447ce9d19d5fb4e03988899f5fb495f9de8d0e` | [draft Wibias #10](https://github.com/Wibias/opencodex/pull/10) | ACCEPTED EARLIER; REBASE + CONTRACT CORRECTION REQUIRED | + +The CL-01 starting SHA is the exact accepted CL-00 tip recorded by CL-01 when +its implementation began. Its PR base ref names the moving CL-00 branch; that +ref is not a substitute for the historical starting SHA above. ## CL-00 acceptance log @@ -42,11 +46,11 @@ whether the next phase is authorized. - `022_protocol_v1_cases.json` - `030_incident_corpus.md` - `040_security_and_privacy.md` - - `050_cl00_acceptance_review.md` (recorded after independent review) -- Baseline verification on clean CL-00 worktree: + - `050_cl00_acceptance_review.md` (refreshed after CodeRabbit re-review) +- Baseline verification on the original clean CL-00 worktree: - `bun x tsc --noEmit`: passed, 0 errors. - `bun run privacy:scan`: passed. - - `bun run test`: **not green** on this Windows/Bun 1.3.14 host. The full + - `bun run test`: **not green** on the Windows/Bun 1.3.14 host. The full run exited 3 after a cache-invalidation failure, an empty Windows effective-account lookup, and a Bun `panic(main thread): index out of bounds: index 0, len 0`. @@ -58,17 +62,63 @@ whether the next phase is authorized. - Focused continuation state semantics: 2 passed, 95 filtered, 0 failed. - A broader focused run including all `responses-state.test.ts` cases had 488 pass and 4 fail; all four failures were Windows `EPERM` creating - symlinks on this host. + symlinks on that host. - `tests/repo-hygiene.test.ts`: 11 passed, 0 failed. -- Documentation verification: complete. JSON authority parses; all 35 cases, - 46 fixture records, eight suites and fixture digests validate; all local - CL-00 links resolve; `git diff --check` passed. -- Independent acceptance review: accepted after correction; all ten required - challenges pass and no Critical/High/Medium findings remain. -- Blockers: none for CL-00. Full-suite green remains unavailable on this host - for the Windows/Bun reasons above. -- CL-00 ending implementation SHA: - `12e50a3502fb4af25283538cc717ead2291edd8b` (delayed-review corrections); - subsequent ledger sync commits may follow without changing that contract set. +- Original documentation verification: JSON authority parsed; all 35 cases, + 46 fixture records, eight suites and fixture digests validated; all local + CL-00 links resolved; `git diff --check` passed. +- CodeRabbit remediation on 2026-08-08 validated and corrected all ten then- + unresolved threads, including outside-diff security findings: + - exact audit metadata; + - deterministic array ordering in `BehaviorFingerprintV1`; + - non-vacuous applicable-required verification; + - source-protocol `[DONE]` semantics; + - actual Chat `messages[].tool_call_id` result selectors; + - immutable `LabDestinationV1` snapshot semantics; + - empty inherited-environment allowlist plus upper/lower proxy rejection; + - bounded custom-header fingerprinting; + - shared contract-artifact retention across invalidation; and + - matching security acceptance-test obligations. +- The selector-only `022` correction changes expanded scenario/suite manifest + digests but no fixture bytes/digests. The acceptance review records this as a + pre-release V1 contract correction. +- Independent CL-00 acceptance review: refreshed at + `163e21c3ca57c6f4a6381d00094e4fc23ae11e89` after validating the CodeRabbit + findings against current production code/contracts. +- Blockers: none for CL-00 contract acceptance. Full-suite green remains + unavailable from the original Windows/Bun run for the documented unrelated + host failures. Final GitHub CI/status and unresolved-thread state are checked + after this ledger sync. +- CL-00 accepted contract SHA: + `163e21c3ca57c6f4a6381d00094e4fc23ae11e89`; this status-ledger sync follows + without changing the accepted contract semantics. - Draft PR: [#1286](https://github.com/lidge-jun/opencodex/pull/1286). -- CL-01 authorized: **NO**. Acceptance of CL-00 does not start CL-01. + +## CL-01 impact of refreshed CL-00 + +CL-01 was independently accepted at +`cc447ce9d19d5fb4e03988899f5fb495f9de8d0e`, but it was built against the +older CL-00 tip `c2113ca47b8a05c5a5f90679e4eaa640ca2c6a66` and copied the old +Protocol V1 authority. + +Before CL-01 can be stacked/merged it must: + +1. rebase onto the refreshed CL-00 accepted contract head; +2. synchronize the two corrected Chat tool-result selectors in its copied + `src/lab/conformance/fixtures/protocol-v1-cases.json`; +3. remove or narrow the harness-only Chat-wire `messages` -> synthetic + Responses `input[]` observation projection that its acceptance review used + to satisfy the old selectors; +4. align the SSE normalizer API/contract with source-protocol `[DONE]` + selection rather than client-surface selection; and +5. rerun the canonical scenarios, negative controls, digest/manifest checks and + CL-01 acceptance review. + +This is a required CL-01 correction/revalidation, not CL-02 work. + +## Authorization + +- CL-00: **ACCEPTED AFTER CODERABBIT REMEDIATION**. +- CL-01: **EXISTS AND WAS ACCEPTED EARLIER, BUT MUST BE REBASED/CORRECTED BEFORE + STACKING OR MERGE**. +- CL-02: **NOT STARTED / NOT AUTHORIZED BY THIS REMEDIATION**. From a519a6cd9568d124e9c6746cdbb042fd43d19f8e Mon Sep 17 00:00:00 2001 From: Wibias <37517432+Wibias@users.noreply.github.com> Date: Sun, 9 Aug 2026 00:13:54 +0200 Subject: [PATCH 15/20] docs(lab): close ledger reproducibility gaps --- .../010_architecture_and_evidence_contract.md | 167 +++++++++++++++--- 1 file changed, 139 insertions(+), 28 deletions(-) diff --git a/devlog/_plan/260807_compatibility_lab/010_architecture_and_evidence_contract.md b/devlog/_plan/260807_compatibility_lab/010_architecture_and_evidence_contract.md index c2f4aa777..f48166fb0 100644 --- a/devlog/_plan/260807_compatibility_lab/010_architecture_and_evidence_contract.md +++ b/devlog/_plan/260807_compatibility_lab/010_architecture_and_evidence_contract.md @@ -118,15 +118,18 @@ event kinds are: observation claim_snapshot invalidation +purge_tombstone ``` An `observation` records one scenario attempt. A `claim_snapshot` captures the local declared-capability inputs and source revisions needed to reproduce `CLAIMED`: registry/config, cached catalog or native metadata, including the adapter inference currently assembled by `src/routing/capability.ts`. An -`invalidation` identifies prior evidence that a later-discovered harness, -fixture, redaction, or integrity defect makes unusable. Invalidations append; -they never delete or edit prior lines. +`invalidation` identifies prior non-sensitive evidence that a later-discovered +harness, fixture, redaction, or integrity defect makes unusable. Invalidations +append; they never delete or edit prior lines. A `purge_tombstone` is the +privacy-safe exceptional record left after confirmed sensitive evidence is +physically removed under the security contract. Each event has: @@ -180,24 +183,129 @@ supersedes[] effectiveAt ``` +An `invalidation` additionally has: + +```text +targetEventIds[] non-empty, maximum 1024 +reason harness_defect | fixture_defect | + redaction_defect | integrity_defect | + contract_artifact_missing | manual_correction +``` + +Invalidation rules: + +- `targetEventIds` is a set encoded as lowercase event IDs sorted + lexicographically by UTF-8 bytes. Duplicate IDs reject the whole invalidation. +- Every target must be an earlier valid `observation` or `claim_snapshot` in + the same ledger. Unknown, malformed, future, self, `invalidation`, or + `purge_tombstone` targets reject the whole invalidation and report ledger + corruption; projection never applies a partial target list. +- The `reason` set above is closed. Unsupported or missing reasons reject the + invalidation. +- Multiple valid invalidation events may name the same prior target; the target + remains excluded. There is no "uninvalidate" mutation: replacement evidence + is a new observation or claim snapshot. +- Sensitive-evidence deletion does not weaken these rules. It uses the separate + `purge_tombstone` contract below because the sensitive target line may no + longer exist by design. + +A `purge_tombstone` additionally has: + +```text +targetEventIds[] sorted unique lowercase event IDs +targetArtifactDigests[] sorted unique lowercase SHA-256 digests +reason sensitive_evidence +purgeActions[] ledger | sqlite | artifact | scratch | export +``` + +At least one target array is non-empty. The reason is exactly +`sensitive_evidence`; action names are a closed set, sorted and unique. A purge +tombstone may refer to an event line or artifact that has already been +physically removed by the same fail-closed purge. Projection applies valid +purge tombstones before ordinary invalidations, excludes every targeted event, +and treats every targeted artifact as typed `purged_unavailable`. Cached SQLite +rows that depended on a targeted event or artifact are invalid and must be +removed/rebuilt. A prior `VERIFIED`, `PROBED`, `DEGRADED`, `UNSUPPORTED`, or +`CLAIMED` result is never retained solely from purged evidence. + +### `ClaimSourceManifestV1` + +`sourceManifestDigest` is reproducible evidence, not an opaque checksum. Every +claim snapshot references one retained canonical `ClaimSourceManifestV1`: + +```text +{ + "schemaVersion": 1, + "subjectId": string, + "providerId": string, + "clientModelId": string, + "capability": string, + "sources": [ClaimSourceV1, ...], + "resolvedEvidence": RouteCapabilityEvidenceV1 +} +``` + +`ClaimSourceV1` is one record for each consulted source that contributed or +could have contributed to the selected capability: + +```text +{ + "kind": "provider_config" | "provider_registry" | "cached_catalog" | + "native_metadata" | "adapter_inference", + "revision": string | null, + "facts": ClaimCapabilityFactsV1 +} +``` + +`sources` is ordered by the closed kind order shown above and contains at most +one record per kind. `ClaimCapabilityFactsV1` is a closed, sanitized projection +of only capability-relevant inputs used by the current resolver: selected +context window, input modalities, reasoning efforts, catalog capability names, +service-tier support, effective adapter/tool-capable classification, +parallel-tool-call enablement, endpoint locality class +`local|private|unknown`, and canonical-OpenAI-forward classification where +applicable. It never stores a base URL, hostname, credential, account identity, +custom header, arbitrary provider config, catalog path, prompt, or repository +data. `RouteCapabilityEvidenceV1` is the closed resolver output currently +represented by `RouteCapabilityEvidence`: context window, image, tools, +reasoning efforts, service tier, local-only, remote-allowed, and encrypted +Codex-task evidence, with absent fields remaining absent. + +The canonical digest is lowercase: + +```text +sha256( + UTF8("ocx-lab:claim-source-manifest:v1\0") || + UTF8(JCS(ClaimSourceManifestV1)) +) +``` + +The exact canonical bytes are retained as a content-addressed +`claim_source_manifest` artifact while any non-purged claim snapshot references +them. Projection must load those retained bytes, recompute the digest, and +verify matching `subjectId` and `capability` before accepting historical +`CLAIMED` state. Missing bytes, digest mismatch, duplicate/unknown source kinds, +unknown facts, or a subject/capability mismatch makes that claim unusable, +projects no `CLAIMED` result from it, and reports ledger/artifact corruption. + Claim rules: - Claims exist only for `live_route_compatibility`; protocol and task verdicts cannot be `CLAIMED`. - `sourceManifestDigest` identifies the canonical, content-addressed snapshot - of all registry/config/catalog/native/adapter inputs used by the existing - capability resolver. Raw secrets are forbidden. + above of all sanitized registry/config/catalog/native/adapter inputs used by + the existing capability resolver. Raw secrets are forbidden. - `supersedes` explicitly lists every previously current claim event for the same `(subjectId, capability)`. The first snapshot uses an empty list. - A source refresh, removal, or changed polarity appends a new snapshot; it never edits a prior event. `withdrawn` means the local declaration no longer exists. `not_supported` suppresses `CLAIMED` but cannot produce `UNSUPPORTED`. -- Projection first applies invalidations, then removes every event named by a - valid later snapshot's `supersedes`. Exactly one unsuperseded claim may - remain current for a claim key. Multiple unsuperseded claims, a missing - referenced predecessor, a cross-key supersession, or a cycle makes claim - state `UNKNOWN` and reports ledger corruption. +- Projection first applies purge tombstones and invalidations, then removes + every event named by a valid later snapshot's `supersedes`. Exactly one + unsuperseded claim may remain current for a claim key. Multiple unsuperseded + claims, a missing referenced predecessor, a cross-key supersession, or a + cycle makes claim state `UNKNOWN` and reports ledger corruption. - Claim ordering is `effectiveAt`, then `recordedAt`, then event ID, but ordering never substitutes for the explicit supersession graph. @@ -233,20 +341,23 @@ Rules: OpenCodex state. - A structurally invalid or partially written line contributes no evidence and is reported as ledger corruption. SQLite must be rebuildable from all valid - complete lines. - -The canonical scenario manifest, suite manifest and synthetic fixture bytes -referenced by an observation are retained as content-addressed -`scenario_manifest`, `suite_manifest`, and `fixture` artifacts. These contract -artifacts have indefinite retention while any non-invalidated observation -references them. A missing or digest-mismatched contract artifact makes that -observation unusable and yields `harness_failure`; projection code must never -substitute the current manifest for the historical one. + complete lines plus privacy-safe purge tombstones. + +The canonical scenario manifest, suite manifest, synthetic fixture bytes, and +claim-source manifest referenced by valid evidence are retained as +content-addressed `scenario_manifest`, `suite_manifest`, `fixture`, and +`claim_source_manifest` artifacts. Scenario/suite/fixture contract artifacts +remain retained while any non-invalidated, non-purged observation references +them. Claim-source manifests remain retained while any non-purged claim +snapshot references them. A missing or digest-mismatched required contract +artifact makes that evidence unusable and reports corruption; projection code +must never substitute the current manifest for the historical one. The SQLite projection may cache derived verdict rows. Such rows must include -their `asOf`, projection spec, scenario/suite/fixture manifest digests, and -contributing event IDs. Deleting SQLite and replaying JSONL plus the referenced -content-addressed contract artifacts must reproduce them. +their `asOf`, projection spec, scenario/suite/fixture manifest digests, claim +source manifest digest where applicable, and contributing event IDs. Deleting +SQLite and replaying JSONL plus the referenced content-addressed contract +artifacts and purge tombstones must reproduce every non-purged row. ## 3. Canonical verdict contract @@ -312,7 +423,7 @@ The projection must expose: - subject ID and full local subject; - projection algorithm version and `asOf`; - freshness calculation; -- any invalidations and contradictory events considered. +- any invalidations, purge tombstones, and contradictory events considered. An LLM judge, user assertion, registry declaration, successful model listing, or mutable `verified=true` flag cannot produce `VERIFIED`. @@ -391,7 +502,7 @@ state outside this set. Contradictory attributable evidence is never overwritten. The projection: 1. filters by exact subject/layer/suite/scenario versions; -2. applies invalidation events; +2. applies purge tombstones and invalidation events; 3. applies freshness; 4. orders observations by completion time and deterministic event-ID tie-break; 5. applies the suite's contradiction rule; @@ -592,10 +703,10 @@ Semantics: provider-instance fingerprint, client/upstream model IDs, effective adapter, upstream protocol, endpoint fingerprint, and behavior fingerprint. It cannot nest. Records sort by this total order of UTF-8 string comparisons: - role, provider ID, upstream model ID, endpoint fingerprint, client model - ID, effective adapter, upstream protocol, then behavior fingerprint. A - duplicate full key makes subject construction `harness_failure`. An empty - list is canonical when no sidecar is invoked. + role, provider ID, provider-instance fingerprint, upstream model ID, + endpoint fingerprint, client model ID, effective adapter, upstream protocol, + then behavior fingerprint. A duplicate full key makes subject construction + `harness_failure`. An empty list is canonical when no sidecar is invoked. ### Behavior fingerprint allowlist From c94fd1442f7cbc74e93198fa1e5d513c59f7177a Mon Sep 17 00:00:00 2001 From: Wibias <37517432+Wibias@users.noreply.github.com> Date: Sun, 9 Aug 2026 00:14:42 +0200 Subject: [PATCH 16/20] docs(lab): close fixture and MCP execution semantics --- .../021_protocol_v1_manifest_authority.md | 64 +++++++++++++++++-- 1 file changed, 58 insertions(+), 6 deletions(-) diff --git a/devlog/_plan/260807_compatibility_lab/021_protocol_v1_manifest_authority.md b/devlog/_plan/260807_compatibility_lab/021_protocol_v1_manifest_authority.md index 4086ee09b..4135ba967 100644 --- a/devlog/_plan/260807_compatibility_lab/021_protocol_v1_manifest_authority.md +++ b/devlog/_plan/260807_compatibility_lab/021_protocol_v1_manifest_authority.md @@ -50,10 +50,25 @@ freshness manifestDefaults.freshness role: x.role, mediaType: x.mediaType, digest: x.digest, - byteLength: UTF8(x.bytesUtf8).byteLength + byteLength: UTF8(x.bytesUtf8).byteLength, + syntheticMarker: "ocx-lab-synthetic-v1", + provenance: { + kind: "lab_authored", + authority: "022_protocol_v1_cases.json", + sourceCommit: source.sourceCommit + } } ``` +The marker/provenance fields are mandatory for every protocol V1 fixture +reference and participate in the scenario manifest digest. Registration rejects +any fixture reference whose marker is absent/different, whose provenance kind +is not `lab_authored`, whose authority is not the exact authority file above, +or whose source commit differs from the parsed authority. This is the +machine-checkable synthetic-fixture boundary: a fixture cannot be substituted +from user input, a repository, MCP config, an external URL, or a live response +merely because its bytes share a valid digest. + `expandFailureRules(case)` copies `failureRuleSets[manifestDefaults.failureRuleSet]`. When `case.expectedFailure` is absent, that copy is the complete rule list and no @@ -203,7 +218,8 @@ Closed V1 fixture roles are: - `adapter_vector`: decode the fixture JSON and feed its documented fields to the selected adapter boundary without network access; - `synthetic_tool`: decode the fixture JSON into the in-memory inert tool/MCP - stub. It never executes model arguments. + stub and execute only the closed harness action selected below. It never + executes model arguments. Closed media types are `application/json`, `text/event-stream`, `application/vnd.opencodex.adapter-vector+json`, and @@ -215,6 +231,41 @@ defined by the literal vector and scenario assertions; unknown keys reject the fixture. CL-01 must encode those fields as a discriminated union keyed by the scenario ID, not a generic callback or dynamic module. +The four protocol V1 MCP cases each carry exactly one additional closed +`requiredHarnessFeatures` action token. These tokens are executable semantics +and participate in the scenario manifest digest: + +- `mcp_namespace_round_trip_v1`: decode `namespace`, `name`, `description`, and + `inputSchema`; register exactly one inert tool; serialize its upstream name + as `namespace + "__" + name`; then synthesize exactly one completed + client-visible function call with ID `call_fixture`, that flattened name, and + arguments `{}`. Run the normal `toolCalls` -> `mcpCalls` projection. No stub + result, process, filesystem, or network action occurs. +- `mcp_schema_bounds_v1`: decode only `limitBytes`, `exactSchema`, and + `overSchema`; stage an otherwise-identical single inert tool first with + `exactSchema` and then in a fresh transaction with `overSchema`. The first + transaction commits only when its UTF-8 schema bytes equal the limit; the + second must reject atomically before commit when it is exactly one byte over. + It emits no tool call and performs no invocation. +- `mcp_call_result_v1`: decode `namespace`, `name`, `arguments`, and `result`; + register exactly one inert tool; synthesize exactly one completed call with + ID `call_fixture`; invoke the in-memory stub exactly once with the decoded + namespace/name/arguments; the stub returns the literal decoded `result`, + which becomes `/client/response/json`. Duplicate/missing calls or any extra + invocation fail the required verifier. +- `mcp_resource_round_trip_v1`: decode only `resources` and `read`; install them + in the in-memory resource stub; perform exactly one list operation followed + by exactly one read for `read.uri`; expose the literal list as + `/client/response/json/resources` and the matching literal contents as + `/client/response/json/contents`. Missing, duplicate, or extra operations + fail the required assertion. + +A protocol V1 MCP case missing its scenario-specific action token, carrying the +wrong token, carrying more than one of these tokens, or providing fixture keys +outside that token's closed schema rejects registration. No generic +`synthetic_tool` callback, implicit model output, or implementation-defined +invocation is permitted. + The MCP cases use only `in_memory_mcp_stub`. No case authorizes stdio, a child process, user MCP configuration, filesystem access, or a user tool. @@ -416,13 +467,14 @@ CL-01 may materialize and execute only the protocol manifests in the case authority. It must: 1. parse the authority as JSON and reject unknown fields; -2. recompute all fixture, scenario, and suite digests; +2. recompute all fixture, scenario, and suite digests, including the mandatory + synthetic fixture marker/provenance fields; 3. retain the exact case fixture bytes and expanded manifests content-addressably; -4. execute only loopback mocks, closed adapter vectors, and in-memory inert MCP - stubs; +4. execute only loopback mocks, closed adapter vectors, and the four exact + in-memory MCP action tokens above; 5. fail registration rather than invent semantics. This document and the JSON authority authorize no runner, mock server, ledger, SQLite projection, fixture extraction from tests, live probe, or Fabric -ingestion implementation in CL-00. \ No newline at end of file +ingestion implementation in CL-00. From 272408928f93f09ad6200794e3bc54233813bf92 Mon Sep 17 00:00:00 2001 From: Wibias <37517432+Wibias@users.noreply.github.com> Date: Sun, 9 Aug 2026 00:16:41 +0200 Subject: [PATCH 17/20] docs(lab): bind MCP cases to closed harness actions --- .../022_protocol_v1_cases.json | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/devlog/_plan/260807_compatibility_lab/022_protocol_v1_cases.json b/devlog/_plan/260807_compatibility_lab/022_protocol_v1_cases.json index 00f196cd1..491aee981 100644 --- a/devlog/_plan/260807_compatibility_lab/022_protocol_v1_cases.json +++ b/devlog/_plan/260807_compatibility_lab/022_protocol_v1_cases.json @@ -416,7 +416,7 @@ "id": "mcp-core.protocol.namespace-mapping", "suite": "mcp-core", "capability": "tools.mcp.core", - "requirements": { "inboundProtocols": ["openai-responses"], "upstreamProtocols": ["cursor-protobuf"], "surfaces": ["responses-http"], "requiredClaims": ["mcp"], "requiredHarnessFeatures": ["adapter_vector","in_memory_mcp_stub"], "platforms": [], "routePreconditions": [] }, + "requirements": { "inboundProtocols": ["openai-responses"], "upstreamProtocols": ["cursor-protobuf"], "surfaces": ["responses-http"], "requiredClaims": ["mcp"], "requiredHarnessFeatures": ["adapter_vector","in_memory_mcp_stub","mcp_namespace_round_trip_v1"], "platforms": [], "routePreconditions": [] }, "fixture": { "id": "mcp-namespace", "role": "synthetic_tool", "mediaType": "application/vnd.opencodex.mcp-stub+json", "bytesUtf8": "{\"namespace\":\"mcp__fixture\",\"name\":\"lookup\",\"description\":\"fixture\",\"inputSchema\":{\"type\":\"object\",\"properties\":{\"q\":{\"type\":\"string\"}}}}", "digest": "91a53f8c580d461d0f5e0d7209e5d4b95249bdfa8bd3fd4f298e18bdeadb0693" }, "assertions": [ { "id": "wire-name", "operator": "json_path_equals", "selector": "/upstream/requests/0/json/tools/0/name", "expected": "mcp__fixture__lookup", "required": true }, @@ -427,7 +427,7 @@ "id": "mcp-core.protocol.schema-and-bounds", "suite": "mcp-core", "capability": "tools.mcp.core", - "requirements": { "inboundProtocols": ["openai-responses"], "upstreamProtocols": ["cursor-protobuf"], "surfaces": ["responses-http"], "requiredClaims": ["mcp"], "requiredHarnessFeatures": ["adapter_vector","in_memory_mcp_stub"], "platforms": [], "routePreconditions": [] }, + "requirements": { "inboundProtocols": ["openai-responses"], "upstreamProtocols": ["cursor-protobuf"], "surfaces": ["responses-http"], "requiredClaims": ["mcp"], "requiredHarnessFeatures": ["adapter_vector","in_memory_mcp_stub","mcp_schema_bounds_v1"], "platforms": [], "routePreconditions": [] }, "fixture": { "id": "mcp-bounds", "role": "synthetic_tool", "mediaType": "application/vnd.opencodex.mcp-stub+json", "bytesUtf8": "{\"limitBytes\":64,\"exactSchema\":\"{\\\"type\\\":\\\"object\\\",\\\"properties\\\":{\\\"x\\\":{\\\"type\\\":\\\"string\\\"}},\\\"a\\\":\\\"xxx\\\"}\",\"overSchema\":\"{\\\"type\\\":\\\"object\\\",\\\"properties\\\":{\\\"x\\\":{\\\"type\\\":\\\"string\\\"}},\\\"a\\\":\\\"xxxx\\\"}\"}", "digest": "34ff4414dc8e196d460390557f4fd74c32418ea00710167baff2a0dc1f3b643c" }, "assertions": [ { "id": "exact", "operator": "verifier_result_equals", "selector": "/verifiers/exact_bound", "expected": "pass", "required": true }, @@ -439,7 +439,7 @@ "id": "mcp-core.protocol.call-result", "suite": "mcp-core", "capability": "tools.mcp.core", - "requirements": { "inboundProtocols": ["openai-responses"], "upstreamProtocols": ["cursor-protobuf"], "surfaces": ["responses-http"], "requiredClaims": ["mcp"], "requiredHarnessFeatures": ["adapter_vector","in_memory_mcp_stub"], "platforms": [], "routePreconditions": [] }, + "requirements": { "inboundProtocols": ["openai-responses"], "upstreamProtocols": ["cursor-protobuf"], "surfaces": ["responses-http"], "requiredClaims": ["mcp"], "requiredHarnessFeatures": ["adapter_vector","in_memory_mcp_stub","mcp_call_result_v1"], "platforms": [], "routePreconditions": [] }, "fixture": { "id": "mcp-call", "role": "synthetic_tool", "mediaType": "application/vnd.opencodex.mcp-stub+json", "bytesUtf8": "{\"namespace\":\"mcp__fixture\",\"name\":\"lookup\",\"arguments\":{\"q\":\"x\"},\"result\":{\"content\":[{\"type\":\"text\",\"text\":\"RESULT\"}],\"isError\":false}}", "digest": "986ef5017fbdb46eb18b30daaffe72aecc93868b7d89b11c3e244dc084f46496" }, "assertions": [ { "id": "call", "operator": "json_path_equals", "selector": "/verifiers/stub_received", "expected": {"namespace":"mcp__fixture","name":"lookup","arguments":{"q":"x"}}, "required": true }, @@ -450,7 +450,7 @@ "id": "mcp-core.protocol.resource-round-trip", "suite": "mcp-core", "capability": "tools.mcp.core", - "requirements": { "inboundProtocols": ["openai-responses"], "upstreamProtocols": ["cursor-protobuf"], "surfaces": ["responses-http"], "requiredClaims": ["mcp"], "requiredHarnessFeatures": ["adapter_vector","in_memory_mcp_stub"], "platforms": [], "routePreconditions": [] }, + "requirements": { "inboundProtocols": ["openai-responses"], "upstreamProtocols": ["cursor-protobuf"], "surfaces": ["responses-http"], "requiredClaims": ["mcp"], "requiredHarnessFeatures": ["adapter_vector","in_memory_mcp_stub","mcp_resource_round_trip_v1"], "platforms": [], "routePreconditions": [] }, "fixture": { "id": "mcp-resource", "role": "synthetic_tool", "mediaType": "application/vnd.opencodex.mcp-stub+json", "bytesUtf8": "{\"resources\":[{\"uri\":\"fixture://one\",\"name\":\"one\"}],\"read\":{\"uri\":\"fixture://one\",\"contents\":[{\"uri\":\"fixture://one\",\"text\":\"RESOURCE\"}]}}", "digest": "a3f6317374ce92da0155dd14bbf0d5822e8687cbe8ef7968221f23acf8b16aa5" }, "assertions": [ { "id": "list", "operator": "json_path_equals", "selector": "/client/response/json/resources", "expected": [{"uri":"fixture://one","name":"one"}], "required": true }, @@ -458,4 +458,4 @@ ] } ] -} \ No newline at end of file +} From 1510b57febf9785bf96670ac1669c735f2fb9d30 Mon Sep 17 00:00:00 2001 From: Wibias <37517432+Wibias@users.noreply.github.com> Date: Sun, 9 Aug 2026 00:19:17 +0200 Subject: [PATCH 18/20] docs(lab): harden probe and artifact security contracts --- .../040_security_and_privacy.md | 237 +++++++++++++----- 1 file changed, 176 insertions(+), 61 deletions(-) diff --git a/devlog/_plan/260807_compatibility_lab/040_security_and_privacy.md b/devlog/_plan/260807_compatibility_lab/040_security_and_privacy.md index e925a401d..ac8ea1d10 100644 --- a/devlog/_plan/260807_compatibility_lab/040_security_and_privacy.md +++ b/devlog/_plan/260807_compatibility_lab/040_security_and_privacy.md @@ -26,6 +26,29 @@ Scenarios contain Lab-authored synthetic prompts, fixtures, tool definitions and results only. They must be recognizable as synthetic and contain no copied customer material. +### Synthetic-input admission + +Synthetic status is machine-checked, not inferred from content or a friendly +name. Every fixture admitted to the Lab has a canonical reference containing: + +```text +syntheticMarker ocx-lab-synthetic-v1 +provenance.kind lab_authored +provenance.authority reviewed manifest/authority identifier +provenance.sourceCommit immutable source revision +digest domain-separated content digest +byteLength exact admitted byte length +``` + +The reference participates in the scenario manifest digest. Admission verifies +the marker, closed provenance kind, expected authority/source revision, digest, +and byte length before fixture bytes reach an adapter, mock, tool stub, event +normalizer, or artifact writer. A request payload, model output, local path, +repository file, MCP object, URL response, or runtime caller cannot label +itself synthetic. Protocol V1 uses the exact `fixtureRef` contract in +`021_protocol_v1_manifest_authority.md`; future live/Fabric suites must define +an equally closed provenance authority before they may run. + ## 2. Future live-probe sandbox A live probe is an explicit background/management/CLI action. It is never @@ -88,12 +111,22 @@ The future runner must enforce a capability-deny sandbox: ### Credentials -- A credential broker resolves the existing route credential immediately - before the request and binds it to the validated destination. -- Credentials exist in memory for the request only and are never included in a - subject, assertion, error, artifact, log, event ID input, or SQLite row. -- Probe code receives no entire auth store and no unrelated provider/account - credential. +- Secret bytes remain owned by the existing trusted provider credential/ + transport layer. Lab runner code never receives an API key, token, cookie, + authorization value, entire auth store, or credential-bearing header. +- Immediately before a live request, the credential broker validates the exact + immutable `LabDestinationV1` snapshot and returns an opaque, + non-serializable `LabCredentialLeaseV1` capability bound to that destination, + auth transport, and one run/request budget. The lease exposes no secret + bytes, string conversion, header map, equality/debug representation, or + credential identity. +- The trusted transport consumes the lease and injects authorization only after + its connection is bound to the same approved destination/address set. A + destination, Host/SNI, address-set, transport, or lease-scope mismatch fails + closed before authorization is emitted. The lease expires after its bounded + request/run and cannot be reused for another destination or sidecar. +- Credentials therefore never enter a subject, assertion, error, artifact, + log, event-ID input, SQLite row, callback, mock, or Lab-owned memory buffer. - Credential absence/rejection produces `authentication_blocked`, never a compatibility failure. @@ -142,7 +175,7 @@ The future runner must enforce a capability-deny sandbox: - No task repository, prompt transcript, worktree, patch body, terminal log, or hidden reasoning is copied into `~/.opencodex/lab/`. -## 3. Artifact contract +## 3. Artifact and resource contract Artifacts are deny-by-default, normalized, sanitized, bounded, and content-addressed after redaction. @@ -150,30 +183,49 @@ content-addressed after redaction. Initial hard ceilings: ```text -maximum artifacts per run 16 -maximum bytes per artifact 256 KiB -maximum aggregate artifact data 1 MiB -maximum normalized events 4,096 -maximum sanitized string field 4 KiB -maximum serialized bytes per event 64 KiB +maximum wall-clock time per run 120,000 ms +maximum connect time 10,000 ms +maximum first-byte time 30,000 ms +maximum inactivity time 30,000 ms +maximum requests per run 16 +maximum aggregate input bytes 8 MiB +maximum aggregate output bytes 16 MiB +maximum output tokens 32,768 +maximum tool calls 32 +maximum runner resident memory 512 MiB +maximum child processes 0 +maximum artifacts per run 16 +maximum bytes per artifact 256 KiB +maximum aggregate artifact data 1 MiB +maximum normalized events 4,096 +maximum sanitized string field 4 KiB +maximum serialized bytes per event 64 KiB maximum aggregate normalized event bytes 1 MiB -maximum event JSON nesting depth 8 -maximum object keys per event object 64 -maximum array elements per event array 256 +maximum event JSON nesting depth 8 +maximum object keys per event object 64 +maximum array elements per event array 256 ``` -These event bounds are enforced while decoding/normalizing, before an event -is buffered into the observation or any artifact. Exceeding a bound fails the -run as `harness_failure` without retaining the oversized fragment. +Scenario limits may be lower. No scenario, CLI flag, profile, provider config, +environment variable, route metadata, or runtime caller may raise a hard +ceiling. Raising one requires a reviewed security-contract version change. +Timeout ceilings classify with the matching typed `timeout`; request/byte/ +token/tool/artifact ceilings classify `budget_exhausted`. A memory ceiling +breach or attempted child-process creation terminates the sandboxed run and is +`budget_exhausted`; inability to enforce any required ceiling is +`harness_failure` before executable evidence is accepted. -Scenario limits may be lower. Raising a hard ceiling requires a reviewed -security-contract change; a scenario manifest alone cannot raise it. +Event bounds are enforced while decoding/normalizing, before an event is +buffered into the observation or any artifact. Exceeding an event structural +bound fails the run as `harness_failure` without retaining the oversized +fragment. Allowed artifact classes: - canonical scenario manifest; - canonical suite manifest; - canonical synthetic fixture; +- canonical claim-source manifest; - assertion report containing normalized expected/observed summaries; - sanitized request shape with content replaced by type/length/digest markers; - sanitized response shape with visible synthetic fixture output only; @@ -181,15 +233,43 @@ Allowed artifact classes: - sanitized error taxonomy/status; - deterministic verifier summary. -Artifact paths are derived from the SHA-256 digest and fixed extension under -`~/.opencodex/lab/artifacts/`. Manifests reject traversal, symlinks, -device/special files, alternate data streams, and digest/size mismatch. The -ledger stores relative content-addressed references, never arbitrary paths. +Artifact names are derived only from the expected lowercase SHA-256 digest and +a fixed extension under `~/.opencodex/lab/artifacts/`. The Lab artifact store +must use descriptor/handle-bound, no-follow I/O rather than validate a pathname +and reopen it later: + +- open and retain a trusted handle to the artifact directory after verifying it + is a directory and not a symlink/reparse-point redirection; +- create writes relative to that directory handle with exclusive/no-follow + semantics, reject special files and `st_nlink != 1`, and write only already + redacted bytes; +- compute size and digest from the same open descriptor/handle that received + the bytes, flush it, then publish by an atomic rename relative to the trusted + directory handle; +- when a content-addressed target already exists, open it with no-follow + semantics and verify regular-file type, single-link status, size, and digest + from that same descriptor before reuse; +- reads open the digest-derived name relative to the trusted directory handle, + verify file type/link count/size, hash the bytes, and return/consume those + exact bytes from the same open descriptor without closing and reopening by + path; +- traversal, separators, absolute paths, alternate data streams, symlinks, + reparse points, hard links, device/special files, digest/size mismatch, or a + directory-handle identity change fail closed; +- on a platform where equivalent descriptor/handle-bound no-follow operations + cannot be enforced, the artifact operation is `harness_failure`; it must not + fall back to path-only `exists/stat/read` validation. + +The ledger stores relative content-addressed references, never arbitrary paths. +This contract applies to the future Lab artifact store; it does not silently +reuse a different existing application artifact reader whose path-race +properties have not been reviewed for Lab evidence. Scenario/suite manifests and synthetic fixtures use the domain-separated digests in the evidence contract and remain retained while referenced by any -non-invalidated observation. Their content is still subject to the same -synthetic-data and size rules. +non-invalidated, non-purged observation. Claim-source manifests remain retained +while a non-purged claim snapshot references them. Their content is still +subject to the same synthetic/sanitized-data and size rules. Redaction occurs before hashing and writing. A redaction failure discards the artifact and marks the run `harness_failure`; "write now, redact later" is @@ -281,7 +361,8 @@ Public publishing is not authorized in CL-00 and remains a later phase. - JSONL is the immutable local authority for non-sensitive evidence, but a user can delete the entire Lab directory. Immutability describes in-ledger - correction semantics, not a promise to resist user deletion. + correction semantics, not a promise to resist user deletion. Confirmed + sensitive evidence is the explicit privacy exception below. - Retention ceilings by class: - scratch/temp run directories: deleted at run end; cleanup retry within 24h; - export staging: maximum 24h; @@ -290,51 +371,85 @@ Public publishing is not authorized in CL-00 and remains a later phase. - sanitized non-contract artifacts (`assertion_report`, shapes, traces, errors): default 90 days, hard ceiling 365 days; - content-addressed scenario/suite/fixture contract artifacts: retained - while any non-invalidated observation references them, because - reproducible `VERIFIED` projection requires the exact historical bytes. - Their content remains synthetic-only and size-bounded. User deletion of - the Lab directory remains absolute. + while any non-invalidated, non-purged observation references them, because + reproducible executable projection requires the exact historical bytes; + - content-addressed claim-source manifests: retained while any non-purged + claim snapshot references them, because reproducible `CLAIMED` projection + requires the exact historical sanitized source bytes. + Contract artifacts remain synthetic/sanitized and size-bounded. User deletion + of the Lab directory remains absolute. - Deleting an expired non-contract artifact leaves its digest/reference and a typed unavailable marker; it does not alter the observation. -- SQLite is disposable and contains no data absent from valid ledger events and - artifact metadata. -- Invalid non-sensitive evidence is neutralized by an appended invalidation. - Event-private non-contract artifacts may then be securely deleted. A shared - scenario, suite, or fixture contract artifact must remain while any other - non-invalidated observation references its digest, and may be deleted only - after the last such reference is gone. +- SQLite is disposable and contains no data absent from valid ledger events, + privacy-safe purge tombstones, and artifact metadata. +- Invalid non-sensitive evidence is neutralized by a valid appended + invalidation under the evidence contract. Event-private non-contract + artifacts may then be securely deleted. A shared scenario, suite, fixture, + or claim-source contract artifact must remain while any other usable event + references its digest, and may be deleted only after the last such reference + is invalidated or purged. - Confirmed sensitive evidence is distinct from ordinary invalidation. It - requires a fail-closed purge of every local copy: JSONL lines containing the - leak, SQLite rows, artifacts, scratch/temp files, and generated exports. The - purge record stores only taxonomy, time, affected event/artifact digests, - and action taken — never the leaked value. Append-only semantics never - override that duty. + requires a fail-closed purge of every local copy: offending JSONL event + lines, SQLite rows, artifacts, scratch/temp files, and generated exports. + The purge first determines the affected event IDs/artifact digests without + retaining the leaked value, then creates a clean replacement ledger that + omits the sensitive event lines and includes the canonical privacy-safe + `purge_tombstone` defined in the evidence contract. The replacement ledger + is flushed and atomically installed; SQLite is deleted/rebuilt from it and + targeted artifacts/temp/exports are removed. If any required replacement or + deletion cannot be completed, the purge remains visibly failed and evidence + projection is disabled rather than serving stale compatibility state. +- Projection applies purge tombstones before ordinary invalidation/supersession. + Targeted events contribute no `CLAIMED`, `PROBED`, `VERIFIED`, `DEGRADED`, or + `UNSUPPORTED` state; targeted artifacts surface only + `purged_unavailable`. A previously cached verdict that depended on purged + material is invalid and must not survive SQLite rebuild. +- A purge tombstone stores only event IDs/artifact digests, the fixed + `sensitive_evidence` taxonomy, time/producer metadata, and closed action + names. It never records the leaked value, raw path, credential, prompt, or + identifying diagnostic. Append-only semantics never override the duty to + remove sensitive bytes. ## 8. Security acceptance tests required later Before any live runner ships, tests must prove: 1. prompt/repository/MCP/user-tool inputs are unreachable from the scenario DSL; -2. redirects and model-supplied URLs cannot widen network access, and Lab +2. every admitted fixture has the required synthetic marker/provenance and a + runtime/user/repository/MCP/network object cannot self-assert synthetic + status; +3. redirects and model-supplied URLs cannot widen network access, and Lab clients pin connections to the validated IP set; -3. the inherited-environment allowlist is empty, all uppercase/lowercase proxy +4. the inherited-environment allowlist is empty, all uppercase/lowercase proxy variables are rejected, and no ambient variable changes Lab behavior; -4. destination-record mutation, replacement, or address-set drift between +5. destination-record mutation, replacement, or address-set drift between authorization, fingerprinting, credential binding, and connect fails closed; -5. credential, account, custom header and endpoint canaries never enter - evidence, errors, SQLite or artifacts; -6. custom-header canonicalization is deterministic; unknown credential +6. credential canaries never enter Lab memory/objects through the broker: the + runner receives only an opaque destination-bound one-run lease, and lease + scope/transport/destination mismatch fails before authorization is sent; +7. account, custom-header and endpoint canaries never enter evidence, errors, + SQLite or artifacts; +8. custom-header canonicalization is deterministic; unknown credential classification and every count/name/value/aggregate bound fail closed; -7. local subject-salt rotation breaks prior correlation and forces +9. local subject-salt rotation breaks prior correlation and forces re-projection/reverification without reverse lookup; -8. tool arguments cannot execute; -9. artifact traversal/symlink/oversize/digest attacks fail closed; -10. normalized event byte/depth/key/array ceilings fail closed before buffering; -11. timeout, quota, auth, DNS and harness failures remain blockers; -12. retention expiry emits typed unavailable markers; cleanup retry/failure is +10. tool arguments cannot execute; +11. every wall-clock/connect/first-byte/inactivity/request/input/output/token/ + tool-call/memory/process/artifact hard ceiling is enforced and cannot be + widened by a manifest, config, profile, environment variable, or caller; +12. artifact traversal, symlink/reparse, hard-link, path-race, oversize and + digest attacks fail closed, and validated artifact bytes are consumed from + the same descriptor/handle that was checked; +13. normalized event byte/depth/key/array ceilings fail closed before buffering; +14. timeout, quota, auth, DNS and harness failures remain blockers; +15. retention expiry emits typed unavailable markers; cleanup retry/failure is visible and bounded; shared contract artifacts survive invalidation while - any non-invalidated observation still references them; -13. confirmed sensitive evidence is purged from JSONL, SQLite, artifacts, temp - files and exports without recording the leaked value; -14. public export rejects unknown/private fields; -15. no probe runs from the production routing path. + any usable event still references them; +16. ordinary invalidations reject unknown/future/cross-kind/partial target + lists and deterministically remove only valid named evidence; +17. confirmed sensitive evidence is removed from JSONL, SQLite, artifacts, + temp files and exports; the replacement ledger retains only a privacy-safe + purge tombstone; replay cannot preserve a verdict that depended on purged + evidence; +18. public export rejects unknown/private fields; +19. no probe runs from the production routing path. From c014464237fd3c95bda08bc18bfab8ba8f532308 Mon Sep 17 00:00:00 2001 From: Wibias <37517432+Wibias@users.noreply.github.com> Date: Sun, 9 Aug 2026 00:26:27 +0200 Subject: [PATCH 19/20] docs(lab): finalize CL-00 CodeRabbit acceptance --- .../050_cl00_acceptance_review.md | 209 +++++++++++------- 1 file changed, 130 insertions(+), 79 deletions(-) diff --git a/devlog/_plan/260807_compatibility_lab/050_cl00_acceptance_review.md b/devlog/_plan/260807_compatibility_lab/050_cl00_acceptance_review.md index 4392be86f..68a6a0e0b 100644 --- a/devlog/_plan/260807_compatibility_lab/050_cl00_acceptance_review.md +++ b/devlog/_plan/260807_compatibility_lab/050_cl00_acceptance_review.md @@ -6,15 +6,14 @@ Scope: the complete CL-00 contract set on `feat/cl-00-compatibility-contracts`, based on `3ad5bb6bd3f76f6879d84b78ea39edd3e01ec296`. The initial delayed-review acceptance was recorded at `12e50a3502fb4af25283538cc717ead2291edd8b`. -A later CodeRabbit re-review was validated against current production code and -contract semantics; its accepted contract corrections are incorporated through -`b62e29395d25b2cd9a1dfd852101dbf7a2c91507` and this record supersedes the -stale acceptance/status statements from the earlier pass. +CodeRabbit re-reviews were validated against current production code and the +contract authority before any change was accepted. This record supersedes the +stale acceptance/status statements from the earlier passes. -The review was read-only and separate from the original authoring pass. Every -validated Critical, High, and Medium finding was corrected and re-reviewed -before this record was finalized. Valid deterministic/security contract defects -were also corrected even when review severity was lower. +The review is contract-focused and separate from the original authoring pass. +Every validated Critical, High, and Medium finding was corrected. Valid +deterministic and security-contract defects were also corrected regardless of +review label. CL-02 was not started. ## Findings and corrections @@ -128,15 +127,14 @@ None. append-only, limited the profile claim to compatibility policy, and noted the explicit state-mutating `ocx doctor --fix-codex-runtime` mode. -## CodeRabbit re-review remediation +## CodeRabbit remediation: first pass -All ten unresolved CodeRabbit threads visible on PR #1286 were inspected against -current branch code/contracts before editing. The valid findings were resolved -as follows: +All ten unresolved CodeRabbit threads in the first remediation pass were +inspected against current branch code/contracts before editing. 1. Stack audit metadata used a non-SHA dependency label where an exact CL-01 - base revision is required. The stack ledger is refreshed after this review - with exact base/head fields and the CL-01 correction requirement. + base revision is required. The stack ledger now records exact base/head + revisions and the CL-01 correction requirement. 2. `BehaviorFingerprintV1` did not define deterministic ordering for every array-valued closed key. V1 now classifies each allowed array as `set` or `ordered`, defines JCS-byte sorting/deduplication for sets, preserves source @@ -147,63 +145,110 @@ as follows: claim/unknown semantics, while attempted environmental blockers remain `BLOCKED`. 4. `[DONE]` handling was selected by the client-facing surface even for a Chat - upstream response fixture. Sentinel interpretation now follows the protocol - of the byte stream being normalized: an `upstream_response` uses its single - resolved upstream protocol, and only OpenAI Chat recognizes exact `[DONE]`. + upstream fixture. Sentinel interpretation now follows the protocol of the + byte stream being normalized; only OpenAI Chat recognizes exact `[DONE]`. 5. Two Chat-backed tool-result assertions incorrectly selected Responses `input[].call_id`. Production `openai-chat` emits the continuation as - `messages[1].tool_call_id`; both canonical selectors now assert that actual - Chat wire shape. + `messages[1].tool_call_id`; both selectors now assert the actual Chat wire. 6. Live-probe destination authorization could drift between endpoint fingerprinting, credential binding and connect. The security contract now requires one immutable per-run `LabDestinationV1` snapshot for every stage and fails closed before credential transmission on mutation/re-resolution or mismatch. -7. Ambient environment/proxy handling was not executable enough. The V1 - inherited-environment allowlist is empty; the runner constructs only - `TZ=UTC` and `NO_COLOR=1`, rejects uppercase and lowercase proxy variables, - and cannot derive behavior from ambient variables. +7. Ambient environment/proxy handling was not executable enough. The inherited + environment allowlist is empty; the runner constructs only `TZ=UTC` and + `NO_COLOR=1`, rejects uppercase/lowercase proxy variables, and cannot derive + behavior from ambient variables. 8. Custom-header fingerprinting lacked resource/canonicalization bounds. The broker now enforces entry, duplicate-value, field-name, per-value and aggregate byte ceilings before JCS/HMAC, with unknown credential classification or overflow failing closed. 9. Ordinary invalidation wording allowed deletion of shared contract artifacts. Only event-private non-contract artifacts may be deleted after invalidation; - shared scenario/suite/fixture artifacts survive until no non-invalidated - observation references them. + shared scenario/suite/fixture artifacts survive until no usable observation + references them. 10. Security acceptance coverage omitted the new invariants. Required tests now cover environment/proxy denial, destination snapshot mutation/address drift, custom-header canonicalization and bounds, subject-salt rotation, and retention expiry/cleanup/unavailable markers. -These corrections remain CL-00 contracts only. No CL-02 implementation or -runtime live-probe feature was started. +## CodeRabbit remediation: second pass + +CodeRabbit reviewed the remediation again and raised additional deterministic +and security-contract issues. Every Major finding in that pass was validated as +material and corrected within CL-00: + +1. `invalidation` had no executable payload. It now has a non-empty bounded, + sorted/unique target-event set, a closed reason set, all-or-nothing target + validation, earlier-event/type constraints, and no implicit uninvalidation. +2. `sourceManifestDigest` was not reproducible. `ClaimSourceManifestV1` now + defines a closed sanitized source snapshot, a domain-separated digest, + content-addressed retention, and replay validation. Missing/mismatched source + bytes cannot produce `CLAIMED`. +3. Sidecar dependency sorting omitted `providerInstanceFingerprint`, so two + otherwise-equal dependencies could compare as equal. The canonical total + ordering now includes it. +4. Synthetic fixture trust was prose-only. Every expanded protocol V1 + `fixtureRef` now includes mandatory `ocx-lab-synthetic-v1` marker and + `lab_authored` provenance bound to the authority and source commit; these + fields participate in the scenario-manifest digest and are validated before + fixture admission. +5. `synthetic_tool` did not say what MCP cases execute. The four MCP V1 cases + now carry exact scenario-specific action tokens with closed fixture schemas, + deterministic invocation/list/read/boundary behavior, and fail-closed + registration for missing/wrong/multiple actions. +6. The credential broker could still expose the selected secret to Lab code. + The contract now keeps secret bytes in trusted credential/transport plumbing + and gives the Lab only a non-serializable, destination/auth-transport-bound, + one-run/request `LabCredentialLeaseV1` capability. +7. Required execution limits had no hard maxima. V1 now freezes hard ceilings + for wall/connect/first-byte/inactivity time, requests, aggregate input/output + bytes, output tokens, tool calls, resident memory, child processes and + artifacts; manifests/config/profiles/environment/callers can only tighten + them. +8. Artifact path validation was vulnerable to path-race/symlink substitution if + implemented like the unrelated current image-artifact helper. The future Lab + store is now required to use trusted-directory-handle, no-follow, + descriptor-bound validation/read/write and atomic publication. CL-00 did not + alter `src/images/artifacts.ts`; that runtime is outside this contract-only + phase. +9. Physical sensitive-evidence purge did not define canonical projection state. + The ledger now defines privacy-safe `purge_tombstone` events, clean atomic + ledger replacement, SQLite rebuild/removal, typed `purged_unavailable` + artifacts, and mandatory exclusion of every verdict/claim that depended on + purged evidence. + +The protocol-authority rewrite also fixed the flagged missing final newline. +Two remaining review notes were editorial-only (`falsey` -> `falsy` in the +incident prose and a master-plan acceptance phrase); they do not change any +contract, deterministic behavior, security boundary, acceptance criterion, or +CL-01 implementation input and are handled as non-blocking review-thread +responses rather than expanding this contract remediation. ## Mechanical review evidence -- `022_protocol_v1_cases.json` parses as JSON. -- 35 unique cases cover all required members of the eight initial suites. -- 46 fixture artifacts are present: 35 primary vectors and 11 initiating - requests. -- Every fixture digest matches - `sha256("ocx-lab:fixture:v1\0" || UTF8(bytesUtf8))`. -- Every response fixture has one initiating request. -- The MCP bound vector is exactly 64/65 UTF-8 bytes. -- All named verifier selectors have one closed deterministic definition. -- `vision-core.protocol.modality-gate` is the sole V1 negative control and is - represented as such in case and suite expansion. -- Base failure rules contain no control effect; the vision case alone expands - the conformance-control rule. +- `022_protocol_v1_cases.json` remains valid JSON by inspection through the + GitHub file API and contains the same 35 case objects / 46 fixture records as + the accepted authority. +- Fixture `bytesUtf8` and fixture digest values were not changed by the + CodeRabbit selector/provenance/MCP-action remediation. - The two corrected Chat continuation selectors target `/upstream/requests/1/json/messages/1/tool_call_id`, matching the current `openai-chat` request builder's assistant-call then tool-result message order. -- The selector-only `022` correction did not change any fixture bytes or fixture - digest; it changes expanded scenario/suite manifest digests as expected for - the corrected pre-release V1 authority. +- `fixtureRef` expansion now adds mandatory marker/provenance fields. Therefore + all expanded protocol scenario-manifest digests and dependent suite-manifest + digests change even though fixture bytes/digests remain unchanged. +- Four MCP `requiredHarnessFeatures` arrays now additionally contain their exact + closed action token. Those four scenario-manifest digests therefore also + change for semantic reasons. +- `vision-core.protocol.modality-gate` remains the sole V1 negative control. +- Base failure rules contain no control effect; the vision case alone expands + the conformance-control rule. +- The MCP bound fixture remains the accepted exact 64/65 UTF-8-byte vector. ## Repository verification -Initial acceptance verification: +Initial acceptance verification remains the last executed local-suite evidence: - `bun run typecheck`: passed. - `bun run privacy:scan`: passed. @@ -211,17 +256,15 @@ Initial acceptance verification: - Focused protocol/compatibility suite excluding Windows privileged-symlink state cases: 395 passed, 0 failed across 24 files. - Focused continuation-state semantics: 2 passed, 95 filtered, 0 failed. -- Serial isolation of failures observed in the full run: - - `tests/codex-models-cache-invalidate.test.ts`: 6 passed, 0 failed. - - `tests/codex-native-residue.test.ts`: 63 passed, 2 platform skips, - 0 failed. -- Local link validation, canonical case/digest validation, and - `git diff --check`: passed. - -The current CodeRabbit remediation is documentation/contract-only. Final branch -CI/status and the refreshed unresolved-thread set are checked after the status -ledger sync; this record must not be read as claiming a new full local Bun test -run from the connector environment. +- `tests/codex-models-cache-invalidate.test.ts`: 6 passed, 0 failed. +- `tests/codex-native-residue.test.ts`: 63 passed, 2 platform skips, 0 failed. +- Original local link/case/digest checks and `git diff --check`: passed. + +The CodeRabbit remediation is documentation/contract-only. The GitHub connector +does not provide a local Bun execution environment, so this review does **not** +claim a new typecheck/privacy/test run after these documentation changes. Final +GitHub status/workflow contexts and unresolved review threads are checked after +the status-ledger sync. The earlier full `bun run test` result was **not green**. On Windows with Bun 1.3.14 it exited 3 after a cache-invalidation failure, an empty effective-account @@ -236,48 +279,56 @@ suite passed. 1. Protocol conformance, live compatibility, and task effectiveness are separated: **PASS**. 2. Environmental failures cannot poison compatibility verdicts: **PASS**. -3. `VERIFIED` is reproducible from immutable evidence and cannot be vacuous: +3. `VERIFIED` is reproducible, non-vacuous, and invalidation/purge aware: + **PASS**. +4. Exact route/dependency identity prevents false evidence reuse: **PASS**. +5. `CLAIMED` is reproducible from retained sanitized source manifests: **PASS**. -4. Exact route identity prevents false evidence reuse: **PASS**. -5. Routing Profiles remain the sole compatibility-policy surface: **PASS**. -6. The Lab cannot become a second router: **PASS**. -7. The Lab cannot become a second provider registry: **PASS**. -8. Probes cannot access user data or arbitrary tools, and live destinations, - environment and custom-header fingerprints are fail-closed: **PASS**. -9. Historical incidents are representable as deterministic versioned +6. Routing Profiles remain the sole compatibility-policy surface: **PASS**. +7. The Lab cannot become a second router or provider registry: **PASS**. +8. Synthetic-fixture admission, credentials, destinations, environment, + resources, artifacts and purge behavior are fail-closed: **PASS**. +9. Historical incidents remain representable as deterministic versioned scenarios: **PASS**. 10. CL-01 remains implementable without semantic invention after synchronizing - the corrected V1 authority: **PASS WITH REQUIRED CL-01 CORRECTION**. + the refreshed V1 authority: **PASS WITH REQUIRED CL-01 REBASE, CORRECTION, + AND REVALIDATION**. ## CL-01 impact The independently accepted CL-01 branch exists at -`feat/cl-01-conformance-harness` and was built on an earlier CL-00 contract tip. +`feat/cl-01-conformance-harness` at accepted head +`cc447ce9d19d5fb4e03988899f5fb495f9de8d0e`. It was built from the older CL-00 +revision `c2113ca47b8a05c5a5f90679e4eaa640ca2c6a66`. + Its acceptance record explicitly documents a harness-only projection of Chat-wire `messages` tool rows into a synthetic Responses-shaped `input[]` to satisfy the old CL-00 selectors. That workaround is no longer authoritative: -CL-00 now selects the actual Chat wire `messages[].tool_call_id` field. +CL-00 selects the actual Chat `messages[].tool_call_id` field. CL-01 also copied +the pre-remediation case authority and its SSE helper retained client-surface +sentinel selection. Before CL-01 is stacked or merged it must therefore: -- rebase onto the refreshed CL-00 accepted contract head; -- synchronize its copied `protocol-v1-cases.json` authority with the two new - Chat selectors; +- rebase onto the final refreshed CL-00 accepted contract head; +- synchronize the two corrected Chat result selectors; - remove or narrow the synthetic Chat-to-Responses `input[]` observation - projection so the asserted upstream JSON remains the actual Chat request; -- align the harness SSE-normalizer contract with source-protocol sentinel - selection (current Chat upstream execution already goes through the - production Chat parser, but the helper/API contract must not retain the stale - client-surface rule); and -- rerun the 24 canonical CL-01 scenarios, negative controls, digest/manifest - checks and its acceptance review. + projection so upstream observations remain the actual Chat request; +- align SSE normalization with source-protocol `[DONE]` selection; +- implement/validate the mandatory synthetic fixture marker/provenance in + expanded `fixtureRef` values and recompute all scenario/suite manifests; +- synchronize the four exact MCP action tokens and their closed execution + semantics; and +- rerun the canonical CL-01 scenarios, negative controls, digest/manifest + checks and independent CL-01 acceptance review. This CL-00 remediation does not modify CL-01 and does not start CL-02. ## Verdict -No validated unresolved Critical, High, Medium, deterministic-contract, or -security-contract finding remains in the CL-00 contract set after the -CodeRabbit remediation above, subject to the final GitHub thread/CI re-check. +All validated Critical, High, Medium, deterministic-contract and +security-contract findings found through the two CodeRabbit remediation passes +are corrected in the CL-00 contract set. Final GitHub thread/status checks are +recorded after the stack ledger is synchronized. **CL-00: ACCEPTED AFTER CODERABBIT REMEDIATION** From fe7ae543aeee1e950f23d08e70be34f9bc8ba88a Mon Sep 17 00:00:00 2001 From: Wibias <37517432+Wibias@users.noreply.github.com> Date: Sun, 9 Aug 2026 00:26:52 +0200 Subject: [PATCH 20/20] docs(lab): sync final CL-00 remediation status --- .../001_pr_stack_status.md | 154 +++++++++--------- 1 file changed, 77 insertions(+), 77 deletions(-) diff --git a/devlog/_plan/260807_compatibility_lab/001_pr_stack_status.md b/devlog/_plan/260807_compatibility_lab/001_pr_stack_status.md index 0ccf56740..382543a24 100644 --- a/devlog/_plan/260807_compatibility_lab/001_pr_stack_status.md +++ b/devlog/_plan/260807_compatibility_lab/001_pr_stack_status.md @@ -19,26 +19,20 @@ independent review, blockers, and whether a later phase is authorized. | Phase | Branch | Starting/base SHA | Accepted head | PR | State | |---|---|---|---|---|---| -| CL-00 | `feat/cl-00-compatibility-contracts` | `3ad5bb6bd3f76f6879d84b78ea39edd3e01ec296` | `163e21c3ca57c6f4a6381d00094e4fc23ae11e89` | [draft #1286](https://github.com/lidge-jun/opencodex/pull/1286) | ACCEPTED AFTER CODERABBIT REMEDIATION | -| CL-01 | `feat/cl-01-conformance-harness` | `c2113ca47b8a05c5a5f90679e4eaa640ca2c6a66` | `cc447ce9d19d5fb4e03988899f5fb495f9de8d0e` | [draft Wibias #10](https://github.com/Wibias/opencodex/pull/10) | ACCEPTED EARLIER; REBASE + CONTRACT CORRECTION REQUIRED | +| CL-00 | `feat/cl-00-compatibility-contracts` | `3ad5bb6bd3f76f6879d84b78ea39edd3e01ec296` | `c014464237fd3c95bda08bc18bfab8ba8f532308` | [#1286](https://github.com/lidge-jun/opencodex/pull/1286) | ACCEPTED AFTER CODERABBIT REMEDIATION | +| CL-01 | `feat/cl-01-conformance-harness` | `c2113ca47b8a05c5a5f90679e4eaa640ca2c6a66` | `cc447ce9d19d5fb4e03988899f5fb495f9de8d0e` | [draft Wibias #10](https://github.com/Wibias/opencodex/pull/10) | ACCEPTED EARLIER; REBASE + CONTRACT CORRECTION + REVALIDATION REQUIRED | -The CL-01 starting SHA is the exact accepted CL-00 tip recorded by CL-01 when -its implementation began. Its PR base ref names the moving CL-00 branch; that -ref is not a substitute for the historical starting SHA above. +The CL-01 starting SHA is the exact CL-00 tip recorded when CL-01 began. Its +moving base-ref name is not a substitute for that historical SHA. ## CL-00 acceptance log -- Live-tree audit: complete against the exact base above. Audited provider - registry/derivation, Routing Profile types/normalization/evaluator/API/UI/ - dry-run, route traces and Why-this-route, usage/request-history/analytics, - doctor/provider connectivity validation, protocol regression tests, and - relevant open/closed devlog incidents. -- Live-tree correction: generated Cursor task/grind protobuf messages exist, - but no native Agent Fabric task persistence or management contract and no - Compatibility Lab implementation exists. CL-00 freezes consumer semantics - without claiming production endpoints. Future compatibility policy must - extend the shipped Routing Profiles system. -- Documents: +- Live-tree audit covered provider registry/derivation, Routing Profiles, + routing traces, request history/analytics, doctor/connectivity validation, + protocol regression tests, and relevant incident/devlog records. +- CL-00 remains contract-only. No Compatibility Lab runtime, live runner, + profile/router implementation, or CL-02 work was added. +- Contract documents: - `000_master_plan.md` - `010_architecture_and_evidence_contract.md` - `020_scenario_contract_and_catalogue.md` @@ -46,79 +40,85 @@ ref is not a substitute for the historical starting SHA above. - `022_protocol_v1_cases.json` - `030_incident_corpus.md` - `040_security_and_privacy.md` - - `050_cl00_acceptance_review.md` (refreshed after CodeRabbit re-review) -- Baseline verification on the original clean CL-00 worktree: - - `bun x tsc --noEmit`: passed, 0 errors. + - `050_cl00_acceptance_review.md` +- Original baseline verification: + - `bun x tsc --noEmit`: passed. - `bun run privacy:scan`: passed. - - `bun run test`: **not green** on the Windows/Bun 1.3.14 host. The full - run exited 3 after a cache-invalidation failure, an empty Windows - effective-account lookup, and a Bun - `panic(main thread): index out of bounds: index 0, len 0`. - - Serial isolation: - `tests/codex-models-cache-invalidate.test.ts` passed 6/6; - `tests/codex-native-residue.test.ts` passed 63 with 2 platform skips. - - Focused protocol/compatibility suite excluding privileged-symlink state - cases: 395 passed, 0 failed across 24 files. - - Focused continuation state semantics: 2 passed, 95 filtered, 0 failed. - - A broader focused run including all `responses-state.test.ts` cases had - 488 pass and 4 fail; all four failures were Windows `EPERM` creating - symlinks on that host. - `tests/repo-hygiene.test.ts`: 11 passed, 0 failed. -- Original documentation verification: JSON authority parsed; all 35 cases, - 46 fixture records, eight suites and fixture digests validated; all local - CL-00 links resolved; `git diff --check` passed. -- CodeRabbit remediation on 2026-08-08 validated and corrected all ten then- - unresolved threads, including outside-diff security findings: - - exact audit metadata; - - deterministic array ordering in `BehaviorFingerprintV1`; - - non-vacuous applicable-required verification; - - source-protocol `[DONE]` semantics; - - actual Chat `messages[].tool_call_id` result selectors; - - immutable `LabDestinationV1` snapshot semantics; - - empty inherited-environment allowlist plus upper/lower proxy rejection; - - bounded custom-header fingerprinting; - - shared contract-artifact retention across invalidation; and - - matching security acceptance-test obligations. -- The selector-only `022` correction changes expanded scenario/suite manifest - digests but no fixture bytes/digests. The acceptance review records this as a - pre-release V1 contract correction. -- Independent CL-00 acceptance review: refreshed at - `163e21c3ca57c6f4a6381d00094e4fc23ae11e89` after validating the CodeRabbit - findings against current production code/contracts. -- Blockers: none for CL-00 contract acceptance. Full-suite green remains - unavailable from the original Windows/Bun run for the documented unrelated - host failures. Final GitHub CI/status and unresolved-thread state are checked - after this ledger sync. -- CL-00 accepted contract SHA: - `163e21c3ca57c6f4a6381d00094e4fc23ae11e89`; this status-ledger sync follows - without changing the accepted contract semantics. -- Draft PR: [#1286](https://github.com/lidge-jun/opencodex/pull/1286). + - focused protocol/compatibility: 395 passed, 0 failed across 24 files. + - continuation semantics: 2 passed, 95 filtered, 0 failed. + - isolated cache invalidation: 6 passed, 0 failed. + - isolated native residue: 63 passed, 2 platform skips, 0 failed. + - full `bun run test` was not green on the Windows/Bun 1.3.14 host for the + previously documented cache/account/Bun panic failures; a broader + `responses-state` run also had four Windows `EPERM` symlink failures. +- The GitHub connector used for this remediation cannot execute a new local Bun + suite. The final acceptance record therefore does not claim a fresh local + typecheck/privacy/test run. + +### CodeRabbit remediation + +The first unresolved-thread pass corrected: + +- exact stack/audit revision metadata; +- deterministic `BehaviorFingerprintV1` array ordering; +- non-vacuous applicable-required verification; +- source-protocol `[DONE]` semantics; +- actual Chat `messages[].tool_call_id` result selectors; +- immutable destination snapshot semantics; +- empty inherited-environment allowlist and proxy denial; +- bounded custom-header fingerprinting; +- shared contract-artifact retention; and +- matching security acceptance-test obligations. + +The second pass corrected additional deterministic/security gaps: + +- closed invalidation payload/target semantics and privacy-safe purge tombstones; +- retained, replay-verifiable `ClaimSourceManifestV1` evidence; +- a total sidecar dependency sort including provider-instance fingerprint; +- machine-checkable synthetic fixture marker/provenance; +- exact closed MCP harness action tokens/semantics; +- destination-bound opaque credential leases that never expose secret bytes; +- hard, non-overridable V1 time/request/byte/token/tool/memory/process ceilings; +- descriptor/handle-bound no-follow Lab artifact validation/consumption; and +- sensitive-purge replay semantics that cannot preserve stale verdicts. + +`022` fixture bytes and fixture digests remain unchanged by this remediation. +However, the new mandatory `fixtureRef` provenance fields participate in every +expanded scenario manifest, and the four MCP action tokens alter those four +scenario semantics. Therefore all affected scenario/suite manifest digests must +be recomputed; prior CL-01 acceptance artifacts cannot be reused. + +Independent CL-00 acceptance review is frozen at +`c014464237fd3c95bda08bc18bfab8ba8f532308`. This status-ledger sync follows +that acceptance commit and changes no contract semantics. ## CL-01 impact of refreshed CL-00 CL-01 was independently accepted at -`cc447ce9d19d5fb4e03988899f5fb495f9de8d0e`, but it was built against the -older CL-00 tip `c2113ca47b8a05c5a5f90679e4eaa640ca2c6a66` and copied the old +`cc447ce9d19d5fb4e03988899f5fb495f9de8d0e`, but it was built against older +CL-00 tip `c2113ca47b8a05c5a5f90679e4eaa640ca2c6a66` and copied the pre-remediation Protocol V1 authority. -Before CL-01 can be stacked/merged it must: +Before CL-01 can be stacked or merged it must: -1. rebase onto the refreshed CL-00 accepted contract head; -2. synchronize the two corrected Chat tool-result selectors in its copied - `src/lab/conformance/fixtures/protocol-v1-cases.json`; -3. remove or narrow the harness-only Chat-wire `messages` -> synthetic - Responses `input[]` observation projection that its acceptance review used - to satisfy the old selectors; -4. align the SSE normalizer API/contract with source-protocol `[DONE]` - selection rather than client-surface selection; and -5. rerun the canonical scenarios, negative controls, digest/manifest checks and - CL-01 acceptance review. +1. rebase onto the final refreshed CL-00 branch; +2. synchronize both corrected Chat tool-result selectors; +3. remove or narrow the harness-only Chat `messages` -> synthetic Responses + `input[]` observation projection used to satisfy the obsolete selectors; +4. select `[DONE]` semantics by source protocol rather than client surface; +5. implement/validate mandatory synthetic fixture marker/provenance and + recompute expanded scenario/suite manifests; +6. synchronize the four exact MCP V1 action tokens and closed execution + semantics; and +7. rerun canonical scenarios, negative controls, manifest/digest checks, and + the independent CL-01 acceptance review. -This is a required CL-01 correction/revalidation, not CL-02 work. +This is a required CL-01 correction/revalidation. It is not CL-02 work. ## Authorization - CL-00: **ACCEPTED AFTER CODERABBIT REMEDIATION**. -- CL-01: **EXISTS AND WAS ACCEPTED EARLIER, BUT MUST BE REBASED/CORRECTED BEFORE - STACKING OR MERGE**. +- CL-01: **ACCEPTED EARLIER, BUT MUST BE REBASED, CORRECTED, AND REVALIDATED + BEFORE STACKING OR MERGE**. - CL-02: **NOT STARTED / NOT AUTHORIZED BY THIS REMEDIATION**.