diff --git a/devlog/_plan/260807_compatibility_lab/000_master_plan.md b/devlog/_plan/260807_compatibility_lab/000_master_plan.md new file mode 100644 index 000000000..f9ee5b833 --- /dev/null +++ b/devlog/_plan/260807_compatibility_lab/000_master_plan.md @@ -0,0 +1,306 @@ +# OpenCodex Compatibility Lab / EvalGrid + +Status: CL-00 architecture authority +Authority baseline: `upstream/dev` at `3ad5bb6bd3f76f6879d84b78ea39edd3e01ec296` +Package/runtime at baseline: OpenCodex `2.10.2`, Bun `1.3.14` + +## Purpose + +Compatibility Lab turns compatibility claims into bounded, reproducible +evidence. It tests OpenCodex protocol behavior, exact configured routes, and +later execution-grounded task outcomes without becoming a provider registry, a +user-policy system, or a production router. + +This directory is the programme authority. The contracts frozen by CL-00 are: + +- [Architecture and evidence](./010_architecture_and_evidence_contract.md) +- [Scenario model and initial catalogue](./020_scenario_contract_and_catalogue.md) +- [Protocol V1 manifest authority](./021_protocol_v1_manifest_authority.md) +- [Protocol V1 canonical cases](./022_protocol_v1_cases.json) +- [Historical incident corpus](./030_incident_corpus.md) +- [Security and privacy](./040_security_and_privacy.md) +- [CL-00 independent acceptance review](./050_cl00_acceptance_review.md) +- [PR stack status](./001_pr_stack_status.md) + +Later phases may add implementation detail, but must amend these contracts +explicitly rather than silently changing their meaning. + +## Live repository truth + +CL-00 audited the live `dev` tree before defining new authority. + +### Shipped and authoritative + +- Provider declarations and model metadata: + `src/providers/registry.ts`, `src/providers/derive.ts`, `src/types.ts`; + route-time claimed capability assembly in `src/routing/capability.ts` also + consumes provider config, cached Codex catalog rows and native metadata. + Generated fallback metadata lives in `src/generated/model-metadata.ts` and is + sourced by `scripts/model-metadata.source.json`. +- Routing Profile public/config types: `OcxRoutingProfileConfig` in + `src/types.ts`; validation, normalization, revision hashing, persistence and + resolution in `src/routing/profile.ts`. +- Deterministic profile evaluation and route traces: + `src/routing/evaluator.ts`, `src/routing/trace.ts`, and + `src/router.ts`. +- Profile management CRUD and dry-run: + `src/server/management/routing-profile-routes.ts`. +- Dashboard profile editor, dry-run and routing analytics: + `gui/src/pages/RoutingProfiles.tsx`, mounted under Models -> Routing. +- Canonical append-only request/usage evidence and rebuildable history + projection: + `src/usage/log.ts`, `src/routing/history/indexer.ts`, and + `src/routing/analytics.ts`. +- Why-this-route evidence: + `RouteDecisionTraceV1`, request-history explain endpoints in + `src/server/management/request-history-routes.ts`, and CLI explain support. + The GUI Logs modal renders only a compact route summary through + `gui/src/pages/log-route-decision.ts`; there is no GUI request-history browser + or full trace + attempts + outcome view on this baseline. +- Existing diagnostics are narrower than Compatibility Lab: + the default `ocx doctor` path is observe-only environment/OAuth/runtime + diagnosis; explicit `--fix-codex-runtime` may persist a repair. In contrast, + `POST /api/providers/test` performs a bounded live `/models` connectivity + check only when applicable; forward providers return configured status and + static catalogues return not-applicable without network access. +- Protocol behavior is already covered by many focused tests under `tests/`, + but those tests are not a versioned scenario catalogue or evidence ledger. + +### Not shipped + +- No Compatibility Lab runner, scenario registry, evidence ledger, SQLite + projection, CLI, management API, or UI exists. +- Generated Cursor agent protobufs include task/grind/subagent message types, + but OpenCodex has no native Agent Fabric task persistence, harness handoff, + portable task-state model, or management API. Router Intelligence's own + master plan explicitly excluded Agent Fabric. +- No Routing Profile compatibility fields exist. + +Current routing nuances that later phases must preserve rather than +over-describe: + +- selection traces and execution attempts are separate; the explain API merges + trace + `attempts[]` + final outcome at read time; +- `optimize.latency` is currently a declaration-priority share, while observed + latency contributes through health evidence rather than an independent + top-level score; +- cost evidence is commonly unknown on the live pre-dispatch path because + request usage is not yet available; +- profile dry-run is evaluation-only and never dispatches upstream; +- an unknown canonical `policy/` currently falls through to ordinary model + routing rather than failing closed. + +Consequently, CL-00 defines future contracts and integration seams only. It +does not rename existing Router Intelligence concepts or describe speculative +Agent Fabric endpoints as current behavior. + +## Architectural invariant + +```text +Provider Registry + ↓ +Compatibility Lab + ↓ +Compatibility Graph / Verified Evidence + ↓ +Routing Profiles + ↓ +Router Intelligence + ↓ +Selected Model / Route + ↓ +Agent Fabric / Real Execution + └──────────────→ execution-grounded outcomes back to Lab +``` + +The arrows are data dependencies, not ownership transfers. + +### Provider Registry + +The Provider Registry declares what a provider/model is believed to support and +supplies defaults used to construct an effective route. The shipped +`candidateCapabilityEvidence()` also combines explicit provider config, cached +catalog rows, adapter-level inference and native-model metadata. These local +declarations are claims. They may seed `CLAIMED`; they cannot by themselves +produce `PROBED`, `VERIFIED`, `DEGRADED`, or `UNSUPPORTED`. + +The Lab may snapshot a registry claim with its source revision for +reproducibility. It must not create a parallel provider catalogue or write +provider declarations back into the registry. + +Registry-owned runtime defaults such as model wire selection, discovery policy, +upstream streaming, reasoning replay, service-tier support, and item-ID repair +are intentionally not all persisted to `config.json`. Claim snapshots capture +the effective sources; they do not freeze runtime defaults into user config. + +### Compatibility Lab + +The Lab owns versioned scenarios, immutable compatibility evidence, failure +attribution, freshness, derived verdicts, and regression history. It may +project evidence into a compatibility graph keyed by exact route subject and +evidence layer. + +The Lab never chooses a production candidate, mutates a Routing Profile, +changes provider metadata, or turns a probe result directly into a route. + +### Routing Profiles + +Routing Profiles remain the sole compatibility-policy surface. Future +compatibility requirements extend `OcxRoutingProfileConfig`, its +normalizer/revision, the existing evaluator, the existing management +CRUD/dry-run endpoints, and the Models dashboard editor. Existing combo and +account-pool controls retain their separate non-compatibility responsibilities. +There will be no compatibility-specific profile store, evaluator, or editor. + +### Router Intelligence + +Router Intelligence combines the selected profile with current capability, +compatibility, health, quota, cost, and latency evidence and makes the +deterministic route decision. Its existing `RouteDecisionTraceV1` remains the +authority for explaining that decision. Compatibility inputs will later add +bounded evidence to that trace rather than introduce a second explanation +record. + +### Agent Fabric + +Agent Fabric is a future producer of task-effectiveness observations. It owns +real task execution and its sandbox. The Lab accepts only structured outcome +data and sanitized content-addressed artifact references; it does not copy task +repositories, prompts, worktrees, or hidden reasoning. + +Because a native Agent Fabric is not present on the CL-00 baseline, this +programme freezes the consumer semantics, not a fictitious production API. A +later producer contract must identify its schema version, task class, exact +route subject, deterministic verifier results, timing, resource limits, +outcome, and sanitized artifact references. Existing request-grounded evidence +may be linked through `RouteDecisionTraceV1`, `PersistedUsageAttempt`, and the +final request outcome; prompt-bearing `responses-state.json` and generated +Cursor task protobufs are not Lab feeds. + +## Evidence-layer invariant + +Every scenario and observation has exactly one layer: + +1. `protocol_conformance`: whether OpenCodex translates and preserves a + protocol contract correctly. +2. `live_route_compatibility`: whether an exact + provider/model/adapter/configuration route works now. +3. `task_effectiveness`: whether that route produces verifier-confirmed + outcomes for a versioned class of coding work. + +Verdicts are projected per `(subject, layer, suite)`. Evidence from one layer +may be shown as a prerequisite or correlated signal, but cannot promote or +degrade another layer's verdict. There is no universal compatibility score. + +## Persistence authority + +Future implementation uses the existing OpenCodex config root returned by +`getConfigDir()` (`OPENCODEX_HOME`, default `~/.opencodex`) and owns: + +```text +~/.opencodex/lab/ + compatibility.jsonl + compatibility.sqlite + artifacts/ +``` + +- `compatibility.jsonl` is the canonical append-only evidence/event ledger. +- `compatibility.sqlite` is a disposable query projection rebuilt from JSONL. +- `artifacts/` contains bounded, sanitized, content-addressed artifacts. +- Scenario/suite manifests and synthetic fixture/source anchors are + content-addressed contract artifacts retained with the observations that + reference them. +- Verdicts are derived projections, never mutable canonical booleans. +- Corrections append invalidation events or a new claim snapshot with explicit + `supersedes[]`; prior bytes are not edited. +- The Lab does not copy `usage.jsonl` or routing-history rows. When useful, an + observation references an existing request ID or route decision ID. +- Agent Fabric supplies structured outcome data/references, never repositories + or prompt transcripts. + +This location follows current repository state-root conventions. No filename +or location change from the proposed architecture was justified by the audit. + +## Routing Profiles boundary for CL-06 + +CL-06 must add optional compatibility controls alongside existing capability, +health, quota, cost, and latency policy: + +- required compatibility suites; +- minimum compatibility status; +- maximum evidence age; +- unknown-evidence behavior; +- degraded-evidence behavior. + +`minimum compatibility status` is not a total ordering across all verdicts. +Only `PROBED` and `VERIFIED` are positive thresholds. `DEGRADED` is governed by +its explicit behavior, `UNKNOWN`/`CLAIMED`/`BLOCKED` by unknown-evidence +behavior, and `UNSUPPORTED` fails a required suite. + +The exact future flow is: + +```text +Routing Profile + ↓ +Configured candidates + ↓ +Hard capability gates + ↓ +Compatibility requirements / penalties + ↓ +Eligible candidates + ↓ +Health / quota / cost / latency scoring + ↓ +Deterministic winner +``` + +All compatibility fields are optional. Profiles that omit them retain their +current validation, revision, eligibility and scoring behavior. A profile +evaluation reads an existing projection only. No compatibility probe, network +request, task, or projection rebuild may run synchronously on the production +request path. + +## Programme phases + +Only CL-00 is authorized by this document at present. + +| Phase | Purpose | Authorization | +|---|---|---| +| CL-00 | Architecture authority, contracts, scenario catalogue, incident corpus | This PR | +| CL-01 | Deterministic protocol-conformance runner and fixtures | Not started; requires CL-00 to be accepted | +| CL-02 | Immutable JSONL ledger, artifacts and SQLite projection | Not started | +| CL-03 | Bounded live-route probes | Not started | +| CL-04 | Lab CLI and management read surfaces | Not started | +| CL-05 | Compatibility Matrix UI | Not started | +| CL-06 | Existing Routing Profile compatibility controls and Router Intelligence consumption | Not started | +| CL-07 | Agent Fabric task-effectiveness ingestion | Not started | +| CL-08 | Shadow/automatic/public evidence workflows | Not started | + +Phase numbering after CL-01 is programme planning, not implementation +authorization. A later accepted plan may split a phase while preserving these +ownership boundaries. + +## CL-00 acceptance criteria + +CL-00 is accepted only when: + +1. all three evidence layers have separate subjects, scenarios and verdicts; +2. every canonical verdict is reproducible from immutable inputs; +3. environmental blockers cannot poison compatibility conclusions; +4. exact route identity prevents evidence reuse across behavior changes; +5. scenario semantics and initial IDs are implementable without an LLM judge; +6. representative historical incidents map to abstract regression scenarios; +7. future compatibility policy extends existing Routing Profiles; +8. probes and task execution are excluded from production request routing; +9. privacy and sandbox ceilings are explicit; +10. an independent review finds no unresolved Critical, High, or Medium issue. + +## CL-00 hard stop + +This phase does not implement a runner, mock upstream, persistence code, live +probe, CLI, management endpoint, UI, profile field, routing behavior, shadow +route, Fabric ingestion, automatic routing, or public publisher. + +Acceptance of CL-00 authorizes discussion and planning of CL-01; it does not +start CL-01 automatically. diff --git a/devlog/_plan/260807_compatibility_lab/001_pr_stack_status.md b/devlog/_plan/260807_compatibility_lab/001_pr_stack_status.md new file mode 100644 index 000000000..382543a24 --- /dev/null +++ b/devlog/_plan/260807_compatibility_lab/001_pr_stack_status.md @@ -0,0 +1,124 @@ +# Compatibility Lab PR stack status + +Updated throughout the programme. Every phase records its branch, exact +starting/base revision, accepted contract/implementation head, PR, verification, +independent review, blockers, and whether a later phase is authorized. + +## Programme facts + +- Repository: `lidge-jun/opencodex` +- Integration target: `dev` +- CL-00 starting `upstream/dev`: + `3ad5bb6bd3f76f6879d84b78ea39edd3e01ec296` +- Package/runtime at start: OpenCodex `2.10.2`, Bun `1.3.14` +- CL-00 branch: `feat/cl-00-compatibility-contracts` +- CL-00 scope: documentation/contracts/incident corpus only +- PR target: `lidge-jun/opencodex:dev` + +## Stack + +| Phase | Branch | Starting/base SHA | Accepted head | PR | State | +|---|---|---|---|---|---| +| CL-00 | `feat/cl-00-compatibility-contracts` | `3ad5bb6bd3f76f6879d84b78ea39edd3e01ec296` | `c014464237fd3c95bda08bc18bfab8ba8f532308` | [#1286](https://github.com/lidge-jun/opencodex/pull/1286) | ACCEPTED AFTER CODERABBIT REMEDIATION | +| CL-01 | `feat/cl-01-conformance-harness` | `c2113ca47b8a05c5a5f90679e4eaa640ca2c6a66` | `cc447ce9d19d5fb4e03988899f5fb495f9de8d0e` | [draft Wibias #10](https://github.com/Wibias/opencodex/pull/10) | ACCEPTED EARLIER; REBASE + CONTRACT CORRECTION + REVALIDATION REQUIRED | + +The CL-01 starting SHA is the exact CL-00 tip recorded when CL-01 began. Its +moving base-ref name is not a substitute for that historical SHA. + +## CL-00 acceptance log + +- Live-tree audit covered provider registry/derivation, Routing Profiles, + routing traces, request history/analytics, doctor/connectivity validation, + protocol regression tests, and relevant incident/devlog records. +- CL-00 remains contract-only. No Compatibility Lab runtime, live runner, + profile/router implementation, or CL-02 work was added. +- Contract documents: + - `000_master_plan.md` + - `010_architecture_and_evidence_contract.md` + - `020_scenario_contract_and_catalogue.md` + - `021_protocol_v1_manifest_authority.md` + - `022_protocol_v1_cases.json` + - `030_incident_corpus.md` + - `040_security_and_privacy.md` + - `050_cl00_acceptance_review.md` +- Original baseline verification: + - `bun x tsc --noEmit`: passed. + - `bun run privacy:scan`: passed. + - `tests/repo-hygiene.test.ts`: 11 passed, 0 failed. + - focused protocol/compatibility: 395 passed, 0 failed across 24 files. + - continuation semantics: 2 passed, 95 filtered, 0 failed. + - isolated cache invalidation: 6 passed, 0 failed. + - isolated native residue: 63 passed, 2 platform skips, 0 failed. + - full `bun run test` was not green on the Windows/Bun 1.3.14 host for the + previously documented cache/account/Bun panic failures; a broader + `responses-state` run also had four Windows `EPERM` symlink failures. +- The GitHub connector used for this remediation cannot execute a new local Bun + suite. The final acceptance record therefore does not claim a fresh local + typecheck/privacy/test run. + +### CodeRabbit remediation + +The first unresolved-thread pass corrected: + +- exact stack/audit revision metadata; +- deterministic `BehaviorFingerprintV1` array ordering; +- non-vacuous applicable-required verification; +- source-protocol `[DONE]` semantics; +- actual Chat `messages[].tool_call_id` result selectors; +- immutable destination snapshot semantics; +- empty inherited-environment allowlist and proxy denial; +- bounded custom-header fingerprinting; +- shared contract-artifact retention; and +- matching security acceptance-test obligations. + +The second pass corrected additional deterministic/security gaps: + +- closed invalidation payload/target semantics and privacy-safe purge tombstones; +- retained, replay-verifiable `ClaimSourceManifestV1` evidence; +- a total sidecar dependency sort including provider-instance fingerprint; +- machine-checkable synthetic fixture marker/provenance; +- exact closed MCP harness action tokens/semantics; +- destination-bound opaque credential leases that never expose secret bytes; +- hard, non-overridable V1 time/request/byte/token/tool/memory/process ceilings; +- descriptor/handle-bound no-follow Lab artifact validation/consumption; and +- sensitive-purge replay semantics that cannot preserve stale verdicts. + +`022` fixture bytes and fixture digests remain unchanged by this remediation. +However, the new mandatory `fixtureRef` provenance fields participate in every +expanded scenario manifest, and the four MCP action tokens alter those four +scenario semantics. Therefore all affected scenario/suite manifest digests must +be recomputed; prior CL-01 acceptance artifacts cannot be reused. + +Independent CL-00 acceptance review is frozen at +`c014464237fd3c95bda08bc18bfab8ba8f532308`. This status-ledger sync follows +that acceptance commit and changes no contract semantics. + +## CL-01 impact of refreshed CL-00 + +CL-01 was independently accepted at +`cc447ce9d19d5fb4e03988899f5fb495f9de8d0e`, but it was built against older +CL-00 tip `c2113ca47b8a05c5a5f90679e4eaa640ca2c6a66` and copied the pre-remediation +Protocol V1 authority. + +Before CL-01 can be stacked or merged it must: + +1. rebase onto the final refreshed CL-00 branch; +2. synchronize both corrected Chat tool-result selectors; +3. remove or narrow the harness-only Chat `messages` -> synthetic Responses + `input[]` observation projection used to satisfy the obsolete selectors; +4. select `[DONE]` semantics by source protocol rather than client surface; +5. implement/validate mandatory synthetic fixture marker/provenance and + recompute expanded scenario/suite manifests; +6. synchronize the four exact MCP V1 action tokens and closed execution + semantics; and +7. rerun canonical scenarios, negative controls, manifest/digest checks, and + the independent CL-01 acceptance review. + +This is a required CL-01 correction/revalidation. It is not CL-02 work. + +## Authorization + +- CL-00: **ACCEPTED AFTER CODERABBIT REMEDIATION**. +- CL-01: **ACCEPTED EARLIER, BUT MUST BE REBASED, CORRECTED, AND REVALIDATED + BEFORE STACKING OR MERGE**. +- CL-02: **NOT STARTED / NOT AUTHORIZED BY THIS REMEDIATION**. diff --git a/devlog/_plan/260807_compatibility_lab/010_architecture_and_evidence_contract.md b/devlog/_plan/260807_compatibility_lab/010_architecture_and_evidence_contract.md new file mode 100644 index 000000000..f48166fb0 --- /dev/null +++ b/devlog/_plan/260807_compatibility_lab/010_architecture_and_evidence_contract.md @@ -0,0 +1,890 @@ +# CL-00 architecture and evidence contract + +This document freezes the semantic model consumed by later Compatibility Lab +phases. Names shown in code blocks are contract names, not claims that +production TypeScript types already exist. + +## 1. Evidence layers + +### `protocol_conformance` + +Question: does this OpenCodex build preserve the declared inbound-to-upstream +and upstream-to-client protocol contract? + +Inputs are deterministic fixture requests and deterministic mock-upstream +responses. The subject includes the OpenCodex compatibility version, adapter, +inbound protocol, upstream protocol, surface, and relevant behavior +fingerprint. A real provider account is neither required nor permitted. + +A pass proves only the exercised OpenCodex translation. It says nothing about a +provider's current availability or a model's coding quality. + +### `live_route_compatibility` + +Question: does this exact configured route satisfy this versioned scenario now? + +The subject is an exact provider/model/effective-adapter/configuration route. +The run may contact only that route's configured upstream under the Lab +sandbox. A pass cannot be reused for a different route fingerprint. + +A pass proves only the exercised route behavior at the observation time. It +does not prove task effectiveness. + +### `task_effectiveness` + +Question: did this exact route produce a successful, deterministically verified +outcome for a versioned task class? + +Agent Fabric, not the Lab, owns real execution. The Lab receives a structured +outcome containing deterministic verifier results and sanitized artifact +references. Human ratings or LLM-judge output may be stored as advisory +annotations in a later phase, but cannot produce a canonical verdict. + +### Non-collapse rule + +Each layer has one executable subject kind: + +```text +protocol_conformance -> ProtocolSubjectV1 +live_route_compatibility -> RouteSubjectV1 +task_effectiveness -> TaskSubjectV1 +``` + +`EvidenceSubjectV1` is that closed discriminated union. An observation whose +layer and `subjectKind` do not match is invalid evidence. Suite manifests also +belong to exactly one evidence layer; a manifest cannot mix protocol, live, and +task scenarios even when their human-facing suite stem is the same. + +The canonical projection key is: + +```text +(subjectId, evidenceLayer, suiteId, suiteVersion, suiteManifestDigest, + projectionSpecVersion) +``` + +There is no projection across all layers and no weighted universal score. +Callers may present multiple layer verdicts next to each other. A prerequisite +failure in one layer may make a later-layer run inapplicable, but it does not +rewrite evidence in the other layer. + +### `ProtocolSubjectV1` + +Protocol evidence identifies OpenCodex translation behavior, not a provider: + +```text +subjectSchemaVersion 1 +subjectKind protocol +opencodexCompatibilityVersion +effectiveAdapter +inboundProtocol +upstreamProtocol +surface +behaviorFingerprint +``` + +The closed `runtime.bunVersion`, `runtime.platform`, and `runtime.arch` behavior +keys cover platform-sensitive paths; there is no second runtime digest. +Provider, credential, account, endpoint, health, quota, cost, and latency +fields are forbidden. + +### `TaskSubjectV1` + +Task evidence identifies one exact route plus one synthetic task/verifier +contract: + +```text +subjectSchemaVersion 1 +subjectKind task +routeSubject RouteSubjectV1 +taskClassId +taskClassVersion +taskFixtureDigest +verifierManifestDigest +fabricCompatibilityVersion +sandboxProfileDigest +``` + +The nested route subject makes execution behavior reproducible. The task +subject never contains repository contents, prompts, user paths, account +identity, or raw artifacts. A changed route, task fixture, verifier, Fabric +runtime, or sandbox profile starts a new task projection. + +## 2. Immutable ledger contract + +The canonical JSONL ledger is a sequence of versioned events. The minimum +event kinds are: + +```text +observation +claim_snapshot +invalidation +purge_tombstone +``` + +An `observation` records one scenario attempt. A `claim_snapshot` captures the +local declared-capability inputs and source revisions needed to reproduce +`CLAIMED`: registry/config, cached catalog or native metadata, including the +adapter inference currently assembled by `src/routing/capability.ts`. An +`invalidation` identifies prior non-sensitive evidence that a later-discovered +harness, fixture, redaction, or integrity defect makes unusable. Invalidations +append; they never delete or edit prior lines. A `purge_tombstone` is the +privacy-safe exceptional record left after confirmed sensitive evidence is +physically removed under the security contract. + +Each event has: + +```text +schemaVersion +eventId +eventKind +recordedAt +producer +producerVersion +``` + +An observation additionally has: + +```text +evidenceLayer +scenarioId +scenarioVersion +scenarioManifestDigest +suiteId +suiteVersion +suiteManifestDigest +fixtureDigests[] +subject +subjectId +startedAt +completedAt +executionMode fixture | live | fabric +attempt +limits +outcome pass | fail | blocked | inconclusive +assertions[] +failure? { class, code, retryable, attribution } +expectedFailure? +environment +artifactRefs[] +sourceRefs? +``` + +A `claim_snapshot` additionally has: + +```text +evidenceLayer live_route_compatibility +subject RouteSubjectV1 +subjectId +capability +polarity supported | not_supported | withdrawn +sourceManifestDigest +sourceEventIds[] +supersedes[] +effectiveAt +``` + +An `invalidation` additionally has: + +```text +targetEventIds[] non-empty, maximum 1024 +reason harness_defect | fixture_defect | + redaction_defect | integrity_defect | + contract_artifact_missing | manual_correction +``` + +Invalidation rules: + +- `targetEventIds` is a set encoded as lowercase event IDs sorted + lexicographically by UTF-8 bytes. Duplicate IDs reject the whole invalidation. +- Every target must be an earlier valid `observation` or `claim_snapshot` in + the same ledger. Unknown, malformed, future, self, `invalidation`, or + `purge_tombstone` targets reject the whole invalidation and report ledger + corruption; projection never applies a partial target list. +- The `reason` set above is closed. Unsupported or missing reasons reject the + invalidation. +- Multiple valid invalidation events may name the same prior target; the target + remains excluded. There is no "uninvalidate" mutation: replacement evidence + is a new observation or claim snapshot. +- Sensitive-evidence deletion does not weaken these rules. It uses the separate + `purge_tombstone` contract below because the sensitive target line may no + longer exist by design. + +A `purge_tombstone` additionally has: + +```text +targetEventIds[] sorted unique lowercase event IDs +targetArtifactDigests[] sorted unique lowercase SHA-256 digests +reason sensitive_evidence +purgeActions[] ledger | sqlite | artifact | scratch | export +``` + +At least one target array is non-empty. The reason is exactly +`sensitive_evidence`; action names are a closed set, sorted and unique. A purge +tombstone may refer to an event line or artifact that has already been +physically removed by the same fail-closed purge. Projection applies valid +purge tombstones before ordinary invalidations, excludes every targeted event, +and treats every targeted artifact as typed `purged_unavailable`. Cached SQLite +rows that depended on a targeted event or artifact are invalid and must be +removed/rebuilt. A prior `VERIFIED`, `PROBED`, `DEGRADED`, `UNSUPPORTED`, or +`CLAIMED` result is never retained solely from purged evidence. + +### `ClaimSourceManifestV1` + +`sourceManifestDigest` is reproducible evidence, not an opaque checksum. Every +claim snapshot references one retained canonical `ClaimSourceManifestV1`: + +```text +{ + "schemaVersion": 1, + "subjectId": string, + "providerId": string, + "clientModelId": string, + "capability": string, + "sources": [ClaimSourceV1, ...], + "resolvedEvidence": RouteCapabilityEvidenceV1 +} +``` + +`ClaimSourceV1` is one record for each consulted source that contributed or +could have contributed to the selected capability: + +```text +{ + "kind": "provider_config" | "provider_registry" | "cached_catalog" | + "native_metadata" | "adapter_inference", + "revision": string | null, + "facts": ClaimCapabilityFactsV1 +} +``` + +`sources` is ordered by the closed kind order shown above and contains at most +one record per kind. `ClaimCapabilityFactsV1` is a closed, sanitized projection +of only capability-relevant inputs used by the current resolver: selected +context window, input modalities, reasoning efforts, catalog capability names, +service-tier support, effective adapter/tool-capable classification, +parallel-tool-call enablement, endpoint locality class +`local|private|unknown`, and canonical-OpenAI-forward classification where +applicable. It never stores a base URL, hostname, credential, account identity, +custom header, arbitrary provider config, catalog path, prompt, or repository +data. `RouteCapabilityEvidenceV1` is the closed resolver output currently +represented by `RouteCapabilityEvidence`: context window, image, tools, +reasoning efforts, service tier, local-only, remote-allowed, and encrypted +Codex-task evidence, with absent fields remaining absent. + +The canonical digest is lowercase: + +```text +sha256( + UTF8("ocx-lab:claim-source-manifest:v1\0") || + UTF8(JCS(ClaimSourceManifestV1)) +) +``` + +The exact canonical bytes are retained as a content-addressed +`claim_source_manifest` artifact while any non-purged claim snapshot references +them. Projection must load those retained bytes, recompute the digest, and +verify matching `subjectId` and `capability` before accepting historical +`CLAIMED` state. Missing bytes, digest mismatch, duplicate/unknown source kinds, +unknown facts, or a subject/capability mismatch makes that claim unusable, +projects no `CLAIMED` result from it, and reports ledger/artifact corruption. + +Claim rules: + +- Claims exist only for `live_route_compatibility`; protocol and task verdicts + cannot be `CLAIMED`. +- `sourceManifestDigest` identifies the canonical, content-addressed snapshot + above of all sanitized registry/config/catalog/native/adapter inputs used by + the existing capability resolver. Raw secrets are forbidden. +- `supersedes` explicitly lists every previously current claim event for the + same `(subjectId, capability)`. The first snapshot uses an empty list. +- A source refresh, removal, or changed polarity appends a new snapshot; it + never edits a prior event. `withdrawn` means the local declaration no longer + exists. `not_supported` suppresses `CLAIMED` but cannot produce + `UNSUPPORTED`. +- Projection first applies purge tombstones and invalidations, then removes + every event named by a valid later snapshot's `supersedes`. Exactly one + unsuperseded claim may remain current for a claim key. Multiple unsuperseded + claims, a missing referenced predecessor, a cross-key supersession, or a + cycle makes claim state `UNKNOWN` and reports ledger corruption. +- Claim ordering is `effectiveAt`, then `recordedAt`, then event ID, but + ordering never substitutes for the explicit supersession graph. + +Rules: + +- Canonical JSON is RFC 8785 JSON Canonicalization Scheme (JCS), encoded as + UTF-8 with no BOM. +- `eventId` is lowercase + `sha256("ocx-lab:event:v1\0" || JCS(event without eventId))`. +- `subjectId` is lowercase + `sha256("ocx-lab:subject:v1\0" || JCS(subject))`. +- Scenario and suite manifests use the same JCS construction with domains + `ocx-lab:scenario-manifest:v1` and `ocx-lab:suite-manifest:v1`. Fixture + digests are lowercase + `sha256("ocx-lab:fixture:v1\0" || exact fixture bytes)`. A manifest's digest + field and storage path are excluded from its preimage. Domain text is UTF-8 + and terminated by one NUL byte. +- Every observation carries the exact scenario, suite and fixture digests it + executed. Version strings without matching digests are invalid evidence. +- Timestamps are UTC epoch milliseconds; duration alone is not sufficient. +- Assertions record expected value/shape, observed normalized value/shape, and + pass/fail. They never require raw prompts or unbounded bodies. +- `failure.attribution` records whether the failure is attributable to + OpenCodex, the exact route, the environment, or the harness. +- Artifact references contain digest, media type, byte count, redaction + policy, and local relative path; never an arbitrary filesystem path. +- `sourceRefs` may contain existing request IDs, route-decision IDs, or future + Fabric outcome IDs. It must not inline the referenced request or task. +- The shipped request-history explain surface already composes selection trace, + `PersistedUsageAttempt[]`, and final outcome. A later Lab consumer should link + that normalized composition instead of reading prompt-bearing + `responses-state.json` or treating generated Cursor task protobufs as durable + OpenCodex state. +- A structurally invalid or partially written line contributes no evidence and + is reported as ledger corruption. SQLite must be rebuildable from all valid + complete lines plus privacy-safe purge tombstones. + +The canonical scenario manifest, suite manifest, synthetic fixture bytes, and +claim-source manifest referenced by valid evidence are retained as +content-addressed `scenario_manifest`, `suite_manifest`, `fixture`, and +`claim_source_manifest` artifacts. Scenario/suite/fixture contract artifacts +remain retained while any non-invalidated, non-purged observation references +them. Claim-source manifests remain retained while any non-purged claim +snapshot references them. A missing or digest-mismatched required contract +artifact makes that evidence unusable and reports corruption; projection code +must never substitute the current manifest for the historical one. + +The SQLite projection may cache derived verdict rows. Such rows must include +their `asOf`, projection spec, scenario/suite/fixture manifest digests, claim +source manifest digest where applicable, and contributing event IDs. Deleting +SQLite and replaying JSONL plus the referenced content-addressed contract +artifacts and purge tombstones must reproduce every non-purged row. + +## 3. Canonical verdict contract + +The closed verdict set is: + +```text +UNKNOWN +CLAIMED +PROBED +VERIFIED +DEGRADED +BLOCKED +UNSUPPORTED +``` + +Verdicts are projections, not mutable evidence fields. + +### `UNKNOWN` + +Produced when no current positive claim snapshot and no current, valid, +attributable observation can classify the projection key. + +It can also result when all prior evidence became stale or was invalidated and +there is no current claim or blocker. Absence of a test is not +`UNSUPPORTED`. + +### `CLAIMED` + +Produced only by a current snapshotted positive local capability declaration +for the exact capability/subject, when no current executable evidence yields a +stronger state. The snapshot records whether the declaration came from explicit +provider config, Provider Registry, cached catalog, native metadata, or current +adapter inference rather than pretending every claim is registry-authored. + +`CLAIMED` applies only to `live_route_compatibility`; protocol and task layers +have no declaration source and project `UNKNOWN` without executable evidence. + +A claim cannot produce `PROBED` or `VERIFIED`. A registry negative declaration +is shown as claim metadata but does not by itself prove `UNSUPPORTED`. + +### `PROBED` + +Produced when at least one required executable scenario completed with an +attributable pass, but the suite's versioned verification rule is not yet +satisfied. Examples are partial required-scenario coverage or a scenario whose +assertions establish reachability/shape but not full suite verification. + +Connectivity-only `/models` checks, registry discovery, doctor output, health +samples, and blocked attempts cannot produce `PROBED`. + +### `VERIFIED` + +Produced only when every requirement in the suite manifest's verification rule +is met by current, valid, attributable observations for the exact projection +key, and no newer current contradictory attributable failure remains +unresolved. A verification rule with zero applicable required scenarios is not +satisfied and cannot produce `PROBED` or `VERIFIED`. + +The projection must expose: + +- exact contributing event IDs; +- suite/scenario versions and manifest digest; +- subject ID and full local subject; +- projection algorithm version and `asOf`; +- freshness calculation; +- any invalidations, purge tombstones, and contradictory events considered. + +An LLM judge, user assertion, registry declaration, successful model listing, +or mutable `verified=true` flag cannot produce `VERIFIED`. + +### `DEGRADED` + +Produced when executable evidence proves that the capability works only +partially, loses required semantics, violates a required assertion while a +usable subset remains, or requires a suite-declared workaround. + +Examples include malformed tool-call/result correlation, dropped reasoning +replay required by the suite, or incomplete stream semantics with otherwise +usable output. Environmental blockers cannot produce `DEGRADED`. + +### `BLOCKED` + +Produced when a current attempt cannot reach a compatibility assertion because +of an environmental or administrative precondition and no current +compatibility-attributable verdict should take precedence. + +Authentication, quota, region policy, local network failure, provider +transients, harness failure, or exhausted Lab budget can yield `BLOCKED`. The +projection retains those classes separately. + +A blocked retry does not erase a still-current `VERIFIED`, `PROBED`, +`DEGRADED`, or `UNSUPPORTED` verdict. Once that prior verdict is stale, +`BLOCKED` may become the current state until a conclusive run succeeds. + +### `UNSUPPORTED` + +Produced only by executable, suite-declared evidence that the exact capability +is unavailable by contract for this subject. The scenario must define an +unambiguous `capability_absence_control`; a conformance negative control, +generic 4xx, timeout, empty output, registry omission, or failed authentication +is insufficient. + +Expected rejection of a deliberately unsupported feature can prove +`UNSUPPORTED` when the rejection itself matches the deterministic contract. It +is not a failed harness run. + +### Projection precedence + +For current valid evidence at the same projection key: + +1. Satisfied full verification rule yields `VERIFIED`. +2. An unresolved attributable required-assertion failure yields `DEGRADED` or + `UNSUPPORTED` according to the scenario's failure rule. +3. Partial positive coverage yields `PROBED`. +4. A blocker yields `BLOCKED` only when 1-3 have no current result. +5. A current positive claim snapshot yields `CLAIMED`. +6. Otherwise the result is `UNKNOWN`. + +This is precedence, not a quality scale. In particular, `DEGRADED`, +`BLOCKED`, and `UNSUPPORTED` are not numeric values below `PROBED`. + +## 4. Transitions, contradiction and freshness + +Because verdicts are recomputed, a "transition" means that new events, time, or +version inputs change a projection. + +Allowed transitions: + +| From | May move to | Cause | +|---|---|---| +| `UNKNOWN` | any state | claim, observation, or blocker | +| `CLAIMED` | `UNKNOWN`, `PROBED`, `VERIFIED`, `DEGRADED`, `BLOCKED`, `UNSUPPORTED` | claim removal/staleness or executable evidence | +| `PROBED` | `UNKNOWN`, `CLAIMED`, `VERIFIED`, `DEGRADED`, `BLOCKED`, `UNSUPPORTED` | coverage, contradiction, staleness, invalidation | +| `VERIFIED` | `UNKNOWN`, `CLAIMED`, `PROBED`, `DEGRADED`, `BLOCKED`, `UNSUPPORTED` | partial remaining coverage after invalidation, contradiction, staleness, or changed inputs | +| `DEGRADED` | `UNKNOWN`, `CLAIMED`, `PROBED`, `VERIFIED`, `BLOCKED`, `UNSUPPORTED` | repair evidence, staleness, invalidation, or reclassification | +| `BLOCKED` | any state | blocker clears, prior evidence becomes current/stale, or claim changes | +| `UNSUPPORTED` | `UNKNOWN`, `CLAIMED`, `PROBED`, `VERIFIED`, `DEGRADED`, `BLOCKED` | route/version/config change, invalidation, or new evidence | + +Direct transitions not listed are forbidden; implementations must not invent a +state outside this set. + +Contradictory attributable evidence is never overwritten. The projection: + +1. filters by exact subject/layer/suite/scenario versions; +2. applies purge tombstones and invalidation events; +3. applies freshness; +4. orders observations by completion time and deterministic event-ID tie-break; +5. applies the suite's contradiction rule; +6. emits the contributing and contradicting event IDs. + +The initial contradiction rule is conservative: a newer required-scenario +failure prevents `VERIFIED` until a newer pass of that scenario and all other +required coverage exists. A newer pass can restore `VERIFIED`; history remains. + +### Freshness + +Each scenario manifest declares its maximum evidence age. The suite manifest +may declare a stricter maximum, and a future Routing Profile may tighten it +again. Effective maximum age is the minimum of all finite scenario, suite and +profile values; `null` means no bound at that layer: + +- deterministic protocol evidence has no wall-clock expiry by default, but is + exact-match bound to scenario, suite, compatibility version, adapter and + behavior fingerprint; +- initial live-route manifests default to seven days; +- initial task-effectiveness manifests default to thirty days; +- a profile's maximum evidence age is an additional upper bound, never an + extension. + +Stale observations remain queryable but cannot support current +`PROBED`/`VERIFIED`/`DEGRADED`/`UNSUPPORTED`. The projection may display a +`lastKnownVerdict` separately. Its current verdict falls to `CLAIMED`, +`BLOCKED`, or `UNKNOWN` according to current inputs. + +### Version and configuration changes + +- Evidence matches an exact scenario and suite version in contract v1. No + implicit semver range reuse is allowed. +- A changed scenario assertion, fixture, requirement, or classification rule + requires a new scenario version and invalidates old evidence for the new + projection key. +- `opencodexCompatibilityVersion` is lowercase + `sha256("ocx-lab:compatibility-version:v1\0" || JCS(manifest))`. + The exact manifest object is: + + ```text + { + "schemaVersion": 1, + "assertionDslVersion": "1.0.0", + "evidenceSchemaVersion": "1.0.0", + "bunRuntimeVersion": , + "files": [ + { + "path": , + "sha256": + }, + ... + ] + } + ``` + + `files` contains every Git-index-tracked regular file under `src/`, plus + `package.json`, `bun.lock`, and `scripts/model-metadata.source.json`, sorted + by UTF-8 path bytes. Generation reads current working-tree bytes so a dirty + behavior change cannot reuse clean-tree evidence. A missing file, a tracked + symlink, a non-regular file, duplicate normalized path, invalid UTF-8 path, + or unreadable file makes the run `harness_failure`; untracked files are not + loaded by the compatibility harness. Release/package builds embed this + generated manifest so an installed runtime does not require Git. This + conservative whole-runtime input set may invalidate unrelated evidence, but + cannot falsely reuse evidence after a behavior change. The package marketing + version remains provenance only. +- A compatibility-version change starts a new subject projection. +- Any behavior-relevant configuration fingerprint change starts a new subject. +- Credential rotation alone does not change the subject. + +## 5. Failure-attribution contract + +Every non-pass observation uses exactly one primary class. Stable secondary +codes may add detail without changing these semantics. + +| Class | Meaning | Affects verdict? | Default action | +|---|---|---:|---| +| `protocol_failure` | OpenCodex or the exact route emitted, accepted, ordered, translated, or terminated protocol data incorrectly | Yes, in the observation's layer | Conclusive; reverify after code/config/version change | +| `capability_failure` | The exact route cannot satisfy a capability assertion that it was expected to support | Yes | Conclusive when the scenario rules out an unsupported contract; otherwise retry once then `inconclusive` | +| `behavioral_failure` | A task-effectiveness deterministic verifier failed although protocol/capability prerequisites completed | Yes, task layer only | Conclusive for that task scenario | +| `authentication_blocked` | Missing, expired, rejected, or insufficient credentials prevented the assertion | No | `BLOCKED`; reauthenticate and retry | +| `quota_blocked` | Rate, credit, token, concurrency, or account quota prevented the assertion | No | `BLOCKED`; retry after reset/backoff | +| `region_blocked` | Region/tenant policy prevented execution | No | `BLOCKED`; retry only when route context changes | +| `network_failure` | DNS, TLS establishment, connect, local proxy, or transport reachability failed without provider response evidence | No | `BLOCKED`; repair environment and retry | +| `provider_transient` | Upstream returned a recognized transient/overload failure or interrupted a previously valid service path | No by default | `BLOCKED`; bounded retry/reverification | +| `timeout` | A versioned scenario deadline expired; secondary code distinguishes connect, first-byte, inactivity, or total budget | No by default | `BLOCKED`; bounded retry; reclassify only with deterministic protocol evidence | +| `harness_failure` | Runner, fixture, mock, sandbox, assertion engine, or artifact writer failed | No | Invalidate affected evidence and fix harness | +| `budget_exhausted` | Lab request/token/tool/byte/time budget ended the run before its assertion | No | `BLOCKED`; revise scenario limits/version or retry | +| `inconclusive` | Observations conflict or lack enough information for another class | No | No promotion/degradation; investigate/reverify | + +Manifest registration enforces this exhaustive class/effect matrix: + +| Class | Legal `verdictEffect` | Additional constraint | +|---|---|---| +| `protocol_failure` | `degraded`, or `none` for a supplemental assertion | Never `unsupported` | +| `capability_failure` | `degraded`, `unsupported`, or `none` for an exact conformance negative control | `unsupported` requires a `capability_absence_control` | +| `behavioral_failure` | `degraded`, or `none` for a supplemental assertion | `task_effectiveness` only | +| `authentication_blocked` | `none` | Environmental blocker only | +| `quota_blocked` | `none` | Environmental blocker only | +| `region_blocked` | `none` | Environmental blocker only | +| `network_failure` | `none` | Environmental blocker only | +| `provider_transient` | `none` | Environmental blocker only | +| `timeout` | `none` | Environmental blocker only | +| `harness_failure` | `none` | Invalidates/blocks evidence only | +| `budget_exhausted` | `none` | Environmental blocker only | +| `inconclusive` | `none` | No verdict promotion or degradation | + +Any other pair rejects the manifest. A retry count cannot change the pair; +reclassification requires a new observation with independently satisfied +classification rules. + +`expectedFailure` is either absent or: + +```text +controlKind conformance_negative_control | + capability_absence_control +expectedClass +expectedCode +assertionIds[] +onMatch pass | unsupported +onMismatch fail | inconclusive +``` + +For `conformance_negative_control`, `onMatch` must be `pass` and the legal +effect is `none`; the expected rejection helps satisfy `VERIFIED`. For +`capability_absence_control`, `onMatch` must be `unsupported` and the class +must be `capability_failure`. No one observation can both satisfy verification +and project `UNSUPPORTED`. + +Safety rules: + +- Expired credentials never imply broken tool support. +- Quota exhaustion never implies model incompatibility. +- Local DNS/TLS/connect failure never degrades provider capability. +- A generic timeout never proves missing terminal semantics. A deterministic + mock stream that closes without its required terminal event is + `protocol_failure`; a live body that simply stalls is `timeout`. +- Malformed tool-call semantics, broken tool-result correlation, or lost + required event ordering may legitimately produce `protocol_failure` and + `DEGRADED`. +- `provider_transient` may be promoted to a compatibility-affecting class only + by a scenario-specific deterministic rule and a new observation; projection + code must not infer promotion from retry count. +- Expected failures follow the closed control contract above. A generic + `negative_control` label never implies capability absence. + +## 6. Canonical route subject + +Live route evidence is never keyed by model name alone. `RouteSubjectV1` +contains: + +```text +subjectSchemaVersion 1 +subjectKind route +providerId +providerInstanceFingerprint +clientModelId +upstreamModelId +effectiveAdapter +inboundProtocol +upstreamProtocol +surface +opencodexCompatibilityVersion +behaviorFingerprint +endpointFingerprint +dependencies[] +``` + +Semantics: + +- `providerId` is the built-in registry ID or `custom`; it is not a display + label. +- `providerInstanceFingerprint` is a locally salted HMAC over the configured + provider identity, allowing two instances of one preset to differ without + leaking a user-selected name. All local opaque fingerprints use lowercase + HMAC-SHA-256 over + `UTF8("ocx-lab:local-fingerprint:v1\0" + fieldName + "\0") || JCS(value)` + with the installation salt as key. +- `clientModelId` is the selected canonical route model; `upstreamModelId` is + the effective wire model after namespace, virtual-model, combo and suffix + resolution. +- `effectiveAdapter` reflects model-specific wire defaults/overrides and wire + pins, not merely the provider-wide configured adapter. +- Protocol values distinguish OpenAI Responses, OpenAI Chat Completions, + Anthropic Messages, and provider-specific wires. +- `surface` distinguishes behaviorally different ingress/transport paths such + as Responses HTTP, Responses WebSocket, Chat HTTP/SSE, and Anthropic + Messages HTTP/SSE. +- The package/build version remains observation provenance in + `producerVersion`; only `opencodexCompatibilityVersion` participates in the + subject so an unrelated release does not discard valid conformance evidence. +- `endpointFingerprint` is a locally salted HMAC of the normalized destination + scheme/host/port/base path. Raw URLs, userinfo, query strings, and fragments + are not evidence fields. +- `dependencies` is an ordered list of flat `RouteDependencyV1` records for + behaviorally invoked sidecars. Each record contains role, provider ID, + provider-instance fingerprint, client/upstream model IDs, effective adapter, + upstream protocol, endpoint fingerprint, and behavior fingerprint. It cannot + nest. Records sort by this total order of UTF-8 string comparisons: + role, provider ID, provider-instance fingerprint, upstream model ID, + endpoint fingerprint, client model ID, effective adapter, upstream protocol, + then behavior fingerprint. A duplicate full key makes subject construction + `harness_failure`. An empty list is canonical when no sidecar is invoked. + +### Behavior fingerprint allowlist + +The fingerprint is SHA-256 over JCS `BehaviorFingerprintV1`: + +```text +{ + "schemaVersion": 1, + "resolverVersion": 1, + "values": { + "": { + "source": "request" | "model_override" | "provider_config" | + "registry_runtime_default" | "generated_model_metadata" | + "global_config" | "adapter_default" | "lab_forced", + "value": + } + } +} +``` + +The production route/model/adapter resolver, not a second Lab merge, emits the +effective value and winning source tag. Thus model-specific wire/config +overrides, registry runtime defaults, generated metadata, global values and +adapter defaults retain the exact live precedence. If precedence is ambiguous, +an effective default cannot be resolved, or scenario execution reads a +behavior-changing input that has no closed key below, subject construction +fails as `harness_failure` with code `unclassified_behavior_input`; no evidence +is emitted. + +Closed V1 keys are: + +```text +wire.adapter +wire.upstreamProtocol +wire.responsesPath +wire.commandCodeVersion +wire.modelSuffixMode +auth.mode +auth.transport +responses.stateful +responses.upstreamStreaming +responses.serviceTier +responses.snapshotRepair +responses.itemIdRepair +limits.contextWindow +limits.maxInputTokens +limits.maxOutputTokens +modalities.input +sampling.omitTemperature +sampling.omitTopP +sampling.omitPenalties +reasoning.supported +reasoning.efforts +reasoning.defaultEffort +reasoning.effortMap +reasoning.wireFormat +reasoning.summaryMode +reasoning.replayMode +reasoning.splitMode +reasoning.toggleMode +reasoning.budgetMode +tools.choiceRestrictions +tools.parallel +tools.hostedPreference +tools.customFreeform +tools.builtinNameEscaping +cache.forwarding +cache.retention +anthropic.eofPolicy +google.mode +google.projectFingerprint +google.locationFingerprint +openrouter.order +openrouter.only +openrouter.allowFallbacks +sidecars.vision +sidecars.webSearch +mcp.maxTools +mcp.maxSchemaBytes +mcp.maxResultBytes +mcp.nativeLocalExec +runtime.bunVersion +runtime.platform +runtime.arch +runtime.streamMode +runtime.fastMode +runtime.effortCap +headers.nonCredentialBehaviorDigest +``` + +`sampling.omitTemperature`, `sampling.omitTopP`, and +`sampling.omitPenalties` are the selected-model effective booleans produced +from `noTemperatureModels`, `noTopPModels`, and `noPenaltyModels`, rather than +hashes of the whole configured arrays. `wire.commandCodeVersion` is the +effective `commandCodeVersion`. `cache.retention` is the effective global +`cacheRetention` value. Sidecar keys contain the effective enablement, backend, +model, reasoning, per-turn limits and timeout/stall limits plus the matching +flat dependency subject ID. + +Values that are inapplicable to the selected model/surface are omitted rather +than copied wholesale. Canonical JSON sorts object keys and normalizes +absent/default values to their effective value. Array handling is closed and +part of the V1 schema: + +| Array-valued key | Ordering mode | Element comparison | +|---|---|---| +| `modalities.input` | `set` | UTF-8 JCS bytes | +| `reasoning.efforts` | `set` | UTF-8 JCS bytes | +| `tools.choiceRestrictions` | `set` | UTF-8 JCS bytes | +| `openrouter.order` | `ordered` | source order preserved | +| `openrouter.only` | `set` | UTF-8 JCS bytes | + +For `set`, each element is JCS-canonicalized, duplicate JCS byte sequences are +rejected, and elements sort lexicographically by their UTF-8 JCS bytes. For +`ordered`, elements are JCS-canonicalized but their resolved source order is +preserved. Any other closed key resolving to an array, or any array-valued key +without one of these declared modes, fails subject construction as +`harness_failure` with code `unclassified_behavior_input`. + +Acceptance tests for the future builder must enumerate every +behavior-changing config/default read by the selected adapter and prove that +changing each effective value changes the fingerprint, while changing every +excluded secret/transient value does not. + +The following never participate: + +- API keys, OAuth/access/refresh tokens, cookies, authorization headers; +- account IDs, emails, labels, aliases, quota balances or plan names; +- raw custom/private header names or values; +- prompts, messages, tool results, repository paths or contents; +- timestamps, transient health, latency, cost, quota or retry state. + +Credential headers are excluded. The config-owner fingerprint broker described +by the security contract supplies only +`headers.nonCredentialBehaviorDigest`; Lab code never receives raw header names +or values. Project/location and custom endpoint values use the same local +opaque treatment. + +Public export replaces all local fingerprints with export-scoped opaque IDs +and redacts custom model IDs unless the export policy explicitly classifies +them as public. + +## 7. Task-effectiveness ingress + +A future Fabric observation must provide, at minimum: + +```text +producerSchemaVersion +outcomeId +taskClassId +taskClassVersion +subject TaskSubjectV1 +taskFixtureDigest +verifierManifestDigest +startedAt +completedAt +resourceLimits +result success | failure | blocked | inconclusive +verifiers[] { id, version, manifestDigest, result, + normalizedMetrics? } +artifactRefs[] +``` + +The Lab rejects an outcome if the subject cannot be reconstructed, a verifier +is nondeterministic for a canonical assertion, or an artifact violates the +security contract. Free-form narrative may be retained only as bounded, +sanitized advisory metadata and never determines the verdict. + +## 8. Consumer boundaries + +- Provider Registry supplies claims; Lab does not rewrite them. +- Request history supplies route/outcome references; Lab does not copy its + ledger. +- Selection and execution stay separate: `RouteDecisionTraceV1` records the + pre-dispatch choice, `attempts[]` records physical execution/fallback, and + final outcome is joined at read time. +- Routing Profiles express user requirements; Lab does not evaluate policy. +- Router Intelligence reads projections; Lab does not rank candidates. +- Route decision traces explain compatibility exclusions/penalties in the + existing trace; Lab does not create a parallel route explanation. +- Agent Fabric executes tasks; Lab does not run arbitrary repository work. diff --git a/devlog/_plan/260807_compatibility_lab/020_scenario_contract_and_catalogue.md b/devlog/_plan/260807_compatibility_lab/020_scenario_contract_and_catalogue.md new file mode 100644 index 000000000..fe53ef6e0 --- /dev/null +++ b/devlog/_plan/260807_compatibility_lab/020_scenario_contract_and_catalogue.md @@ -0,0 +1,465 @@ +# CL-00 scenario contract and initial catalogue + +This document freezes the scenario schema and the initial IDs. CL-01 implements +the `*.protocol.*` scenarios only. Entries marked live or Fabric-reserved define +future semantics and are not authorization to execute them. + +The normative V1 selector/operator semantics, immutable fixture anchors, +expanded defaults and complete protocol scenario/suite manifest records are in +[the protocol V1 manifest authority](./021_protocol_v1_manifest_authority.md); +its canonical fixture vectors and literal expectations are in +[`022_protocol_v1_cases.json`](./022_protocol_v1_cases.json). + +## 1. Versioned scenario model + +A `CompatibilityScenarioV1` has: + +```text +schemaVersion 1 +id stable lowercase dotted ID +version exact semver +suite { id, version, evidenceLayer } +evidenceLayer protocol_conformance | + live_route_compatibility | + task_effectiveness +capability stable capability ID +verificationRole required | supplemental | negative_control +requirements +fixtures +executionLimits +assertions[] +failureRules[] +artifactPolicy +freshness +``` + +### Identity and versioning + +- `id` names semantics and does not contain a provider, model, or version. +- `version` is exact-match in contract v1. +- Any assertion, fixture, limit that affects expected behavior, failure rule, + artifact exposure, or requirement change increments the scenario version. +- Editorial description changes do not require a version change. +- A suite manifest has its own exact version and lists scenario IDs, versions, + roles, and its verification rule. +- A suite manifest belongs to exactly one `evidenceLayer`; a scenario and suite + with different layers are invalid. Human-facing suite stems may recur across + layers, but their manifest keys are + `(id, evidenceLayer, version, manifestDigest)`. +- Scenario and suite manifests are RFC 8785 canonical JSON with the + domain-separated digest defined by the evidence contract. + +### Requirements + +Requirements are declarative and may include: + +```text +inboundProtocols[] +upstreamProtocols[] +surfaces[] +requiredClaims[] +requiredHarnessFeatures[] +platforms[] +routePreconditions[] +``` + +An unmet deterministic fixture requirement is `harness_failure`. An unmet live +route precondition is either inapplicable or a typed blocker; it is never +silently counted as a capability failure. + +### Fixtures + +Fixture references include ID, content digest, media type, generator version +where generated, and role (`client_request`, `upstream_response`, +`adapter_vector`, `synthetic_tool`, `synthetic_image`, or `task_fixture`). A +fixture never embeds credentials, user data, or an external mutable URL. + +Protocol fixtures run against a deterministic mock upstream. Live scenarios use +only Lab-owned synthetic requests and inert tools. Fabric scenarios refer to a +versioned synthetic task class; they do not place a repository in the Lab +ledger. + +### Execution limits + +Every scenario states: + +```text +totalTimeoutMs +connectTimeoutMs? +firstByteTimeoutMs? +inactivityTimeoutMs? +maxRequests +maxInputBytes +maxOutputBytes +maxOutputTokens? +maxToolCalls +maxArtifactBytes +``` + +Absent limits are invalid. Limits may be stricter than Lab-wide ceilings but +not wider without a scenario version change and security review. Expiry of a +limit classifies as `timeout` or `budget_exhausted` according to the failed +limit; it does not imply incompatibility. + +### Deterministic assertion DSL + +Canonical assertions use a closed set of observable operators: + +```text +http_status_equals +header_present +header_absent +header_value_equals +json_schema_matches +json_path_equals +json_path_present +json_path_absent +sse_field_equals +sse_event_sequence +sse_event_count +terminal_signal_equals +id_matches +id_stable_across_events +id_correlates +tool_call_equals +tool_result_correlates +fixture_request_matches +normalized_text_equals +byte_limit_observed +process_exit_equals +verifier_result_equals +``` + +Each assertion has an ID, operator, selector, expected value, required flag, +and redaction-safe observed summary. Implementations must exhaustively reject +unknown operators. Protocol V1 permits no arbitrary regular expressions; ID +grammars and all operator type/missing-value behavior are closed in the +manifest authority. + +Core verdicts cannot use an LLM judge, free-form human interpretation, or a +snapshot that contains unstable timestamps/IDs without normalization. + +### Failure rules + +Rules are ordered and explicit: + +```text +match assertion IDs, normalized status/error/event/timeout +classification canonical failure class +secondaryCode +verdictEffect none | degraded | unsupported +retry never | bounded | after_precondition_change +expected boolean +``` + +The first exact rule wins. If no rule establishes a compatibility-attributable +class, the attempt is `inconclusive`. Generic HTTP 4xx/5xx rules may classify a +blocker or transient but cannot prove `UNSUPPORTED`. + +Manifest registration must enforce the exhaustive classification/effect matrix +in `010_architecture_and_evidence_contract.md`. Environmental, timeout, +budget, harness, transient, and inconclusive classes permit only +`verdictEffect: none`; a manifest that maps any of them to `degraded` or +`unsupported` is invalid. + +`expectedFailure`, when present, includes `controlKind`, exact class/code, +assertion IDs, `onMatch`, and `onMismatch`. A +`conformance_negative_control` exact rejection is a verification pass with no +verdict effect. A `capability_absence_control` may produce `UNSUPPORTED`. These +meanings cannot be combined in one scenario observation. + +### Artifact policy + +The policy is deny-by-default and names allowed normalized artifacts: + +```text +assertion_report +sanitized_request_shape +sanitized_response_shape +normalized_event_trace +sanitized_error +verifier_summary +``` + +It states per-artifact and aggregate byte limits, retention class, local/public +visibility, and redaction profile. Raw credentials, prompts, hidden reasoning, +full task repositories, arbitrary headers, and arbitrary response bodies are +not valid artifact kinds. + +Each scenario declares `freshness.maxAgeMs`; its suite may declare a stricter +bound, and a future profile may tighten it again. Effective maximum age is the +minimum finite bound, with `null` meaning unbounded at that layer. + +## 2. Suite projection rules + +For each exact suite manifest, first compute the required scenarios whose +manifest requirements are applicable to the exact subject. That applicable +required set must be non-empty for any positive executable verdict. + +- `VERIFIED`: the applicable required set is non-empty; every applicable + `required` scenario has a current pass; and every applicable + `conformance_negative_control` observed its exact required rejection. +- `PROBED`: at least one applicable required scenario passed, with no current + compatibility-attributable required-scenario failure, but the suite's + verification rule is not fully satisfied. +- `DEGRADED`: a failure rule on an applicable required scenario yields + `degraded`. +- `UNSUPPORTED`: a required scenario's exact + `capability_absence_control` proves the capability unavailable. +- `BLOCKED`: only blockers exist and no current attributable verdict takes + precedence. +- `CLAIMED`/`UNKNOWN`: follow the evidence contract. + +An inapplicable required scenario contributes neither a pass nor a failure. If +no required scenario is applicable, the suite cannot project `PROBED` or +`VERIFIED`; it falls through to `CLAIMED` or `UNKNOWN` under the evidence +contract. If execution was attempted but an environmental or administrative +precondition prevented reaching assertions, that attempt is a typed blocker +and may project `BLOCKED` under normal precedence; it is not treated as +inapplicability. + +Supplemental scenarios never block `VERIFIED` unless a new suite version makes +them required. + +## 3. Initial suite catalogue + +All initial scenario versions and suite versions are `1.0.0`. + +For protocol V1, the literal fixtures/assertions in `022` are the complete +verification boundary. Descriptions below summarize those exact vectors; they +do not silently incorporate every historical incident mapped to the same +scenario ID. An incident absent from `022` is candidate coverage for a reviewed +scenario/suite version amendment and cannot be claimed by a V1 `VERIFIED` +verdict. + +### `responses-core` + +Purpose: preserve the OpenAI Responses request, output-item lifecycle, stream +framing, IDs, terminal state, and JSON/SSE equivalence. + +Capability: `protocol.responses.core`. + +| Scenario ID | Layer/applicability | Required observable assertions | +|---|---|---| +| `responses-core.protocol.request-shape` | Deterministic; CL-01 | Mock receives the exact model, first user text and zero temperature | +| `responses-core.protocol.sse-framing` | Deterministic; CL-01 | Spaced and unspaced data fields, a non-record `null` frame, data-only event inference, exact text and completion terminal | +| `responses-core.protocol.item-lifecycle` | Deterministic; CL-01 | One added/done/completed lifecycle with stable valid message ID | +| `responses-core.protocol.terminal-state` | Deterministic; CL-01 | One explicit failed terminal is preserved exactly | +| `responses-core.protocol.json-sse-equivalence` | Deterministic; CL-01 | One normalized JSON/SSE pair has equal text and completion terminal | +| `responses-core.live.basic-turn` | Live-reserved | 2xx, bounded output, valid lifecycle and terminal state from exact route | + +Unsupported means a route deterministically rejects the Responses surface with +a suite-recognized unsupported signal. Semantic loss, invalid IDs, malformed +event order, or missing terminal state is degraded. Auth/quota/region/network, +transient upstream errors, and body stalls are blocked. + +### `chat-core` + +Purpose: preserve OpenAI Chat Completions request/response semantics for JSON +and streaming routes. + +Capability: `protocol.chat.core`. + +| Scenario ID | Layer/applicability | Required observable assertions | +|---|---|---| +| `chat-core.protocol.request-mapping` | Deterministic; CL-01 | System/developer/user order and JSON-object response format match the fixture | +| `chat-core.protocol.nonstream-envelope` | Deterministic; CL-01 | One valid choice/message/finish/usage envelope yields exact text and terminal | +| `chat-core.protocol.stream-assembly` | Deterministic; CL-01 | Fragmented/interleaved deltas assemble in order and emit one finish | +| `chat-core.protocol.stream-terminal` | Deterministic; CL-01 | One stop finish plus `[DONE]` yields exact text and one terminal | +| `chat-core.live.basic-turn` | Live-reserved | Exact route returns bounded text and a valid finish contract | + +Unsupported is a deterministic surface rejection. Incorrect role mapping, +malformed choices, lost stream fragments, or invalid finish semantics is +degraded. Environmental and transient failures are blocked. + +### `anthropic-core` + +Purpose: preserve Anthropic Messages roles/content blocks, tool/thinking block +ordering, stop reasons, usage, and SSE lifecycle. + +Capability: `protocol.anthropic.messages.core`. + +| Scenario ID | Layer/applicability | Required observable assertions | +|---|---|---| +| `anthropic-core.protocol.request-mapping` | Deterministic; CL-01 | Model, system instruction and first user text map exactly | +| `anthropic-core.protocol.content-sequence` | Deterministic; CL-01 | `message_start`, monotonic content blocks/deltas/stops, `message_delta`, `message_stop` | +| `anthropic-core.protocol.tool-round-trip` | Deterministic; CL-01 | One `tool_use`/`tool_result` pair preserves its ID correlation and result text | +| `anthropic-core.protocol.terminal-errors` | Deterministic; CL-01 | One explicit Responses failure maps to the Anthropic error/failed terminal | +| `anthropic-core.live.basic-turn` | Live-reserved | Exact route returns a valid bounded Messages lifecycle and terminal | + +Unsupported is a recognized Messages-surface rejection. Wrong block ordering, +lost tool correlation, invalid stop reason, or clean EOF accepted without the +suite's terminal contract is degraded. Authentication, quota, region, network, +transient failure, and silence timeout are blocked. + +### `tools-core` + +Purpose: prove deterministic function/custom tool declaration, call assembly, +parallel correlation, and result continuation. + +Capability: `tools.round_trip`. + +| Scenario ID | Layer/applicability | Required observable assertions | +|---|---|---| +| `tools-core.protocol.function-round-trip` | Deterministic; CL-01 | One function call preserves ID/name/parsed arguments and its continuation result correlates | +| `tools-core.protocol.custom-freeform-round-trip` | Deterministic; CL-01 | One `apply_patch` call preserves exact freeform input and its continuation result correlates | +| `tools-core.protocol.parallel-correlation` | Deterministic; CL-01 | Two interleaved calls assemble once in first-seen order without overlap | +| `tools-core.protocol.result-content` | Deterministic; CL-01 | One result preserves exact text and data-image parts | +| `tools-core.protocol.choice-and-allowed-set` | Deterministic; CL-01 | One required single-tool allowed set narrows exactly without widening | +| `tools-core.live.function-round-trip` | Live-reserved | Inert deterministic function is called once with schema-valid args and static result is continued | +| `tools-core.live.custom-freeform-round-trip` | Live-reserved | Route emits exact custom/freeform call and accepts static result continuation | + +An explicit route `capability_absence_control` that rejects a tool kind can +prove unsupported. +Malformed arguments, dangling IDs, widened choice, dropped calls/results, or +incorrect parallel assembly is degraded. A model choosing not to call an +`auto` tool is inconclusive; required tool choice is used for conclusive live +coverage. Environmental failures are blocked. + +### `codex-core` + +Purpose: establish the minimum end-to-end semantics required to advertise a +route as usable by Codex. A basic Responses text request is insufficient. + +Capability: `client.codex.core`. + +| Scenario ID | Layer/applicability | Required observable assertions | +|---|---|---| +| `codex-core.protocol.streaming-turn` | Deterministic; CL-01 | One Chat-backed stream yields exact text, final-answer phase and one completed terminal | +| `codex-core.protocol.apply-patch-turn` | Deterministic; CL-01 | One custom `apply_patch` call preserves exact patch text and its result ID correlates | +| `codex-core.protocol.tool-continuation` | Deterministic; CL-01 | One function result follows and correlates with its prior call | +| `codex-core.protocol.previous-response-replay` | Deterministic; CL-01 | One local expansion preserves stored input/output/new-input order and strips `previous_response_id` | +| `codex-core.protocol.structured-output` | Deterministic; CL-01 | One JSON-schema request maps to the exact Chat `response_format` | +| `codex-core.protocol.compaction-and-special-items` | Deterministic; CL-01 | Compaction, local shell, tool search and hosted-tool items are normalized without leaking opaque raw data | +| `codex-core.live.tool-turn` | Live-reserved | Valid stream plus required inert tool call/result continuation and terminal | +| `codex-core.live.custom-tool-turn` | Live-reserved | Required custom/freeform call/result continuation and valid terminal | + +The `codex-core` manifest requires all six protocol scenarios for conformance +verification. Future live verification requires both live scenarios plus +current `responses-core.live.basic-turn`. Text-only success is at most partial +coverage, never `codex-core: VERIFIED`. + +An exact route may be unsupported when it deterministically lacks a mandatory +Codex surface or tool kind. Lossy lifecycle, call/result correlation, +continuation, or special-item behavior is degraded. Environmental failures are +blocked. + +### `vision-core` + +Purpose: preserve declared image input and tool-result image behavior and prove +exact-route image understanding without user media. + +Capability: `modalities.image.input`. + +| Scenario ID | Layer/applicability | Required observable assertions | +|---|---|---| +| `vision-core.protocol.input-image` | Deterministic; CL-01 | One data-URL image preserves detail and text/image ordering | +| `vision-core.protocol.tool-result-image` | Deterministic; CL-01 | Image content in function/tool result remains structured and correlated | +| `vision-core.protocol.modality-gate` | Deterministic negative control; CL-01 | A text-only/no-sidecar synthetic vector produces the typed unsupported path without silent image drop | +| `vision-core.live.synthetic-ocr` | Live-reserved | Lab-generated image nonce is returned in an exact JSON schema | + +A deterministic declared no-image `capability_absence_control` may prove +unsupported. The protocol V1 modality gate is instead a conformance negative +control whose exact rejection is a verification pass. Dropping, textifying +without a declared sidecar, corrupting, or misordering image content is +degraded. Failure of the optional sidecar route is attributed to that exact +subject. Auth/quota/network/transient failures are blocked. + +### `reasoning-core` + +Purpose: preserve supported reasoning controls, summaries, signatures and +replay while preventing provider-private reasoning material from crossing an +incompatible boundary. + +Capability: `reasoning.round_trip`. + +| Scenario ID | Layer/applicability | Required observable assertions | +|---|---|---| +| `reasoning-core.protocol.effort-mapping` | Deterministic; CL-01 | Effective effort maps to the declared wire form and unsupported parameters are omitted | +| `reasoning-core.protocol.summary-stream` | Deterministic; CL-01 | One summary-part/delta/completed sequence preserves ordering and reasoning ID | +| `reasoning-core.protocol.replay` | Deterministic; CL-01 | One synthetic plaintext/signature replay reaches the second turn exactly | +| `reasoning-core.protocol.private-content-isolation` | Deterministic; CL-01 | One provider-private value is absent from an incompatible upstream and client response | +| `reasoning-core.live.replay` | Live-reserved | Synthetic two-turn route accepts its declared replay form and completes | + +An explicit no-reasoning route can prove unsupported. Rejected or corrupted +declared replay, lost required signatures, or private content sent across an +incompatible provider boundary is degraded (and the latter is also a security +finding). Environmental failures are blocked. + +### `mcp-core` + +Purpose: preserve MCP namespace, schema, tool/resource invocation and result +contracts through supported adapters without touching user MCP servers. + +Capability: `tools.mcp.core`. + +| Scenario ID | Layer/applicability | Required observable assertions | +|---|---|---| +| `mcp-core.protocol.namespace-mapping` | Deterministic; CL-01 | One namespace/name pair flattens and reverses exactly | +| `mcp-core.protocol.schema-and-bounds` | Deterministic; CL-01 | Tool schemas encode correctly; exact configured bounds admit and one-byte-over rejects atomically | +| `mcp-core.protocol.call-result` | Deterministic; CL-01 | Lab stub receives one exact call and returns one successful text result | +| `mcp-core.protocol.resource-round-trip` | Deterministic; CL-01 | One list/read resource success shape preserves URI, name and text | +| `mcp-core.live.synthetic-tool` | Live-reserved | Lab-owned loopback pure-function MCP tool is advertised, called and correlated | + +Only a Lab-owned in-memory or loopback fixture is allowed. A route that +deterministically cannot expose MCP may be unsupported. Namespace loss, schema +corruption, partial bound commits, or result miscorrelation is degraded. +User-server unavailability is never tested; environmental failures are blocked. + +### `fabric-core` task-effectiveness reservation + +This is a distinct `task_effectiveness` suite manifest, not an extension of a +protocol or live-route manifest: + +```text +suite.id fabric-core +suite.version 1.0.0 +suite.evidenceLayer task_effectiveness +verificationRule all-applicable-required-pass-v1 +freshness.maxAgeMs 2592000000 +``` + +The first reserved scenario is +`fabric-core.task.synthetic-patch@1.0.0`: + +- subject: exact `TaskSubjectV1`; +- verification role: `required`; +- fixture: a content-addressed synthetic scratch tree containing + `src/value.txt` with UTF-8 bytes `before\n`, plus a task-class manifest that + requests the exact final bytes `after\n`; +- execution: Fabric-owned, no user repository/prompt, no network, no user MCP, + no shell, and filesystem access restricted to that synthetic scratch tree; +- limits: one file, 64 KiB aggregate input/output, one patch operation, + 30-second total, 5-second inactivity, and 1 MiB aggregate artifacts; +- verifier manifest: `exact-tree-diff-v1`, whose digest participates in + `TaskSubjectV1`; +- deterministic verifier: sort repository-relative POSIX paths by UTF-8 bytes, + reject symlinks/special files/path traversal, hash exact file bytes, and pass + only when the sole diff changes `src/value.txt` from `before\n` to `after\n` + with no added/deleted/renamed file; +- success assertion: verifier result `pass`; +- failure rules: verifier `fail` is `behavioral_failure -> degraded`; + unavailable Fabric/sandbox is `harness_failure -> none`; exhausted time/bytes + is the corresponding blocker with effect `none`; +- artifact policy: retain only the bounded normalized path/digest diff and + verifier summary, never file bodies. + +This reservation freezes task-subject and verifier semantics for a later Fabric +phase. It does not authorize CL-01 to implement or execute the task. + +## 4. CL-01 implementation boundary + +CL-01 may implement the scenario registry, deterministic mock-upstream harness, +closed assertion DSL, and only the `protocol_conformance` manifests frozen in +`021_protocol_v1_manifest_authority.md`. It must not: + +- contact a real provider; +- write the planned production evidence ledger or SQLite projection; +- add profile/routing controls; +- execute a user tool, shell, filesystem, repository, MCP server, or external + network action; +- implement live/Fabric scenarios merely because their IDs are reserved here. + +If CL-01 discovers that an observable assertion cannot be implemented without +new semantics, it must amend this contract in a reviewed change rather than +quietly inventing behavior. diff --git a/devlog/_plan/260807_compatibility_lab/021_protocol_v1_manifest_authority.md b/devlog/_plan/260807_compatibility_lab/021_protocol_v1_manifest_authority.md new file mode 100644 index 000000000..4135ba967 --- /dev/null +++ b/devlog/_plan/260807_compatibility_lab/021_protocol_v1_manifest_authority.md @@ -0,0 +1,480 @@ +# CL-00 protocol V1 manifest authority + +This document closes the executable semantics for the initial +`protocol_conformance` scenarios. It is normative for CL-01 and does not +implement a runner. + +The machine-readable source of truth is +[`022_protocol_v1_cases.json`](./022_protocol_v1_cases.json). It contains 35 +provider-independent canonical fixture vectors, literal expected values, +row-specific requirements, exact roles/media types, execution limits, artifact +policy, failure rules, and domain-separated fixture digests. Historical tests +in the [incident corpus](./030_incident_corpus.md) are provenance and coverage +guidance only; they are not executable manifest semantics. + +## 1. Exact manifest expansion + +For each entry in `cases`, CL-01 constructs `CompatibilityScenarioV1` in this +field order before RFC 8785 canonicalization: + +```text +schemaVersion source.schemaVersion +id case.id +version manifestDefaults.version +suite { id: case.suite, + version: manifestDefaults.suiteVersion, + evidenceLayer: + manifestDefaults.evidenceLayer } +evidenceLayer manifestDefaults.evidenceLayer +capability case.capability +verificationRole case.verificationRole when present, + otherwise manifestDefaults.verificationRole +requirements case.requirements +fixtures when case.initiatingRequest is present: + [fixtureRef(case.initiatingRequest), + fixtureRef(case.fixture)] + otherwise: [fixtureRef(case.fixture)] +executionLimits manifestDefaults.executionLimits +assertions case.assertions +expectedFailure case.expectedFailure, only when present +failureRules expandFailureRules(case) +artifactPolicy manifestDefaults.artifactPolicy +freshness manifestDefaults.freshness +``` + +`fixtureRef(x)` is exactly: + +```text +{ + id: x.id, + role: x.role, + mediaType: x.mediaType, + digest: x.digest, + byteLength: UTF8(x.bytesUtf8).byteLength, + syntheticMarker: "ocx-lab-synthetic-v1", + provenance: { + kind: "lab_authored", + authority: "022_protocol_v1_cases.json", + sourceCommit: source.sourceCommit + } +} +``` + +The marker/provenance fields are mandatory for every protocol V1 fixture +reference and participate in the scenario manifest digest. Registration rejects +any fixture reference whose marker is absent/different, whose provenance kind +is not `lab_authored`, whose authority is not the exact authority file above, +or whose source commit differs from the parsed authority. This is the +machine-checkable synthetic-fixture boundary: a fixture cannot be substituted +from user input, a repository, MCP config, an external URL, or a live response +merely because its bytes share a valid digest. + +`expandFailureRules(case)` copies +`failureRuleSets[manifestDefaults.failureRuleSet]`. When +`case.expectedFailure` is absent, that copy is the complete rule list and no +control-specific rule exists. When present, registration validates the closed +control matrix in the evidence contract and inserts this materialized record +immediately before `required-assertion`: + +```text +id expectedFailureRuleTemplate.id +match expectedFailureRuleTemplate.match +classification case.expectedFailure.expectedClass +secondaryCode case.expectedFailure.expectedCode +verdictEffect none when onMatch is pass, + unsupported when onMatch is unsupported +retry expectedFailureRuleTemplate.retry +expected expectedFailureRuleTemplate.expected +``` + +`expected_failure_exact_match` exists only when the observed class and code +equal `expectedClass`/`expectedCode` and every listed assertion ID passed. An +`onMismatch: fail` falls through to `required-assertion`; an +`onMismatch: inconclusive` falls through to `fallback`. No V1 case declares a +`capability_absence_control`, so no V1 expanded manifest contains an +`unsupported` effect. + +JSON object field order has no digest effect, but the field set above is +closed. Unknown fields reject registration. Arrays preserve source order. +Empty arrays remain present. No default may be read from runtime code. + +The scenario digest is: + +```text +sha256( + UTF8("ocx-lab:scenario-manifest:v1\0") || + UTF8(JCS(expanded scenario)) +) +``` + +The exact fixture bytes are UTF-8 encoding of each `bytesUtf8` field; every +published `fixture` and `initiatingRequest` digest is: + +```text +sha256(UTF8("ocx-lab:fixture:v1\0") || fixture bytes) +``` + +Registration recomputes all digests and rejects mismatch. Every `bytesUtf8` is +retained as a content-addressed fixture artifact, not duplicated into the +expanded manifest. An `upstream_response` case without an initiating +`client_request` rejects registration. + +## 2. Exact suite manifests + +All suite versions are `1.0.0`. Every listed scenario has role `required` +except `vision-core.protocol.modality-gate`, whose role is `negative_control`. +No supplemental protocol V1 scenario exists. Order is the order in the case +authority: + +| Suite | Capability | Required member suffixes | +|---|---|---| +| `responses-core` | `protocol.responses.core` | `request-shape`, `sse-framing`, `item-lifecycle`, `terminal-state`, `json-sse-equivalence` | +| `chat-core` | `protocol.chat.core` | `request-mapping`, `nonstream-envelope`, `stream-assembly`, `stream-terminal` | +| `anthropic-core` | `protocol.anthropic.messages.core` | `request-mapping`, `content-sequence`, `tool-round-trip`, `terminal-errors` | +| `tools-core` | `tools.round_trip` | `function-round-trip`, `custom-freeform-round-trip`, `parallel-correlation`, `result-content`, `choice-and-allowed-set` | +| `codex-core` | `client.codex.core` | `streaming-turn`, `apply-patch-turn`, `tool-continuation`, `previous-response-replay`, `structured-output`, `compaction-and-special-items` | +| `vision-core` | `modalities.image.input` | `input-image`, `tool-result-image`, `modality-gate` | +| `reasoning-core` | `reasoning.round_trip` | `effort-mapping`, `summary-stream`, `replay`, `private-content-isolation` | +| `mcp-core` | `tools.mcp.core` | `namespace-mapping`, `schema-and-bounds`, `call-result`, `resource-round-trip` | + +For each row, the expanded suite manifest is: + +```text +schemaVersion 1 +id table suite +version 1.0.0 +evidenceLayer protocol_conformance +capability table capability +assertionDslVersion 1.0.0 +evidenceSchemaVersion 1.0.0 +freshness { maxAgeMs: null } +contradictionRule newest-required-observation-v1 +scenarios [{ + id: full case ID, + version: 1.0.0, + role: expanded scenario verificationRole, + manifestDigest: expanded scenario digest + }, ...] +verificationRule all-applicable-required-pass-v1 +``` + +Unknown fields reject registration. The suite digest uses +`ocx-lab:suite-manifest:v1` plus JCS exactly as defined by the evidence +contract. `VERIFIED` requires at least one applicable required member, a +current pass for every applicable required member, and the exact +suite/scenario/fixture digests above. + +## 3. Observation selector model + +Assertions use absolute RFC 6901 JSON Pointers into this closed normalized +observation: + +```text +{ + "client": { + "request": { "status": 0, "headers": {}, "json": null, "rawBytes": 0 }, + "response": { + "status": 0, + "headers": {}, + "json": null, + "events": [], + "toolCalls": [], + "mcpCalls": [], + "terminal": null, + "normalizedText": "" + } + }, + "upstream": { + "requests": [ + { "status": 0, "headers": {}, "json": null, "rawBytes": 0 } + ], + "responses": [] + }, + "process": { "exitCode": null }, + "verifiers": {} +} +``` + +Header names are lowercase. JSON pointers use `~0` and `~1` escaping. Array +indexes are decimal with no leading zero except `0`; `-` is forbidden. +Wildcard, filter, recursive descent, script expression, URI, and filesystem +selectors do not exist in V1. + +Unless the operator is `json_path_absent`, a missing selector fails with +`selector_missing`. Unless the operator checks presence/absence, a wrong JSON +type fails with `selector_type_mismatch`. Both are required-assertion failures, +not harness failures. + +Objects compare by JCS bytes. Arrays are order-sensitive. Strings compare +without trimming or Unicode normalization. Numbers use JCS representation. +`null`, missing, empty string, empty array, and empty object are distinct. + +## 4. Fixture roles and execution + +Closed V1 fixture roles are: + +- `client_request`: inject exact bytes at the named inbound protocol surface; +- `upstream_response`: return exact bytes from the loopback mock; +- `adapter_vector`: decode the fixture JSON and feed its documented fields to + the selected adapter boundary without network access; +- `synthetic_tool`: decode the fixture JSON into the in-memory inert tool/MCP + stub and execute only the closed harness action selected below. It never + executes model arguments. + +Closed media types are `application/json`, `text/event-stream`, +`application/vnd.opencodex.adapter-vector+json`, and +`application/vnd.opencodex.mcp-stub+json`. + +Each case's exact `requirements` selects the adapter/surface and harness +features. `adapter_vector` keys are scenario-specific closed input fields +defined by the literal vector and scenario assertions; unknown keys reject the +fixture. CL-01 must encode those fields as a discriminated union keyed by the +scenario ID, not a generic callback or dynamic module. + +The four protocol V1 MCP cases each carry exactly one additional closed +`requiredHarnessFeatures` action token. These tokens are executable semantics +and participate in the scenario manifest digest: + +- `mcp_namespace_round_trip_v1`: decode `namespace`, `name`, `description`, and + `inputSchema`; register exactly one inert tool; serialize its upstream name + as `namespace + "__" + name`; then synthesize exactly one completed + client-visible function call with ID `call_fixture`, that flattened name, and + arguments `{}`. Run the normal `toolCalls` -> `mcpCalls` projection. No stub + result, process, filesystem, or network action occurs. +- `mcp_schema_bounds_v1`: decode only `limitBytes`, `exactSchema`, and + `overSchema`; stage an otherwise-identical single inert tool first with + `exactSchema` and then in a fresh transaction with `overSchema`. The first + transaction commits only when its UTF-8 schema bytes equal the limit; the + second must reject atomically before commit when it is exactly one byte over. + It emits no tool call and performs no invocation. +- `mcp_call_result_v1`: decode `namespace`, `name`, `arguments`, and `result`; + register exactly one inert tool; synthesize exactly one completed call with + ID `call_fixture`; invoke the in-memory stub exactly once with the decoded + namespace/name/arguments; the stub returns the literal decoded `result`, + which becomes `/client/response/json`. Duplicate/missing calls or any extra + invocation fail the required verifier. +- `mcp_resource_round_trip_v1`: decode only `resources` and `read`; install them + in the in-memory resource stub; perform exactly one list operation followed + by exactly one read for `read.uri`; expose the literal list as + `/client/response/json/resources` and the matching literal contents as + `/client/response/json/contents`. Missing, duplicate, or extra operations + fail the required assertion. + +A protocol V1 MCP case missing its scenario-specific action token, carrying the +wrong token, carrying more than one of these tokens, or providing fixture keys +outside that token's closed schema rejects registration. No generic +`synthetic_tool` callback, implicit model output, or implementation-defined +invocation is permitted. + +The MCP cases use only `in_memory_mcp_stub`. No case authorizes stdio, a child +process, user MCP configuration, filesystem access, or a user tool. + +## 5. SSE normalization + +This section is the Lab harness normalizer for protocol V1 observations. It +does not rewrite production sidecar parsers such as +`src/vision/anthropic-describe.ts`; those remain out of CL-00/CL-01 scope and +may keep closed product-local semantics until a later shared-normalizer phase. + +The harness retains exact fixture bytes and normalizes only for assertions: + +1. UTF-8 must decode without replacement. A BOM is allowed only at byte zero + and is removed. +2. CRLF and CR become LF. +3. An empty line terminates a frame. Comment lines beginning `:` are ignored. +4. The first `:` separates field and value. No colon means an empty value. + Exactly one optional leading U+0020 after `:` is removed; no other + whitespace is trimmed. +5. Repeated `data` fields join with LF. The last `event` field wins. +6. Sentinel interpretation follows the protocol of the byte stream being + normalized, not the client-facing `requirements.surfaces` label. For an + `upstream_response` fixture, the source is the case's single resolved + `requirements.upstreamProtocols` entry: only `openai-chat` recognizes a + data value exactly equal to `[DONE]` as a sentinel. `openai-responses` and + `anthropic-messages` do not. A raw fixture with zero or multiple resolved + source protocols is invalid. Client-output normalization instead follows + the emitted client protocol; a Responses bridge's transport `[DONE]` + padding is not reclassified as an upstream Chat sentinel. +7. When `event` is absent and parsed `data` is an object with string `type`, + Responses/Anthropic normalization infers that `type`. Explicit event wins. + Parsed `null`, scalar, array, or empty data is padding and emits no event. + Syntactically malformed nonempty JSON is terminal. +8. Arrival order is preserved; events are never sorted or deduplicated. + +Each normalized event is: + +```text +{ "event": string, "data": JSON value, "ordinal": integer } +``` + +### Semantic call projections + +`/client/response/events` always contains the event records above and is never +treated as an array of bare calls. The normalized observation additionally +contains `/client/response/toolCalls` and `/client/response/mcpCalls`. + +`toolCalls` is built from completed client-visible semantic output items in +response order. For non-streaming Responses, use `output[]`; for Responses SSE, +use each `response.output_item.done.data.item` and reject a completed response +whose added item lacks exactly one done item. Chat fragments are first +translated by the adapter into those client-visible Responses items; the +projection never reads the upstream Chat deltas directly. + +Accepted item shapes and exact projections are: + +```text +function_call: + { id: item.call_id, name: item.name, + arguments: JSON.parse(item.arguments), + kind: "function", ordinal: output order } +custom_tool_call: + { id: item.call_id, name: item.name, + arguments: item.input, + kind: "custom", ordinal: output order } +``` + +Missing/non-string IDs or names, malformed function JSON, duplicate IDs, an +unknown item type, or an added/done mismatch emits no repaired call and causes +the corresponding required assertion to fail. Ordinals are contiguous from +zero; they are not SSE event ordinals. + +`mcpCalls` is derived from `toolCalls` whose name starts with `mcp__` and +contains a final `__` separator. Split at the final separator: + +```text +{ namespace: bytes before final "__", name: bytes after final "__" } +``` + +Empty components, more than 64 UTF-8 bytes per component, invalid UTF-8, or a +non-MCP name emits no MCP call. Order matches `toolCalls`. + +## 6. Assertion operators + +- `http_status_equals`: selected integer equals expected integer. +- `header_present` / `header_absent`: selected lowercase header key exists/does + not exist. +- `header_value_equals`: selected normalized header string equals expected. +- `json_schema_matches`: selected value validates against embedded JSON Schema + draft 2020-12. Only local `$defs`/`$ref` are allowed; coercion, defaults, + custom formats, and network resolution are forbidden. +- `json_path_equals`: selected value equals literal expected under JCS rules. +- `json_path_present` / `json_path_absent`: pointer succeeds/fails; a present + `null` is present. Expected must be literal `true`. +- `sse_field_equals`: selected normalized field equals expected string. +- `sse_event_sequence`: exact event-name array; no subsequence or extras. +- `sse_event_count`: expected is `{event,count}` and exact count is required. +- `terminal_signal_equals`: expected is `completed`, `failed`, `incomplete`, + `done`, `message_stop`, `eof_tolerated`, or `none`. Exactly one terminal is + required unless expected is `none`. +- `id_matches`: expected is a closed grammar: + - `responses_message`: `msg_` plus 1..128 ASCII alphanumeric/underscore/dash; + - `responses_reasoning`: `rs_` plus 1..128 of that set; + - `responses_call`: `call_` plus 1..128 of that set; + - `nonempty_128`: 1..128 printable non-whitespace ASCII characters. + Arbitrary regular expressions are forbidden. +- `id_stable_across_events`: expected is an ordered pointer list. Every + resolved string is byte-equal. +- `id_correlates`: expected is exactly two pointers resolving to byte-equal + strings. +- `tool_call_equals`: selected normalized call equals + `{id,name,arguments,kind,ordinal}`. Function arguments are parsed JSON; + custom/freeform arguments are exact strings. +- `tool_result_correlates`: expected is `{call,result}` pointers. IDs match, + result follows call, and no intervening call reuses the ID. +- `fixture_request_matches`: method, normalized path, allowlisted headers and + JSON body equal the literal fixture expectation. +- `normalized_text_equals`: selected string equals expected exactly. +- `byte_limit_observed`: selected nonnegative integer is `<=` expected. +- `process_exit_equals`: selected integer equals expected. +- `verifier_result_equals`: selected value `pass|fail|blocked|inconclusive` + equals expected. + +Unknown operators, selectors, expected shapes, fixture roles, media types, or +requirements reject registration. + +## 7. Closed verifier derivations + +`/verifiers` is populated only by these pure V1 functions. They may read the +current case's decoded synthetic fixture and normalized observation, but no +clock, random source, network, filesystem, environment, runtime callback, or +model output outside that observation. + +- `json_sse_equivalence`: build the JSON projection + `{text,terminal}` where `text` concatenates, in order, every + `output[].content[]` `output_text.text`, and `terminal` is top-level + `status`. Build the SSE projection where `text` concatenates every + `response.output_text.delta.data.delta`, and `terminal` is the normalized + terminal. Return `pass` iff the two JCS objects are equal, else `fail`. +- `nonoverlap_order`: read `/client/response/toolCalls` in array order. Return + their IDs only when every ID occurs once, each record has a complete + arguments value, and ordinals are contiguous from zero and equal the array + indexes. Otherwise return an empty array. +- `call_result_order`: over the normalized two-turn input, return `pass` iff a + `function_call` occurs in turn 1, exactly one `function_call_output` with the + same `call_id` occurs in turn 2, and no result precedes its call; otherwise + `fail`. +- `compaction_replayed`: return `true` iff the one + `context_compaction.encrypted_content` value is accepted into the parser's + normalized compaction slot and is absent from user-visible output; otherwise + `false`. The synthetic value is never decrypted or executed. +- `local_shell_correlated`: return `true` iff the `local_shell_call.call_id` + equals the following `function_call_output.call_id` and neither item invokes + a process; otherwise `false`. +- `tool_search_error`: for the one failed `tool_search_output`, return its exact + `error` string; missing, duplicate, or non-failed items return `null`. +- `modality_path`: return `native` when `requestHasImage` is true and + `modelInputModalities` contains `image`; otherwise return `sidecar` when an + enabled authorized vision sidecar exists; otherwise return `unsupported`. +- `silent_image_drop`: return `true` only when an image-bearing input is + omitted from adapter output without either a `native`/`sidecar` path or the + typed unsupported rejection; otherwise `false`. +- `exact_bound`: UTF-8 encode `exactSchema`; return `pass` iff its byte length + equals `limitBytes`, JSON parsing succeeds, and the complete staged catalogue + commits; otherwise `fail`. +- `one_over_rejected`: UTF-8 encode `overSchema`; return `pass` iff its byte + length equals `limitBytes + 1` and admission rejects it before commit; + otherwise `fail`. +- `partial_commit`: return `true` iff any tool from the rejected one-byte-over + staging transaction is visible in the committed catalogue; otherwise + `false`. +- `stub_received`: the in-memory MCP stub records exactly + `{namespace,name,arguments}` from the one decoded invocation. Duplicate or + missing invocations produce `null`. + +Verifier outputs use only the literal types above. A missing or type-invalid +input returns `fail`, `false`, `[]`, or `null` as specified and therefore fails +the corresponding required assertion; it is not silently repaired. + +## 8. Failure rules and freshness + +The exact ordered base `protocol-v1-default` records and the one +expected-failure rule template are in the case authority. Control-specific +rules are included only by `expandFailureRules(case)`. Fixture/manifest +integrity and harness failures do not affect compatibility. Time/resource +limits are environmental blockers. The exact expected rejection of the V1 +conformance negative control satisfies that control without producing +`UNSUPPORTED`; no protocol V1 case can produce `UNSUPPORTED`. Other required +deterministic mismatches are `protocol_failure`/`DEGRADED`. + +Protocol V1 scenario and suite freshness are both unbounded (`null`) because +their exact scenario, suite, fixture, compatibility-version, adapter, and +behavior digests invalidate behavior changes. A future profile may still set a +stricter maximum age. + +## 9. CL-01 boundary + +CL-01 may materialize and execute only the protocol manifests in the case +authority. It must: + +1. parse the authority as JSON and reject unknown fields; +2. recompute all fixture, scenario, and suite digests, including the mandatory + synthetic fixture marker/provenance fields; +3. retain the exact case fixture bytes and expanded manifests + content-addressably; +4. execute only loopback mocks, closed adapter vectors, and the four exact + in-memory MCP action tokens above; +5. fail registration rather than invent semantics. + +This document and the JSON authority authorize no runner, mock server, ledger, +SQLite projection, fixture extraction from tests, live probe, or Fabric +ingestion implementation in CL-00. diff --git a/devlog/_plan/260807_compatibility_lab/022_protocol_v1_cases.json b/devlog/_plan/260807_compatibility_lab/022_protocol_v1_cases.json new file mode 100644 index 000000000..491aee981 --- /dev/null +++ b/devlog/_plan/260807_compatibility_lab/022_protocol_v1_cases.json @@ -0,0 +1,461 @@ +{ + "schemaVersion": 1, + "authority": "CL-00 design contract; not a runtime registry", + "sourceCommit": "3ad5bb6bd3f76f6879d84b78ea39edd3e01ec296", + "assertionDslVersion": "1.0.0", + "evidenceSchemaVersion": "1.0.0", + "failureRuleSets": { + "protocol-v1-default": [ + { "id": "contract-integrity", "match": ["fixture_digest_mismatch", "manifest_digest_mismatch", "fixture_decode_failure", "harness_failure", "sanitizer_failure"], "classification": "harness_failure", "secondaryCode": "contract_integrity", "verdictEffect": "none", "retry": "never", "expected": false }, + { "id": "time-limit", "match": ["connect_timeout", "first_byte_timeout", "inactivity_timeout", "total_timeout"], "classification": "timeout", "secondaryCode": "scenario_time_limit", "verdictEffect": "none", "retry": "never", "expected": false }, + { "id": "resource-limit", "match": ["request_limit", "input_byte_limit", "output_byte_limit", "output_token_limit", "tool_call_limit", "artifact_byte_limit"], "classification": "budget_exhausted", "secondaryCode": "scenario_resource_limit", "verdictEffect": "none", "retry": "never", "expected": false }, + { "id": "required-assertion", "match": ["required_assertion_failed"], "classification": "protocol_failure", "secondaryCode": "deterministic_assertion", "verdictEffect": "degraded", "retry": "never", "expected": false }, + { "id": "fallback", "match": ["no_prior_rule"], "classification": "inconclusive", "secondaryCode": "unclassified", "verdictEffect": "none", "retry": "never", "expected": false } + ] + }, + "expectedFailureRuleTemplate": { + "id": "expected-failure-exact-match", + "match": ["expected_failure_exact_match"], + "retry": "never", + "expected": true + }, + "manifestDefaults": { + "version": "1.0.0", + "suiteVersion": "1.0.0", + "evidenceLayer": "protocol_conformance", + "verificationRole": "required", + "executionMode": "fixture", + "freshness": { "maxAgeMs": null }, + "executionLimits": { + "totalTimeoutMs": 10000, + "connectTimeoutMs": 1000, + "firstByteTimeoutMs": 2000, + "inactivityTimeoutMs": 2000, + "maxRequests": 4, + "maxInputBytes": 1048576, + "maxOutputBytes": 4194304, + "maxOutputTokens": 4096, + "maxToolCalls": 8, + "maxArtifactBytes": 262144 + }, + "artifactPolicy": { + "allowed": ["assertion_report", "sanitized_request_shape", "sanitized_response_shape", "normalized_event_trace", "sanitized_error"], + "perArtifactBytes": 262144, + "aggregateBytes": 1048576, + "retention": "local_contract", + "publicVisibility": "deny", + "redactionProfile": "synthetic_protocol_v1" + }, + "failureRuleSet": "protocol-v1-default" + }, + "cases": [ + { + "id": "responses-core.protocol.request-shape", + "suite": "responses-core", + "capability": "protocol.responses.core", + "requirements": { "inboundProtocols": ["openai-responses"], "upstreamProtocols": ["openai-chat"], "surfaces": ["responses-http"], "requiredClaims": [], "requiredHarnessFeatures": ["adapter_vector"], "platforms": [], "routePreconditions": [] }, + "fixture": { "id": "rsp-request-shape", "role": "adapter_vector", "mediaType": "application/vnd.opencodex.adapter-vector+json", "bytesUtf8": "{\"modelId\":\"fixture-model\",\"context\":{\"messages\":[{\"role\":\"user\",\"content\":\"PING\",\"timestamp\":0}]},\"stream\":false,\"options\":{\"temperature\":0}}", "digest": "ccc7549e8bcfe4e28d0d4a87c14e622ecfb75973600b5eef830d83620c5bd0f8" }, + "assertions": [ + { "id": "method", "operator": "json_path_equals", "selector": "/upstream/requests/0/json/model", "expected": "fixture-model", "required": true }, + { "id": "message", "operator": "json_path_equals", "selector": "/upstream/requests/0/json/messages/0/content", "expected": "PING", "required": true }, + { "id": "temperature", "operator": "json_path_equals", "selector": "/upstream/requests/0/json/temperature", "expected": 0, "required": true } + ] + }, + { + "id": "responses-core.protocol.sse-framing", + "suite": "responses-core", + "capability": "protocol.responses.core", + "requirements": { "inboundProtocols": ["openai-responses"], "upstreamProtocols": ["openai-responses"], "surfaces": ["responses-sse"], "requiredClaims": [], "requiredHarnessFeatures": ["raw_sse_capture"], "platforms": [], "routePreconditions": [] }, + "initiatingRequest": { "id": "rsp-sse-framing-request", "role": "client_request", "mediaType": "application/json", "bytesUtf8": "{\"model\":\"fixture-model\",\"input\":\"PING\",\"stream\":true}", "digest": "c2c39a78b939e6c5182d3206f86aa86d6fd706959de9c37072db759aa510a1f6" }, + "fixture": { "id": "rsp-sse-framing", "role": "upstream_response", "mediaType": "text/event-stream", "bytesUtf8": "data:{\"type\":\"response.output_text.delta\",\"delta\":\"A\"}\n\ndata: null\n\ndata: {\"type\":\"response.output_text.delta\",\"delta\":\"B\"}\n\ndata:{\"type\":\"response.completed\",\"response\":{\"id\":\"resp_fixture\",\"status\":\"completed\"}}\n\n", "digest": "1c384ef32886054d8f15c14cbcbcc9af4a3bed845d6f820691368614d61515e7" }, + "assertions": [ + { "id": "events", "operator": "sse_event_sequence", "selector": "/client/response/events", "expected": ["response.output_text.delta", "response.output_text.delta", "response.completed"], "required": true }, + { "id": "text", "operator": "normalized_text_equals", "selector": "/client/response/normalizedText", "expected": "AB", "required": true }, + { "id": "terminal", "operator": "terminal_signal_equals", "selector": "/client/response/terminal", "expected": "completed", "required": true } + ] + }, + { + "id": "responses-core.protocol.item-lifecycle", + "suite": "responses-core", + "capability": "protocol.responses.core", + "requirements": { "inboundProtocols": ["openai-responses"], "upstreamProtocols": ["openai-responses"], "surfaces": ["responses-sse"], "requiredClaims": [], "requiredHarnessFeatures": ["raw_sse_capture"], "platforms": [], "routePreconditions": [] }, + "initiatingRequest": { "id": "rsp-item-lifecycle-request", "role": "client_request", "mediaType": "application/json", "bytesUtf8": "{\"model\":\"fixture-model\",\"input\":\"PING\",\"stream\":true}", "digest": "c2c39a78b939e6c5182d3206f86aa86d6fd706959de9c37072db759aa510a1f6" }, + "fixture": { "id": "rsp-item-lifecycle", "role": "upstream_response", "mediaType": "text/event-stream", "bytesUtf8": "event: response.output_item.added\ndata: {\"type\":\"response.output_item.added\",\"output_index\":0,\"item\":{\"id\":\"msg_fixture\",\"type\":\"message\",\"status\":\"in_progress\",\"role\":\"assistant\",\"content\":[]}}\n\nevent: response.output_item.done\ndata: {\"type\":\"response.output_item.done\",\"output_index\":0,\"item\":{\"id\":\"msg_fixture\",\"type\":\"message\",\"status\":\"completed\",\"role\":\"assistant\",\"content\":[]}}\n\nevent: response.completed\ndata: {\"type\":\"response.completed\",\"response\":{\"id\":\"resp_fixture\",\"status\":\"completed\",\"output\":[]}}\n\n", "digest": "ef271e8aaa1d63d51d4e7e0d47facadf39603c1ffa2e871865ace3684feead08" }, + "assertions": [ + { "id": "events", "operator": "sse_event_sequence", "selector": "/client/response/events", "expected": ["response.output_item.added", "response.output_item.done", "response.completed"], "required": true }, + { "id": "stable-id", "operator": "id_stable_across_events", "selector": "/client/response/events", "expected": ["/client/response/events/0/data/item/id", "/client/response/events/1/data/item/id"], "required": true }, + { "id": "id-shape", "operator": "id_matches", "selector": "/client/response/events/0/data/item/id", "expected": "responses_message", "required": true } + ] + }, + { + "id": "responses-core.protocol.terminal-state", + "suite": "responses-core", + "capability": "protocol.responses.core", + "requirements": { "inboundProtocols": ["openai-responses"], "upstreamProtocols": ["openai-responses"], "surfaces": ["responses-sse"], "requiredClaims": [], "requiredHarnessFeatures": ["raw_sse_capture"], "platforms": [], "routePreconditions": [] }, + "initiatingRequest": { "id": "rsp-terminal-state-request", "role": "client_request", "mediaType": "application/json", "bytesUtf8": "{\"model\":\"fixture-model\",\"input\":\"PING\",\"stream\":true}", "digest": "c2c39a78b939e6c5182d3206f86aa86d6fd706959de9c37072db759aa510a1f6" }, + "fixture": { "id": "rsp-terminal-state", "role": "upstream_response", "mediaType": "text/event-stream", "bytesUtf8": "event: response.failed\ndata: {\"type\":\"response.failed\",\"response\":{\"id\":\"resp_fixture\",\"status\":\"failed\",\"error\":{\"type\":\"server_error\",\"code\":\"fixture_failure\"}}}\n\n", "digest": "472735364ce0ee28e68192d478ccb658ec8d6a149dba6fe914e5ab35cc1a41d7" }, + "assertions": [ + { "id": "events", "operator": "sse_event_sequence", "selector": "/client/response/events", "expected": ["response.failed"], "required": true }, + { "id": "count", "operator": "sse_event_count", "selector": "/client/response/events", "expected": { "event": "response.failed", "count": 1 }, "required": true }, + { "id": "terminal", "operator": "terminal_signal_equals", "selector": "/client/response/terminal", "expected": "failed", "required": true } + ] + }, + { + "id": "responses-core.protocol.json-sse-equivalence", + "suite": "responses-core", + "capability": "protocol.responses.core", + "requirements": { "inboundProtocols": ["openai-responses"], "upstreamProtocols": ["openai-responses"], "surfaces": ["responses-http", "responses-sse"], "requiredClaims": [], "requiredHarnessFeatures": ["adapter_vector", "raw_sse_capture"], "platforms": [], "routePreconditions": [] }, + "fixture": { "id": "rsp-json-sse-equivalence", "role": "adapter_vector", "mediaType": "application/vnd.opencodex.adapter-vector+json", "bytesUtf8": "{\"json\":{\"id\":\"resp_fixture\",\"status\":\"completed\",\"output\":[{\"id\":\"msg_fixture\",\"type\":\"message\",\"role\":\"assistant\",\"status\":\"completed\",\"content\":[{\"type\":\"output_text\",\"text\":\"OK\"}]}]},\"sse\":\"event: response.output_text.delta\\ndata: {\\\"type\\\":\\\"response.output_text.delta\\\",\\\"delta\\\":\\\"OK\\\"}\\n\\nevent: response.completed\\ndata: {\\\"type\\\":\\\"response.completed\\\",\\\"response\\\":{\\\"id\\\":\\\"resp_fixture\\\",\\\"status\\\":\\\"completed\\\"}}\\n\\n\"}", "digest": "b7288170258b91361530d1dd5a0a818859ff9b6176793554ec8b0ae1177d87cf" }, + "assertions": [ + { "id": "text", "operator": "normalized_text_equals", "selector": "/client/response/normalizedText", "expected": "OK", "required": true }, + { "id": "terminal", "operator": "terminal_signal_equals", "selector": "/client/response/terminal", "expected": "completed", "required": true }, + { "id": "equivalent", "operator": "verifier_result_equals", "selector": "/verifiers/json_sse_equivalence", "expected": "pass", "required": true } + ] + }, + { + "id": "chat-core.protocol.request-mapping", + "suite": "chat-core", + "capability": "protocol.chat.core", + "requirements": { "inboundProtocols": ["openai-responses"], "upstreamProtocols": ["openai-chat"], "surfaces": ["responses-http"], "requiredClaims": [], "requiredHarnessFeatures": ["adapter_vector"], "platforms": [], "routePreconditions": [] }, + "fixture": { "id": "chat-request-mapping", "role": "adapter_vector", "mediaType": "application/vnd.opencodex.adapter-vector+json", "bytesUtf8": "{\"context\":{\"systemPrompt\":[\"SYS\"],\"messages\":[{\"role\":\"developer\",\"content\":\"DEV\",\"timestamp\":0},{\"role\":\"user\",\"content\":\"PING\",\"timestamp\":1}]},\"options\":{\"textFormat\":{\"type\":\"json_object\"}}}", "digest": "0a9c319b3a6dadbf581d0d2185f57527cd28aa57127e0eac91a421735b4c2ad9" }, + "assertions": [ + { "id": "roles", "operator": "json_path_equals", "selector": "/upstream/requests/0/json/messages", "expected": [{"role":"system","content":"SYS"},{"role":"developer","content":"DEV"},{"role":"user","content":"PING"}], "required": true }, + { "id": "format", "operator": "json_path_equals", "selector": "/upstream/requests/0/json/response_format", "expected": {"type":"json_object"}, "required": true } + ] + }, + { + "id": "chat-core.protocol.nonstream-envelope", + "suite": "chat-core", + "capability": "protocol.chat.core", + "requirements": { "inboundProtocols": ["openai-responses"], "upstreamProtocols": ["openai-chat"], "surfaces": ["responses-http"], "requiredClaims": [], "requiredHarnessFeatures": ["adapter_vector"], "platforms": [], "routePreconditions": [] }, + "initiatingRequest": { "id": "chat-nonstream-envelope-request", "role": "client_request", "mediaType": "application/json", "bytesUtf8": "{\"model\":\"fixture-model\",\"input\":\"PING\",\"stream\":false}", "digest": "4f6e495840e4fc80f833aa8cc09c09ee765ae9f8134db70565f3445989892db7" }, + "fixture": { "id": "chat-nonstream-envelope", "role": "upstream_response", "mediaType": "application/json", "bytesUtf8": "{\"id\":\"chatcmpl_fixture\",\"choices\":[{\"index\":0,\"message\":{\"role\":\"assistant\",\"content\":\"OK\"},\"finish_reason\":\"stop\"}],\"usage\":{\"prompt_tokens\":1,\"completion_tokens\":1,\"total_tokens\":2}}", "digest": "4a9a0352daa284e73850ce613b1cc939a534d6a930c7b68c8eb28e3fcca5b248" }, + "assertions": [ + { "id": "status", "operator": "http_status_equals", "selector": "/client/response/status", "expected": 200, "required": true }, + { "id": "text", "operator": "normalized_text_equals", "selector": "/client/response/normalizedText", "expected": "OK", "required": true }, + { "id": "terminal", "operator": "terminal_signal_equals", "selector": "/client/response/terminal", "expected": "completed", "required": true } + ] + }, + { + "id": "chat-core.protocol.stream-assembly", + "suite": "chat-core", + "capability": "protocol.chat.core", + "requirements": { "inboundProtocols": ["openai-responses"], "upstreamProtocols": ["openai-chat"], "surfaces": ["responses-sse"], "requiredClaims": [], "requiredHarnessFeatures": ["raw_sse_capture"], "platforms": [], "routePreconditions": [] }, + "initiatingRequest": { "id": "chat-stream-assembly-request", "role": "client_request", "mediaType": "application/json", "bytesUtf8": "{\"model\":\"fixture-model\",\"input\":\"PING\",\"stream\":true}", "digest": "c2c39a78b939e6c5182d3206f86aa86d6fd706959de9c37072db759aa510a1f6" }, + "fixture": { "id": "chat-stream-assembly", "role": "upstream_response", "mediaType": "text/event-stream", "bytesUtf8": "data: {\"choices\":[{\"index\":0,\"delta\":{\"tool_calls\":[{\"index\":0,\"id\":\"call_a\",\"function\":{\"name\":\"alpha\",\"arguments\":\"{\\\"x\\\":\"}},{\"index\":1,\"id\":\"call_b\",\"function\":{\"name\":\"beta\",\"arguments\":\"{\\\"y\\\":\"}}]}}]}\n\ndata: {\"choices\":[{\"index\":0,\"delta\":{\"tool_calls\":[{\"index\":1,\"function\":{\"arguments\":\"2}\"}},{\"index\":0,\"function\":{\"arguments\":\"1}\"}}]},\"finish_reason\":\"tool_calls\"}]}\n\ndata: [DONE]\n\n", "digest": "0085f298a8690aefb74bb09ea2e0cb77703aaf6cfd822c6ce0d4d334ad4b9b3f" }, + "assertions": [ + { "id": "alpha", "operator": "tool_call_equals", "selector": "/client/response/toolCalls/0", "expected": {"id":"call_a","name":"alpha","arguments":{"x":1},"kind":"function","ordinal":0}, "required": true }, + { "id": "beta", "operator": "tool_call_equals", "selector": "/client/response/toolCalls/1", "expected": {"id":"call_b","name":"beta","arguments":{"y":2},"kind":"function","ordinal":1}, "required": true }, + { "id": "terminal", "operator": "terminal_signal_equals", "selector": "/client/response/terminal", "expected": "completed", "required": true } + ] + }, + { + "id": "chat-core.protocol.stream-terminal", + "suite": "chat-core", + "capability": "protocol.chat.core", + "requirements": { "inboundProtocols": ["openai-responses"], "upstreamProtocols": ["openai-chat"], "surfaces": ["responses-sse"], "requiredClaims": [], "requiredHarnessFeatures": ["raw_sse_capture"], "platforms": [], "routePreconditions": [] }, + "initiatingRequest": { "id": "chat-stream-terminal-request", "role": "client_request", "mediaType": "application/json", "bytesUtf8": "{\"model\":\"fixture-model\",\"input\":\"PING\",\"stream\":true}", "digest": "c2c39a78b939e6c5182d3206f86aa86d6fd706959de9c37072db759aa510a1f6" }, + "fixture": { "id": "chat-stream-terminal", "role": "upstream_response", "mediaType": "text/event-stream", "bytesUtf8": "data: {\"choices\":[{\"index\":0,\"delta\":{\"content\":\"OK\"},\"finish_reason\":\"stop\"}]}\n\ndata: [DONE]\n\n", "digest": "6e0e4e8d32d8575db6a09e89c222b16338e1499e940e038599f7a6b5332e59e6" }, + "assertions": [ + { "id": "text", "operator": "normalized_text_equals", "selector": "/client/response/normalizedText", "expected": "OK", "required": true }, + { "id": "terminal", "operator": "terminal_signal_equals", "selector": "/client/response/terminal", "expected": "completed", "required": true } + ] + }, + { + "id": "anthropic-core.protocol.request-mapping", + "suite": "anthropic-core", + "capability": "protocol.anthropic.messages.core", + "requirements": { "inboundProtocols": ["anthropic-messages"], "upstreamProtocols": ["openai-responses"], "surfaces": ["anthropic-http"], "requiredClaims": [], "requiredHarnessFeatures": ["adapter_vector"], "platforms": [], "routePreconditions": [] }, + "fixture": { "id": "anthropic-request-mapping", "role": "client_request", "mediaType": "application/json", "bytesUtf8": "{\"model\":\"fixture-model\",\"system\":\"SYS\",\"messages\":[{\"role\":\"user\",\"content\":\"PING\"}],\"max_tokens\":32,\"stream\":false}", "digest": "deeca799f660f413d0cb85263aa332bdc05aa995ccf8c7322f6af43e9bf6a627" }, + "assertions": [ + { "id": "model", "operator": "json_path_equals", "selector": "/upstream/requests/0/json/model", "expected": "fixture-model", "required": true }, + { "id": "system", "operator": "json_path_equals", "selector": "/upstream/requests/0/json/instructions", "expected": "SYS", "required": true }, + { "id": "input", "operator": "json_path_equals", "selector": "/upstream/requests/0/json/input/0/content/0/text", "expected": "PING", "required": true } + ] + }, + { + "id": "anthropic-core.protocol.content-sequence", + "suite": "anthropic-core", + "capability": "protocol.anthropic.messages.core", + "requirements": { "inboundProtocols": ["anthropic-messages"], "upstreamProtocols": ["openai-responses"], "surfaces": ["anthropic-sse"], "requiredClaims": [], "requiredHarnessFeatures": ["raw_sse_capture"], "platforms": [], "routePreconditions": [] }, + "initiatingRequest": { "id": "anthropic-content-sequence-request", "role": "client_request", "mediaType": "application/json", "bytesUtf8": "{\"model\":\"fixture-model\",\"messages\":[{\"role\":\"user\",\"content\":\"PING\"}],\"max_tokens\":32,\"stream\":true}", "digest": "96e15d2044ccaacca81d32bda4157e4baf82ef98c7640f27034e10285f5de8f3" }, + "fixture": { "id": "anthropic-content-sequence", "role": "upstream_response", "mediaType": "text/event-stream", "bytesUtf8": "data:{\"type\":\"response.output_text.delta\",\"delta\":\"OK\"}\n\ndata:{\"type\":\"response.completed\",\"response\":{\"id\":\"resp_fixture\",\"status\":\"completed\",\"usage\":{\"input_tokens\":1,\"output_tokens\":1}}}\n\n", "digest": "1f8148d142038f42fadf4b3e938b45f4313986cbbd6338feac3b8db8f355299a" }, + "assertions": [ + { "id": "events", "operator": "sse_event_sequence", "selector": "/client/response/events", "expected": ["message_start","content_block_start","content_block_delta","content_block_stop","message_delta","message_stop"], "required": true }, + { "id": "terminal", "operator": "terminal_signal_equals", "selector": "/client/response/terminal", "expected": "message_stop", "required": true } + ] + }, + { + "id": "anthropic-core.protocol.tool-round-trip", + "suite": "anthropic-core", + "capability": "protocol.anthropic.messages.core", + "requirements": { "inboundProtocols": ["anthropic-messages"], "upstreamProtocols": ["openai-responses"], "surfaces": ["anthropic-http"], "requiredClaims": ["tools"], "requiredHarnessFeatures": ["adapter_vector"], "platforms": [], "routePreconditions": [] }, + "fixture": { "id": "anthropic-tool-roundtrip", "role": "client_request", "mediaType": "application/json", "bytesUtf8": "{\"model\":\"fixture-model\",\"messages\":[{\"role\":\"assistant\",\"content\":[{\"type\":\"tool_use\",\"id\":\"call_fixture\",\"name\":\"lookup\",\"input\":{\"q\":\"x\"}}]},{\"role\":\"user\",\"content\":[{\"type\":\"tool_result\",\"tool_use_id\":\"call_fixture\",\"content\":\"RESULT\"}]}],\"tools\":[{\"name\":\"lookup\",\"input_schema\":{\"type\":\"object\",\"properties\":{\"q\":{\"type\":\"string\"}},\"required\":[\"q\"]}}],\"max_tokens\":32}", "digest": "f8dfefb427ce81fb4570f83d8d24c91e79350a7a256ecde7e636e0d70fbdff64" }, + "assertions": [ + { "id": "call-id", "operator": "id_correlates", "selector": "/upstream/requests", "expected": ["/upstream/requests/0/json/input/0/call_id","/upstream/requests/0/json/input/1/call_id"], "required": true }, + { "id": "result", "operator": "json_path_equals", "selector": "/upstream/requests/0/json/input/1/output", "expected": "RESULT", "required": true } + ] + }, + { + "id": "anthropic-core.protocol.terminal-errors", + "suite": "anthropic-core", + "capability": "protocol.anthropic.messages.core", + "requirements": { "inboundProtocols": ["anthropic-messages"], "upstreamProtocols": ["openai-responses"], "surfaces": ["anthropic-sse"], "requiredClaims": [], "requiredHarnessFeatures": ["raw_sse_capture"], "platforms": [], "routePreconditions": [] }, + "initiatingRequest": { "id": "anthropic-terminal-error-request", "role": "client_request", "mediaType": "application/json", "bytesUtf8": "{\"model\":\"fixture-model\",\"messages\":[{\"role\":\"user\",\"content\":\"PING\"}],\"max_tokens\":32,\"stream\":true}", "digest": "96e15d2044ccaacca81d32bda4157e4baf82ef98c7640f27034e10285f5de8f3" }, + "fixture": { "id": "anthropic-terminal-error", "role": "upstream_response", "mediaType": "text/event-stream", "bytesUtf8": "event: response.failed\ndata: {\"type\":\"response.failed\",\"response\":{\"status\":\"failed\",\"error\":{\"type\":\"server_error\",\"code\":\"overloaded\",\"message\":\"fixture\"}}}\n\n", "digest": "fad0d0edca35d066e89de5488635a2912930d5dedc79d047759fb6ecc6567718" }, + "assertions": [ + { "id": "events", "operator": "sse_event_sequence", "selector": "/client/response/events", "expected": ["error"], "required": true }, + { "id": "terminal", "operator": "terminal_signal_equals", "selector": "/client/response/terminal", "expected": "failed", "required": true } + ] + }, + { + "id": "tools-core.protocol.function-round-trip", + "suite": "tools-core", + "capability": "tools.round_trip", + "requirements": { "inboundProtocols": ["openai-responses"], "upstreamProtocols": ["openai-chat"], "surfaces": ["responses-http"], "requiredClaims": ["tools"], "requiredHarnessFeatures": ["adapter_vector"], "platforms": [], "routePreconditions": [] }, + "fixture": { "id": "tools-function", "role": "adapter_vector", "mediaType": "application/vnd.opencodex.adapter-vector+json", "bytesUtf8": "{\"tools\":[{\"name\":\"lookup\",\"parameters\":{\"type\":\"object\",\"properties\":{\"q\":{\"type\":\"string\"}},\"required\":[\"q\"]}}],\"upstreamToolCall\":{\"id\":\"call_fixture\",\"name\":\"lookup\",\"arguments\":\"{\\\"q\\\":\\\"x\\\"}\"},\"toolResult\":{\"toolCallId\":\"call_fixture\",\"content\":\"RESULT\"}}", "digest": "9107f4dfdd7da8340c866c9fb6f42854437cebb98592d0510969c810c1eeb0ad" }, + "assertions": [ + { "id": "call", "operator": "tool_call_equals", "selector": "/client/response/toolCalls/0", "expected": {"id":"call_fixture","name":"lookup","arguments":{"q":"x"},"kind":"function","ordinal":0}, "required": true }, + { "id": "result", "operator": "tool_result_correlates", "selector": "/upstream/requests", "expected": {"call":"/client/response/toolCalls/0/id","result":"/upstream/requests/1/json/messages/1/tool_call_id"}, "required": true } + ] + }, + { + "id": "tools-core.protocol.custom-freeform-round-trip", + "suite": "tools-core", + "capability": "tools.round_trip", + "requirements": { "inboundProtocols": ["openai-responses"], "upstreamProtocols": ["openai-responses"], "surfaces": ["responses-http"], "requiredClaims": ["custom_tools"], "requiredHarnessFeatures": ["adapter_vector"], "platforms": [], "routePreconditions": [] }, + "fixture": { "id": "tools-custom", "role": "adapter_vector", "mediaType": "application/vnd.opencodex.adapter-vector+json", "bytesUtf8": "{\"tool\":{\"type\":\"custom\",\"name\":\"apply_patch\",\"format\":{\"type\":\"grammar\",\"syntax\":\"lark\",\"definition\":\"start: /[\\\\s\\\\S]+/\"}},\"call\":{\"id\":\"call_patch\",\"name\":\"apply_patch\",\"input\":\"*** Begin Patch\\n*** End Patch\\n\"},\"output\":{\"call_id\":\"call_patch\",\"output\":\"Done\"}}", "digest": "752750104e99602d9160feaa591bcbfcfd0c8c53fc9feda4a48c3b6813b74d44" }, + "assertions": [ + { "id": "call", "operator": "tool_call_equals", "selector": "/client/response/toolCalls/0", "expected": {"id":"call_patch","name":"apply_patch","arguments":"*** Begin Patch\n*** End Patch\n","kind":"custom","ordinal":0}, "required": true }, + { "id": "result", "operator": "tool_result_correlates", "selector": "/upstream/requests", "expected": {"call":"/client/response/toolCalls/0/id","result":"/upstream/requests/1/json/input/0/call_id"}, "required": true } + ] + }, + { + "id": "tools-core.protocol.parallel-correlation", + "suite": "tools-core", + "capability": "tools.round_trip", + "requirements": { "inboundProtocols": ["openai-responses"], "upstreamProtocols": ["openai-chat"], "surfaces": ["responses-sse"], "requiredClaims": ["parallel_tools"], "requiredHarnessFeatures": ["raw_sse_capture"], "platforms": [], "routePreconditions": [] }, + "initiatingRequest": { "id": "tools-parallel-request", "role": "client_request", "mediaType": "application/json", "bytesUtf8": "{\"model\":\"fixture-model\",\"input\":\"PING\",\"stream\":true}", "digest": "c2c39a78b939e6c5182d3206f86aa86d6fd706959de9c37072db759aa510a1f6" }, + "fixture": { "id": "tools-parallel", "role": "upstream_response", "mediaType": "text/event-stream", "bytesUtf8": "data:{\"choices\":[{\"delta\":{\"tool_calls\":[{\"index\":0,\"id\":\"call_a\",\"function\":{\"name\":\"a\",\"arguments\":\"{\"}},{\"index\":1,\"id\":\"call_b\",\"function\":{\"name\":\"b\",\"arguments\":\"{\"}}]}}]}\n\ndata:{\"choices\":[{\"delta\":{\"tool_calls\":[{\"index\":1,\"function\":{\"arguments\":\"}\"}},{\"index\":0,\"function\":{\"arguments\":\"}\"}}]},\"finish_reason\":\"tool_calls\"}]}\n\ndata:[DONE]\n\n", "digest": "7a954d390bdf48d0dec3ed2515a5bbbedd4165656fcb7f0643ce743d17bb39f0" }, + "assertions": [ + { "id": "calls", "operator": "json_path_equals", "selector": "/client/response/toolCalls", "expected": [{"id":"call_a","name":"a","arguments":{},"kind":"function","ordinal":0},{"id":"call_b","name":"b","arguments":{},"kind":"function","ordinal":1}], "required": true }, + { "id": "order", "operator": "json_path_equals", "selector": "/verifiers/nonoverlap_order", "expected": ["call_a","call_b"], "required": true } + ] + }, + { + "id": "tools-core.protocol.result-content", + "suite": "tools-core", + "capability": "tools.round_trip", + "requirements": { "inboundProtocols": ["openai-responses"], "upstreamProtocols": ["openai-chat"], "surfaces": ["responses-http"], "requiredClaims": ["tools","image"], "requiredHarnessFeatures": ["adapter_vector"], "platforms": [], "routePreconditions": [] }, + "fixture": { "id": "tools-result-content", "role": "adapter_vector", "mediaType": "application/vnd.opencodex.adapter-vector+json", "bytesUtf8": "{\"callId\":\"call_fixture\",\"content\":[{\"type\":\"input_text\",\"text\":\"RESULT\"},{\"type\":\"input_image\",\"image_url\":\"data:image/png;base64,iVBORw0KGgo=\",\"detail\":\"high\"}],\"isError\":false}", "digest": "ec81d47d3d6a67254afcc21b55f458269d8dd34ab3b3d52a6c12fec9bec814ab" }, + "assertions": [ + { "id": "text", "operator": "json_path_equals", "selector": "/upstream/requests/0/json/messages/0/content", "expected": "RESULT", "required": true }, + { "id": "image", "operator": "json_path_equals", "selector": "/upstream/requests/0/json/messages/1/content/0/image_url/url", "expected": "data:image/png;base64,iVBORw0KGgo=", "required": true } + ] + }, + { + "id": "tools-core.protocol.choice-and-allowed-set", + "suite": "tools-core", + "capability": "tools.round_trip", + "requirements": { "inboundProtocols": ["openai-responses"], "upstreamProtocols": ["openai-chat"], "surfaces": ["responses-http"], "requiredClaims": ["tools"], "requiredHarnessFeatures": ["adapter_vector"], "platforms": [], "routePreconditions": [] }, + "fixture": { "id": "tools-choice", "role": "client_request", "mediaType": "application/json", "bytesUtf8": "{\"model\":\"fixture-model\",\"input\":\"PING\",\"tools\":[{\"type\":\"function\",\"name\":\"alpha\",\"parameters\":{\"type\":\"object\"}},{\"type\":\"function\",\"name\":\"beta\",\"parameters\":{\"type\":\"object\"}}],\"tool_choice\":{\"type\":\"allowed_tools\",\"mode\":\"required\",\"tools\":[{\"type\":\"function\",\"name\":\"beta\"}]}}", "digest": "fe8b6dde44f88cb9e9a7c6b2bb290e2ee57e7ed425ca8249fb4b7804feff148a" }, + "assertions": [ + { "id": "choice", "operator": "json_path_equals", "selector": "/upstream/requests/0/json/tool_choice", "expected": {"type":"function","function":{"name":"beta"}}, "required": true }, + { "id": "set", "operator": "json_path_equals", "selector": "/upstream/requests/0/json/tools", "expected": [{"type":"function","function":{"name":"beta","parameters":{"type":"object"}}}], "required": true } + ] + }, + { + "id": "codex-core.protocol.streaming-turn", + "suite": "codex-core", + "capability": "client.codex.core", + "requirements": { "inboundProtocols": ["openai-responses"], "upstreamProtocols": ["openai-chat"], "surfaces": ["responses-sse"], "requiredClaims": [], "requiredHarnessFeatures": ["raw_sse_capture"], "platforms": [], "routePreconditions": [] }, + "initiatingRequest": { "id": "codex-streaming-request", "role": "client_request", "mediaType": "application/json", "bytesUtf8": "{\"model\":\"fixture-model\",\"input\":\"PING\",\"stream\":true}", "digest": "c2c39a78b939e6c5182d3206f86aa86d6fd706959de9c37072db759aa510a1f6" }, + "fixture": { "id": "codex-streaming", "role": "upstream_response", "mediaType": "text/event-stream", "bytesUtf8": "data:{\"choices\":[{\"delta\":{\"content\":\"OK\"},\"finish_reason\":\"stop\"}],\"usage\":{\"prompt_tokens\":1,\"completion_tokens\":1}}\n\ndata:[DONE]\n\n", "digest": "f109d35734ecca8e71226ff739b6a0783aca283a9b0d0238a7974b2d7fd9af53" }, + "assertions": [ + { "id": "text", "operator": "normalized_text_equals", "selector": "/client/response/normalizedText", "expected": "OK", "required": true }, + { "id": "terminal", "operator": "terminal_signal_equals", "selector": "/client/response/terminal", "expected": "completed", "required": true }, + { "id": "phase", "operator": "json_path_equals", "selector": "/client/response/events/0/data/phase", "expected": "final_answer", "required": true } + ] + }, + { + "id": "codex-core.protocol.apply-patch-turn", + "suite": "codex-core", + "capability": "client.codex.core", + "requirements": { "inboundProtocols": ["openai-responses"], "upstreamProtocols": ["openai-chat"], "surfaces": ["responses-http"], "requiredClaims": ["custom_tools"], "requiredHarnessFeatures": ["adapter_vector"], "platforms": [], "routePreconditions": [] }, + "fixture": { "id": "codex-patch", "role": "adapter_vector", "mediaType": "application/vnd.opencodex.adapter-vector+json", "bytesUtf8": "{\"name\":\"apply_patch\",\"input\":\"*** Begin Patch\\n*** Add File: x\\n+x\\n*** End Patch\\n\",\"callId\":\"call_patch\",\"result\":\"Done\"}", "digest": "668baa1fbea1d7a6556f717467fc3b90a47b2edfaa2ccf0c7950fd30dfe27a81" }, + "assertions": [ + { "id": "call", "operator": "tool_call_equals", "selector": "/client/response/toolCalls/0", "expected": {"id":"call_patch","name":"apply_patch","arguments":"*** Begin Patch\n*** Add File: x\n+x\n*** End Patch\n","kind":"custom","ordinal":0}, "required": true }, + { "id": "result", "operator": "tool_result_correlates", "selector": "/upstream/requests", "expected": {"call":"/client/response/toolCalls/0/id","result":"/upstream/requests/1/json/messages/1/tool_call_id"}, "required": true } + ] + }, + { + "id": "codex-core.protocol.tool-continuation", + "suite": "codex-core", + "capability": "client.codex.core", + "requirements": { "inboundProtocols": ["openai-responses"], "upstreamProtocols": ["openai-responses"], "surfaces": ["responses-http"], "requiredClaims": ["tools"], "requiredHarnessFeatures": ["adapter_vector"], "platforms": [], "routePreconditions": [] }, + "fixture": { "id": "codex-tool-continuation", "role": "adapter_vector", "mediaType": "application/vnd.opencodex.adapter-vector+json", "bytesUtf8": "{\"turn1\":{\"output\":[{\"type\":\"function_call\",\"id\":\"fc_fixture\",\"call_id\":\"call_fixture\",\"name\":\"lookup\",\"arguments\":\"{}\"}]},\"turn2\":{\"input\":[{\"type\":\"function_call_output\",\"call_id\":\"call_fixture\",\"output\":\"RESULT\"}]}}", "digest": "0b1e955829282c51e056e0bd1d6eb88d62fbae1accd52bdda579c3fce9eac205" }, + "assertions": [ + { "id": "correlation", "operator": "id_correlates", "selector": "/upstream/requests", "expected": ["/upstream/requests/0/json/input/0/call_id","/upstream/requests/0/json/input/1/call_id"], "required": true }, + { "id": "order", "operator": "json_path_equals", "selector": "/verifiers/call_result_order", "expected": "pass", "required": true } + ] + }, + { + "id": "codex-core.protocol.previous-response-replay", + "suite": "codex-core", + "capability": "client.codex.core", + "requirements": { "inboundProtocols": ["openai-responses"], "upstreamProtocols": ["openai-responses"], "surfaces": ["responses-http"], "requiredClaims": [], "requiredHarnessFeatures": ["adapter_vector"], "platforms": [], "routePreconditions": [] }, + "fixture": { "id": "codex-replay", "role": "adapter_vector", "mediaType": "application/vnd.opencodex.adapter-vector+json", "bytesUtf8": "{\"stored\":{\"id\":\"resp_prev\",\"input\":[{\"role\":\"user\",\"content\":\"ONE\"}],\"output\":[{\"role\":\"assistant\",\"content\":\"TWO\"}]},\"next\":{\"previous_response_id\":\"resp_prev\",\"input\":[{\"role\":\"user\",\"content\":\"THREE\"}]}}", "digest": "e849a72d9772616a5ca8853bef48fd2f0884fd006b9ac747bd442513ad05e0f4" }, + "assertions": [ + { "id": "expanded", "operator": "json_path_equals", "selector": "/upstream/requests/0/json/input", "expected": [{"role":"user","content":"ONE"},{"role":"assistant","content":"TWO"},{"role":"user","content":"THREE"}], "required": true }, + { "id": "private-id", "operator": "json_path_absent", "selector": "/upstream/requests/0/json/previous_response_id", "expected": true, "required": true } + ] + }, + { + "id": "codex-core.protocol.structured-output", + "suite": "codex-core", + "capability": "client.codex.core", + "requirements": { "inboundProtocols": ["openai-responses"], "upstreamProtocols": ["openai-chat"], "surfaces": ["responses-http"], "requiredClaims": ["structured_output"], "requiredHarnessFeatures": ["adapter_vector"], "platforms": [], "routePreconditions": [] }, + "fixture": { "id": "codex-structured", "role": "client_request", "mediaType": "application/json", "bytesUtf8": "{\"model\":\"fixture-model\",\"input\":\"PING\",\"text\":{\"format\":{\"type\":\"json_schema\",\"name\":\"answer\",\"schema\":{\"type\":\"object\",\"properties\":{\"ok\":{\"type\":\"boolean\"}},\"required\":[\"ok\"],\"additionalProperties\":false},\"strict\":true}}}", "digest": "e6278954535f4d482a9bb1f6c0189ef7aed00294a7bde695747b7898886cf937" }, + "assertions": [ + { "id": "format", "operator": "json_path_equals", "selector": "/upstream/requests/0/json/response_format", "expected": {"type":"json_schema","json_schema":{"name":"answer","schema":{"type":"object","properties":{"ok":{"type":"boolean"}},"required":["ok"],"additionalProperties":false},"strict":true}}, "required": true } + ] + }, + { + "id": "codex-core.protocol.compaction-and-special-items", + "suite": "codex-core", + "capability": "client.codex.core", + "requirements": { "inboundProtocols": ["openai-responses"], "upstreamProtocols": ["openai-chat"], "surfaces": ["responses-http"], "requiredClaims": [], "requiredHarnessFeatures": ["adapter_vector"], "platforms": [], "routePreconditions": [] }, + "fixture": { "id": "codex-special-items", "role": "client_request", "mediaType": "application/json", "bytesUtf8": "{\"model\":\"fixture-model\",\"input\":[{\"type\":\"context_compaction\",\"encrypted_content\":\"ocx1:fixture\"},{\"type\":\"local_shell_call\",\"id\":\"shell_fixture\",\"call_id\":\"call_shell\",\"status\":\"completed\",\"action\":{\"type\":\"exec\",\"command\":[\"echo\",\"ok\"]}},{\"type\":\"function_call_output\",\"call_id\":\"call_shell\",\"output\":\"ok\"},{\"type\":\"tool_search_output\",\"status\":\"failed\",\"error\":\"fixture\"}]}", "digest": "bf61cb0783f288a4dd6b0b8c0f9a2ddb60f02d0f8ea1d2875ea7e3fe1740b043" }, + "assertions": [ + { "id": "compaction", "operator": "json_path_equals", "selector": "/verifiers/compaction_replayed", "expected": true, "required": true }, + { "id": "shell", "operator": "json_path_equals", "selector": "/verifiers/local_shell_correlated", "expected": true, "required": true }, + { "id": "search", "operator": "json_path_equals", "selector": "/verifiers/tool_search_error", "expected": "fixture", "required": true } + ] + }, + { + "id": "vision-core.protocol.input-image", + "suite": "vision-core", + "capability": "modalities.image.input", + "requirements": { "inboundProtocols": ["openai-responses"], "upstreamProtocols": ["openai-chat"], "surfaces": ["responses-http"], "requiredClaims": ["image"], "requiredHarnessFeatures": ["adapter_vector"], "platforms": [], "routePreconditions": [] }, + "fixture": { "id": "vision-input", "role": "client_request", "mediaType": "application/json", "bytesUtf8": "{\"model\":\"fixture-model\",\"input\":[{\"role\":\"user\",\"content\":[{\"type\":\"input_text\",\"text\":\"READ\"},{\"type\":\"input_image\",\"image_url\":\"data:image/png;base64,iVBORw0KGgo=\",\"detail\":\"high\"}]}]}", "digest": "a26ba5209858c3658d698c1dcb6c92845b2e6aae6bab70a7b9ad1cba1d8aa6a5" }, + "assertions": [ + { "id": "text", "operator": "json_path_equals", "selector": "/upstream/requests/0/json/messages/0/content/0/text", "expected": "READ", "required": true }, + { "id": "image", "operator": "json_path_equals", "selector": "/upstream/requests/0/json/messages/0/content/1/image_url", "expected": {"url":"data:image/png;base64,iVBORw0KGgo=","detail":"high"}, "required": true } + ] + }, + { + "id": "vision-core.protocol.tool-result-image", + "suite": "vision-core", + "capability": "modalities.image.input", + "requirements": { "inboundProtocols": ["openai-responses"], "upstreamProtocols": ["openai-chat"], "surfaces": ["responses-http"], "requiredClaims": ["tools","image"], "requiredHarnessFeatures": ["adapter_vector"], "platforms": [], "routePreconditions": [] }, + "fixture": { "id": "vision-tool-result", "role": "adapter_vector", "mediaType": "application/vnd.opencodex.adapter-vector+json", "bytesUtf8": "{\"callId\":\"call_fixture\",\"result\":[{\"type\":\"input_text\",\"text\":\"RESULT\"},{\"type\":\"input_image\",\"image_url\":\"data:image/png;base64,iVBORw0KGgo=\"}]}", "digest": "02c724259bb3c98002842cafad6d890d3dab7db287f1803fd9ce97ec79630a6d" }, + "assertions": [ + { "id": "tool-text", "operator": "json_path_equals", "selector": "/upstream/requests/0/json/messages/0", "expected": {"role":"tool","tool_call_id":"call_fixture","content":"RESULT"}, "required": true }, + { "id": "image-carrier", "operator": "json_path_equals", "selector": "/upstream/requests/0/json/messages/1/content/0/image_url/url", "expected": "data:image/png;base64,iVBORw0KGgo=", "required": true } + ] + }, + { + "id": "vision-core.protocol.modality-gate", + "suite": "vision-core", + "capability": "modalities.image.input", + "verificationRole": "negative_control", + "requirements": { "inboundProtocols": ["openai-responses"], "upstreamProtocols": ["openai-chat"], "surfaces": ["responses-http"], "requiredClaims": [], "requiredHarnessFeatures": ["adapter_vector"], "platforms": [], "routePreconditions": [] }, + "fixture": { "id": "vision-gate", "role": "adapter_vector", "mediaType": "application/vnd.opencodex.adapter-vector+json", "bytesUtf8": "{\"model\":\"text-only\",\"modelInputModalities\":[\"text\"],\"visionSidecar\":{\"enabled\":false},\"requestHasImage\":true}", "digest": "d5b438fb3fad873b0a1bb1b6c91539862e4f3aa8690a963eb6121c5a3229818a" }, + "assertions": [ + { "id": "path", "operator": "json_path_equals", "selector": "/verifiers/modality_path", "expected": "unsupported", "required": true }, + { "id": "no-drop", "operator": "json_path_equals", "selector": "/verifiers/silent_image_drop", "expected": false, "required": true } + ], + "expectedFailure": { "controlKind": "conformance_negative_control", "expectedClass": "capability_failure", "expectedCode": "image_input_unsupported", "assertionIds": ["path", "no-drop"], "onMatch": "pass", "onMismatch": "fail" } + }, + { + "id": "reasoning-core.protocol.effort-mapping", + "suite": "reasoning-core", + "capability": "reasoning.round_trip", + "requirements": { "inboundProtocols": ["openai-responses"], "upstreamProtocols": ["openai-chat"], "surfaces": ["responses-http"], "requiredClaims": ["reasoning"], "requiredHarnessFeatures": ["adapter_vector"], "platforms": [], "routePreconditions": [] }, + "fixture": { "id": "reasoning-effort", "role": "adapter_vector", "mediaType": "application/vnd.opencodex.adapter-vector+json", "bytesUtf8": "{\"requested\":\"high\",\"reasoningEffortMap\":{\"high\":\"adaptive\"},\"reasoningWireFormat\":\"gateway-object\"}", "digest": "d9d5cce104809764d5edbc833088a0a9bb3b4d678a4f135353cc5fecf62e8b57" }, + "assertions": [ + { "id": "wire", "operator": "json_path_equals", "selector": "/upstream/requests/0/json/reasoning", "expected": {"effort":"adaptive"}, "required": true }, + { "id": "legacy-absent", "operator": "json_path_absent", "selector": "/upstream/requests/0/json/reasoning_effort", "expected": true, "required": true } + ] + }, + { + "id": "reasoning-core.protocol.summary-stream", + "suite": "reasoning-core", + "capability": "reasoning.round_trip", + "requirements": { "inboundProtocols": ["openai-responses"], "upstreamProtocols": ["openai-responses"], "surfaces": ["responses-sse"], "requiredClaims": ["reasoning"], "requiredHarnessFeatures": ["raw_sse_capture"], "platforms": [], "routePreconditions": [] }, + "initiatingRequest": { "id": "reasoning-summary-request", "role": "client_request", "mediaType": "application/json", "bytesUtf8": "{\"model\":\"fixture-model\",\"input\":\"PING\",\"stream\":true}", "digest": "c2c39a78b939e6c5182d3206f86aa86d6fd706959de9c37072db759aa510a1f6" }, + "fixture": { "id": "reasoning-summary", "role": "upstream_response", "mediaType": "text/event-stream", "bytesUtf8": "event: response.reasoning_summary_part.added\ndata: {\"type\":\"response.reasoning_summary_part.added\",\"item_id\":\"rs_fixture\",\"summary_index\":0,\"part\":{\"type\":\"summary_text\",\"text\":\"\"}}\n\nevent: response.reasoning_summary_text.delta\ndata: {\"type\":\"response.reasoning_summary_text.delta\",\"item_id\":\"rs_fixture\",\"summary_index\":0,\"delta\":\"WHY\"}\n\nevent: response.completed\ndata: {\"type\":\"response.completed\",\"response\":{\"status\":\"completed\"}}\n\n", "digest": "d9c4fa73b67f7a92d9ec55af7ff12b16ddc9870059e5e530ee04006b171367e7" }, + "assertions": [ + { "id": "events", "operator": "sse_event_sequence", "selector": "/client/response/events", "expected": ["response.reasoning_summary_part.added","response.reasoning_summary_text.delta","response.completed"], "required": true }, + { "id": "id", "operator": "id_matches", "selector": "/client/response/events/0/data/item_id", "expected": "responses_reasoning", "required": true } + ] + }, + { + "id": "reasoning-core.protocol.replay", + "suite": "reasoning-core", + "capability": "reasoning.round_trip", + "requirements": { "inboundProtocols": ["openai-responses"], "upstreamProtocols": ["openai-responses"], "surfaces": ["responses-http"], "requiredClaims": ["reasoning"], "requiredHarnessFeatures": ["adapter_vector"], "platforms": [], "routePreconditions": [] }, + "fixture": { "id": "reasoning-replay", "role": "adapter_vector", "mediaType": "application/vnd.opencodex.adapter-vector+json", "bytesUtf8": "{\"turn1\":{\"reasoning\":{\"id\":\"rs_fixture\",\"text\":\"PLAN\",\"signature\":\"sig_fixture\"},\"toolCall\":{\"callId\":\"call_fixture\"}},\"turn2\":{\"toolResult\":{\"callId\":\"call_fixture\",\"output\":\"RESULT\"}}}", "digest": "6e137e06f52c32e9f7d394b92343a8b849103328e958ab7c9b2825a799ea60c3" }, + "assertions": [ + { "id": "text", "operator": "json_path_equals", "selector": "/upstream/requests/1/json/input/0/content/0/text", "expected": "PLAN", "required": true }, + { "id": "signature", "operator": "json_path_equals", "selector": "/upstream/requests/1/json/input/0/signature", "expected": "sig_fixture", "required": true } + ] + }, + { + "id": "reasoning-core.protocol.private-content-isolation", + "suite": "reasoning-core", + "capability": "reasoning.round_trip", + "requirements": { "inboundProtocols": ["openai-responses"], "upstreamProtocols": ["openai-chat"], "surfaces": ["responses-http"], "requiredClaims": ["reasoning"], "requiredHarnessFeatures": ["adapter_vector"], "platforms": [], "routePreconditions": [] }, + "fixture": { "id": "reasoning-private", "role": "adapter_vector", "mediaType": "application/vnd.opencodex.adapter-vector+json", "bytesUtf8": "{\"origin\":{\"provider\":\"alpha\",\"encrypted\":\"opaque_fixture\"},\"destination\":{\"provider\":\"beta\",\"adapter\":\"openai-chat\"}}", "digest": "3519bd299fe2cd5b8069e0cd1c3b65b61d5ce9588c66e54b49d42af5ccf0c81e" }, + "assertions": [ + { "id": "upstream-absent", "operator": "json_path_absent", "selector": "/upstream/requests/0/json/encrypted_content", "expected": true, "required": true }, + { "id": "client-absent", "operator": "json_path_absent", "selector": "/client/response/json/hidden_reasoning", "expected": true, "required": true } + ] + }, + { + "id": "mcp-core.protocol.namespace-mapping", + "suite": "mcp-core", + "capability": "tools.mcp.core", + "requirements": { "inboundProtocols": ["openai-responses"], "upstreamProtocols": ["cursor-protobuf"], "surfaces": ["responses-http"], "requiredClaims": ["mcp"], "requiredHarnessFeatures": ["adapter_vector","in_memory_mcp_stub","mcp_namespace_round_trip_v1"], "platforms": [], "routePreconditions": [] }, + "fixture": { "id": "mcp-namespace", "role": "synthetic_tool", "mediaType": "application/vnd.opencodex.mcp-stub+json", "bytesUtf8": "{\"namespace\":\"mcp__fixture\",\"name\":\"lookup\",\"description\":\"fixture\",\"inputSchema\":{\"type\":\"object\",\"properties\":{\"q\":{\"type\":\"string\"}}}}", "digest": "91a53f8c580d461d0f5e0d7209e5d4b95249bdfa8bd3fd4f298e18bdeadb0693" }, + "assertions": [ + { "id": "wire-name", "operator": "json_path_equals", "selector": "/upstream/requests/0/json/tools/0/name", "expected": "mcp__fixture__lookup", "required": true }, + { "id": "reverse", "operator": "json_path_equals", "selector": "/client/response/mcpCalls/0", "expected": {"namespace":"mcp__fixture","name":"lookup"}, "required": true } + ] + }, + { + "id": "mcp-core.protocol.schema-and-bounds", + "suite": "mcp-core", + "capability": "tools.mcp.core", + "requirements": { "inboundProtocols": ["openai-responses"], "upstreamProtocols": ["cursor-protobuf"], "surfaces": ["responses-http"], "requiredClaims": ["mcp"], "requiredHarnessFeatures": ["adapter_vector","in_memory_mcp_stub","mcp_schema_bounds_v1"], "platforms": [], "routePreconditions": [] }, + "fixture": { "id": "mcp-bounds", "role": "synthetic_tool", "mediaType": "application/vnd.opencodex.mcp-stub+json", "bytesUtf8": "{\"limitBytes\":64,\"exactSchema\":\"{\\\"type\\\":\\\"object\\\",\\\"properties\\\":{\\\"x\\\":{\\\"type\\\":\\\"string\\\"}},\\\"a\\\":\\\"xxx\\\"}\",\"overSchema\":\"{\\\"type\\\":\\\"object\\\",\\\"properties\\\":{\\\"x\\\":{\\\"type\\\":\\\"string\\\"}},\\\"a\\\":\\\"xxxx\\\"}\"}", "digest": "34ff4414dc8e196d460390557f4fd74c32418ea00710167baff2a0dc1f3b643c" }, + "assertions": [ + { "id": "exact", "operator": "verifier_result_equals", "selector": "/verifiers/exact_bound", "expected": "pass", "required": true }, + { "id": "over", "operator": "verifier_result_equals", "selector": "/verifiers/one_over_rejected", "expected": "pass", "required": true }, + { "id": "atomic", "operator": "json_path_equals", "selector": "/verifiers/partial_commit", "expected": false, "required": true } + ] + }, + { + "id": "mcp-core.protocol.call-result", + "suite": "mcp-core", + "capability": "tools.mcp.core", + "requirements": { "inboundProtocols": ["openai-responses"], "upstreamProtocols": ["cursor-protobuf"], "surfaces": ["responses-http"], "requiredClaims": ["mcp"], "requiredHarnessFeatures": ["adapter_vector","in_memory_mcp_stub","mcp_call_result_v1"], "platforms": [], "routePreconditions": [] }, + "fixture": { "id": "mcp-call", "role": "synthetic_tool", "mediaType": "application/vnd.opencodex.mcp-stub+json", "bytesUtf8": "{\"namespace\":\"mcp__fixture\",\"name\":\"lookup\",\"arguments\":{\"q\":\"x\"},\"result\":{\"content\":[{\"type\":\"text\",\"text\":\"RESULT\"}],\"isError\":false}}", "digest": "986ef5017fbdb46eb18b30daaffe72aecc93868b7d89b11c3e244dc084f46496" }, + "assertions": [ + { "id": "call", "operator": "json_path_equals", "selector": "/verifiers/stub_received", "expected": {"namespace":"mcp__fixture","name":"lookup","arguments":{"q":"x"}}, "required": true }, + { "id": "result", "operator": "json_path_equals", "selector": "/client/response/json", "expected": {"content":[{"type":"text","text":"RESULT"}],"isError":false}, "required": true } + ] + }, + { + "id": "mcp-core.protocol.resource-round-trip", + "suite": "mcp-core", + "capability": "tools.mcp.core", + "requirements": { "inboundProtocols": ["openai-responses"], "upstreamProtocols": ["cursor-protobuf"], "surfaces": ["responses-http"], "requiredClaims": ["mcp"], "requiredHarnessFeatures": ["adapter_vector","in_memory_mcp_stub","mcp_resource_round_trip_v1"], "platforms": [], "routePreconditions": [] }, + "fixture": { "id": "mcp-resource", "role": "synthetic_tool", "mediaType": "application/vnd.opencodex.mcp-stub+json", "bytesUtf8": "{\"resources\":[{\"uri\":\"fixture://one\",\"name\":\"one\"}],\"read\":{\"uri\":\"fixture://one\",\"contents\":[{\"uri\":\"fixture://one\",\"text\":\"RESOURCE\"}]}}", "digest": "a3f6317374ce92da0155dd14bbf0d5822e8687cbe8ef7968221f23acf8b16aa5" }, + "assertions": [ + { "id": "list", "operator": "json_path_equals", "selector": "/client/response/json/resources", "expected": [{"uri":"fixture://one","name":"one"}], "required": true }, + { "id": "read", "operator": "json_path_equals", "selector": "/client/response/json/contents", "expected": [{"uri":"fixture://one","text":"RESOURCE"}], "required": true } + ] + } + ] +} diff --git a/devlog/_plan/260807_compatibility_lab/030_incident_corpus.md b/devlog/_plan/260807_compatibility_lab/030_incident_corpus.md new file mode 100644 index 000000000..2e17f89e3 --- /dev/null +++ b/devlog/_plan/260807_compatibility_lab/030_incident_corpus.md @@ -0,0 +1,457 @@ +# CL-00 compatibility incident corpus + +These are abstract regression specifications distilled from shipped tests, +public issues, and devlog records. Provider names identify historical evidence, +not special cases to encode in the scenario model. + +Each future fixture must reproduce the observable wire condition with a mock +upstream. CL-00 does not reopen or fix the production incidents. + +`Future mapping` identifies the suite/scenario family that should own the +regression. It does not claim protocol V1 already covers the incident. Only a +literal vector in `022_protocol_v1_cases.json` gates a V1 verdict. +Unrepresented incidents below are reviewed inputs to a later scenario/suite +version amendment; prose and source-test references cannot be inferred into +V1. + +## IC-001 - legal SSE field spacing + +- Incident class: valid SSE framing rejected. +- Historical source: + [#1170](https://github.com/lidge-jun/opencodex/issues/1170), + `devlog/_plan/260807_untouched_bug_stack/010_sse_unspaced_data_fields.md`, + `tests/sse-unspaced-data-fields.test.ts`. +- Observable failure: a parser accepts `data: {...}` but rejects legal + `data:{...}`/`event:name`, trims payload whitespace, or treats a bare + `data:` as malformed. +- Expected behavior: accept both legal forms, strip at most one optional space, + preserve payload whitespace, and handle empty field values consistently on + Responses, Chat, Anthropic and sidecar paths. +- Future mapping: `responses-core.protocol.sse-framing`, + `chat-core.protocol.stream-terminal`, + `anthropic-core.protocol.content-sequence`. +- Classification: `protocol_failure` -> `DEGRADED`. +- Deterministic fixture: yes; paired frames differing only in legal spacing. + +## IC-002 - null and empty SSE data frames + +- Incident class: ignorable frame mishandled as payload or terminal error. +- Historical source: + `devlog/_plan/260808_bug_campaign/020_wp2_sse_frame_contract.md`, + `tests/sse-null-data-frame.test.ts`. +- Observable failure: `data: null`, a bare `data:` field, or a comment frame + crashes decoding, creates a synthetic event, or hides a later valid event. +- Expected behavior: apply each surface's explicit ignorable-frame contract; + continue parsing without fabricating output, while malformed non-null JSON + still fails closed. +- Future mapping: `responses-core.protocol.sse-framing`, + `chat-core.protocol.stream-terminal`, + `anthropic-core.protocol.content-sequence`. +- Classification: `protocol_failure` -> `DEGRADED`. +- Deterministic fixture: yes; null/empty/comment/malformed controls. + +## IC-003 - missing or incorrect terminal stream signal + +- Incident class: clean EOF, `[DONE]`, completion event, and terminal state + confused. +- Historical source: `tests/openai-chat-eof.test.ts`, + `tests/sse-failed-tail.test.ts`, `tests/claude-outbound.test.ts`, + `tests/responses-stream-tool-events.test.ts`, + [#658](https://github.com/lidge-jun/opencodex/issues/658), + [#735](https://github.com/lidge-jun/opencodex/issues/735). +- Observable failure: a stream ends without the protocol-required terminal, + emits more than one terminal, accepts `[DONE]` as a Responses completion + without a terminal event, or maps failed/incomplete to successful end-turn. +- Expected behavior: exactly one surface-correct terminal. Responses remains + strict. A fingerprinted Chat/Anthropic EOF-tolerance contract may complete + only after visible output or a fully assembled tool call and only when no + incomplete call remains; every other deterministic EOF fails closed and + preserves the typed failed/incomplete reason. +- Future mapping: `responses-core.protocol.terminal-state`, + `chat-core.protocol.stream-terminal`, + `anthropic-core.protocol.terminal-errors`, + `codex-core.protocol.streaming-turn`. +- Classification: deterministic close is `protocol_failure` -> `DEGRADED`. +- Deterministic fixture: yes; completed, failed, incomplete, duplicate and + missing-terminal tails. + +## IC-004 - body stall versus protocol truncation + +- Incident class: environmental/provider timeout misreported as incompatibility. +- Historical source: + [#875](https://github.com/lidge-jun/opencodex/issues/875), + [#1065](https://github.com/lidge-jun/opencodex/issues/1065), + `devlog/_plan/260805_bug_stack_campaign/050_issue875_deepseek_flash_stall.md`, + `devlog/_plan/260806_overnight_triage_round2/020_bounded_body_first_byte.md`. +- Observable failure: no first byte or no later body byte arrives before the + deadline; the system labels the model's protocol unsupported, or waits + without a bound. +- Expected behavior: connect, first-byte, inactivity and total deadlines remain + distinguishable. A silent live stall is a blocker, not proof of malformed + protocol. A mock that deliberately closes without terminal data remains + IC-003. +- Future mapping: supplemental timeout controls for + `responses-core.protocol.terminal-state` and every future live suite. +- Classification: `timeout` -> `BLOCKED`; `provider_transient` when an + authoritative transient response exists. +- Deterministic fixture: yes for timeout attribution; no deterministic fixture + can convert an arbitrary live stall into incompatibility evidence. + +## IC-005 - sparse lifecycle snapshots + +- Incident class: incomplete Responses lifecycle snapshots forwarded as valid. +- Historical source: + [#893](https://github.com/lidge-jun/opencodex/issues/893), + `devlog/_plan/260805_bug_stack_campaign/040_issue893_sparse_snapshot_repair.md`, + `tests/responses-snapshot-repair.test.ts`, + `tests/responses-snapshot-repair-server.test.ts`. +- Observable failure: added/done snapshots omit required ID, type, role, status, + output index or closing item, producing a client-invalid lifecycle. +- Expected behavior: preserve a complete canonical lifecycle or apply an + explicitly configured, assertion-visible repair; never claim a sparse stream + is valid without proving the repaired output. +- Future mapping: `responses-core.protocol.item-lifecycle`, + `codex-core.protocol.streaming-turn`. +- Classification: `protocol_failure` -> `DEGRADED`. +- Deterministic fixture: yes; sparse permutations plus no-repair control. + +## IC-006 - invalid, reused, or missing item IDs + +- Incident class: client-facing Responses item identity violates its grammar or + event correlation. +- Historical source: + [#938](https://github.com/lidge-jun/opencodex/issues/938), + `devlog/_plan/260805_bug_stack_campaign/060_issue938_uuid_item_ids.md`, + `tests/responses-item-id-repair.test.ts`, + `tests/deepseek-responses-item-id-repair.test.ts`. +- Observable failure: UUID/placeholder/missing IDs reach a client contract that + requires typed IDs, or added/done events use inconsistent IDs. +- Expected behavior: valid stable IDs on the client surface; any configured + repair is deterministic, type-scoped, and never rewrites function call IDs + or breaks `call_id` correlation. +- Future mapping: `responses-core.protocol.item-lifecycle`, + `codex-core.protocol.streaming-turn`. +- Classification: `protocol_failure` -> `DEGRADED`. +- Deterministic fixture: yes; valid, invalid, reused, missing-terminal and + correlation controls. + +## IC-007 - function schema root normalization + +- Incident class: valid tool rejected because its schema root is missing or + non-object. +- Historical source: + [PR #745](https://github.com/lidge-jun/opencodex/pull/745), + `tests/responses-parser.test.ts`. +- Observable failure: tool definition reaches an object-schema-only upstream + with absent/invalid root shape, or normalization corrupts an already valid + schema. +- Expected behavior: produce the required object root without changing valid + properties/required/additionalProperties semantics. +- Future mapping: `tools-core.protocol.function-round-trip`, + `mcp-core.protocol.schema-and-bounds`. +- Classification: `protocol_failure` -> `DEGRADED`. +- Deterministic fixture: yes; absent, malformed and valid schema controls. + +## IC-008 - custom/freeform tool envelope mismatch + +- Incident class: function-only route or token preset rejects a valid + custom/freeform tool. +- Historical source: + `devlog/_plan/260807_untouched_bug_stack/070_mimo_token_plan_preset.md`, + `tests/responses-parser.test.ts` (exact `apply_patch` envelope), + `tests/responses-tool-groups.test.ts`. +- Observable failure: a custom tool is serialized as a function, its freeform + input/output is JSON-wrapped or dropped, or the route rejects the tool without + an honest unsupported result. +- Expected behavior: preserve the exact custom tool declaration, call and + output grammar, or deterministically classify the exact route unsupported for + custom tools. +- Future mapping: `tools-core.protocol.custom-freeform-round-trip`, + `codex-core.protocol.apply-patch-turn`. +- Classification: malformed translation is `protocol_failure` -> `DEGRADED`; + a proven route contract is `capability_failure` -> `UNSUPPORTED`. +- Deterministic fixture: yes for translation; a future live negative control + proves route support. + +## IC-009 - dangling tool calls and result correlation + +- Incident class: tool call/result pair becomes orphaned or misassociated. +- Historical source: + `devlog/_fin/260718_dangling_toolcall_hardening/010_record.md`, + `tests/openai-chat-dangling-toolcalls.test.ts`, + `tests/issue-702-expired-replay-state.test.ts`, + [#334](https://github.com/lidge-jun/opencodex/issues/334), + [#620](https://github.com/lidge-jun/opencodex/issues/620). +- Observable failure: an assistant tool call is forwarded without a matching + result, a result is attached to the wrong ID, or expired continuation state + resurrects an unrelated call. +- Expected behavior: preserve exact call/result identity and order; repair only + the narrowly declared orphan case; otherwise fail closed without fabricating + a successful tool result. +- Future mapping: `tools-core.protocol.function-round-trip`, + `codex-core.protocol.tool-continuation`, + `codex-core.protocol.previous-response-replay`. +- Classification: `protocol_failure` -> `DEGRADED`. +- Deterministic fixture: yes; missing, duplicate, out-of-order, expired and + mismatched IDs. + +## IC-010 - parallel tool fragment assembly + +- Incident class: interleaved calls merged, lost, reordered, or correlated to + the wrong result. +- Historical source: + `devlog/_fin/260709_parallel_tool_calls/000_plan.md`, + `tests/openai-chat-parallel-stream.test.ts`, + `tests/parallel-tool-calls-optin.test.ts`, + [#361](https://github.com/lidge-jun/opencodex/issues/361). +- Observable failure: fragmented deltas from two calls produce one argument + buffer, unstable ordering, duplicate completion, or incorrect result IDs. +- Expected behavior: assemble each indexed call independently, never duplicate + argument fragments, preserve stable order/identity, and advertise parallel + capability only when the effective adapter contract supports it. Provider + interleaving does not require overlapping canonical adapter events; atomic + sequential emission is a valid compatibility-preserving bridge contract. +- Future mapping: `tools-core.protocol.parallel-correlation`. +- Classification: `protocol_failure` -> `DEGRADED`; explicit no-parallel + contract -> `UNSUPPORTED` for that capability only. +- Deterministic fixture: yes; interleaved, fragmented, out-of-order and + single-call controls. + +## IC-011 - wrong upstream wire for a model + +- Incident class: Responses-capable and Chat-only models behind one gateway use + the provider-wide wire indiscriminately. +- Historical source: `src/types.ts` and `src/providers/registry.ts` model-wire + contract for [#404](https://github.com/lidge-jun/opencodex/issues/404), + `tests/adapter-resolve.test.ts`, `tests/deepseek-inbound-wire.test.ts`, + `tests/chat-completions-endpoint.test.ts`. +- Observable failure: an exact model is sent to the wrong endpoint/request + shape, producing rejection or silent semantic loss. +- Expected behavior: resolve the effective model-specific adapter before + subject identity and send the declared wire shape. Evidence for one wire is + never reused for the other. +- Future mapping: `responses-core.protocol.request-shape`, + `chat-core.protocol.request-mapping`; future live route variants. +- Classification: deterministic resolver/translation error is + `protocol_failure` -> `DEGRADED`; a correctly selected but unsupported route + is `capability_failure` -> `UNSUPPORTED`. +- Deterministic fixture: yes; mixed gateway with endpoint-specific fixtures. + +## IC-012 - reasoning replay form mismatch + +- Incident class: plaintext reasoning, signature, redacted block, or thought + signature is dropped or replayed in the wrong form. +- Historical source: `tests/deepseek-reasoning-replay.test.ts`, + `tests/deepseek-reasoning-replay-gaps.test.ts`, + `tests/google-antigravity-replay.test.ts`, + `tests/anthropic-thinking-signature.test.ts`, + `tests/kiro-reasoning-roundtrip.test.ts`. +- Observable failure: a second turn is rejected, reasoning text leaks into + visible output, required signature data is lost, or incompatible replay data + is forwarded. +- Expected behavior: use the exact selected adapter's replay contract, preserve + opaque data only on its compatible route, and omit/normalize it safely + elsewhere. +- Future mapping: `reasoning-core.protocol.replay`, + `reasoning-core.protocol.summary-stream`. +- Classification: `protocol_failure` -> `DEGRADED`. +- Deterministic fixture: yes; two-turn fixtures for each abstract replay form. + +## IC-013 - provider-private content crosses a route boundary + +- Incident class: encrypted/task/reasoning content from one provider is sent to + an incompatible provider or exposed as ordinary text. +- Historical source: + [#92](https://github.com/lidge-jun/opencodex/issues/92), + `devlog/_fin/260706_previous-response-id-400/000_plan.md`, + `tests/responses-parser.test.ts` encrypted-content case, + `tests/bridge-raw-reasoning-hidden.test.ts`, + `tests/v2-agent-message-failfast.test.ts`. +- Observable failure: opaque encrypted content is forwarded where it cannot be + decrypted, causes a 400, or becomes user-visible/private evidence. +- Expected behavior: provider-private envelopes remain origin-scoped; cross + route replay fails closed or uses a bounded opaque marker expressly allowed + by the protocol, never raw private data. +- Future mapping: `reasoning-core.protocol.private-content-isolation`, + `codex-core.protocol.previous-response-replay`; a later encrypted-task + capability scenario when its upstream contract is implementable. +- Classification: unsafe translation is `protocol_failure` -> `DEGRADED`; a + route proven unable to consume the encrypted task capability is + `UNSUPPORTED`; the current explicit fail-fast is safe `UNSUPPORTED` evidence + only when the scenario's exact route and encrypted-task preconditions match. + Raw disclosure is also a security Critical independent of compatibility + verdict. +- Deterministic fixture: yes for local origin isolation and fail-fast + mitigation; partial for true cross-provider encrypted task execution. + +## IC-014 - image modality or tool-result image mismatch + +- Incident class: structured image content is dropped, stringified, sent to a + text-only route, or advertised inaccurately. +- Historical source: + [#888](https://github.com/lidge-jun/opencodex/issues/888), + `tests/openai-chat-tool-result-images.test.ts`, + `tests/responses-parser.test.ts`, `tests/vision-anthropic.test.ts`, + `tests/vision-fail-closed.test.ts`, `tests/request-evidence.test.ts`. +- Observable failure: image order/detail/MIME is lost, a tool-result image + becomes raw JSON/text, or capability gating disagrees with the effective + native/sidecar path. +- Expected behavior: preserve structured image parts and honestly choose + native, declared sidecar, or unsupported behavior without silent loss. +- Future mapping: `vision-core.protocol.input-image`, + `vision-core.protocol.tool-result-image`, + `vision-core.protocol.modality-gate`. +- Classification: `protocol_failure` -> `DEGRADED`; proven no-image route -> + `UNSUPPORTED`; sidecar/network unavailability -> `BLOCKED`. +- Deterministic fixture: yes; synthetic data image and text-only controls. + +## IC-015 - malformed continuation and previous-response state + +- Incident class: stateful continuation is forwarded to a stateless/incompatible + route or local replay is incomplete. +- Historical source: + [#702](https://github.com/lidge-jun/opencodex/issues/702), + `devlog/_fin/260706_previous-response-id-400/000_plan.md`, + `tests/responses-state.test.ts`, + `tests/issue-702-expired-replay-state.test.ts`, + `tests/grok-orphan-adoption.test.ts`. +- Observable failure: upstream 400, duplicate history, missing prior tool call, + orphaned result, or continuation state reused after expiry/route change. +- Expected behavior: use valid provider-private continuation only on its exact + compatible subject; otherwise perform bounded ordered local expansion or fail + closed. +- Future mapping: `codex-core.protocol.previous-response-replay`, + `codex-core.protocol.tool-continuation`. +- Classification: `protocol_failure` -> `DEGRADED`. +- Deterministic fixture: yes; stateful, stateless, expired and route-change + matrices. + +## IC-016 - Anthropic terminal/error taxonomy corruption + +- Incident class: failed/incomplete/upstream-overload response appears as + successful `end_turn` or wrong Anthropic error type. +- Historical source: `tests/claude-outbound.test.ts`, + `tests/anthropic-eof-tolerance.test.ts`, + `tests/anthropic-compatible-stream.test.ts`. +- Observable failure: missing `message_stop` is accepted outside a declared + tolerance, transient 502 becomes a normal message, or content-filter/max-token + stop reason is mapped incorrectly. +- Expected behavior: preserve exact content-block and message terminal + sequence; map failure classes and stop reasons deterministically; apply any + EOF tolerance only to its exact fingerprinted route. +- Future mapping: `anthropic-core.protocol.terminal-errors`, + `anthropic-core.protocol.content-sequence`. +- Classification: `protocol_failure` -> `DEGRADED`; live transient -> + `provider_transient` -> `BLOCKED`. +- Deterministic fixture: yes; strict/tolerant, failed, incomplete and transient + controls. + +## IC-017 - MCP namespace, bound, and result atomicity + +- Incident class: namespace collision, oversized schema/result partial commit, + or result type loss. +- Historical source: `tests/cursor-mcp-manager.test.ts`, + `tests/cursor-mcp-stdio.test.ts`. +- Observable failure: flattened names cannot map back, one-byte-over input + leaves a partial catalogue, image/error result changes type, or unknown tool + becomes an untyped exception. +- Expected behavior: collision-safe namespace mapping, exact atomic bounds, and + typed result/error/resource behavior through a Lab-owned stub. +- Future mapping: all `mcp-core.protocol.*` scenarios. +- Classification: `protocol_failure` -> `DEGRADED`; declared no-MCP route -> + `UNSUPPORTED`. +- Deterministic fixture: yes; in-memory/loopback stub only. + +## IC-018 - DNS/connect failure poisons account or capability evidence + +- Incident class: pre-connection transport failure attributed to credentials, + account, model, or capability. +- Historical source: + [#914](https://github.com/lidge-jun/opencodex/issues/914), + `devlog/_plan/260805_bug_stack_campaign/030_issue914_dns_transport_attribution.md`, + `devlog/_plan/260803_transport_attribution/000_plan.md`, + `tests/upstream-connect-error.test.ts`. +- Observable failure: DNS/connect/TLS setup rotates account state, marks a + route capability degraded, or becomes authentication evidence. +- Expected behavior: classify pre-response transport evidence as environment/ + network, leave compatibility and credential capability unchanged, and permit + retry after environment repair. +- Future mapping: blocker controls shared by every future live suite. +- Classification: `network_failure` -> `BLOCKED`. +- Deterministic fixture: yes for attribution using an injected connect failure; + it never contributes a compatibility failure. + +## IC-019 - malformed error or empty success envelope + +- Incident class: upstream error/empty payload accepted as a successful model + response. +- Historical source: `tests/openai-chat-hardening.test.ts`, + `tests/error-fidelity.test.ts`, `tests/upstream-http-error.test.ts`. +- Observable failure: falsey error payload, empty choices, null choice, missing + message, or malformed SSE data is emitted as success or hidden by a terminal. +- Expected behavior: fail closed with a typed normalized error while preserving + any safe usage/status evidence. +- Future mapping: `chat-core.protocol.nonstream-envelope`, + `chat-core.protocol.stream-terminal`. +- Classification: `protocol_failure` -> `DEGRADED` for deterministic malformed + protocol; recognized live transient remains `provider_transient`. +- Deterministic fixture: yes. + +## IC-020 - structured-output wire mismatch + +- Incident class: Responses `text.format` is lost, malformed, or sent to an + upstream in the wrong shape. +- Historical source: `tests/responses-parser.test.ts`, + `tests/openai-chat-hardening.test.ts`, + `tests/deepseek-inbound-wire.test.ts`. +- Observable failure: JSON schema/object request widens to plain text, schema + nesting changes, or a strict unsupported route receives an invalid parameter. +- Expected behavior: preserve the known equivalent wire form, or return a + deterministic unsupported result without pretending structured output was + honored. +- Future mapping: `codex-core.protocol.structured-output`, + `responses-core.protocol.request-shape`, + `chat-core.protocol.request-mapping`. +- Classification: translation error is `protocol_failure` -> `DEGRADED`; + proven route limitation is `capability_failure` -> `UNSUPPORTED`. +- Deterministic fixture: yes for wire translation; future live negative control + for route support. + +## IC-021 - data-only Responses SSE + +- Incident class: valid Responses events rejected because the producer omits + the redundant `event:` field. +- Historical source: + [#700](https://github.com/lidge-jun/opencodex/issues/700), + `tests/claude-outbound.test.ts`. +- Observable failure: a payload with a valid typed Responses JSON record in + `data:` is ignored or treated as a truncated stream when no `event:` line is + present. +- Expected behavior: infer the event name from the payload's canonical `type` + when the surface permits data-only events, permit explicit and inferred + frames to interleave, and keep untyped data-only records ignored/fail-closed + according to the scenario. +- Future mapping: `responses-core.protocol.sse-framing`, + `anthropic-core.protocol.content-sequence`. +- Classification: `protocol_failure` -> `DEGRADED`. +- Deterministic fixture: yes; explicit-only, data-only, mixed and untyped + controls. + +## Corpus maintenance rule + +New incidents enter this corpus only when they add a reusable wire condition, +assertion, or attribution boundary. A provider-specific workaround is not a +scenario. The abstraction must state: + +```text +incident class +historical source/reference +observable failure +expected correct behavior +future scenario/suite mapping +expected failure classification +deterministic fixture feasibility +``` + +When a future fix changes the expected contract, bump the mapped scenario +version and preserve this historical record. diff --git a/devlog/_plan/260807_compatibility_lab/040_security_and_privacy.md b/devlog/_plan/260807_compatibility_lab/040_security_and_privacy.md new file mode 100644 index 000000000..ac8ea1d10 --- /dev/null +++ b/devlog/_plan/260807_compatibility_lab/040_security_and_privacy.md @@ -0,0 +1,455 @@ +# CL-00 security, privacy, and probe sandbox contract + +Compatibility evidence is useful only if collecting it does not turn the Lab +into a data-exfiltration or arbitrary-execution surface. These requirements are +release blockers for later implementation. + +## 1. Data prohibition + +The Lab must not read, accept, persist, or export: + +- user prompts or conversation history; +- real user repositories, worktrees, patches, source files, or file paths; +- user MCP server definitions, resources, results, or credentials, with no + Lab mode, CLI flag, profile, or override that can load them; +- arbitrary shell commands or process output; +- arbitrary filesystem contents; +- arbitrary external-network tool requests or responses; +- API keys, OAuth/access/refresh tokens, cookies, authorization material, or + raw credential errors; +- account IDs, account emails, aliases, plan labels, tenant IDs, or other PII; +- raw private/custom headers; +- hidden reasoning, chain of thought, encrypted reasoning payloads, provider + thought signatures, or decrypted private task content. + +Scenarios contain Lab-authored synthetic prompts, fixtures, tool definitions +and results only. They must be recognizable as synthetic and contain no copied +customer material. + +### Synthetic-input admission + +Synthetic status is machine-checked, not inferred from content or a friendly +name. Every fixture admitted to the Lab has a canonical reference containing: + +```text +syntheticMarker ocx-lab-synthetic-v1 +provenance.kind lab_authored +provenance.authority reviewed manifest/authority identifier +provenance.sourceCommit immutable source revision +digest domain-separated content digest +byteLength exact admitted byte length +``` + +The reference participates in the scenario manifest digest. Admission verifies +the marker, closed provenance kind, expected authority/source revision, digest, +and byte length before fixture bytes reach an adapter, mock, tool stub, event +normalizer, or artifact writer. A request payload, model output, local path, +repository file, MCP object, URL response, or runtime caller cannot label +itself synthetic. Protocol V1 uses the exact `fixtureRef` contract in +`021_protocol_v1_manifest_authority.md`; future live/Fabric suites must define +an equally closed provenance authority before they may run. + +## 2. Future live-probe sandbox + +A live probe is an explicit background/management/CLI action. It is never +started by the production request path, profile evaluator, Router Intelligence, +request-history read, dashboard render, or provider discovery. + +The future runner must enforce a capability-deny sandbox: + +### Network + +- The immutable scenario manifest may authorize only fixed dependency roles + and protocol classes, never a route-local URL. +- Network authorization uses a trusted in-memory `LabDestinationV1` record + owned by the existing provider destination/credential plumbing: scheme, + host, port, base path, resolved IP family/address set after policy checks, + TLS SNI/Host values, and private-network opt-in. That record is never + written to JSONL, SQLite, artifacts, or export. +- Provider destination/credential plumbing creates one per-run immutable + `LabDestinationV1` snapshot before endpoint fingerprinting. The exact same + snapshot must be used unchanged for endpoint fingerprinting, destination + authorization, credential binding, and connection. Mutation, replacement, + re-resolution to a different address set, or any mismatch between those + stages fails closed as `harness_failure` before credentials are sent. +- The composite route subject stores only the keyed opaque + `endpointFingerprint` derived from the normalized destination. Raw URLs are + never evidence fields. +- The only remote destinations are the exact primary and flat sidecar + destinations named by those in-memory records after existing provider + destination-policy validation. +- DNS is resolved once before the policy check. The HTTP client must connect + to the validated IP set (pin/connect to the approved addresses) while + preserving the intended Host/SNI. A later resolution that differs fails + closed as `harness_failure`. +- Redirects are rejected by default for Lab probes, matching the existing + SSRF fail-closed posture. A future scenario that explicitly opts into + redirects must authorize every hop with the same destination policy, IP + pinning, and Host/SNI preservation; redirects still cannot widen scheme, + host, port, or private-network access. +- Private/loopback endpoints require the route's existing explicit private + network opt-in and an explicit Lab-run confirmation. Metadata endpoints + remain blocked. +- No scenario-supplied URL, model output, tool argument, or redirect may add a + destination. +- A sidecar dependency is allowed only when the scenario explicitly authorizes + its role/protocol class, the composite subject names the exact dependency + fingerprint, an in-memory destination record exists for that fingerprint, + the operator approves the composite live probe, and its credential is + destination-bound independently. Unmanifested roles or subject-external/ + dynamically widened endpoints make the run `harness_failure`. +- Inherited `HTTP_PROXY`, `HTTPS_PROXY`, `ALL_PROXY`, `NO_PROXY`, `http_proxy`, + `https_proxy`, `all_proxy`, and `no_proxy` values are rejected for Lab runs. + If a future reviewed scenario requires a proxy, that proxy endpoint is + authorized as its own exact destination under the same SSRF checks and never + inherits ambient proxy environment variables. +- Tools have no network capability. A model-requested web search, image + generation, URL fetch, computer use, or hosted external tool is disabled or + classified inapplicable unless a future separately reviewed scenario owns a + fixed synthetic sidecar. +- Deterministic protocol tests may contact only a Lab-owned loopback mock. + +### Credentials + +- Secret bytes remain owned by the existing trusted provider credential/ + transport layer. Lab runner code never receives an API key, token, cookie, + authorization value, entire auth store, or credential-bearing header. +- Immediately before a live request, the credential broker validates the exact + immutable `LabDestinationV1` snapshot and returns an opaque, + non-serializable `LabCredentialLeaseV1` capability bound to that destination, + auth transport, and one run/request budget. The lease exposes no secret + bytes, string conversion, header map, equality/debug representation, or + credential identity. +- The trusted transport consumes the lease and injects authorization only after + its connection is bound to the same approved destination/address set. A + destination, Host/SNI, address-set, transport, or lease-scope mismatch fails + closed before authorization is emitted. The lease expires after its bounded + request/run and cannot be reused for another destination or sidecar. +- Credentials therefore never enter a subject, assertion, error, artifact, + log, event-ID input, SQLite row, callback, mock, or Lab-owned memory buffer. +- Credential absence/rejection produces `authentication_blocked`, never a + compatibility failure. + +### Process and system access + +- The scenario DSL cannot express a shell command, executable, arbitrary + module, callback, script, filesystem path, or dynamic import. +- The runner receives no general shell/process API and no inherited stdin. +- Filesystem access is restricted to a fresh Lab scratch directory, read-only + packaged synthetic fixtures, and the bounded artifact writer. +- The V1 inherited-environment allowlist is empty: Lab code must not read an + ambient variable to determine behavior, routing, credentials, destinations, + paths, locale, or proxying. The runner may expose only a constructed, + non-inherited environment view with exact constants `TZ=UTC` and + `NO_COLOR=1`; every other name is absent. In particular all uppercase and + lowercase proxy variables are rejected as stated in Network above. Secrets + are supplied only through reviewed destination/credential plumbing. +- The run has enforced wall-clock, inactivity, byte, request, token, tool-call, + memory, process and artifact limits. If the platform cannot enforce a + required boundary, the run fails as `harness_failure`. +- Scratch data is deleted after artifact sanitization. Cleanup failure is + visible and retried by bounded maintenance; it does not silently retain user + data because none was admitted. + +### Tools and MCP + +- Function/custom tool scenarios expose inert Lab-authored definitions. The + harness returns static or pure-function results and never executes model + arguments. +- `apply_patch`, shell, file, browser, web-search, image-generation, computer + use and similar names are protocol tokens only. They do not invoke the real + facility. +- MCP scenarios use an in-memory or Lab-owned loopback stub with fixed schemas, + resources and pure results. User MCP configuration is not loaded. +- Cursor `nativeLocalExec`, `unsafeAllowNativeLocalExec`, desktop executors and + configured `mcpServers` are forced off for the Lab subject. Their disabled + state participates in the behavior fingerprint. + +### Agent Fabric + +- Real task execution remains in Agent Fabric's separately reviewed sandbox. +- The Lab accepts a structured outcome and sanitized content-addressed + references only. +- Outcome ingestion cannot dereference an arbitrary path or URL. Artifact + transfer uses an allowlisted broker and re-runs Lab validation. +- No task repository, prompt transcript, worktree, patch body, terminal log, or + hidden reasoning is copied into `~/.opencodex/lab/`. + +## 3. Artifact and resource contract + +Artifacts are deny-by-default, normalized, sanitized, bounded, and +content-addressed after redaction. + +Initial hard ceilings: + +```text +maximum wall-clock time per run 120,000 ms +maximum connect time 10,000 ms +maximum first-byte time 30,000 ms +maximum inactivity time 30,000 ms +maximum requests per run 16 +maximum aggregate input bytes 8 MiB +maximum aggregate output bytes 16 MiB +maximum output tokens 32,768 +maximum tool calls 32 +maximum runner resident memory 512 MiB +maximum child processes 0 +maximum artifacts per run 16 +maximum bytes per artifact 256 KiB +maximum aggregate artifact data 1 MiB +maximum normalized events 4,096 +maximum sanitized string field 4 KiB +maximum serialized bytes per event 64 KiB +maximum aggregate normalized event bytes 1 MiB +maximum event JSON nesting depth 8 +maximum object keys per event object 64 +maximum array elements per event array 256 +``` + +Scenario limits may be lower. No scenario, CLI flag, profile, provider config, +environment variable, route metadata, or runtime caller may raise a hard +ceiling. Raising one requires a reviewed security-contract version change. +Timeout ceilings classify with the matching typed `timeout`; request/byte/ +token/tool/artifact ceilings classify `budget_exhausted`. A memory ceiling +breach or attempted child-process creation terminates the sandboxed run and is +`budget_exhausted`; inability to enforce any required ceiling is +`harness_failure` before executable evidence is accepted. + +Event bounds are enforced while decoding/normalizing, before an event is +buffered into the observation or any artifact. Exceeding an event structural +bound fails the run as `harness_failure` without retaining the oversized +fragment. + +Allowed artifact classes: + +- canonical scenario manifest; +- canonical suite manifest; +- canonical synthetic fixture; +- canonical claim-source manifest; +- assertion report containing normalized expected/observed summaries; +- sanitized request shape with content replaced by type/length/digest markers; +- sanitized response shape with visible synthetic fixture output only; +- normalized bounded event trace; +- sanitized error taxonomy/status; +- deterministic verifier summary. + +Artifact names are derived only from the expected lowercase SHA-256 digest and +a fixed extension under `~/.opencodex/lab/artifacts/`. The Lab artifact store +must use descriptor/handle-bound, no-follow I/O rather than validate a pathname +and reopen it later: + +- open and retain a trusted handle to the artifact directory after verifying it + is a directory and not a symlink/reparse-point redirection; +- create writes relative to that directory handle with exclusive/no-follow + semantics, reject special files and `st_nlink != 1`, and write only already + redacted bytes; +- compute size and digest from the same open descriptor/handle that received + the bytes, flush it, then publish by an atomic rename relative to the trusted + directory handle; +- when a content-addressed target already exists, open it with no-follow + semantics and verify regular-file type, single-link status, size, and digest + from that same descriptor before reuse; +- reads open the digest-derived name relative to the trusted directory handle, + verify file type/link count/size, hash the bytes, and return/consume those + exact bytes from the same open descriptor without closing and reopening by + path; +- traversal, separators, absolute paths, alternate data streams, symlinks, + reparse points, hard links, device/special files, digest/size mismatch, or a + directory-handle identity change fail closed; +- on a platform where equivalent descriptor/handle-bound no-follow operations + cannot be enforced, the artifact operation is `harness_failure`; it must not + fall back to path-only `exists/stat/read` validation. + +The ledger stores relative content-addressed references, never arbitrary paths. +This contract applies to the future Lab artifact store; it does not silently +reuse a different existing application artifact reader whose path-race +properties have not been reviewed for Lab evidence. + +Scenario/suite manifests and synthetic fixtures use the domain-separated +digests in the evidence contract and remain retained while referenced by any +non-invalidated, non-purged observation. Claim-source manifests remain retained +while a non-purged claim snapshot references them. Their content is still +subject to the same synthetic/sanitized-data and size rules. + +Redaction occurs before hashing and writing. A redaction failure discards the +artifact and marks the run `harness_failure`; "write now, redact later" is +forbidden. + +## 4. Diagnostic sanitization + +Provider diagnostics retain only: + +- normalized HTTP status; +- allowlisted non-sensitive error type/code; +- coarse phase (`dns`, `connect`, `tls`, `first_byte`, `stream`, `terminal`); +- bounded latency/duration; +- redacted, bounded message selected by an explicit provider sanitizer. + +They remove URLs, query strings, authorization values, header dumps, request/ +response bodies, account identifiers, project/tenant names, local paths, IPs +where identifying, and token-like strings. Unknown provider diagnostics are +reduced to taxonomy and phase rather than persisted verbatim. + +Sanitizers are tested with seeded canary secrets and common credential forms. +`bun run privacy:scan` remains required but is defense in depth, not the +redaction mechanism. + +## 5. Subject privacy + +The local route subject distinguishes exact behavior without raw secrets: + +- configured instance, endpoint, custom-header behavior, project and location + use a per-installation keyed HMAC; +- credential/account identity does not participate; +- raw base URLs and private/custom headers are absent; +- model IDs are retained locally because they are required route identity, but + custom model IDs are private-by-default for export; +- rotating the local subject salt invalidates local correlation and requires + re-projection/reverification, never reverse lookup. + +The salt is stored with secret-file permissions outside the JSONL/artifact +tree. It is not exported. + +### Custom-header fingerprint broker + +Raw custom headers remain owned by the provider/config request builder and are +never passed to Lab code. That owner computes +`headers.nonCredentialBehaviorDigest` through a narrow fingerprint broker: + +1. resolve the effective static custom headers after preset/config merge but + before request-specific or credential injection; +2. remove every credential-bearing header according to the same auth transport + classification used by the request builder; +3. before canonicalization, enforce at most 64 non-credential header entries, + at most 16 duplicate values for one lowercase name, at most 256 ASCII bytes + per field name, at most 8 KiB UTF-8 bytes per value, and at most 64 KiB of + aggregate normalized name/value bytes; exceeding any bound is + `harness_failure` and no digest is emitted; +4. lowercase valid ASCII field names, reject invalid names, preserve duplicate + value order, and preserve exact UTF-8 value bytes without trimming; +5. sort entries by lowercase name while retaining duplicate order and encode + JCS `[{"name": string, "values": string[]}, ...]`; +6. return lowercase HMAC-SHA-256 with installation salt and domain + `ocx-lab:local-fingerprint:v1\0customHeaderBehavior\0`. + +The broker returns only the digest. Its API cannot return normalized names, +values, intermediate bytes, the salt, or the credential classification. +Unknown classification fails subject construction; it never falls back to +hashing or logging the raw header. Canary tests must prove raw names/values do +not enter Lab events, errors, SQLite, or artifacts. + +## 6. Local evidence versus public export + +Local evidence is already sanitized. Public export is stricter and uses a new, +allowlist-only schema: + +- include suite/scenario versions, evidence layer, verdict, observation time + bucket, public registry provider/model where permitted, assertion summaries, + and public incident/scenario references; +- replace local subject/event/artifact IDs with export-scoped opaque IDs; +- omit endpoint and provider-instance fingerprints, local request/decision/ + Fabric references, precise local paths, custom headers, project/location, + custom provider/model names, account context, raw latency traces, and local + errors; +- include artifact content only when its policy explicitly says + `public_export`; local visibility does not imply export permission; +- run export-specific secret/PII scanning and fail closed on an unknown field. + +Public publishing is not authorized in CL-00 and remains a later phase. + +## 7. Retention and deletion + +- JSONL is the immutable local authority for non-sensitive evidence, but a + user can delete the entire Lab directory. Immutability describes in-ledger + correction semantics, not a promise to resist user deletion. Confirmed + sensitive evidence is the explicit privacy exception below. +- Retention ceilings by class: + - scratch/temp run directories: deleted at run end; cleanup retry within 24h; + - export staging: maximum 24h; + - disposable SQLite projection: rebuildable anytime; may be deleted at any + time and must be deleted during a sensitive purge; + - sanitized non-contract artifacts (`assertion_report`, shapes, traces, + errors): default 90 days, hard ceiling 365 days; + - content-addressed scenario/suite/fixture contract artifacts: retained + while any non-invalidated, non-purged observation references them, because + reproducible executable projection requires the exact historical bytes; + - content-addressed claim-source manifests: retained while any non-purged + claim snapshot references them, because reproducible `CLAIMED` projection + requires the exact historical sanitized source bytes. + Contract artifacts remain synthetic/sanitized and size-bounded. User deletion + of the Lab directory remains absolute. +- Deleting an expired non-contract artifact leaves its digest/reference and a + typed unavailable marker; it does not alter the observation. +- SQLite is disposable and contains no data absent from valid ledger events, + privacy-safe purge tombstones, and artifact metadata. +- Invalid non-sensitive evidence is neutralized by a valid appended + invalidation under the evidence contract. Event-private non-contract + artifacts may then be securely deleted. A shared scenario, suite, fixture, + or claim-source contract artifact must remain while any other usable event + references its digest, and may be deleted only after the last such reference + is invalidated or purged. +- Confirmed sensitive evidence is distinct from ordinary invalidation. It + requires a fail-closed purge of every local copy: offending JSONL event + lines, SQLite rows, artifacts, scratch/temp files, and generated exports. + The purge first determines the affected event IDs/artifact digests without + retaining the leaked value, then creates a clean replacement ledger that + omits the sensitive event lines and includes the canonical privacy-safe + `purge_tombstone` defined in the evidence contract. The replacement ledger + is flushed and atomically installed; SQLite is deleted/rebuilt from it and + targeted artifacts/temp/exports are removed. If any required replacement or + deletion cannot be completed, the purge remains visibly failed and evidence + projection is disabled rather than serving stale compatibility state. +- Projection applies purge tombstones before ordinary invalidation/supersession. + Targeted events contribute no `CLAIMED`, `PROBED`, `VERIFIED`, `DEGRADED`, or + `UNSUPPORTED` state; targeted artifacts surface only + `purged_unavailable`. A previously cached verdict that depended on purged + material is invalid and must not survive SQLite rebuild. +- A purge tombstone stores only event IDs/artifact digests, the fixed + `sensitive_evidence` taxonomy, time/producer metadata, and closed action + names. It never records the leaked value, raw path, credential, prompt, or + identifying diagnostic. Append-only semantics never override the duty to + remove sensitive bytes. + +## 8. Security acceptance tests required later + +Before any live runner ships, tests must prove: + +1. prompt/repository/MCP/user-tool inputs are unreachable from the scenario DSL; +2. every admitted fixture has the required synthetic marker/provenance and a + runtime/user/repository/MCP/network object cannot self-assert synthetic + status; +3. redirects and model-supplied URLs cannot widen network access, and Lab + clients pin connections to the validated IP set; +4. the inherited-environment allowlist is empty, all uppercase/lowercase proxy + variables are rejected, and no ambient variable changes Lab behavior; +5. destination-record mutation, replacement, or address-set drift between + authorization, fingerprinting, credential binding, and connect fails closed; +6. credential canaries never enter Lab memory/objects through the broker: the + runner receives only an opaque destination-bound one-run lease, and lease + scope/transport/destination mismatch fails before authorization is sent; +7. account, custom-header and endpoint canaries never enter evidence, errors, + SQLite or artifacts; +8. custom-header canonicalization is deterministic; unknown credential + classification and every count/name/value/aggregate bound fail closed; +9. local subject-salt rotation breaks prior correlation and forces + re-projection/reverification without reverse lookup; +10. tool arguments cannot execute; +11. every wall-clock/connect/first-byte/inactivity/request/input/output/token/ + tool-call/memory/process/artifact hard ceiling is enforced and cannot be + widened by a manifest, config, profile, environment variable, or caller; +12. artifact traversal, symlink/reparse, hard-link, path-race, oversize and + digest attacks fail closed, and validated artifact bytes are consumed from + the same descriptor/handle that was checked; +13. normalized event byte/depth/key/array ceilings fail closed before buffering; +14. timeout, quota, auth, DNS and harness failures remain blockers; +15. retention expiry emits typed unavailable markers; cleanup retry/failure is + visible and bounded; shared contract artifacts survive invalidation while + any usable event still references them; +16. ordinary invalidations reject unknown/future/cross-kind/partial target + lists and deterministically remove only valid named evidence; +17. confirmed sensitive evidence is removed from JSONL, SQLite, artifacts, + temp files and exports; the replacement ledger retains only a privacy-safe + purge tombstone; replay cannot preserve a verdict that depended on purged + evidence; +18. public export rejects unknown/private fields; +19. no probe runs from the production routing path. diff --git a/devlog/_plan/260807_compatibility_lab/050_cl00_acceptance_review.md b/devlog/_plan/260807_compatibility_lab/050_cl00_acceptance_review.md new file mode 100644 index 000000000..68a6a0e0b --- /dev/null +++ b/devlog/_plan/260807_compatibility_lab/050_cl00_acceptance_review.md @@ -0,0 +1,334 @@ +# CL-00 independent acceptance review + +Date: 2026-08-08 + +Scope: the complete CL-00 contract set on +`feat/cl-00-compatibility-contracts`, based on +`3ad5bb6bd3f76f6879d84b78ea39edd3e01ec296`. The initial delayed-review +acceptance was recorded at `12e50a3502fb4af25283538cc717ead2291edd8b`. +CodeRabbit re-reviews were validated against current production code and the +contract authority before any change was accepted. This record supersedes the +stale acceptance/status statements from the earlier passes. + +The review is contract-focused and separate from the original authoring pass. +Every validated Critical, High, and Medium finding was corrected. Valid +deterministic and security-contract defects were also corrected regardless of +review label. CL-02 was not started. + +## Findings and corrections + +### Critical + +None. + +### High + +1. Initial scenario prose did not define executable selector/operator semantics + or canonical per-case manifests. + - Correction: added the closed assertion/selector/SSE contract in `021` and + the 35-case machine-readable authority in `022`, with literal fixtures, + expected values, row-specific requirements, roles, media types, limits, + artifact policy, and failure rules. +2. Immutable observations lacked enough manifest/fixture provenance to + reproduce `VERIFIED`. + - Correction: observations now carry scenario, suite, and fixture digests; + domain-separated digest preimages are exact; referenced manifests and + fixtures are retained content-addressably and cannot be replaced by the + current version during replay. +3. Route identity did not close compatibility-version and sidecar-dependent + behavior. + - Correction: froze the compatibility-version manifest/preimage, included + effective runtime and sidecar settings, added flat dependency identities, + and kept route-local endpoints in the composite subject rather than the + provider-independent scenario manifest. +4. Response-only protocol vectors did not identify an initiating client + request. + - Correction: added 11 explicit initiating request fixtures. All + `upstream_response` cases now have one request that fixes model, input, + stream mode, and inbound surface. +5. Named verifier values had no deterministic derivation. + - Correction: defined every V1 verifier as a closed pure function over the + current synthetic fixture and normalized observation. +6. Catalogue prose and incident mappings initially implied coverage beyond the + literal V1 assertions. + - Correction: narrowed every protocol V1 row to its exact `022` evidence and + made incident mappings explicit future scenario/version inputs when no + literal V1 vector exists. +7. The three evidence layers lacked separate executable subject identities, + and suites could span layers. + - Correction: added the closed `ProtocolSubjectV1`, `RouteSubjectV1`, and + `TaskSubjectV1` union, exact layer/subject matching, layer-qualified suite + manifests, and one reserved deterministic Fabric task/verifier contract. +8. Behavior identity did not explicitly close effective + `commandCodeVersion`, sampling-parameter omission sets, `cacheRetention`, or + unknown future behavior inputs. + - Correction: froze `BehaviorFingerprintV1`, its closed keys and source + tags, required values from the production resolver, and fail-closed + handling/tests for an unclassified behavior input. +9. A global protocol failure-rule list attached both control effects to every + case, violating the legal control matrix. + - Correction: the base rule set contains no control rule; expansion adds + exactly one materialized rule only for a case with `expectedFailure`. + Protocol V1 has one conformance control and no unsupported effect. +10. Tool/MCP assertions selected bare calls from the normalized SSE event array. + - Correction: froze separate canonical `toolCalls[]` and `mcpCalls[]` + semantic projections and moved all call/correlation selectors to them. +11. Chat-backed Responses cases expected adapter `done` instead of the + client-visible `completed` terminal, and parallel-tool counting still used + nonexistent `tool_call` SSE events. + - Correction: those terminals now expect `completed`; parallel count/order + derive from `/client/response/toolCalls`, and `toolCalls`/`mcpCalls` are + part of the closed observation schema. + +### Medium + +1. `VERIFIED -> PROBED` was missing after partial invalidation. + - Correction: added the transition for remaining partial coverage. +2. Scenario and suite freshness authorities conflicted. + - Correction: effective age is the minimum finite scenario, suite, and + profile bound. +3. Compatibility-version file hashing and dirty/missing/symlink behavior were + underspecified. + - Correction: froze the canonical object, file set, raw-byte hashes, sort + order, current-working-tree behavior, and fail-closed cases. +4. Sidecar network wording incorrectly put route-local endpoints in scenario + manifests. + - Correction: manifests authorize only dependency roles/protocol classes; + the composite subject owns exact destination fingerprints. +5. The MCP exact-bound vector was not actually at its stated boundary. + - Correction: replaced it with exact 64-byte and 65-byte UTF-8 JSON schema + payloads and a recomputed fixture digest. +6. The vision modality control could have made a compatible suite + `UNSUPPORTED`. + - Correction: made it a `negative_control`; its exact rejection satisfies + the suite without projecting route-level `UNSUPPORTED`. +7. The compaction assertion tested presence rather than truth. + - Correction: changed it to exact equality with `true`. +8. One result-content description claimed call correlation absent from its + assertions. + - Correction: removed the claim. +9. Environmental failure effects, claim supersession/currentness, and + custom-header fingerprint ownership were not mechanically closed. + - Correction: added the exhaustive class/effect matrix, formal + `claim_snapshot`/`supersedes[]` schema and currentness algorithm, and a + config-owner broker that exposes only a domain-separated header digest. +10. `ProtocolSubjectV1` named a second `runtimeFingerprint` without a schema. + - Correction: removed it; the closed `runtime.*` behavior keys are the sole + platform-sensitive identity inputs. + +### Low + +- Corrected the provider-test description: forward/static providers do not + always perform a live `/models` request. +- Corrected historical reference `#745` from issue to pull request. +- Added `021`/`022` to the stack ledger and created this review record, closing + all local document links. +- Corrected request-history evidence from “immutable” to canonical + append-only, limited the profile claim to compatibility policy, and noted the + explicit state-mutating `ocx doctor --fix-codex-runtime` mode. + +## CodeRabbit remediation: first pass + +All ten unresolved CodeRabbit threads in the first remediation pass were +inspected against current branch code/contracts before editing. + +1. Stack audit metadata used a non-SHA dependency label where an exact CL-01 + base revision is required. The stack ledger now records exact base/head + revisions and the CL-01 correction requirement. +2. `BehaviorFingerprintV1` did not define deterministic ordering for every + array-valued closed key. V1 now classifies each allowed array as `set` or + `ordered`, defines JCS-byte sorting/deduplication for sets, preserves source + order for ordered arrays, and fails closed for any undeclared array input. +3. `all-applicable-required-pass-v1` admitted a vacuous `VERIFIED` result when + zero required scenarios applied. Positive executable verdicts now require a + non-empty applicable required set; zero-applicable falls through to current + claim/unknown semantics, while attempted environmental blockers remain + `BLOCKED`. +4. `[DONE]` handling was selected by the client-facing surface even for a Chat + upstream fixture. Sentinel interpretation now follows the protocol of the + byte stream being normalized; only OpenAI Chat recognizes exact `[DONE]`. +5. Two Chat-backed tool-result assertions incorrectly selected Responses + `input[].call_id`. Production `openai-chat` emits the continuation as + `messages[1].tool_call_id`; both selectors now assert the actual Chat wire. +6. Live-probe destination authorization could drift between endpoint + fingerprinting, credential binding and connect. The security contract now + requires one immutable per-run `LabDestinationV1` snapshot for every stage + and fails closed before credential transmission on mutation/re-resolution or + mismatch. +7. Ambient environment/proxy handling was not executable enough. The inherited + environment allowlist is empty; the runner constructs only `TZ=UTC` and + `NO_COLOR=1`, rejects uppercase/lowercase proxy variables, and cannot derive + behavior from ambient variables. +8. Custom-header fingerprinting lacked resource/canonicalization bounds. The + broker now enforces entry, duplicate-value, field-name, per-value and + aggregate byte ceilings before JCS/HMAC, with unknown credential + classification or overflow failing closed. +9. Ordinary invalidation wording allowed deletion of shared contract artifacts. + Only event-private non-contract artifacts may be deleted after invalidation; + shared scenario/suite/fixture artifacts survive until no usable observation + references them. +10. Security acceptance coverage omitted the new invariants. Required tests now + cover environment/proxy denial, destination snapshot mutation/address drift, + custom-header canonicalization and bounds, subject-salt rotation, and + retention expiry/cleanup/unavailable markers. + +## CodeRabbit remediation: second pass + +CodeRabbit reviewed the remediation again and raised additional deterministic +and security-contract issues. Every Major finding in that pass was validated as +material and corrected within CL-00: + +1. `invalidation` had no executable payload. It now has a non-empty bounded, + sorted/unique target-event set, a closed reason set, all-or-nothing target + validation, earlier-event/type constraints, and no implicit uninvalidation. +2. `sourceManifestDigest` was not reproducible. `ClaimSourceManifestV1` now + defines a closed sanitized source snapshot, a domain-separated digest, + content-addressed retention, and replay validation. Missing/mismatched source + bytes cannot produce `CLAIMED`. +3. Sidecar dependency sorting omitted `providerInstanceFingerprint`, so two + otherwise-equal dependencies could compare as equal. The canonical total + ordering now includes it. +4. Synthetic fixture trust was prose-only. Every expanded protocol V1 + `fixtureRef` now includes mandatory `ocx-lab-synthetic-v1` marker and + `lab_authored` provenance bound to the authority and source commit; these + fields participate in the scenario-manifest digest and are validated before + fixture admission. +5. `synthetic_tool` did not say what MCP cases execute. The four MCP V1 cases + now carry exact scenario-specific action tokens with closed fixture schemas, + deterministic invocation/list/read/boundary behavior, and fail-closed + registration for missing/wrong/multiple actions. +6. The credential broker could still expose the selected secret to Lab code. + The contract now keeps secret bytes in trusted credential/transport plumbing + and gives the Lab only a non-serializable, destination/auth-transport-bound, + one-run/request `LabCredentialLeaseV1` capability. +7. Required execution limits had no hard maxima. V1 now freezes hard ceilings + for wall/connect/first-byte/inactivity time, requests, aggregate input/output + bytes, output tokens, tool calls, resident memory, child processes and + artifacts; manifests/config/profiles/environment/callers can only tighten + them. +8. Artifact path validation was vulnerable to path-race/symlink substitution if + implemented like the unrelated current image-artifact helper. The future Lab + store is now required to use trusted-directory-handle, no-follow, + descriptor-bound validation/read/write and atomic publication. CL-00 did not + alter `src/images/artifacts.ts`; that runtime is outside this contract-only + phase. +9. Physical sensitive-evidence purge did not define canonical projection state. + The ledger now defines privacy-safe `purge_tombstone` events, clean atomic + ledger replacement, SQLite rebuild/removal, typed `purged_unavailable` + artifacts, and mandatory exclusion of every verdict/claim that depended on + purged evidence. + +The protocol-authority rewrite also fixed the flagged missing final newline. +Two remaining review notes were editorial-only (`falsey` -> `falsy` in the +incident prose and a master-plan acceptance phrase); they do not change any +contract, deterministic behavior, security boundary, acceptance criterion, or +CL-01 implementation input and are handled as non-blocking review-thread +responses rather than expanding this contract remediation. + +## Mechanical review evidence + +- `022_protocol_v1_cases.json` remains valid JSON by inspection through the + GitHub file API and contains the same 35 case objects / 46 fixture records as + the accepted authority. +- Fixture `bytesUtf8` and fixture digest values were not changed by the + CodeRabbit selector/provenance/MCP-action remediation. +- The two corrected Chat continuation selectors target + `/upstream/requests/1/json/messages/1/tool_call_id`, matching the current + `openai-chat` request builder's assistant-call then tool-result message order. +- `fixtureRef` expansion now adds mandatory marker/provenance fields. Therefore + all expanded protocol scenario-manifest digests and dependent suite-manifest + digests change even though fixture bytes/digests remain unchanged. +- Four MCP `requiredHarnessFeatures` arrays now additionally contain their exact + closed action token. Those four scenario-manifest digests therefore also + change for semantic reasons. +- `vision-core.protocol.modality-gate` remains the sole V1 negative control. +- Base failure rules contain no control effect; the vision case alone expands + the conformance-control rule. +- The MCP bound fixture remains the accepted exact 64/65 UTF-8-byte vector. + +## Repository verification + +Initial acceptance verification remains the last executed local-suite evidence: + +- `bun run typecheck`: passed. +- `bun run privacy:scan`: passed. +- `bun test tests/repo-hygiene.test.ts`: 11 passed, 0 failed. +- Focused protocol/compatibility suite excluding Windows privileged-symlink + state cases: 395 passed, 0 failed across 24 files. +- Focused continuation-state semantics: 2 passed, 95 filtered, 0 failed. +- `tests/codex-models-cache-invalidate.test.ts`: 6 passed, 0 failed. +- `tests/codex-native-residue.test.ts`: 63 passed, 2 platform skips, 0 failed. +- Original local link/case/digest checks and `git diff --check`: passed. + +The CodeRabbit remediation is documentation/contract-only. The GitHub connector +does not provide a local Bun execution environment, so this review does **not** +claim a new typecheck/privacy/test run after these documentation changes. Final +GitHub status/workflow contexts and unresolved review threads are checked after +the status-ledger sync. + +The earlier full `bun run test` result was **not green**. On Windows with Bun +1.3.14 it exited 3 after a cache-invalidation failure, an empty effective-account +lookup, and a Bun `index out of bounds` panic. A broader focused run separately +found four `responses-state.test.ts` failures, all Windows `EPERM` errors +creating symlinks (488 passed, 4 failed). The isolated cache/native tests and +the non-privileged protocol suite passed; this review does not claim the full +suite passed. + +## Required challenge results + +1. Protocol conformance, live compatibility, and task effectiveness are + separated: **PASS**. +2. Environmental failures cannot poison compatibility verdicts: **PASS**. +3. `VERIFIED` is reproducible, non-vacuous, and invalidation/purge aware: + **PASS**. +4. Exact route/dependency identity prevents false evidence reuse: **PASS**. +5. `CLAIMED` is reproducible from retained sanitized source manifests: + **PASS**. +6. Routing Profiles remain the sole compatibility-policy surface: **PASS**. +7. The Lab cannot become a second router or provider registry: **PASS**. +8. Synthetic-fixture admission, credentials, destinations, environment, + resources, artifacts and purge behavior are fail-closed: **PASS**. +9. Historical incidents remain representable as deterministic versioned + scenarios: **PASS**. +10. CL-01 remains implementable without semantic invention after synchronizing + the refreshed V1 authority: **PASS WITH REQUIRED CL-01 REBASE, CORRECTION, + AND REVALIDATION**. + +## CL-01 impact + +The independently accepted CL-01 branch exists at +`feat/cl-01-conformance-harness` at accepted head +`cc447ce9d19d5fb4e03988899f5fb495f9de8d0e`. It was built from the older CL-00 +revision `c2113ca47b8a05c5a5f90679e4eaa640ca2c6a66`. + +Its acceptance record explicitly documents a harness-only projection of +Chat-wire `messages` tool rows into a synthetic Responses-shaped `input[]` to +satisfy the old CL-00 selectors. That workaround is no longer authoritative: +CL-00 selects the actual Chat `messages[].tool_call_id` field. CL-01 also copied +the pre-remediation case authority and its SSE helper retained client-surface +sentinel selection. + +Before CL-01 is stacked or merged it must therefore: + +- rebase onto the final refreshed CL-00 accepted contract head; +- synchronize the two corrected Chat result selectors; +- remove or narrow the synthetic Chat-to-Responses `input[]` observation + projection so upstream observations remain the actual Chat request; +- align SSE normalization with source-protocol `[DONE]` selection; +- implement/validate the mandatory synthetic fixture marker/provenance in + expanded `fixtureRef` values and recompute all scenario/suite manifests; +- synchronize the four exact MCP action tokens and their closed execution + semantics; and +- rerun the canonical CL-01 scenarios, negative controls, digest/manifest + checks and independent CL-01 acceptance review. + +This CL-00 remediation does not modify CL-01 and does not start CL-02. + +## Verdict + +All validated Critical, High, Medium, deterministic-contract and +security-contract findings found through the two CodeRabbit remediation passes +are corrected in the CL-00 contract set. Final GitHub thread/status checks are +recorded after the stack ledger is synchronized. + +**CL-00: ACCEPTED AFTER CODERABBIT REMEDIATION**