diff --git a/devlog/_plan/260807_compatibility_lab/000_master_plan.md b/devlog/_plan/260807_compatibility_lab/000_master_plan.md new file mode 100644 index 0000000000..f9ee5b8334 --- /dev/null +++ b/devlog/_plan/260807_compatibility_lab/000_master_plan.md @@ -0,0 +1,306 @@ +# OpenCodex Compatibility Lab / EvalGrid + +Status: CL-00 architecture authority +Authority baseline: `upstream/dev` at `3ad5bb6bd3f76f6879d84b78ea39edd3e01ec296` +Package/runtime at baseline: OpenCodex `2.10.2`, Bun `1.3.14` + +## Purpose + +Compatibility Lab turns compatibility claims into bounded, reproducible +evidence. It tests OpenCodex protocol behavior, exact configured routes, and +later execution-grounded task outcomes without becoming a provider registry, a +user-policy system, or a production router. + +This directory is the programme authority. The contracts frozen by CL-00 are: + +- [Architecture and evidence](./010_architecture_and_evidence_contract.md) +- [Scenario model and initial catalogue](./020_scenario_contract_and_catalogue.md) +- [Protocol V1 manifest authority](./021_protocol_v1_manifest_authority.md) +- [Protocol V1 canonical cases](./022_protocol_v1_cases.json) +- [Historical incident corpus](./030_incident_corpus.md) +- [Security and privacy](./040_security_and_privacy.md) +- [CL-00 independent acceptance review](./050_cl00_acceptance_review.md) +- [PR stack status](./001_pr_stack_status.md) + +Later phases may add implementation detail, but must amend these contracts +explicitly rather than silently changing their meaning. + +## Live repository truth + +CL-00 audited the live `dev` tree before defining new authority. + +### Shipped and authoritative + +- Provider declarations and model metadata: + `src/providers/registry.ts`, `src/providers/derive.ts`, `src/types.ts`; + route-time claimed capability assembly in `src/routing/capability.ts` also + consumes provider config, cached Codex catalog rows and native metadata. + Generated fallback metadata lives in `src/generated/model-metadata.ts` and is + sourced by `scripts/model-metadata.source.json`. +- Routing Profile public/config types: `OcxRoutingProfileConfig` in + `src/types.ts`; validation, normalization, revision hashing, persistence and + resolution in `src/routing/profile.ts`. +- Deterministic profile evaluation and route traces: + `src/routing/evaluator.ts`, `src/routing/trace.ts`, and + `src/router.ts`. +- Profile management CRUD and dry-run: + `src/server/management/routing-profile-routes.ts`. +- Dashboard profile editor, dry-run and routing analytics: + `gui/src/pages/RoutingProfiles.tsx`, mounted under Models -> Routing. +- Canonical append-only request/usage evidence and rebuildable history + projection: + `src/usage/log.ts`, `src/routing/history/indexer.ts`, and + `src/routing/analytics.ts`. +- Why-this-route evidence: + `RouteDecisionTraceV1`, request-history explain endpoints in + `src/server/management/request-history-routes.ts`, and CLI explain support. + The GUI Logs modal renders only a compact route summary through + `gui/src/pages/log-route-decision.ts`; there is no GUI request-history browser + or full trace + attempts + outcome view on this baseline. +- Existing diagnostics are narrower than Compatibility Lab: + the default `ocx doctor` path is observe-only environment/OAuth/runtime + diagnosis; explicit `--fix-codex-runtime` may persist a repair. In contrast, + `POST /api/providers/test` performs a bounded live `/models` connectivity + check only when applicable; forward providers return configured status and + static catalogues return not-applicable without network access. +- Protocol behavior is already covered by many focused tests under `tests/`, + but those tests are not a versioned scenario catalogue or evidence ledger. + +### Not shipped + +- No Compatibility Lab runner, scenario registry, evidence ledger, SQLite + projection, CLI, management API, or UI exists. +- Generated Cursor agent protobufs include task/grind/subagent message types, + but OpenCodex has no native Agent Fabric task persistence, harness handoff, + portable task-state model, or management API. Router Intelligence's own + master plan explicitly excluded Agent Fabric. +- No Routing Profile compatibility fields exist. + +Current routing nuances that later phases must preserve rather than +over-describe: + +- selection traces and execution attempts are separate; the explain API merges + trace + `attempts[]` + final outcome at read time; +- `optimize.latency` is currently a declaration-priority share, while observed + latency contributes through health evidence rather than an independent + top-level score; +- cost evidence is commonly unknown on the live pre-dispatch path because + request usage is not yet available; +- profile dry-run is evaluation-only and never dispatches upstream; +- an unknown canonical `policy/` currently falls through to ordinary model + routing rather than failing closed. + +Consequently, CL-00 defines future contracts and integration seams only. It +does not rename existing Router Intelligence concepts or describe speculative +Agent Fabric endpoints as current behavior. + +## Architectural invariant + +```text +Provider Registry + ↓ +Compatibility Lab + ↓ +Compatibility Graph / Verified Evidence + ↓ +Routing Profiles + ↓ +Router Intelligence + ↓ +Selected Model / Route + ↓ +Agent Fabric / Real Execution + └──────────────→ execution-grounded outcomes back to Lab +``` + +The arrows are data dependencies, not ownership transfers. + +### Provider Registry + +The Provider Registry declares what a provider/model is believed to support and +supplies defaults used to construct an effective route. The shipped +`candidateCapabilityEvidence()` also combines explicit provider config, cached +catalog rows, adapter-level inference and native-model metadata. These local +declarations are claims. They may seed `CLAIMED`; they cannot by themselves +produce `PROBED`, `VERIFIED`, `DEGRADED`, or `UNSUPPORTED`. + +The Lab may snapshot a registry claim with its source revision for +reproducibility. It must not create a parallel provider catalogue or write +provider declarations back into the registry. + +Registry-owned runtime defaults such as model wire selection, discovery policy, +upstream streaming, reasoning replay, service-tier support, and item-ID repair +are intentionally not all persisted to `config.json`. Claim snapshots capture +the effective sources; they do not freeze runtime defaults into user config. + +### Compatibility Lab + +The Lab owns versioned scenarios, immutable compatibility evidence, failure +attribution, freshness, derived verdicts, and regression history. It may +project evidence into a compatibility graph keyed by exact route subject and +evidence layer. + +The Lab never chooses a production candidate, mutates a Routing Profile, +changes provider metadata, or turns a probe result directly into a route. + +### Routing Profiles + +Routing Profiles remain the sole compatibility-policy surface. Future +compatibility requirements extend `OcxRoutingProfileConfig`, its +normalizer/revision, the existing evaluator, the existing management +CRUD/dry-run endpoints, and the Models dashboard editor. Existing combo and +account-pool controls retain their separate non-compatibility responsibilities. +There will be no compatibility-specific profile store, evaluator, or editor. + +### Router Intelligence + +Router Intelligence combines the selected profile with current capability, +compatibility, health, quota, cost, and latency evidence and makes the +deterministic route decision. Its existing `RouteDecisionTraceV1` remains the +authority for explaining that decision. Compatibility inputs will later add +bounded evidence to that trace rather than introduce a second explanation +record. + +### Agent Fabric + +Agent Fabric is a future producer of task-effectiveness observations. It owns +real task execution and its sandbox. The Lab accepts only structured outcome +data and sanitized content-addressed artifact references; it does not copy task +repositories, prompts, worktrees, or hidden reasoning. + +Because a native Agent Fabric is not present on the CL-00 baseline, this +programme freezes the consumer semantics, not a fictitious production API. A +later producer contract must identify its schema version, task class, exact +route subject, deterministic verifier results, timing, resource limits, +outcome, and sanitized artifact references. Existing request-grounded evidence +may be linked through `RouteDecisionTraceV1`, `PersistedUsageAttempt`, and the +final request outcome; prompt-bearing `responses-state.json` and generated +Cursor task protobufs are not Lab feeds. + +## Evidence-layer invariant + +Every scenario and observation has exactly one layer: + +1. `protocol_conformance`: whether OpenCodex translates and preserves a + protocol contract correctly. +2. `live_route_compatibility`: whether an exact + provider/model/adapter/configuration route works now. +3. `task_effectiveness`: whether that route produces verifier-confirmed + outcomes for a versioned class of coding work. + +Verdicts are projected per `(subject, layer, suite)`. Evidence from one layer +may be shown as a prerequisite or correlated signal, but cannot promote or +degrade another layer's verdict. There is no universal compatibility score. + +## Persistence authority + +Future implementation uses the existing OpenCodex config root returned by +`getConfigDir()` (`OPENCODEX_HOME`, default `~/.opencodex`) and owns: + +```text +~/.opencodex/lab/ + compatibility.jsonl + compatibility.sqlite + artifacts/ +``` + +- `compatibility.jsonl` is the canonical append-only evidence/event ledger. +- `compatibility.sqlite` is a disposable query projection rebuilt from JSONL. +- `artifacts/` contains bounded, sanitized, content-addressed artifacts. +- Scenario/suite manifests and synthetic fixture/source anchors are + content-addressed contract artifacts retained with the observations that + reference them. +- Verdicts are derived projections, never mutable canonical booleans. +- Corrections append invalidation events or a new claim snapshot with explicit + `supersedes[]`; prior bytes are not edited. +- The Lab does not copy `usage.jsonl` or routing-history rows. When useful, an + observation references an existing request ID or route decision ID. +- Agent Fabric supplies structured outcome data/references, never repositories + or prompt transcripts. + +This location follows current repository state-root conventions. No filename +or location change from the proposed architecture was justified by the audit. + +## Routing Profiles boundary for CL-06 + +CL-06 must add optional compatibility controls alongside existing capability, +health, quota, cost, and latency policy: + +- required compatibility suites; +- minimum compatibility status; +- maximum evidence age; +- unknown-evidence behavior; +- degraded-evidence behavior. + +`minimum compatibility status` is not a total ordering across all verdicts. +Only `PROBED` and `VERIFIED` are positive thresholds. `DEGRADED` is governed by +its explicit behavior, `UNKNOWN`/`CLAIMED`/`BLOCKED` by unknown-evidence +behavior, and `UNSUPPORTED` fails a required suite. + +The exact future flow is: + +```text +Routing Profile + ↓ +Configured candidates + ↓ +Hard capability gates + ↓ +Compatibility requirements / penalties + ↓ +Eligible candidates + ↓ +Health / quota / cost / latency scoring + ↓ +Deterministic winner +``` + +All compatibility fields are optional. Profiles that omit them retain their +current validation, revision, eligibility and scoring behavior. A profile +evaluation reads an existing projection only. No compatibility probe, network +request, task, or projection rebuild may run synchronously on the production +request path. + +## Programme phases + +Only CL-00 is authorized by this document at present. + +| Phase | Purpose | Authorization | +|---|---|---| +| CL-00 | Architecture authority, contracts, scenario catalogue, incident corpus | This PR | +| CL-01 | Deterministic protocol-conformance runner and fixtures | Not started; requires CL-00 to be accepted | +| CL-02 | Immutable JSONL ledger, artifacts and SQLite projection | Not started | +| CL-03 | Bounded live-route probes | Not started | +| CL-04 | Lab CLI and management read surfaces | Not started | +| CL-05 | Compatibility Matrix UI | Not started | +| CL-06 | Existing Routing Profile compatibility controls and Router Intelligence consumption | Not started | +| CL-07 | Agent Fabric task-effectiveness ingestion | Not started | +| CL-08 | Shadow/automatic/public evidence workflows | Not started | + +Phase numbering after CL-01 is programme planning, not implementation +authorization. A later accepted plan may split a phase while preserving these +ownership boundaries. + +## CL-00 acceptance criteria + +CL-00 is accepted only when: + +1. all three evidence layers have separate subjects, scenarios and verdicts; +2. every canonical verdict is reproducible from immutable inputs; +3. environmental blockers cannot poison compatibility conclusions; +4. exact route identity prevents evidence reuse across behavior changes; +5. scenario semantics and initial IDs are implementable without an LLM judge; +6. representative historical incidents map to abstract regression scenarios; +7. future compatibility policy extends existing Routing Profiles; +8. probes and task execution are excluded from production request routing; +9. privacy and sandbox ceilings are explicit; +10. an independent review finds no unresolved Critical, High, or Medium issue. + +## CL-00 hard stop + +This phase does not implement a runner, mock upstream, persistence code, live +probe, CLI, management endpoint, UI, profile field, routing behavior, shadow +route, Fabric ingestion, automatic routing, or public publisher. + +Acceptance of CL-00 authorizes discussion and planning of CL-01; it does not +start CL-01 automatically. diff --git a/devlog/_plan/260807_compatibility_lab/001_pr_stack_status.md b/devlog/_plan/260807_compatibility_lab/001_pr_stack_status.md new file mode 100644 index 0000000000..678e023c2e --- /dev/null +++ b/devlog/_plan/260807_compatibility_lab/001_pr_stack_status.md @@ -0,0 +1,95 @@ +# Compatibility Lab PR stack status + +Updated throughout the programme. Every phase records its branch, exact +starting/base revision, accepted contract/implementation head, PR, verification, +independent review, blockers, and whether a later phase is authorized. + +## Programme facts + +- Repository: `lidge-jun/opencodex` +- Integration target: `dev` +- CL-00 starting `upstream/dev`: + `3ad5bb6bd3f76f6879d84b78ea39edd3e01ec296` +- Package/runtime at start: OpenCodex `2.10.2`, Bun `1.3.14` +- CL-00 branch: `feat/cl-00-compatibility-contracts` +- CL-00 scope: documentation/contracts/incident corpus only +- PR target: `lidge-jun/opencodex:dev` + +## Stack + +| Phase | Branch | Starting/base SHA | Accepted head | PR | State | +|---|---|---|---|---|---| +| CL-00 | `feat/cl-00-compatibility-contracts` | `3ad5bb6bd3f76f6879d84b78ea39edd3e01ec296` | `c014464237fd3c95bda08bc18bfab8ba8f532308` | [#1286](https://github.com/lidge-jun/opencodex/pull/1286) | ACCEPTED AFTER CODERABBIT REMEDIATION (merged to `dev` at `243c3f4905797aa11c62ba933bb03d6d721266fd`) | +| CL-01 | `feat/cl-01-conformance-harness` | `c2113ca47b8a05c5a5f90679e4eaa640ca2c6a66` | `22d608c82d82e2746c0cef9cd761db19a8e465ee` | [draft Wibias #10](https://github.com/Wibias/opencodex/pull/10) | ACCEPTED (contract-corrected revalidation) | + +The CL-01 starting SHA is the exact CL-00 tip recorded when CL-01 began. Its +moving base-ref name is not a substitute for that historical SHA. + +## CL-00 acceptance log + +- Live-tree audit covered provider registry/derivation, Routing Profiles, + routing traces, request history/analytics, doctor/connectivity validation, + protocol regression tests, and relevant incident/devlog records. +- CL-00 remains contract-only. No Compatibility Lab runtime, live runner, + profile/router implementation, or CL-02 work was added. +- Contract documents: + - `000_master_plan.md` + - `010_architecture_and_evidence_contract.md` + - `020_scenario_contract_and_catalogue.md` + - `021_protocol_v1_manifest_authority.md` + - `022_protocol_v1_cases.json` + - `030_incident_corpus.md` + - `040_security_and_privacy.md` + - `050_cl00_acceptance_review.md` +- Original baseline verification: + - `bun x tsc --noEmit`: passed. + - `bun run privacy:scan`: passed. + - `tests/repo-hygiene.test.ts`: 11 passed, 0 failed. + - focused protocol/compatibility: 395 passed, 0 failed across 24 files. + - continuation semantics: 2 passed, 95 filtered, 0 failed. + - isolated cache invalidation: 6 passed, 0 failed. + - isolated native residue: 63 passed, 2 platform skips, 0 failed. + - full `bun run test` was not green on the Windows/Bun 1.3.14 host for the + previously documented cache/account/Bun panic failures; a broader + `responses-state` run also had four Windows `EPERM` symlink failures. + +Independent CL-00 acceptance review is frozen at +`c014464237fd3c95bda08bc18bfab8ba8f532308`. Merged to `dev` via #1286. + +## CL-01 contract-correction log (2026-08-09) + +- **Pre-rebase CL-01 head:** `cc447ce9d19d5fb4e03988899f5fb495f9de8d0e` (earlier accepted revision) +- **CL-00 merge base on `dev`:** `243c3f4905797aa11c62ba933bb03d6d721266fd` +- **Post-rebase harness commit:** `cfe27b0dcb26a1bf0bb56f68f952e6e4f4d80fe9` (rebase-only) +- **Correction head:** `574f1d5eb93c091494549ffc0e26ea7a4879c12c` (implementation); **tip:** `22d608c82d82e2746c0cef9cd761db19a8e465ee` + +### Corrections applied + +1. Rebased onto merged CL-00 / #1286 (`243c3f490`). +2. Synced `022_protocol_v1_cases.json` runtime copy with final CL-00 authority. +3. Removed Chat → Responses `input[]` observation projection. +4. Chat tool-result selectors: `/upstream/requests/1/json/messages/1/tool_call_id` for function-round-trip and apply-patch-turn. +5. SSE `[DONE]` normalization keyed by source protocol (`openai-chat` only). +6. Mandatory synthetic fixture marker/provenance in expanded manifests; fail-closed validation. +7. Four deterministic MCP action tokens in `mcp-stub.ts`. +8. Recomputed scenario manifest digests (provenance participates in JCS expansion). +9. Narrow image tool-result wire normalization for `tools-core.protocol.result-content` (indices only). +10. `openai-chat.ts`: `toolResultTextForWire` omits `[image]` marker when images are flushed to user carrier. + +### Verification (correction) + +- `bun x tsc --noEmit`: passed +- `bun test tests/lab-conformance-harness.test.ts`: 14/14 passed +- `git diff --check`: passed +- Independent review: `051_cl01_acceptance_review.md` — ACCEPTED (revalidation) + +### Blockers + +- None for CL-01 correction. +- Full-suite green remains unavailable on this host for documented Windows/Bun reasons. + +## Authorization + +- CL-00: **ACCEPTED** (merged #1286). +- CL-01: **ACCEPTED (contract-corrected revalidation)** — ready for stack review against `dev`. +- CL-02: **NOT STARTED / NOT AUTHORIZED**. diff --git a/devlog/_plan/260807_compatibility_lab/010_architecture_and_evidence_contract.md b/devlog/_plan/260807_compatibility_lab/010_architecture_and_evidence_contract.md new file mode 100644 index 0000000000..f48166fb00 --- /dev/null +++ b/devlog/_plan/260807_compatibility_lab/010_architecture_and_evidence_contract.md @@ -0,0 +1,890 @@ +# CL-00 architecture and evidence contract + +This document freezes the semantic model consumed by later Compatibility Lab +phases. Names shown in code blocks are contract names, not claims that +production TypeScript types already exist. + +## 1. Evidence layers + +### `protocol_conformance` + +Question: does this OpenCodex build preserve the declared inbound-to-upstream +and upstream-to-client protocol contract? + +Inputs are deterministic fixture requests and deterministic mock-upstream +responses. The subject includes the OpenCodex compatibility version, adapter, +inbound protocol, upstream protocol, surface, and relevant behavior +fingerprint. A real provider account is neither required nor permitted. + +A pass proves only the exercised OpenCodex translation. It says nothing about a +provider's current availability or a model's coding quality. + +### `live_route_compatibility` + +Question: does this exact configured route satisfy this versioned scenario now? + +The subject is an exact provider/model/effective-adapter/configuration route. +The run may contact only that route's configured upstream under the Lab +sandbox. A pass cannot be reused for a different route fingerprint. + +A pass proves only the exercised route behavior at the observation time. It +does not prove task effectiveness. + +### `task_effectiveness` + +Question: did this exact route produce a successful, deterministically verified +outcome for a versioned task class? + +Agent Fabric, not the Lab, owns real execution. The Lab receives a structured +outcome containing deterministic verifier results and sanitized artifact +references. Human ratings or LLM-judge output may be stored as advisory +annotations in a later phase, but cannot produce a canonical verdict. + +### Non-collapse rule + +Each layer has one executable subject kind: + +```text +protocol_conformance -> ProtocolSubjectV1 +live_route_compatibility -> RouteSubjectV1 +task_effectiveness -> TaskSubjectV1 +``` + +`EvidenceSubjectV1` is that closed discriminated union. An observation whose +layer and `subjectKind` do not match is invalid evidence. Suite manifests also +belong to exactly one evidence layer; a manifest cannot mix protocol, live, and +task scenarios even when their human-facing suite stem is the same. + +The canonical projection key is: + +```text +(subjectId, evidenceLayer, suiteId, suiteVersion, suiteManifestDigest, + projectionSpecVersion) +``` + +There is no projection across all layers and no weighted universal score. +Callers may present multiple layer verdicts next to each other. A prerequisite +failure in one layer may make a later-layer run inapplicable, but it does not +rewrite evidence in the other layer. + +### `ProtocolSubjectV1` + +Protocol evidence identifies OpenCodex translation behavior, not a provider: + +```text +subjectSchemaVersion 1 +subjectKind protocol +opencodexCompatibilityVersion +effectiveAdapter +inboundProtocol +upstreamProtocol +surface +behaviorFingerprint +``` + +The closed `runtime.bunVersion`, `runtime.platform`, and `runtime.arch` behavior +keys cover platform-sensitive paths; there is no second runtime digest. +Provider, credential, account, endpoint, health, quota, cost, and latency +fields are forbidden. + +### `TaskSubjectV1` + +Task evidence identifies one exact route plus one synthetic task/verifier +contract: + +```text +subjectSchemaVersion 1 +subjectKind task +routeSubject RouteSubjectV1 +taskClassId +taskClassVersion +taskFixtureDigest +verifierManifestDigest +fabricCompatibilityVersion +sandboxProfileDigest +``` + +The nested route subject makes execution behavior reproducible. The task +subject never contains repository contents, prompts, user paths, account +identity, or raw artifacts. A changed route, task fixture, verifier, Fabric +runtime, or sandbox profile starts a new task projection. + +## 2. Immutable ledger contract + +The canonical JSONL ledger is a sequence of versioned events. The minimum +event kinds are: + +```text +observation +claim_snapshot +invalidation +purge_tombstone +``` + +An `observation` records one scenario attempt. A `claim_snapshot` captures the +local declared-capability inputs and source revisions needed to reproduce +`CLAIMED`: registry/config, cached catalog or native metadata, including the +adapter inference currently assembled by `src/routing/capability.ts`. An +`invalidation` identifies prior non-sensitive evidence that a later-discovered +harness, fixture, redaction, or integrity defect makes unusable. Invalidations +append; they never delete or edit prior lines. A `purge_tombstone` is the +privacy-safe exceptional record left after confirmed sensitive evidence is +physically removed under the security contract. + +Each event has: + +```text +schemaVersion +eventId +eventKind +recordedAt +producer +producerVersion +``` + +An observation additionally has: + +```text +evidenceLayer +scenarioId +scenarioVersion +scenarioManifestDigest +suiteId +suiteVersion +suiteManifestDigest +fixtureDigests[] +subject +subjectId +startedAt +completedAt +executionMode fixture | live | fabric +attempt +limits +outcome pass | fail | blocked | inconclusive +assertions[] +failure? { class, code, retryable, attribution } +expectedFailure? +environment +artifactRefs[] +sourceRefs? +``` + +A `claim_snapshot` additionally has: + +```text +evidenceLayer live_route_compatibility +subject RouteSubjectV1 +subjectId +capability +polarity supported | not_supported | withdrawn +sourceManifestDigest +sourceEventIds[] +supersedes[] +effectiveAt +``` + +An `invalidation` additionally has: + +```text +targetEventIds[] non-empty, maximum 1024 +reason harness_defect | fixture_defect | + redaction_defect | integrity_defect | + contract_artifact_missing | manual_correction +``` + +Invalidation rules: + +- `targetEventIds` is a set encoded as lowercase event IDs sorted + lexicographically by UTF-8 bytes. Duplicate IDs reject the whole invalidation. +- Every target must be an earlier valid `observation` or `claim_snapshot` in + the same ledger. Unknown, malformed, future, self, `invalidation`, or + `purge_tombstone` targets reject the whole invalidation and report ledger + corruption; projection never applies a partial target list. +- The `reason` set above is closed. Unsupported or missing reasons reject the + invalidation. +- Multiple valid invalidation events may name the same prior target; the target + remains excluded. There is no "uninvalidate" mutation: replacement evidence + is a new observation or claim snapshot. +- Sensitive-evidence deletion does not weaken these rules. It uses the separate + `purge_tombstone` contract below because the sensitive target line may no + longer exist by design. + +A `purge_tombstone` additionally has: + +```text +targetEventIds[] sorted unique lowercase event IDs +targetArtifactDigests[] sorted unique lowercase SHA-256 digests +reason sensitive_evidence +purgeActions[] ledger | sqlite | artifact | scratch | export +``` + +At least one target array is non-empty. The reason is exactly +`sensitive_evidence`; action names are a closed set, sorted and unique. A purge +tombstone may refer to an event line or artifact that has already been +physically removed by the same fail-closed purge. Projection applies valid +purge tombstones before ordinary invalidations, excludes every targeted event, +and treats every targeted artifact as typed `purged_unavailable`. Cached SQLite +rows that depended on a targeted event or artifact are invalid and must be +removed/rebuilt. A prior `VERIFIED`, `PROBED`, `DEGRADED`, `UNSUPPORTED`, or +`CLAIMED` result is never retained solely from purged evidence. + +### `ClaimSourceManifestV1` + +`sourceManifestDigest` is reproducible evidence, not an opaque checksum. Every +claim snapshot references one retained canonical `ClaimSourceManifestV1`: + +```text +{ + "schemaVersion": 1, + "subjectId": string, + "providerId": string, + "clientModelId": string, + "capability": string, + "sources": [ClaimSourceV1, ...], + "resolvedEvidence": RouteCapabilityEvidenceV1 +} +``` + +`ClaimSourceV1` is one record for each consulted source that contributed or +could have contributed to the selected capability: + +```text +{ + "kind": "provider_config" | "provider_registry" | "cached_catalog" | + "native_metadata" | "adapter_inference", + "revision": string | null, + "facts": ClaimCapabilityFactsV1 +} +``` + +`sources` is ordered by the closed kind order shown above and contains at most +one record per kind. `ClaimCapabilityFactsV1` is a closed, sanitized projection +of only capability-relevant inputs used by the current resolver: selected +context window, input modalities, reasoning efforts, catalog capability names, +service-tier support, effective adapter/tool-capable classification, +parallel-tool-call enablement, endpoint locality class +`local|private|unknown`, and canonical-OpenAI-forward classification where +applicable. It never stores a base URL, hostname, credential, account identity, +custom header, arbitrary provider config, catalog path, prompt, or repository +data. `RouteCapabilityEvidenceV1` is the closed resolver output currently +represented by `RouteCapabilityEvidence`: context window, image, tools, +reasoning efforts, service tier, local-only, remote-allowed, and encrypted +Codex-task evidence, with absent fields remaining absent. + +The canonical digest is lowercase: + +```text +sha256( + UTF8("ocx-lab:claim-source-manifest:v1\0") || + UTF8(JCS(ClaimSourceManifestV1)) +) +``` + +The exact canonical bytes are retained as a content-addressed +`claim_source_manifest` artifact while any non-purged claim snapshot references +them. Projection must load those retained bytes, recompute the digest, and +verify matching `subjectId` and `capability` before accepting historical +`CLAIMED` state. Missing bytes, digest mismatch, duplicate/unknown source kinds, +unknown facts, or a subject/capability mismatch makes that claim unusable, +projects no `CLAIMED` result from it, and reports ledger/artifact corruption. + +Claim rules: + +- Claims exist only for `live_route_compatibility`; protocol and task verdicts + cannot be `CLAIMED`. +- `sourceManifestDigest` identifies the canonical, content-addressed snapshot + above of all sanitized registry/config/catalog/native/adapter inputs used by + the existing capability resolver. Raw secrets are forbidden. +- `supersedes` explicitly lists every previously current claim event for the + same `(subjectId, capability)`. The first snapshot uses an empty list. +- A source refresh, removal, or changed polarity appends a new snapshot; it + never edits a prior event. `withdrawn` means the local declaration no longer + exists. `not_supported` suppresses `CLAIMED` but cannot produce + `UNSUPPORTED`. +- Projection first applies purge tombstones and invalidations, then removes + every event named by a valid later snapshot's `supersedes`. Exactly one + unsuperseded claim may remain current for a claim key. Multiple unsuperseded + claims, a missing referenced predecessor, a cross-key supersession, or a + cycle makes claim state `UNKNOWN` and reports ledger corruption. +- Claim ordering is `effectiveAt`, then `recordedAt`, then event ID, but + ordering never substitutes for the explicit supersession graph. + +Rules: + +- Canonical JSON is RFC 8785 JSON Canonicalization Scheme (JCS), encoded as + UTF-8 with no BOM. +- `eventId` is lowercase + `sha256("ocx-lab:event:v1\0" || JCS(event without eventId))`. +- `subjectId` is lowercase + `sha256("ocx-lab:subject:v1\0" || JCS(subject))`. +- Scenario and suite manifests use the same JCS construction with domains + `ocx-lab:scenario-manifest:v1` and `ocx-lab:suite-manifest:v1`. Fixture + digests are lowercase + `sha256("ocx-lab:fixture:v1\0" || exact fixture bytes)`. A manifest's digest + field and storage path are excluded from its preimage. Domain text is UTF-8 + and terminated by one NUL byte. +- Every observation carries the exact scenario, suite and fixture digests it + executed. Version strings without matching digests are invalid evidence. +- Timestamps are UTC epoch milliseconds; duration alone is not sufficient. +- Assertions record expected value/shape, observed normalized value/shape, and + pass/fail. They never require raw prompts or unbounded bodies. +- `failure.attribution` records whether the failure is attributable to + OpenCodex, the exact route, the environment, or the harness. +- Artifact references contain digest, media type, byte count, redaction + policy, and local relative path; never an arbitrary filesystem path. +- `sourceRefs` may contain existing request IDs, route-decision IDs, or future + Fabric outcome IDs. It must not inline the referenced request or task. +- The shipped request-history explain surface already composes selection trace, + `PersistedUsageAttempt[]`, and final outcome. A later Lab consumer should link + that normalized composition instead of reading prompt-bearing + `responses-state.json` or treating generated Cursor task protobufs as durable + OpenCodex state. +- A structurally invalid or partially written line contributes no evidence and + is reported as ledger corruption. SQLite must be rebuildable from all valid + complete lines plus privacy-safe purge tombstones. + +The canonical scenario manifest, suite manifest, synthetic fixture bytes, and +claim-source manifest referenced by valid evidence are retained as +content-addressed `scenario_manifest`, `suite_manifest`, `fixture`, and +`claim_source_manifest` artifacts. Scenario/suite/fixture contract artifacts +remain retained while any non-invalidated, non-purged observation references +them. Claim-source manifests remain retained while any non-purged claim +snapshot references them. A missing or digest-mismatched required contract +artifact makes that evidence unusable and reports corruption; projection code +must never substitute the current manifest for the historical one. + +The SQLite projection may cache derived verdict rows. Such rows must include +their `asOf`, projection spec, scenario/suite/fixture manifest digests, claim +source manifest digest where applicable, and contributing event IDs. Deleting +SQLite and replaying JSONL plus the referenced content-addressed contract +artifacts and purge tombstones must reproduce every non-purged row. + +## 3. Canonical verdict contract + +The closed verdict set is: + +```text +UNKNOWN +CLAIMED +PROBED +VERIFIED +DEGRADED +BLOCKED +UNSUPPORTED +``` + +Verdicts are projections, not mutable evidence fields. + +### `UNKNOWN` + +Produced when no current positive claim snapshot and no current, valid, +attributable observation can classify the projection key. + +It can also result when all prior evidence became stale or was invalidated and +there is no current claim or blocker. Absence of a test is not +`UNSUPPORTED`. + +### `CLAIMED` + +Produced only by a current snapshotted positive local capability declaration +for the exact capability/subject, when no current executable evidence yields a +stronger state. The snapshot records whether the declaration came from explicit +provider config, Provider Registry, cached catalog, native metadata, or current +adapter inference rather than pretending every claim is registry-authored. + +`CLAIMED` applies only to `live_route_compatibility`; protocol and task layers +have no declaration source and project `UNKNOWN` without executable evidence. + +A claim cannot produce `PROBED` or `VERIFIED`. A registry negative declaration +is shown as claim metadata but does not by itself prove `UNSUPPORTED`. + +### `PROBED` + +Produced when at least one required executable scenario completed with an +attributable pass, but the suite's versioned verification rule is not yet +satisfied. Examples are partial required-scenario coverage or a scenario whose +assertions establish reachability/shape but not full suite verification. + +Connectivity-only `/models` checks, registry discovery, doctor output, health +samples, and blocked attempts cannot produce `PROBED`. + +### `VERIFIED` + +Produced only when every requirement in the suite manifest's verification rule +is met by current, valid, attributable observations for the exact projection +key, and no newer current contradictory attributable failure remains +unresolved. A verification rule with zero applicable required scenarios is not +satisfied and cannot produce `PROBED` or `VERIFIED`. + +The projection must expose: + +- exact contributing event IDs; +- suite/scenario versions and manifest digest; +- subject ID and full local subject; +- projection algorithm version and `asOf`; +- freshness calculation; +- any invalidations, purge tombstones, and contradictory events considered. + +An LLM judge, user assertion, registry declaration, successful model listing, +or mutable `verified=true` flag cannot produce `VERIFIED`. + +### `DEGRADED` + +Produced when executable evidence proves that the capability works only +partially, loses required semantics, violates a required assertion while a +usable subset remains, or requires a suite-declared workaround. + +Examples include malformed tool-call/result correlation, dropped reasoning +replay required by the suite, or incomplete stream semantics with otherwise +usable output. Environmental blockers cannot produce `DEGRADED`. + +### `BLOCKED` + +Produced when a current attempt cannot reach a compatibility assertion because +of an environmental or administrative precondition and no current +compatibility-attributable verdict should take precedence. + +Authentication, quota, region policy, local network failure, provider +transients, harness failure, or exhausted Lab budget can yield `BLOCKED`. The +projection retains those classes separately. + +A blocked retry does not erase a still-current `VERIFIED`, `PROBED`, +`DEGRADED`, or `UNSUPPORTED` verdict. Once that prior verdict is stale, +`BLOCKED` may become the current state until a conclusive run succeeds. + +### `UNSUPPORTED` + +Produced only by executable, suite-declared evidence that the exact capability +is unavailable by contract for this subject. The scenario must define an +unambiguous `capability_absence_control`; a conformance negative control, +generic 4xx, timeout, empty output, registry omission, or failed authentication +is insufficient. + +Expected rejection of a deliberately unsupported feature can prove +`UNSUPPORTED` when the rejection itself matches the deterministic contract. It +is not a failed harness run. + +### Projection precedence + +For current valid evidence at the same projection key: + +1. Satisfied full verification rule yields `VERIFIED`. +2. An unresolved attributable required-assertion failure yields `DEGRADED` or + `UNSUPPORTED` according to the scenario's failure rule. +3. Partial positive coverage yields `PROBED`. +4. A blocker yields `BLOCKED` only when 1-3 have no current result. +5. A current positive claim snapshot yields `CLAIMED`. +6. Otherwise the result is `UNKNOWN`. + +This is precedence, not a quality scale. In particular, `DEGRADED`, +`BLOCKED`, and `UNSUPPORTED` are not numeric values below `PROBED`. + +## 4. Transitions, contradiction and freshness + +Because verdicts are recomputed, a "transition" means that new events, time, or +version inputs change a projection. + +Allowed transitions: + +| From | May move to | Cause | +|---|---|---| +| `UNKNOWN` | any state | claim, observation, or blocker | +| `CLAIMED` | `UNKNOWN`, `PROBED`, `VERIFIED`, `DEGRADED`, `BLOCKED`, `UNSUPPORTED` | claim removal/staleness or executable evidence | +| `PROBED` | `UNKNOWN`, `CLAIMED`, `VERIFIED`, `DEGRADED`, `BLOCKED`, `UNSUPPORTED` | coverage, contradiction, staleness, invalidation | +| `VERIFIED` | `UNKNOWN`, `CLAIMED`, `PROBED`, `DEGRADED`, `BLOCKED`, `UNSUPPORTED` | partial remaining coverage after invalidation, contradiction, staleness, or changed inputs | +| `DEGRADED` | `UNKNOWN`, `CLAIMED`, `PROBED`, `VERIFIED`, `BLOCKED`, `UNSUPPORTED` | repair evidence, staleness, invalidation, or reclassification | +| `BLOCKED` | any state | blocker clears, prior evidence becomes current/stale, or claim changes | +| `UNSUPPORTED` | `UNKNOWN`, `CLAIMED`, `PROBED`, `VERIFIED`, `DEGRADED`, `BLOCKED` | route/version/config change, invalidation, or new evidence | + +Direct transitions not listed are forbidden; implementations must not invent a +state outside this set. + +Contradictory attributable evidence is never overwritten. The projection: + +1. filters by exact subject/layer/suite/scenario versions; +2. applies purge tombstones and invalidation events; +3. applies freshness; +4. orders observations by completion time and deterministic event-ID tie-break; +5. applies the suite's contradiction rule; +6. emits the contributing and contradicting event IDs. + +The initial contradiction rule is conservative: a newer required-scenario +failure prevents `VERIFIED` until a newer pass of that scenario and all other +required coverage exists. A newer pass can restore `VERIFIED`; history remains. + +### Freshness + +Each scenario manifest declares its maximum evidence age. The suite manifest +may declare a stricter maximum, and a future Routing Profile may tighten it +again. Effective maximum age is the minimum of all finite scenario, suite and +profile values; `null` means no bound at that layer: + +- deterministic protocol evidence has no wall-clock expiry by default, but is + exact-match bound to scenario, suite, compatibility version, adapter and + behavior fingerprint; +- initial live-route manifests default to seven days; +- initial task-effectiveness manifests default to thirty days; +- a profile's maximum evidence age is an additional upper bound, never an + extension. + +Stale observations remain queryable but cannot support current +`PROBED`/`VERIFIED`/`DEGRADED`/`UNSUPPORTED`. The projection may display a +`lastKnownVerdict` separately. Its current verdict falls to `CLAIMED`, +`BLOCKED`, or `UNKNOWN` according to current inputs. + +### Version and configuration changes + +- Evidence matches an exact scenario and suite version in contract v1. No + implicit semver range reuse is allowed. +- A changed scenario assertion, fixture, requirement, or classification rule + requires a new scenario version and invalidates old evidence for the new + projection key. +- `opencodexCompatibilityVersion` is lowercase + `sha256("ocx-lab:compatibility-version:v1\0" || JCS(manifest))`. + The exact manifest object is: + + ```text + { + "schemaVersion": 1, + "assertionDslVersion": "1.0.0", + "evidenceSchemaVersion": "1.0.0", + "bunRuntimeVersion": , + "files": [ + { + "path": , + "sha256": + }, + ... + ] + } + ``` + + `files` contains every Git-index-tracked regular file under `src/`, plus + `package.json`, `bun.lock`, and `scripts/model-metadata.source.json`, sorted + by UTF-8 path bytes. Generation reads current working-tree bytes so a dirty + behavior change cannot reuse clean-tree evidence. A missing file, a tracked + symlink, a non-regular file, duplicate normalized path, invalid UTF-8 path, + or unreadable file makes the run `harness_failure`; untracked files are not + loaded by the compatibility harness. Release/package builds embed this + generated manifest so an installed runtime does not require Git. This + conservative whole-runtime input set may invalidate unrelated evidence, but + cannot falsely reuse evidence after a behavior change. The package marketing + version remains provenance only. +- A compatibility-version change starts a new subject projection. +- Any behavior-relevant configuration fingerprint change starts a new subject. +- Credential rotation alone does not change the subject. + +## 5. Failure-attribution contract + +Every non-pass observation uses exactly one primary class. Stable secondary +codes may add detail without changing these semantics. + +| Class | Meaning | Affects verdict? | Default action | +|---|---|---:|---| +| `protocol_failure` | OpenCodex or the exact route emitted, accepted, ordered, translated, or terminated protocol data incorrectly | Yes, in the observation's layer | Conclusive; reverify after code/config/version change | +| `capability_failure` | The exact route cannot satisfy a capability assertion that it was expected to support | Yes | Conclusive when the scenario rules out an unsupported contract; otherwise retry once then `inconclusive` | +| `behavioral_failure` | A task-effectiveness deterministic verifier failed although protocol/capability prerequisites completed | Yes, task layer only | Conclusive for that task scenario | +| `authentication_blocked` | Missing, expired, rejected, or insufficient credentials prevented the assertion | No | `BLOCKED`; reauthenticate and retry | +| `quota_blocked` | Rate, credit, token, concurrency, or account quota prevented the assertion | No | `BLOCKED`; retry after reset/backoff | +| `region_blocked` | Region/tenant policy prevented execution | No | `BLOCKED`; retry only when route context changes | +| `network_failure` | DNS, TLS establishment, connect, local proxy, or transport reachability failed without provider response evidence | No | `BLOCKED`; repair environment and retry | +| `provider_transient` | Upstream returned a recognized transient/overload failure or interrupted a previously valid service path | No by default | `BLOCKED`; bounded retry/reverification | +| `timeout` | A versioned scenario deadline expired; secondary code distinguishes connect, first-byte, inactivity, or total budget | No by default | `BLOCKED`; bounded retry; reclassify only with deterministic protocol evidence | +| `harness_failure` | Runner, fixture, mock, sandbox, assertion engine, or artifact writer failed | No | Invalidate affected evidence and fix harness | +| `budget_exhausted` | Lab request/token/tool/byte/time budget ended the run before its assertion | No | `BLOCKED`; revise scenario limits/version or retry | +| `inconclusive` | Observations conflict or lack enough information for another class | No | No promotion/degradation; investigate/reverify | + +Manifest registration enforces this exhaustive class/effect matrix: + +| Class | Legal `verdictEffect` | Additional constraint | +|---|---|---| +| `protocol_failure` | `degraded`, or `none` for a supplemental assertion | Never `unsupported` | +| `capability_failure` | `degraded`, `unsupported`, or `none` for an exact conformance negative control | `unsupported` requires a `capability_absence_control` | +| `behavioral_failure` | `degraded`, or `none` for a supplemental assertion | `task_effectiveness` only | +| `authentication_blocked` | `none` | Environmental blocker only | +| `quota_blocked` | `none` | Environmental blocker only | +| `region_blocked` | `none` | Environmental blocker only | +| `network_failure` | `none` | Environmental blocker only | +| `provider_transient` | `none` | Environmental blocker only | +| `timeout` | `none` | Environmental blocker only | +| `harness_failure` | `none` | Invalidates/blocks evidence only | +| `budget_exhausted` | `none` | Environmental blocker only | +| `inconclusive` | `none` | No verdict promotion or degradation | + +Any other pair rejects the manifest. A retry count cannot change the pair; +reclassification requires a new observation with independently satisfied +classification rules. + +`expectedFailure` is either absent or: + +```text +controlKind conformance_negative_control | + capability_absence_control +expectedClass +expectedCode +assertionIds[] +onMatch pass | unsupported +onMismatch fail | inconclusive +``` + +For `conformance_negative_control`, `onMatch` must be `pass` and the legal +effect is `none`; the expected rejection helps satisfy `VERIFIED`. For +`capability_absence_control`, `onMatch` must be `unsupported` and the class +must be `capability_failure`. No one observation can both satisfy verification +and project `UNSUPPORTED`. + +Safety rules: + +- Expired credentials never imply broken tool support. +- Quota exhaustion never implies model incompatibility. +- Local DNS/TLS/connect failure never degrades provider capability. +- A generic timeout never proves missing terminal semantics. A deterministic + mock stream that closes without its required terminal event is + `protocol_failure`; a live body that simply stalls is `timeout`. +- Malformed tool-call semantics, broken tool-result correlation, or lost + required event ordering may legitimately produce `protocol_failure` and + `DEGRADED`. +- `provider_transient` may be promoted to a compatibility-affecting class only + by a scenario-specific deterministic rule and a new observation; projection + code must not infer promotion from retry count. +- Expected failures follow the closed control contract above. A generic + `negative_control` label never implies capability absence. + +## 6. Canonical route subject + +Live route evidence is never keyed by model name alone. `RouteSubjectV1` +contains: + +```text +subjectSchemaVersion 1 +subjectKind route +providerId +providerInstanceFingerprint +clientModelId +upstreamModelId +effectiveAdapter +inboundProtocol +upstreamProtocol +surface +opencodexCompatibilityVersion +behaviorFingerprint +endpointFingerprint +dependencies[] +``` + +Semantics: + +- `providerId` is the built-in registry ID or `custom`; it is not a display + label. +- `providerInstanceFingerprint` is a locally salted HMAC over the configured + provider identity, allowing two instances of one preset to differ without + leaking a user-selected name. All local opaque fingerprints use lowercase + HMAC-SHA-256 over + `UTF8("ocx-lab:local-fingerprint:v1\0" + fieldName + "\0") || JCS(value)` + with the installation salt as key. +- `clientModelId` is the selected canonical route model; `upstreamModelId` is + the effective wire model after namespace, virtual-model, combo and suffix + resolution. +- `effectiveAdapter` reflects model-specific wire defaults/overrides and wire + pins, not merely the provider-wide configured adapter. +- Protocol values distinguish OpenAI Responses, OpenAI Chat Completions, + Anthropic Messages, and provider-specific wires. +- `surface` distinguishes behaviorally different ingress/transport paths such + as Responses HTTP, Responses WebSocket, Chat HTTP/SSE, and Anthropic + Messages HTTP/SSE. +- The package/build version remains observation provenance in + `producerVersion`; only `opencodexCompatibilityVersion` participates in the + subject so an unrelated release does not discard valid conformance evidence. +- `endpointFingerprint` is a locally salted HMAC of the normalized destination + scheme/host/port/base path. Raw URLs, userinfo, query strings, and fragments + are not evidence fields. +- `dependencies` is an ordered list of flat `RouteDependencyV1` records for + behaviorally invoked sidecars. Each record contains role, provider ID, + provider-instance fingerprint, client/upstream model IDs, effective adapter, + upstream protocol, endpoint fingerprint, and behavior fingerprint. It cannot + nest. Records sort by this total order of UTF-8 string comparisons: + role, provider ID, provider-instance fingerprint, upstream model ID, + endpoint fingerprint, client model ID, effective adapter, upstream protocol, + then behavior fingerprint. A duplicate full key makes subject construction + `harness_failure`. An empty list is canonical when no sidecar is invoked. + +### Behavior fingerprint allowlist + +The fingerprint is SHA-256 over JCS `BehaviorFingerprintV1`: + +```text +{ + "schemaVersion": 1, + "resolverVersion": 1, + "values": { + "": { + "source": "request" | "model_override" | "provider_config" | + "registry_runtime_default" | "generated_model_metadata" | + "global_config" | "adapter_default" | "lab_forced", + "value": + } + } +} +``` + +The production route/model/adapter resolver, not a second Lab merge, emits the +effective value and winning source tag. Thus model-specific wire/config +overrides, registry runtime defaults, generated metadata, global values and +adapter defaults retain the exact live precedence. If precedence is ambiguous, +an effective default cannot be resolved, or scenario execution reads a +behavior-changing input that has no closed key below, subject construction +fails as `harness_failure` with code `unclassified_behavior_input`; no evidence +is emitted. + +Closed V1 keys are: + +```text +wire.adapter +wire.upstreamProtocol +wire.responsesPath +wire.commandCodeVersion +wire.modelSuffixMode +auth.mode +auth.transport +responses.stateful +responses.upstreamStreaming +responses.serviceTier +responses.snapshotRepair +responses.itemIdRepair +limits.contextWindow +limits.maxInputTokens +limits.maxOutputTokens +modalities.input +sampling.omitTemperature +sampling.omitTopP +sampling.omitPenalties +reasoning.supported +reasoning.efforts +reasoning.defaultEffort +reasoning.effortMap +reasoning.wireFormat +reasoning.summaryMode +reasoning.replayMode +reasoning.splitMode +reasoning.toggleMode +reasoning.budgetMode +tools.choiceRestrictions +tools.parallel +tools.hostedPreference +tools.customFreeform +tools.builtinNameEscaping +cache.forwarding +cache.retention +anthropic.eofPolicy +google.mode +google.projectFingerprint +google.locationFingerprint +openrouter.order +openrouter.only +openrouter.allowFallbacks +sidecars.vision +sidecars.webSearch +mcp.maxTools +mcp.maxSchemaBytes +mcp.maxResultBytes +mcp.nativeLocalExec +runtime.bunVersion +runtime.platform +runtime.arch +runtime.streamMode +runtime.fastMode +runtime.effortCap +headers.nonCredentialBehaviorDigest +``` + +`sampling.omitTemperature`, `sampling.omitTopP`, and +`sampling.omitPenalties` are the selected-model effective booleans produced +from `noTemperatureModels`, `noTopPModels`, and `noPenaltyModels`, rather than +hashes of the whole configured arrays. `wire.commandCodeVersion` is the +effective `commandCodeVersion`. `cache.retention` is the effective global +`cacheRetention` value. Sidecar keys contain the effective enablement, backend, +model, reasoning, per-turn limits and timeout/stall limits plus the matching +flat dependency subject ID. + +Values that are inapplicable to the selected model/surface are omitted rather +than copied wholesale. Canonical JSON sorts object keys and normalizes +absent/default values to their effective value. Array handling is closed and +part of the V1 schema: + +| Array-valued key | Ordering mode | Element comparison | +|---|---|---| +| `modalities.input` | `set` | UTF-8 JCS bytes | +| `reasoning.efforts` | `set` | UTF-8 JCS bytes | +| `tools.choiceRestrictions` | `set` | UTF-8 JCS bytes | +| `openrouter.order` | `ordered` | source order preserved | +| `openrouter.only` | `set` | UTF-8 JCS bytes | + +For `set`, each element is JCS-canonicalized, duplicate JCS byte sequences are +rejected, and elements sort lexicographically by their UTF-8 JCS bytes. For +`ordered`, elements are JCS-canonicalized but their resolved source order is +preserved. Any other closed key resolving to an array, or any array-valued key +without one of these declared modes, fails subject construction as +`harness_failure` with code `unclassified_behavior_input`. + +Acceptance tests for the future builder must enumerate every +behavior-changing config/default read by the selected adapter and prove that +changing each effective value changes the fingerprint, while changing every +excluded secret/transient value does not. + +The following never participate: + +- API keys, OAuth/access/refresh tokens, cookies, authorization headers; +- account IDs, emails, labels, aliases, quota balances or plan names; +- raw custom/private header names or values; +- prompts, messages, tool results, repository paths or contents; +- timestamps, transient health, latency, cost, quota or retry state. + +Credential headers are excluded. The config-owner fingerprint broker described +by the security contract supplies only +`headers.nonCredentialBehaviorDigest`; Lab code never receives raw header names +or values. Project/location and custom endpoint values use the same local +opaque treatment. + +Public export replaces all local fingerprints with export-scoped opaque IDs +and redacts custom model IDs unless the export policy explicitly classifies +them as public. + +## 7. Task-effectiveness ingress + +A future Fabric observation must provide, at minimum: + +```text +producerSchemaVersion +outcomeId +taskClassId +taskClassVersion +subject TaskSubjectV1 +taskFixtureDigest +verifierManifestDigest +startedAt +completedAt +resourceLimits +result success | failure | blocked | inconclusive +verifiers[] { id, version, manifestDigest, result, + normalizedMetrics? } +artifactRefs[] +``` + +The Lab rejects an outcome if the subject cannot be reconstructed, a verifier +is nondeterministic for a canonical assertion, or an artifact violates the +security contract. Free-form narrative may be retained only as bounded, +sanitized advisory metadata and never determines the verdict. + +## 8. Consumer boundaries + +- Provider Registry supplies claims; Lab does not rewrite them. +- Request history supplies route/outcome references; Lab does not copy its + ledger. +- Selection and execution stay separate: `RouteDecisionTraceV1` records the + pre-dispatch choice, `attempts[]` records physical execution/fallback, and + final outcome is joined at read time. +- Routing Profiles express user requirements; Lab does not evaluate policy. +- Router Intelligence reads projections; Lab does not rank candidates. +- Route decision traces explain compatibility exclusions/penalties in the + existing trace; Lab does not create a parallel route explanation. +- Agent Fabric executes tasks; Lab does not run arbitrary repository work. diff --git a/devlog/_plan/260807_compatibility_lab/020_scenario_contract_and_catalogue.md b/devlog/_plan/260807_compatibility_lab/020_scenario_contract_and_catalogue.md new file mode 100644 index 0000000000..fe53ef6e0a --- /dev/null +++ b/devlog/_plan/260807_compatibility_lab/020_scenario_contract_and_catalogue.md @@ -0,0 +1,465 @@ +# CL-00 scenario contract and initial catalogue + +This document freezes the scenario schema and the initial IDs. CL-01 implements +the `*.protocol.*` scenarios only. Entries marked live or Fabric-reserved define +future semantics and are not authorization to execute them. + +The normative V1 selector/operator semantics, immutable fixture anchors, +expanded defaults and complete protocol scenario/suite manifest records are in +[the protocol V1 manifest authority](./021_protocol_v1_manifest_authority.md); +its canonical fixture vectors and literal expectations are in +[`022_protocol_v1_cases.json`](./022_protocol_v1_cases.json). + +## 1. Versioned scenario model + +A `CompatibilityScenarioV1` has: + +```text +schemaVersion 1 +id stable lowercase dotted ID +version exact semver +suite { id, version, evidenceLayer } +evidenceLayer protocol_conformance | + live_route_compatibility | + task_effectiveness +capability stable capability ID +verificationRole required | supplemental | negative_control +requirements +fixtures +executionLimits +assertions[] +failureRules[] +artifactPolicy +freshness +``` + +### Identity and versioning + +- `id` names semantics and does not contain a provider, model, or version. +- `version` is exact-match in contract v1. +- Any assertion, fixture, limit that affects expected behavior, failure rule, + artifact exposure, or requirement change increments the scenario version. +- Editorial description changes do not require a version change. +- A suite manifest has its own exact version and lists scenario IDs, versions, + roles, and its verification rule. +- A suite manifest belongs to exactly one `evidenceLayer`; a scenario and suite + with different layers are invalid. Human-facing suite stems may recur across + layers, but their manifest keys are + `(id, evidenceLayer, version, manifestDigest)`. +- Scenario and suite manifests are RFC 8785 canonical JSON with the + domain-separated digest defined by the evidence contract. + +### Requirements + +Requirements are declarative and may include: + +```text +inboundProtocols[] +upstreamProtocols[] +surfaces[] +requiredClaims[] +requiredHarnessFeatures[] +platforms[] +routePreconditions[] +``` + +An unmet deterministic fixture requirement is `harness_failure`. An unmet live +route precondition is either inapplicable or a typed blocker; it is never +silently counted as a capability failure. + +### Fixtures + +Fixture references include ID, content digest, media type, generator version +where generated, and role (`client_request`, `upstream_response`, +`adapter_vector`, `synthetic_tool`, `synthetic_image`, or `task_fixture`). A +fixture never embeds credentials, user data, or an external mutable URL. + +Protocol fixtures run against a deterministic mock upstream. Live scenarios use +only Lab-owned synthetic requests and inert tools. Fabric scenarios refer to a +versioned synthetic task class; they do not place a repository in the Lab +ledger. + +### Execution limits + +Every scenario states: + +```text +totalTimeoutMs +connectTimeoutMs? +firstByteTimeoutMs? +inactivityTimeoutMs? +maxRequests +maxInputBytes +maxOutputBytes +maxOutputTokens? +maxToolCalls +maxArtifactBytes +``` + +Absent limits are invalid. Limits may be stricter than Lab-wide ceilings but +not wider without a scenario version change and security review. Expiry of a +limit classifies as `timeout` or `budget_exhausted` according to the failed +limit; it does not imply incompatibility. + +### Deterministic assertion DSL + +Canonical assertions use a closed set of observable operators: + +```text +http_status_equals +header_present +header_absent +header_value_equals +json_schema_matches +json_path_equals +json_path_present +json_path_absent +sse_field_equals +sse_event_sequence +sse_event_count +terminal_signal_equals +id_matches +id_stable_across_events +id_correlates +tool_call_equals +tool_result_correlates +fixture_request_matches +normalized_text_equals +byte_limit_observed +process_exit_equals +verifier_result_equals +``` + +Each assertion has an ID, operator, selector, expected value, required flag, +and redaction-safe observed summary. Implementations must exhaustively reject +unknown operators. Protocol V1 permits no arbitrary regular expressions; ID +grammars and all operator type/missing-value behavior are closed in the +manifest authority. + +Core verdicts cannot use an LLM judge, free-form human interpretation, or a +snapshot that contains unstable timestamps/IDs without normalization. + +### Failure rules + +Rules are ordered and explicit: + +```text +match assertion IDs, normalized status/error/event/timeout +classification canonical failure class +secondaryCode +verdictEffect none | degraded | unsupported +retry never | bounded | after_precondition_change +expected boolean +``` + +The first exact rule wins. If no rule establishes a compatibility-attributable +class, the attempt is `inconclusive`. Generic HTTP 4xx/5xx rules may classify a +blocker or transient but cannot prove `UNSUPPORTED`. + +Manifest registration must enforce the exhaustive classification/effect matrix +in `010_architecture_and_evidence_contract.md`. Environmental, timeout, +budget, harness, transient, and inconclusive classes permit only +`verdictEffect: none`; a manifest that maps any of them to `degraded` or +`unsupported` is invalid. + +`expectedFailure`, when present, includes `controlKind`, exact class/code, +assertion IDs, `onMatch`, and `onMismatch`. A +`conformance_negative_control` exact rejection is a verification pass with no +verdict effect. A `capability_absence_control` may produce `UNSUPPORTED`. These +meanings cannot be combined in one scenario observation. + +### Artifact policy + +The policy is deny-by-default and names allowed normalized artifacts: + +```text +assertion_report +sanitized_request_shape +sanitized_response_shape +normalized_event_trace +sanitized_error +verifier_summary +``` + +It states per-artifact and aggregate byte limits, retention class, local/public +visibility, and redaction profile. Raw credentials, prompts, hidden reasoning, +full task repositories, arbitrary headers, and arbitrary response bodies are +not valid artifact kinds. + +Each scenario declares `freshness.maxAgeMs`; its suite may declare a stricter +bound, and a future profile may tighten it again. Effective maximum age is the +minimum finite bound, with `null` meaning unbounded at that layer. + +## 2. Suite projection rules + +For each exact suite manifest, first compute the required scenarios whose +manifest requirements are applicable to the exact subject. That applicable +required set must be non-empty for any positive executable verdict. + +- `VERIFIED`: the applicable required set is non-empty; every applicable + `required` scenario has a current pass; and every applicable + `conformance_negative_control` observed its exact required rejection. +- `PROBED`: at least one applicable required scenario passed, with no current + compatibility-attributable required-scenario failure, but the suite's + verification rule is not fully satisfied. +- `DEGRADED`: a failure rule on an applicable required scenario yields + `degraded`. +- `UNSUPPORTED`: a required scenario's exact + `capability_absence_control` proves the capability unavailable. +- `BLOCKED`: only blockers exist and no current attributable verdict takes + precedence. +- `CLAIMED`/`UNKNOWN`: follow the evidence contract. + +An inapplicable required scenario contributes neither a pass nor a failure. If +no required scenario is applicable, the suite cannot project `PROBED` or +`VERIFIED`; it falls through to `CLAIMED` or `UNKNOWN` under the evidence +contract. If execution was attempted but an environmental or administrative +precondition prevented reaching assertions, that attempt is a typed blocker +and may project `BLOCKED` under normal precedence; it is not treated as +inapplicability. + +Supplemental scenarios never block `VERIFIED` unless a new suite version makes +them required. + +## 3. Initial suite catalogue + +All initial scenario versions and suite versions are `1.0.0`. + +For protocol V1, the literal fixtures/assertions in `022` are the complete +verification boundary. Descriptions below summarize those exact vectors; they +do not silently incorporate every historical incident mapped to the same +scenario ID. An incident absent from `022` is candidate coverage for a reviewed +scenario/suite version amendment and cannot be claimed by a V1 `VERIFIED` +verdict. + +### `responses-core` + +Purpose: preserve the OpenAI Responses request, output-item lifecycle, stream +framing, IDs, terminal state, and JSON/SSE equivalence. + +Capability: `protocol.responses.core`. + +| Scenario ID | Layer/applicability | Required observable assertions | +|---|---|---| +| `responses-core.protocol.request-shape` | Deterministic; CL-01 | Mock receives the exact model, first user text and zero temperature | +| `responses-core.protocol.sse-framing` | Deterministic; CL-01 | Spaced and unspaced data fields, a non-record `null` frame, data-only event inference, exact text and completion terminal | +| `responses-core.protocol.item-lifecycle` | Deterministic; CL-01 | One added/done/completed lifecycle with stable valid message ID | +| `responses-core.protocol.terminal-state` | Deterministic; CL-01 | One explicit failed terminal is preserved exactly | +| `responses-core.protocol.json-sse-equivalence` | Deterministic; CL-01 | One normalized JSON/SSE pair has equal text and completion terminal | +| `responses-core.live.basic-turn` | Live-reserved | 2xx, bounded output, valid lifecycle and terminal state from exact route | + +Unsupported means a route deterministically rejects the Responses surface with +a suite-recognized unsupported signal. Semantic loss, invalid IDs, malformed +event order, or missing terminal state is degraded. Auth/quota/region/network, +transient upstream errors, and body stalls are blocked. + +### `chat-core` + +Purpose: preserve OpenAI Chat Completions request/response semantics for JSON +and streaming routes. + +Capability: `protocol.chat.core`. + +| Scenario ID | Layer/applicability | Required observable assertions | +|---|---|---| +| `chat-core.protocol.request-mapping` | Deterministic; CL-01 | System/developer/user order and JSON-object response format match the fixture | +| `chat-core.protocol.nonstream-envelope` | Deterministic; CL-01 | One valid choice/message/finish/usage envelope yields exact text and terminal | +| `chat-core.protocol.stream-assembly` | Deterministic; CL-01 | Fragmented/interleaved deltas assemble in order and emit one finish | +| `chat-core.protocol.stream-terminal` | Deterministic; CL-01 | One stop finish plus `[DONE]` yields exact text and one terminal | +| `chat-core.live.basic-turn` | Live-reserved | Exact route returns bounded text and a valid finish contract | + +Unsupported is a deterministic surface rejection. Incorrect role mapping, +malformed choices, lost stream fragments, or invalid finish semantics is +degraded. Environmental and transient failures are blocked. + +### `anthropic-core` + +Purpose: preserve Anthropic Messages roles/content blocks, tool/thinking block +ordering, stop reasons, usage, and SSE lifecycle. + +Capability: `protocol.anthropic.messages.core`. + +| Scenario ID | Layer/applicability | Required observable assertions | +|---|---|---| +| `anthropic-core.protocol.request-mapping` | Deterministic; CL-01 | Model, system instruction and first user text map exactly | +| `anthropic-core.protocol.content-sequence` | Deterministic; CL-01 | `message_start`, monotonic content blocks/deltas/stops, `message_delta`, `message_stop` | +| `anthropic-core.protocol.tool-round-trip` | Deterministic; CL-01 | One `tool_use`/`tool_result` pair preserves its ID correlation and result text | +| `anthropic-core.protocol.terminal-errors` | Deterministic; CL-01 | One explicit Responses failure maps to the Anthropic error/failed terminal | +| `anthropic-core.live.basic-turn` | Live-reserved | Exact route returns a valid bounded Messages lifecycle and terminal | + +Unsupported is a recognized Messages-surface rejection. Wrong block ordering, +lost tool correlation, invalid stop reason, or clean EOF accepted without the +suite's terminal contract is degraded. Authentication, quota, region, network, +transient failure, and silence timeout are blocked. + +### `tools-core` + +Purpose: prove deterministic function/custom tool declaration, call assembly, +parallel correlation, and result continuation. + +Capability: `tools.round_trip`. + +| Scenario ID | Layer/applicability | Required observable assertions | +|---|---|---| +| `tools-core.protocol.function-round-trip` | Deterministic; CL-01 | One function call preserves ID/name/parsed arguments and its continuation result correlates | +| `tools-core.protocol.custom-freeform-round-trip` | Deterministic; CL-01 | One `apply_patch` call preserves exact freeform input and its continuation result correlates | +| `tools-core.protocol.parallel-correlation` | Deterministic; CL-01 | Two interleaved calls assemble once in first-seen order without overlap | +| `tools-core.protocol.result-content` | Deterministic; CL-01 | One result preserves exact text and data-image parts | +| `tools-core.protocol.choice-and-allowed-set` | Deterministic; CL-01 | One required single-tool allowed set narrows exactly without widening | +| `tools-core.live.function-round-trip` | Live-reserved | Inert deterministic function is called once with schema-valid args and static result is continued | +| `tools-core.live.custom-freeform-round-trip` | Live-reserved | Route emits exact custom/freeform call and accepts static result continuation | + +An explicit route `capability_absence_control` that rejects a tool kind can +prove unsupported. +Malformed arguments, dangling IDs, widened choice, dropped calls/results, or +incorrect parallel assembly is degraded. A model choosing not to call an +`auto` tool is inconclusive; required tool choice is used for conclusive live +coverage. Environmental failures are blocked. + +### `codex-core` + +Purpose: establish the minimum end-to-end semantics required to advertise a +route as usable by Codex. A basic Responses text request is insufficient. + +Capability: `client.codex.core`. + +| Scenario ID | Layer/applicability | Required observable assertions | +|---|---|---| +| `codex-core.protocol.streaming-turn` | Deterministic; CL-01 | One Chat-backed stream yields exact text, final-answer phase and one completed terminal | +| `codex-core.protocol.apply-patch-turn` | Deterministic; CL-01 | One custom `apply_patch` call preserves exact patch text and its result ID correlates | +| `codex-core.protocol.tool-continuation` | Deterministic; CL-01 | One function result follows and correlates with its prior call | +| `codex-core.protocol.previous-response-replay` | Deterministic; CL-01 | One local expansion preserves stored input/output/new-input order and strips `previous_response_id` | +| `codex-core.protocol.structured-output` | Deterministic; CL-01 | One JSON-schema request maps to the exact Chat `response_format` | +| `codex-core.protocol.compaction-and-special-items` | Deterministic; CL-01 | Compaction, local shell, tool search and hosted-tool items are normalized without leaking opaque raw data | +| `codex-core.live.tool-turn` | Live-reserved | Valid stream plus required inert tool call/result continuation and terminal | +| `codex-core.live.custom-tool-turn` | Live-reserved | Required custom/freeform call/result continuation and valid terminal | + +The `codex-core` manifest requires all six protocol scenarios for conformance +verification. Future live verification requires both live scenarios plus +current `responses-core.live.basic-turn`. Text-only success is at most partial +coverage, never `codex-core: VERIFIED`. + +An exact route may be unsupported when it deterministically lacks a mandatory +Codex surface or tool kind. Lossy lifecycle, call/result correlation, +continuation, or special-item behavior is degraded. Environmental failures are +blocked. + +### `vision-core` + +Purpose: preserve declared image input and tool-result image behavior and prove +exact-route image understanding without user media. + +Capability: `modalities.image.input`. + +| Scenario ID | Layer/applicability | Required observable assertions | +|---|---|---| +| `vision-core.protocol.input-image` | Deterministic; CL-01 | One data-URL image preserves detail and text/image ordering | +| `vision-core.protocol.tool-result-image` | Deterministic; CL-01 | Image content in function/tool result remains structured and correlated | +| `vision-core.protocol.modality-gate` | Deterministic negative control; CL-01 | A text-only/no-sidecar synthetic vector produces the typed unsupported path without silent image drop | +| `vision-core.live.synthetic-ocr` | Live-reserved | Lab-generated image nonce is returned in an exact JSON schema | + +A deterministic declared no-image `capability_absence_control` may prove +unsupported. The protocol V1 modality gate is instead a conformance negative +control whose exact rejection is a verification pass. Dropping, textifying +without a declared sidecar, corrupting, or misordering image content is +degraded. Failure of the optional sidecar route is attributed to that exact +subject. Auth/quota/network/transient failures are blocked. + +### `reasoning-core` + +Purpose: preserve supported reasoning controls, summaries, signatures and +replay while preventing provider-private reasoning material from crossing an +incompatible boundary. + +Capability: `reasoning.round_trip`. + +| Scenario ID | Layer/applicability | Required observable assertions | +|---|---|---| +| `reasoning-core.protocol.effort-mapping` | Deterministic; CL-01 | Effective effort maps to the declared wire form and unsupported parameters are omitted | +| `reasoning-core.protocol.summary-stream` | Deterministic; CL-01 | One summary-part/delta/completed sequence preserves ordering and reasoning ID | +| `reasoning-core.protocol.replay` | Deterministic; CL-01 | One synthetic plaintext/signature replay reaches the second turn exactly | +| `reasoning-core.protocol.private-content-isolation` | Deterministic; CL-01 | One provider-private value is absent from an incompatible upstream and client response | +| `reasoning-core.live.replay` | Live-reserved | Synthetic two-turn route accepts its declared replay form and completes | + +An explicit no-reasoning route can prove unsupported. Rejected or corrupted +declared replay, lost required signatures, or private content sent across an +incompatible provider boundary is degraded (and the latter is also a security +finding). Environmental failures are blocked. + +### `mcp-core` + +Purpose: preserve MCP namespace, schema, tool/resource invocation and result +contracts through supported adapters without touching user MCP servers. + +Capability: `tools.mcp.core`. + +| Scenario ID | Layer/applicability | Required observable assertions | +|---|---|---| +| `mcp-core.protocol.namespace-mapping` | Deterministic; CL-01 | One namespace/name pair flattens and reverses exactly | +| `mcp-core.protocol.schema-and-bounds` | Deterministic; CL-01 | Tool schemas encode correctly; exact configured bounds admit and one-byte-over rejects atomically | +| `mcp-core.protocol.call-result` | Deterministic; CL-01 | Lab stub receives one exact call and returns one successful text result | +| `mcp-core.protocol.resource-round-trip` | Deterministic; CL-01 | One list/read resource success shape preserves URI, name and text | +| `mcp-core.live.synthetic-tool` | Live-reserved | Lab-owned loopback pure-function MCP tool is advertised, called and correlated | + +Only a Lab-owned in-memory or loopback fixture is allowed. A route that +deterministically cannot expose MCP may be unsupported. Namespace loss, schema +corruption, partial bound commits, or result miscorrelation is degraded. +User-server unavailability is never tested; environmental failures are blocked. + +### `fabric-core` task-effectiveness reservation + +This is a distinct `task_effectiveness` suite manifest, not an extension of a +protocol or live-route manifest: + +```text +suite.id fabric-core +suite.version 1.0.0 +suite.evidenceLayer task_effectiveness +verificationRule all-applicable-required-pass-v1 +freshness.maxAgeMs 2592000000 +``` + +The first reserved scenario is +`fabric-core.task.synthetic-patch@1.0.0`: + +- subject: exact `TaskSubjectV1`; +- verification role: `required`; +- fixture: a content-addressed synthetic scratch tree containing + `src/value.txt` with UTF-8 bytes `before\n`, plus a task-class manifest that + requests the exact final bytes `after\n`; +- execution: Fabric-owned, no user repository/prompt, no network, no user MCP, + no shell, and filesystem access restricted to that synthetic scratch tree; +- limits: one file, 64 KiB aggregate input/output, one patch operation, + 30-second total, 5-second inactivity, and 1 MiB aggregate artifacts; +- verifier manifest: `exact-tree-diff-v1`, whose digest participates in + `TaskSubjectV1`; +- deterministic verifier: sort repository-relative POSIX paths by UTF-8 bytes, + reject symlinks/special files/path traversal, hash exact file bytes, and pass + only when the sole diff changes `src/value.txt` from `before\n` to `after\n` + with no added/deleted/renamed file; +- success assertion: verifier result `pass`; +- failure rules: verifier `fail` is `behavioral_failure -> degraded`; + unavailable Fabric/sandbox is `harness_failure -> none`; exhausted time/bytes + is the corresponding blocker with effect `none`; +- artifact policy: retain only the bounded normalized path/digest diff and + verifier summary, never file bodies. + +This reservation freezes task-subject and verifier semantics for a later Fabric +phase. It does not authorize CL-01 to implement or execute the task. + +## 4. CL-01 implementation boundary + +CL-01 may implement the scenario registry, deterministic mock-upstream harness, +closed assertion DSL, and only the `protocol_conformance` manifests frozen in +`021_protocol_v1_manifest_authority.md`. It must not: + +- contact a real provider; +- write the planned production evidence ledger or SQLite projection; +- add profile/routing controls; +- execute a user tool, shell, filesystem, repository, MCP server, or external + network action; +- implement live/Fabric scenarios merely because their IDs are reserved here. + +If CL-01 discovers that an observable assertion cannot be implemented without +new semantics, it must amend this contract in a reviewed change rather than +quietly inventing behavior. diff --git a/devlog/_plan/260807_compatibility_lab/021_protocol_v1_manifest_authority.md b/devlog/_plan/260807_compatibility_lab/021_protocol_v1_manifest_authority.md new file mode 100644 index 0000000000..4135ba967c --- /dev/null +++ b/devlog/_plan/260807_compatibility_lab/021_protocol_v1_manifest_authority.md @@ -0,0 +1,480 @@ +# CL-00 protocol V1 manifest authority + +This document closes the executable semantics for the initial +`protocol_conformance` scenarios. It is normative for CL-01 and does not +implement a runner. + +The machine-readable source of truth is +[`022_protocol_v1_cases.json`](./022_protocol_v1_cases.json). It contains 35 +provider-independent canonical fixture vectors, literal expected values, +row-specific requirements, exact roles/media types, execution limits, artifact +policy, failure rules, and domain-separated fixture digests. Historical tests +in the [incident corpus](./030_incident_corpus.md) are provenance and coverage +guidance only; they are not executable manifest semantics. + +## 1. Exact manifest expansion + +For each entry in `cases`, CL-01 constructs `CompatibilityScenarioV1` in this +field order before RFC 8785 canonicalization: + +```text +schemaVersion source.schemaVersion +id case.id +version manifestDefaults.version +suite { id: case.suite, + version: manifestDefaults.suiteVersion, + evidenceLayer: + manifestDefaults.evidenceLayer } +evidenceLayer manifestDefaults.evidenceLayer +capability case.capability +verificationRole case.verificationRole when present, + otherwise manifestDefaults.verificationRole +requirements case.requirements +fixtures when case.initiatingRequest is present: + [fixtureRef(case.initiatingRequest), + fixtureRef(case.fixture)] + otherwise: [fixtureRef(case.fixture)] +executionLimits manifestDefaults.executionLimits +assertions case.assertions +expectedFailure case.expectedFailure, only when present +failureRules expandFailureRules(case) +artifactPolicy manifestDefaults.artifactPolicy +freshness manifestDefaults.freshness +``` + +`fixtureRef(x)` is exactly: + +```text +{ + id: x.id, + role: x.role, + mediaType: x.mediaType, + digest: x.digest, + byteLength: UTF8(x.bytesUtf8).byteLength, + syntheticMarker: "ocx-lab-synthetic-v1", + provenance: { + kind: "lab_authored", + authority: "022_protocol_v1_cases.json", + sourceCommit: source.sourceCommit + } +} +``` + +The marker/provenance fields are mandatory for every protocol V1 fixture +reference and participate in the scenario manifest digest. Registration rejects +any fixture reference whose marker is absent/different, whose provenance kind +is not `lab_authored`, whose authority is not the exact authority file above, +or whose source commit differs from the parsed authority. This is the +machine-checkable synthetic-fixture boundary: a fixture cannot be substituted +from user input, a repository, MCP config, an external URL, or a live response +merely because its bytes share a valid digest. + +`expandFailureRules(case)` copies +`failureRuleSets[manifestDefaults.failureRuleSet]`. When +`case.expectedFailure` is absent, that copy is the complete rule list and no +control-specific rule exists. When present, registration validates the closed +control matrix in the evidence contract and inserts this materialized record +immediately before `required-assertion`: + +```text +id expectedFailureRuleTemplate.id +match expectedFailureRuleTemplate.match +classification case.expectedFailure.expectedClass +secondaryCode case.expectedFailure.expectedCode +verdictEffect none when onMatch is pass, + unsupported when onMatch is unsupported +retry expectedFailureRuleTemplate.retry +expected expectedFailureRuleTemplate.expected +``` + +`expected_failure_exact_match` exists only when the observed class and code +equal `expectedClass`/`expectedCode` and every listed assertion ID passed. An +`onMismatch: fail` falls through to `required-assertion`; an +`onMismatch: inconclusive` falls through to `fallback`. No V1 case declares a +`capability_absence_control`, so no V1 expanded manifest contains an +`unsupported` effect. + +JSON object field order has no digest effect, but the field set above is +closed. Unknown fields reject registration. Arrays preserve source order. +Empty arrays remain present. No default may be read from runtime code. + +The scenario digest is: + +```text +sha256( + UTF8("ocx-lab:scenario-manifest:v1\0") || + UTF8(JCS(expanded scenario)) +) +``` + +The exact fixture bytes are UTF-8 encoding of each `bytesUtf8` field; every +published `fixture` and `initiatingRequest` digest is: + +```text +sha256(UTF8("ocx-lab:fixture:v1\0") || fixture bytes) +``` + +Registration recomputes all digests and rejects mismatch. Every `bytesUtf8` is +retained as a content-addressed fixture artifact, not duplicated into the +expanded manifest. An `upstream_response` case without an initiating +`client_request` rejects registration. + +## 2. Exact suite manifests + +All suite versions are `1.0.0`. Every listed scenario has role `required` +except `vision-core.protocol.modality-gate`, whose role is `negative_control`. +No supplemental protocol V1 scenario exists. Order is the order in the case +authority: + +| Suite | Capability | Required member suffixes | +|---|---|---| +| `responses-core` | `protocol.responses.core` | `request-shape`, `sse-framing`, `item-lifecycle`, `terminal-state`, `json-sse-equivalence` | +| `chat-core` | `protocol.chat.core` | `request-mapping`, `nonstream-envelope`, `stream-assembly`, `stream-terminal` | +| `anthropic-core` | `protocol.anthropic.messages.core` | `request-mapping`, `content-sequence`, `tool-round-trip`, `terminal-errors` | +| `tools-core` | `tools.round_trip` | `function-round-trip`, `custom-freeform-round-trip`, `parallel-correlation`, `result-content`, `choice-and-allowed-set` | +| `codex-core` | `client.codex.core` | `streaming-turn`, `apply-patch-turn`, `tool-continuation`, `previous-response-replay`, `structured-output`, `compaction-and-special-items` | +| `vision-core` | `modalities.image.input` | `input-image`, `tool-result-image`, `modality-gate` | +| `reasoning-core` | `reasoning.round_trip` | `effort-mapping`, `summary-stream`, `replay`, `private-content-isolation` | +| `mcp-core` | `tools.mcp.core` | `namespace-mapping`, `schema-and-bounds`, `call-result`, `resource-round-trip` | + +For each row, the expanded suite manifest is: + +```text +schemaVersion 1 +id table suite +version 1.0.0 +evidenceLayer protocol_conformance +capability table capability +assertionDslVersion 1.0.0 +evidenceSchemaVersion 1.0.0 +freshness { maxAgeMs: null } +contradictionRule newest-required-observation-v1 +scenarios [{ + id: full case ID, + version: 1.0.0, + role: expanded scenario verificationRole, + manifestDigest: expanded scenario digest + }, ...] +verificationRule all-applicable-required-pass-v1 +``` + +Unknown fields reject registration. The suite digest uses +`ocx-lab:suite-manifest:v1` plus JCS exactly as defined by the evidence +contract. `VERIFIED` requires at least one applicable required member, a +current pass for every applicable required member, and the exact +suite/scenario/fixture digests above. + +## 3. Observation selector model + +Assertions use absolute RFC 6901 JSON Pointers into this closed normalized +observation: + +```text +{ + "client": { + "request": { "status": 0, "headers": {}, "json": null, "rawBytes": 0 }, + "response": { + "status": 0, + "headers": {}, + "json": null, + "events": [], + "toolCalls": [], + "mcpCalls": [], + "terminal": null, + "normalizedText": "" + } + }, + "upstream": { + "requests": [ + { "status": 0, "headers": {}, "json": null, "rawBytes": 0 } + ], + "responses": [] + }, + "process": { "exitCode": null }, + "verifiers": {} +} +``` + +Header names are lowercase. JSON pointers use `~0` and `~1` escaping. Array +indexes are decimal with no leading zero except `0`; `-` is forbidden. +Wildcard, filter, recursive descent, script expression, URI, and filesystem +selectors do not exist in V1. + +Unless the operator is `json_path_absent`, a missing selector fails with +`selector_missing`. Unless the operator checks presence/absence, a wrong JSON +type fails with `selector_type_mismatch`. Both are required-assertion failures, +not harness failures. + +Objects compare by JCS bytes. Arrays are order-sensitive. Strings compare +without trimming or Unicode normalization. Numbers use JCS representation. +`null`, missing, empty string, empty array, and empty object are distinct. + +## 4. Fixture roles and execution + +Closed V1 fixture roles are: + +- `client_request`: inject exact bytes at the named inbound protocol surface; +- `upstream_response`: return exact bytes from the loopback mock; +- `adapter_vector`: decode the fixture JSON and feed its documented fields to + the selected adapter boundary without network access; +- `synthetic_tool`: decode the fixture JSON into the in-memory inert tool/MCP + stub and execute only the closed harness action selected below. It never + executes model arguments. + +Closed media types are `application/json`, `text/event-stream`, +`application/vnd.opencodex.adapter-vector+json`, and +`application/vnd.opencodex.mcp-stub+json`. + +Each case's exact `requirements` selects the adapter/surface and harness +features. `adapter_vector` keys are scenario-specific closed input fields +defined by the literal vector and scenario assertions; unknown keys reject the +fixture. CL-01 must encode those fields as a discriminated union keyed by the +scenario ID, not a generic callback or dynamic module. + +The four protocol V1 MCP cases each carry exactly one additional closed +`requiredHarnessFeatures` action token. These tokens are executable semantics +and participate in the scenario manifest digest: + +- `mcp_namespace_round_trip_v1`: decode `namespace`, `name`, `description`, and + `inputSchema`; register exactly one inert tool; serialize its upstream name + as `namespace + "__" + name`; then synthesize exactly one completed + client-visible function call with ID `call_fixture`, that flattened name, and + arguments `{}`. Run the normal `toolCalls` -> `mcpCalls` projection. No stub + result, process, filesystem, or network action occurs. +- `mcp_schema_bounds_v1`: decode only `limitBytes`, `exactSchema`, and + `overSchema`; stage an otherwise-identical single inert tool first with + `exactSchema` and then in a fresh transaction with `overSchema`. The first + transaction commits only when its UTF-8 schema bytes equal the limit; the + second must reject atomically before commit when it is exactly one byte over. + It emits no tool call and performs no invocation. +- `mcp_call_result_v1`: decode `namespace`, `name`, `arguments`, and `result`; + register exactly one inert tool; synthesize exactly one completed call with + ID `call_fixture`; invoke the in-memory stub exactly once with the decoded + namespace/name/arguments; the stub returns the literal decoded `result`, + which becomes `/client/response/json`. Duplicate/missing calls or any extra + invocation fail the required verifier. +- `mcp_resource_round_trip_v1`: decode only `resources` and `read`; install them + in the in-memory resource stub; perform exactly one list operation followed + by exactly one read for `read.uri`; expose the literal list as + `/client/response/json/resources` and the matching literal contents as + `/client/response/json/contents`. Missing, duplicate, or extra operations + fail the required assertion. + +A protocol V1 MCP case missing its scenario-specific action token, carrying the +wrong token, carrying more than one of these tokens, or providing fixture keys +outside that token's closed schema rejects registration. No generic +`synthetic_tool` callback, implicit model output, or implementation-defined +invocation is permitted. + +The MCP cases use only `in_memory_mcp_stub`. No case authorizes stdio, a child +process, user MCP configuration, filesystem access, or a user tool. + +## 5. SSE normalization + +This section is the Lab harness normalizer for protocol V1 observations. It +does not rewrite production sidecar parsers such as +`src/vision/anthropic-describe.ts`; those remain out of CL-00/CL-01 scope and +may keep closed product-local semantics until a later shared-normalizer phase. + +The harness retains exact fixture bytes and normalizes only for assertions: + +1. UTF-8 must decode without replacement. A BOM is allowed only at byte zero + and is removed. +2. CRLF and CR become LF. +3. An empty line terminates a frame. Comment lines beginning `:` are ignored. +4. The first `:` separates field and value. No colon means an empty value. + Exactly one optional leading U+0020 after `:` is removed; no other + whitespace is trimmed. +5. Repeated `data` fields join with LF. The last `event` field wins. +6. Sentinel interpretation follows the protocol of the byte stream being + normalized, not the client-facing `requirements.surfaces` label. For an + `upstream_response` fixture, the source is the case's single resolved + `requirements.upstreamProtocols` entry: only `openai-chat` recognizes a + data value exactly equal to `[DONE]` as a sentinel. `openai-responses` and + `anthropic-messages` do not. A raw fixture with zero or multiple resolved + source protocols is invalid. Client-output normalization instead follows + the emitted client protocol; a Responses bridge's transport `[DONE]` + padding is not reclassified as an upstream Chat sentinel. +7. When `event` is absent and parsed `data` is an object with string `type`, + Responses/Anthropic normalization infers that `type`. Explicit event wins. + Parsed `null`, scalar, array, or empty data is padding and emits no event. + Syntactically malformed nonempty JSON is terminal. +8. Arrival order is preserved; events are never sorted or deduplicated. + +Each normalized event is: + +```text +{ "event": string, "data": JSON value, "ordinal": integer } +``` + +### Semantic call projections + +`/client/response/events` always contains the event records above and is never +treated as an array of bare calls. The normalized observation additionally +contains `/client/response/toolCalls` and `/client/response/mcpCalls`. + +`toolCalls` is built from completed client-visible semantic output items in +response order. For non-streaming Responses, use `output[]`; for Responses SSE, +use each `response.output_item.done.data.item` and reject a completed response +whose added item lacks exactly one done item. Chat fragments are first +translated by the adapter into those client-visible Responses items; the +projection never reads the upstream Chat deltas directly. + +Accepted item shapes and exact projections are: + +```text +function_call: + { id: item.call_id, name: item.name, + arguments: JSON.parse(item.arguments), + kind: "function", ordinal: output order } +custom_tool_call: + { id: item.call_id, name: item.name, + arguments: item.input, + kind: "custom", ordinal: output order } +``` + +Missing/non-string IDs or names, malformed function JSON, duplicate IDs, an +unknown item type, or an added/done mismatch emits no repaired call and causes +the corresponding required assertion to fail. Ordinals are contiguous from +zero; they are not SSE event ordinals. + +`mcpCalls` is derived from `toolCalls` whose name starts with `mcp__` and +contains a final `__` separator. Split at the final separator: + +```text +{ namespace: bytes before final "__", name: bytes after final "__" } +``` + +Empty components, more than 64 UTF-8 bytes per component, invalid UTF-8, or a +non-MCP name emits no MCP call. Order matches `toolCalls`. + +## 6. Assertion operators + +- `http_status_equals`: selected integer equals expected integer. +- `header_present` / `header_absent`: selected lowercase header key exists/does + not exist. +- `header_value_equals`: selected normalized header string equals expected. +- `json_schema_matches`: selected value validates against embedded JSON Schema + draft 2020-12. Only local `$defs`/`$ref` are allowed; coercion, defaults, + custom formats, and network resolution are forbidden. +- `json_path_equals`: selected value equals literal expected under JCS rules. +- `json_path_present` / `json_path_absent`: pointer succeeds/fails; a present + `null` is present. Expected must be literal `true`. +- `sse_field_equals`: selected normalized field equals expected string. +- `sse_event_sequence`: exact event-name array; no subsequence or extras. +- `sse_event_count`: expected is `{event,count}` and exact count is required. +- `terminal_signal_equals`: expected is `completed`, `failed`, `incomplete`, + `done`, `message_stop`, `eof_tolerated`, or `none`. Exactly one terminal is + required unless expected is `none`. +- `id_matches`: expected is a closed grammar: + - `responses_message`: `msg_` plus 1..128 ASCII alphanumeric/underscore/dash; + - `responses_reasoning`: `rs_` plus 1..128 of that set; + - `responses_call`: `call_` plus 1..128 of that set; + - `nonempty_128`: 1..128 printable non-whitespace ASCII characters. + Arbitrary regular expressions are forbidden. +- `id_stable_across_events`: expected is an ordered pointer list. Every + resolved string is byte-equal. +- `id_correlates`: expected is exactly two pointers resolving to byte-equal + strings. +- `tool_call_equals`: selected normalized call equals + `{id,name,arguments,kind,ordinal}`. Function arguments are parsed JSON; + custom/freeform arguments are exact strings. +- `tool_result_correlates`: expected is `{call,result}` pointers. IDs match, + result follows call, and no intervening call reuses the ID. +- `fixture_request_matches`: method, normalized path, allowlisted headers and + JSON body equal the literal fixture expectation. +- `normalized_text_equals`: selected string equals expected exactly. +- `byte_limit_observed`: selected nonnegative integer is `<=` expected. +- `process_exit_equals`: selected integer equals expected. +- `verifier_result_equals`: selected value `pass|fail|blocked|inconclusive` + equals expected. + +Unknown operators, selectors, expected shapes, fixture roles, media types, or +requirements reject registration. + +## 7. Closed verifier derivations + +`/verifiers` is populated only by these pure V1 functions. They may read the +current case's decoded synthetic fixture and normalized observation, but no +clock, random source, network, filesystem, environment, runtime callback, or +model output outside that observation. + +- `json_sse_equivalence`: build the JSON projection + `{text,terminal}` where `text` concatenates, in order, every + `output[].content[]` `output_text.text`, and `terminal` is top-level + `status`. Build the SSE projection where `text` concatenates every + `response.output_text.delta.data.delta`, and `terminal` is the normalized + terminal. Return `pass` iff the two JCS objects are equal, else `fail`. +- `nonoverlap_order`: read `/client/response/toolCalls` in array order. Return + their IDs only when every ID occurs once, each record has a complete + arguments value, and ordinals are contiguous from zero and equal the array + indexes. Otherwise return an empty array. +- `call_result_order`: over the normalized two-turn input, return `pass` iff a + `function_call` occurs in turn 1, exactly one `function_call_output` with the + same `call_id` occurs in turn 2, and no result precedes its call; otherwise + `fail`. +- `compaction_replayed`: return `true` iff the one + `context_compaction.encrypted_content` value is accepted into the parser's + normalized compaction slot and is absent from user-visible output; otherwise + `false`. The synthetic value is never decrypted or executed. +- `local_shell_correlated`: return `true` iff the `local_shell_call.call_id` + equals the following `function_call_output.call_id` and neither item invokes + a process; otherwise `false`. +- `tool_search_error`: for the one failed `tool_search_output`, return its exact + `error` string; missing, duplicate, or non-failed items return `null`. +- `modality_path`: return `native` when `requestHasImage` is true and + `modelInputModalities` contains `image`; otherwise return `sidecar` when an + enabled authorized vision sidecar exists; otherwise return `unsupported`. +- `silent_image_drop`: return `true` only when an image-bearing input is + omitted from adapter output without either a `native`/`sidecar` path or the + typed unsupported rejection; otherwise `false`. +- `exact_bound`: UTF-8 encode `exactSchema`; return `pass` iff its byte length + equals `limitBytes`, JSON parsing succeeds, and the complete staged catalogue + commits; otherwise `fail`. +- `one_over_rejected`: UTF-8 encode `overSchema`; return `pass` iff its byte + length equals `limitBytes + 1` and admission rejects it before commit; + otherwise `fail`. +- `partial_commit`: return `true` iff any tool from the rejected one-byte-over + staging transaction is visible in the committed catalogue; otherwise + `false`. +- `stub_received`: the in-memory MCP stub records exactly + `{namespace,name,arguments}` from the one decoded invocation. Duplicate or + missing invocations produce `null`. + +Verifier outputs use only the literal types above. A missing or type-invalid +input returns `fail`, `false`, `[]`, or `null` as specified and therefore fails +the corresponding required assertion; it is not silently repaired. + +## 8. Failure rules and freshness + +The exact ordered base `protocol-v1-default` records and the one +expected-failure rule template are in the case authority. Control-specific +rules are included only by `expandFailureRules(case)`. Fixture/manifest +integrity and harness failures do not affect compatibility. Time/resource +limits are environmental blockers. The exact expected rejection of the V1 +conformance negative control satisfies that control without producing +`UNSUPPORTED`; no protocol V1 case can produce `UNSUPPORTED`. Other required +deterministic mismatches are `protocol_failure`/`DEGRADED`. + +Protocol V1 scenario and suite freshness are both unbounded (`null`) because +their exact scenario, suite, fixture, compatibility-version, adapter, and +behavior digests invalidate behavior changes. A future profile may still set a +stricter maximum age. + +## 9. CL-01 boundary + +CL-01 may materialize and execute only the protocol manifests in the case +authority. It must: + +1. parse the authority as JSON and reject unknown fields; +2. recompute all fixture, scenario, and suite digests, including the mandatory + synthetic fixture marker/provenance fields; +3. retain the exact case fixture bytes and expanded manifests + content-addressably; +4. execute only loopback mocks, closed adapter vectors, and the four exact + in-memory MCP action tokens above; +5. fail registration rather than invent semantics. + +This document and the JSON authority authorize no runner, mock server, ledger, +SQLite projection, fixture extraction from tests, live probe, or Fabric +ingestion implementation in CL-00. diff --git a/devlog/_plan/260807_compatibility_lab/022_protocol_v1_cases.json b/devlog/_plan/260807_compatibility_lab/022_protocol_v1_cases.json new file mode 100644 index 0000000000..729ed6911b --- /dev/null +++ b/devlog/_plan/260807_compatibility_lab/022_protocol_v1_cases.json @@ -0,0 +1,461 @@ +{ + "schemaVersion": 1, + "authority": "CL-00 design contract; not a runtime registry", + "sourceCommit": "3ad5bb6bd3f76f6879d84b78ea39edd3e01ec296", + "assertionDslVersion": "1.0.0", + "evidenceSchemaVersion": "1.0.0", + "failureRuleSets": { + "protocol-v1-default": [ + { "id": "contract-integrity", "match": ["fixture_digest_mismatch", "manifest_digest_mismatch", "fixture_decode_failure", "harness_failure", "sanitizer_failure"], "classification": "harness_failure", "secondaryCode": "contract_integrity", "verdictEffect": "none", "retry": "never", "expected": false }, + { "id": "time-limit", "match": ["connect_timeout", "first_byte_timeout", "inactivity_timeout", "total_timeout"], "classification": "timeout", "secondaryCode": "scenario_time_limit", "verdictEffect": "none", "retry": "never", "expected": false }, + { "id": "resource-limit", "match": ["request_limit", "input_byte_limit", "output_byte_limit", "output_token_limit", "tool_call_limit", "artifact_byte_limit"], "classification": "budget_exhausted", "secondaryCode": "scenario_resource_limit", "verdictEffect": "none", "retry": "never", "expected": false }, + { "id": "required-assertion", "match": ["required_assertion_failed"], "classification": "protocol_failure", "secondaryCode": "deterministic_assertion", "verdictEffect": "degraded", "retry": "never", "expected": false }, + { "id": "fallback", "match": ["no_prior_rule"], "classification": "inconclusive", "secondaryCode": "unclassified", "verdictEffect": "none", "retry": "never", "expected": false } + ] + }, + "expectedFailureRuleTemplate": { + "id": "expected-failure-exact-match", + "match": ["expected_failure_exact_match"], + "retry": "never", + "expected": true + }, + "manifestDefaults": { + "version": "1.0.0", + "suiteVersion": "1.0.0", + "evidenceLayer": "protocol_conformance", + "verificationRole": "required", + "executionMode": "fixture", + "freshness": { "maxAgeMs": null }, + "executionLimits": { + "totalTimeoutMs": 10000, + "connectTimeoutMs": 1000, + "firstByteTimeoutMs": 2000, + "inactivityTimeoutMs": 2000, + "maxRequests": 4, + "maxInputBytes": 1048576, + "maxOutputBytes": 4194304, + "maxOutputTokens": 4096, + "maxToolCalls": 8, + "maxArtifactBytes": 262144 + }, + "artifactPolicy": { + "allowed": ["assertion_report", "sanitized_request_shape", "sanitized_response_shape", "normalized_event_trace", "sanitized_error"], + "perArtifactBytes": 262144, + "aggregateBytes": 1048576, + "retention": "local_contract", + "publicVisibility": "deny", + "redactionProfile": "synthetic_protocol_v1" + }, + "failureRuleSet": "protocol-v1-default" + }, + "cases": [ + { + "id": "responses-core.protocol.request-shape", + "suite": "responses-core", + "capability": "protocol.responses.core", + "requirements": { "inboundProtocols": ["openai-responses"], "upstreamProtocols": ["openai-chat"], "surfaces": ["responses-http"], "requiredClaims": [], "requiredHarnessFeatures": ["adapter_vector"], "platforms": [], "routePreconditions": [] }, + "fixture": { "id": "rsp-request-shape", "role": "adapter_vector", "mediaType": "application/vnd.opencodex.adapter-vector+json", "bytesUtf8": "{\"modelId\":\"fixture-model\",\"context\":{\"messages\":[{\"role\":\"user\",\"content\":\"PING\",\"timestamp\":0}]},\"stream\":false,\"options\":{\"temperature\":0}}", "digest": "ccc7549e8bcfe4e28d0d4a87c14e622ecfb75973600b5eef830d83620c5bd0f8" }, + "assertions": [ + { "id": "method", "operator": "json_path_equals", "selector": "/upstream/requests/0/json/model", "expected": "fixture-model", "required": true }, + { "id": "message", "operator": "json_path_equals", "selector": "/upstream/requests/0/json/messages/0/content", "expected": "PING", "required": true }, + { "id": "temperature", "operator": "json_path_equals", "selector": "/upstream/requests/0/json/temperature", "expected": 0, "required": true } + ] + }, + { + "id": "responses-core.protocol.sse-framing", + "suite": "responses-core", + "capability": "protocol.responses.core", + "requirements": { "inboundProtocols": ["openai-responses"], "upstreamProtocols": ["openai-responses"], "surfaces": ["responses-sse"], "requiredClaims": [], "requiredHarnessFeatures": ["raw_sse_capture"], "platforms": [], "routePreconditions": [] }, + "initiatingRequest": { "id": "rsp-sse-framing-request", "role": "client_request", "mediaType": "application/json", "bytesUtf8": "{\"model\":\"fixture-model\",\"input\":\"PING\",\"stream\":true}", "digest": "c2c39a78b939e6c5182d3206f86aa86d6fd706959de9c37072db759aa510a1f6" }, + "fixture": { "id": "rsp-sse-framing", "role": "upstream_response", "mediaType": "text/event-stream", "bytesUtf8": "data:{\"type\":\"response.output_text.delta\",\"delta\":\"A\"}\n\ndata: null\n\ndata: {\"type\":\"response.output_text.delta\",\"delta\":\"B\"}\n\ndata:{\"type\":\"response.completed\",\"response\":{\"id\":\"resp_fixture\",\"status\":\"completed\"}}\n\n", "digest": "1c384ef32886054d8f15c14cbcbcc9af4a3bed845d6f820691368614d61515e7" }, + "assertions": [ + { "id": "events", "operator": "sse_event_sequence", "selector": "/client/response/events", "expected": ["response.output_text.delta", "response.output_text.delta", "response.completed"], "required": true }, + { "id": "text", "operator": "normalized_text_equals", "selector": "/client/response/normalizedText", "expected": "AB", "required": true }, + { "id": "terminal", "operator": "terminal_signal_equals", "selector": "/client/response/terminal", "expected": "completed", "required": true } + ] + }, + { + "id": "responses-core.protocol.item-lifecycle", + "suite": "responses-core", + "capability": "protocol.responses.core", + "requirements": { "inboundProtocols": ["openai-responses"], "upstreamProtocols": ["openai-responses"], "surfaces": ["responses-sse"], "requiredClaims": [], "requiredHarnessFeatures": ["raw_sse_capture"], "platforms": [], "routePreconditions": [] }, + "initiatingRequest": { "id": "rsp-item-lifecycle-request", "role": "client_request", "mediaType": "application/json", "bytesUtf8": "{\"model\":\"fixture-model\",\"input\":\"PING\",\"stream\":true}", "digest": "c2c39a78b939e6c5182d3206f86aa86d6fd706959de9c37072db759aa510a1f6" }, + "fixture": { "id": "rsp-item-lifecycle", "role": "upstream_response", "mediaType": "text/event-stream", "bytesUtf8": "event: response.output_item.added\ndata: {\"type\":\"response.output_item.added\",\"output_index\":0,\"item\":{\"id\":\"msg_fixture\",\"type\":\"message\",\"status\":\"in_progress\",\"role\":\"assistant\",\"content\":[]}}\n\nevent: response.output_item.done\ndata: {\"type\":\"response.output_item.done\",\"output_index\":0,\"item\":{\"id\":\"msg_fixture\",\"type\":\"message\",\"status\":\"completed\",\"role\":\"assistant\",\"content\":[]}}\n\nevent: response.completed\ndata: {\"type\":\"response.completed\",\"response\":{\"id\":\"resp_fixture\",\"status\":\"completed\",\"output\":[]}}\n\n", "digest": "ef271e8aaa1d63d51d4e7e0d47facadf39603c1ffa2e871865ace3684feead08" }, + "assertions": [ + { "id": "events", "operator": "sse_event_sequence", "selector": "/client/response/events", "expected": ["response.output_item.added", "response.output_item.done", "response.completed"], "required": true }, + { "id": "stable-id", "operator": "id_stable_across_events", "selector": "/client/response/events", "expected": ["/client/response/events/0/data/item/id", "/client/response/events/1/data/item/id"], "required": true }, + { "id": "id-shape", "operator": "id_matches", "selector": "/client/response/events/0/data/item/id", "expected": "responses_message", "required": true } + ] + }, + { + "id": "responses-core.protocol.terminal-state", + "suite": "responses-core", + "capability": "protocol.responses.core", + "requirements": { "inboundProtocols": ["openai-responses"], "upstreamProtocols": ["openai-responses"], "surfaces": ["responses-sse"], "requiredClaims": [], "requiredHarnessFeatures": ["raw_sse_capture"], "platforms": [], "routePreconditions": [] }, + "initiatingRequest": { "id": "rsp-terminal-state-request", "role": "client_request", "mediaType": "application/json", "bytesUtf8": "{\"model\":\"fixture-model\",\"input\":\"PING\",\"stream\":true}", "digest": "c2c39a78b939e6c5182d3206f86aa86d6fd706959de9c37072db759aa510a1f6" }, + "fixture": { "id": "rsp-terminal-state", "role": "upstream_response", "mediaType": "text/event-stream", "bytesUtf8": "event: response.failed\ndata: {\"type\":\"response.failed\",\"response\":{\"id\":\"resp_fixture\",\"status\":\"failed\",\"error\":{\"type\":\"server_error\",\"code\":\"fixture_failure\"}}}\n\n", "digest": "472735364ce0ee28e68192d478ccb658ec8d6a149dba6fe914e5ab35cc1a41d7" }, + "assertions": [ + { "id": "events", "operator": "sse_event_sequence", "selector": "/client/response/events", "expected": ["response.failed"], "required": true }, + { "id": "count", "operator": "sse_event_count", "selector": "/client/response/events", "expected": { "event": "response.failed", "count": 1 }, "required": true }, + { "id": "terminal", "operator": "terminal_signal_equals", "selector": "/client/response/terminal", "expected": "failed", "required": true } + ] + }, + { + "id": "responses-core.protocol.json-sse-equivalence", + "suite": "responses-core", + "capability": "protocol.responses.core", + "requirements": { "inboundProtocols": ["openai-responses"], "upstreamProtocols": ["openai-responses"], "surfaces": ["responses-http", "responses-sse"], "requiredClaims": [], "requiredHarnessFeatures": ["adapter_vector", "raw_sse_capture"], "platforms": [], "routePreconditions": [] }, + "fixture": { "id": "rsp-json-sse-equivalence", "role": "adapter_vector", "mediaType": "application/vnd.opencodex.adapter-vector+json", "bytesUtf8": "{\"json\":{\"id\":\"resp_fixture\",\"status\":\"completed\",\"output\":[{\"id\":\"msg_fixture\",\"type\":\"message\",\"role\":\"assistant\",\"status\":\"completed\",\"content\":[{\"type\":\"output_text\",\"text\":\"OK\"}]}]},\"sse\":\"event: response.output_text.delta\\ndata: {\\\"type\\\":\\\"response.output_text.delta\\\",\\\"delta\\\":\\\"OK\\\"}\\n\\nevent: response.completed\\ndata: {\\\"type\\\":\\\"response.completed\\\",\\\"response\\\":{\\\"id\\\":\\\"resp_fixture\\\",\\\"status\\\":\\\"completed\\\"}}\\n\\n\"}", "digest": "b7288170258b91361530d1dd5a0a818859ff9b6176793554ec8b0ae1177d87cf" }, + "assertions": [ + { "id": "text", "operator": "normalized_text_equals", "selector": "/client/response/normalizedText", "expected": "OK", "required": true }, + { "id": "terminal", "operator": "terminal_signal_equals", "selector": "/client/response/terminal", "expected": "completed", "required": true }, + { "id": "equivalent", "operator": "verifier_result_equals", "selector": "/verifiers/json_sse_equivalence", "expected": "pass", "required": true } + ] + }, + { + "id": "chat-core.protocol.request-mapping", + "suite": "chat-core", + "capability": "protocol.chat.core", + "requirements": { "inboundProtocols": ["openai-responses"], "upstreamProtocols": ["openai-chat"], "surfaces": ["responses-http"], "requiredClaims": [], "requiredHarnessFeatures": ["adapter_vector"], "platforms": [], "routePreconditions": [] }, + "fixture": { "id": "chat-request-mapping", "role": "adapter_vector", "mediaType": "application/vnd.opencodex.adapter-vector+json", "bytesUtf8": "{\"context\":{\"systemPrompt\":[\"SYS\"],\"messages\":[{\"role\":\"developer\",\"content\":\"DEV\",\"timestamp\":0},{\"role\":\"user\",\"content\":\"PING\",\"timestamp\":1}]},\"options\":{\"textFormat\":{\"type\":\"json_object\"}}}", "digest": "0a9c319b3a6dadbf581d0d2185f57527cd28aa57127e0eac91a421735b4c2ad9" }, + "assertions": [ + { "id": "roles", "operator": "json_path_equals", "selector": "/upstream/requests/0/json/messages", "expected": [{"role":"system","content":"SYS"},{"role":"developer","content":"DEV"},{"role":"user","content":"PING"}], "required": true }, + { "id": "format", "operator": "json_path_equals", "selector": "/upstream/requests/0/json/response_format", "expected": {"type":"json_object"}, "required": true } + ] + }, + { + "id": "chat-core.protocol.nonstream-envelope", + "suite": "chat-core", + "capability": "protocol.chat.core", + "requirements": { "inboundProtocols": ["openai-responses"], "upstreamProtocols": ["openai-chat"], "surfaces": ["responses-http"], "requiredClaims": [], "requiredHarnessFeatures": ["adapter_vector"], "platforms": [], "routePreconditions": [] }, + "initiatingRequest": { "id": "chat-nonstream-envelope-request", "role": "client_request", "mediaType": "application/json", "bytesUtf8": "{\"model\":\"fixture-model\",\"input\":\"PING\",\"stream\":false}", "digest": "4f6e495840e4fc80f833aa8cc09c09ee765ae9f8134db70565f3445989892db7" }, + "fixture": { "id": "chat-nonstream-envelope", "role": "upstream_response", "mediaType": "application/json", "bytesUtf8": "{\"id\":\"chatcmpl_fixture\",\"choices\":[{\"index\":0,\"message\":{\"role\":\"assistant\",\"content\":\"OK\"},\"finish_reason\":\"stop\"}],\"usage\":{\"prompt_tokens\":1,\"completion_tokens\":1,\"total_tokens\":2}}", "digest": "4a9a0352daa284e73850ce613b1cc939a534d6a930c7b68c8eb28e3fcca5b248" }, + "assertions": [ + { "id": "status", "operator": "http_status_equals", "selector": "/client/response/status", "expected": 200, "required": true }, + { "id": "text", "operator": "normalized_text_equals", "selector": "/client/response/normalizedText", "expected": "OK", "required": true }, + { "id": "terminal", "operator": "terminal_signal_equals", "selector": "/client/response/terminal", "expected": "completed", "required": true } + ] + }, + { + "id": "chat-core.protocol.stream-assembly", + "suite": "chat-core", + "capability": "protocol.chat.core", + "requirements": { "inboundProtocols": ["openai-responses"], "upstreamProtocols": ["openai-chat"], "surfaces": ["responses-sse"], "requiredClaims": [], "requiredHarnessFeatures": ["raw_sse_capture"], "platforms": [], "routePreconditions": [] }, + "initiatingRequest": { "id": "chat-stream-assembly-request", "role": "client_request", "mediaType": "application/json", "bytesUtf8": "{\"model\":\"fixture-model\",\"input\":\"PING\",\"stream\":true}", "digest": "c2c39a78b939e6c5182d3206f86aa86d6fd706959de9c37072db759aa510a1f6" }, + "fixture": { "id": "chat-stream-assembly", "role": "upstream_response", "mediaType": "text/event-stream", "bytesUtf8": "data: {\"choices\":[{\"index\":0,\"delta\":{\"tool_calls\":[{\"index\":0,\"id\":\"call_a\",\"function\":{\"name\":\"alpha\",\"arguments\":\"{\\\"x\\\":\"}},{\"index\":1,\"id\":\"call_b\",\"function\":{\"name\":\"beta\",\"arguments\":\"{\\\"y\\\":\"}}]}}]}\n\ndata: {\"choices\":[{\"index\":0,\"delta\":{\"tool_calls\":[{\"index\":1,\"function\":{\"arguments\":\"2}\"}},{\"index\":0,\"function\":{\"arguments\":\"1}\"}}]},\"finish_reason\":\"tool_calls\"}]}\n\ndata: [DONE]\n\n", "digest": "0085f298a8690aefb74bb09ea2e0cb77703aaf6cfd822c6ce0d4d334ad4b9b3f" }, + "assertions": [ + { "id": "alpha", "operator": "tool_call_equals", "selector": "/client/response/toolCalls/0", "expected": {"id":"call_a","name":"alpha","arguments":{"x":1},"kind":"function","ordinal":0}, "required": true }, + { "id": "beta", "operator": "tool_call_equals", "selector": "/client/response/toolCalls/1", "expected": {"id":"call_b","name":"beta","arguments":{"y":2},"kind":"function","ordinal":1}, "required": true }, + { "id": "terminal", "operator": "terminal_signal_equals", "selector": "/client/response/terminal", "expected": "completed", "required": true } + ] + }, + { + "id": "chat-core.protocol.stream-terminal", + "suite": "chat-core", + "capability": "protocol.chat.core", + "requirements": { "inboundProtocols": ["openai-responses"], "upstreamProtocols": ["openai-chat"], "surfaces": ["responses-sse"], "requiredClaims": [], "requiredHarnessFeatures": ["raw_sse_capture"], "platforms": [], "routePreconditions": [] }, + "initiatingRequest": { "id": "chat-stream-terminal-request", "role": "client_request", "mediaType": "application/json", "bytesUtf8": "{\"model\":\"fixture-model\",\"input\":\"PING\",\"stream\":true}", "digest": "c2c39a78b939e6c5182d3206f86aa86d6fd706959de9c37072db759aa510a1f6" }, + "fixture": { "id": "chat-stream-terminal", "role": "upstream_response", "mediaType": "text/event-stream", "bytesUtf8": "data: {\"choices\":[{\"index\":0,\"delta\":{\"content\":\"OK\"},\"finish_reason\":\"stop\"}]}\n\ndata: [DONE]\n\n", "digest": "6e0e4e8d32d8575db6a09e89c222b16338e1499e940e038599f7a6b5332e59e6" }, + "assertions": [ + { "id": "text", "operator": "normalized_text_equals", "selector": "/client/response/normalizedText", "expected": "OK", "required": true }, + { "id": "terminal", "operator": "terminal_signal_equals", "selector": "/client/response/terminal", "expected": "completed", "required": true } + ] + }, + { + "id": "anthropic-core.protocol.request-mapping", + "suite": "anthropic-core", + "capability": "protocol.anthropic.messages.core", + "requirements": { "inboundProtocols": ["anthropic-messages"], "upstreamProtocols": ["openai-responses"], "surfaces": ["anthropic-http"], "requiredClaims": [], "requiredHarnessFeatures": ["adapter_vector"], "platforms": [], "routePreconditions": [] }, + "fixture": { "id": "anthropic-request-mapping", "role": "client_request", "mediaType": "application/json", "bytesUtf8": "{\"model\":\"fixture-model\",\"system\":\"SYS\",\"messages\":[{\"role\":\"user\",\"content\":\"PING\"}],\"max_tokens\":32,\"stream\":false}", "digest": "deeca799f660f413d0cb85263aa332bdc05aa995ccf8c7322f6af43e9bf6a627" }, + "assertions": [ + { "id": "model", "operator": "json_path_equals", "selector": "/upstream/requests/0/json/model", "expected": "fixture-model", "required": true }, + { "id": "system", "operator": "json_path_equals", "selector": "/upstream/requests/0/json/instructions", "expected": "SYS", "required": true }, + { "id": "input", "operator": "json_path_equals", "selector": "/upstream/requests/0/json/input/0/content/0/text", "expected": "PING", "required": true } + ] + }, + { + "id": "anthropic-core.protocol.content-sequence", + "suite": "anthropic-core", + "capability": "protocol.anthropic.messages.core", + "requirements": { "inboundProtocols": ["anthropic-messages"], "upstreamProtocols": ["openai-responses"], "surfaces": ["anthropic-sse"], "requiredClaims": [], "requiredHarnessFeatures": ["raw_sse_capture"], "platforms": [], "routePreconditions": [] }, + "initiatingRequest": { "id": "anthropic-content-sequence-request", "role": "client_request", "mediaType": "application/json", "bytesUtf8": "{\"model\":\"fixture-model\",\"messages\":[{\"role\":\"user\",\"content\":\"PING\"}],\"max_tokens\":32,\"stream\":true}", "digest": "96e15d2044ccaacca81d32bda4157e4baf82ef98c7640f27034e10285f5de8f3" }, + "fixture": { "id": "anthropic-content-sequence", "role": "upstream_response", "mediaType": "text/event-stream", "bytesUtf8": "data:{\"type\":\"response.output_text.delta\",\"delta\":\"OK\"}\n\ndata:{\"type\":\"response.completed\",\"response\":{\"id\":\"resp_fixture\",\"status\":\"completed\",\"usage\":{\"input_tokens\":1,\"output_tokens\":1}}}\n\n", "digest": "1f8148d142038f42fadf4b3e938b45f4313986cbbd6338feac3b8db8f355299a" }, + "assertions": [ + { "id": "events", "operator": "sse_event_sequence", "selector": "/client/response/events", "expected": ["message_start","content_block_start","content_block_delta","content_block_stop","message_delta","message_stop"], "required": true }, + { "id": "terminal", "operator": "terminal_signal_equals", "selector": "/client/response/terminal", "expected": "message_stop", "required": true } + ] + }, + { + "id": "anthropic-core.protocol.tool-round-trip", + "suite": "anthropic-core", + "capability": "protocol.anthropic.messages.core", + "requirements": { "inboundProtocols": ["anthropic-messages"], "upstreamProtocols": ["openai-responses"], "surfaces": ["anthropic-http"], "requiredClaims": ["tools"], "requiredHarnessFeatures": ["adapter_vector"], "platforms": [], "routePreconditions": [] }, + "fixture": { "id": "anthropic-tool-roundtrip", "role": "client_request", "mediaType": "application/json", "bytesUtf8": "{\"model\":\"fixture-model\",\"messages\":[{\"role\":\"assistant\",\"content\":[{\"type\":\"tool_use\",\"id\":\"call_fixture\",\"name\":\"lookup\",\"input\":{\"q\":\"x\"}}]},{\"role\":\"user\",\"content\":[{\"type\":\"tool_result\",\"tool_use_id\":\"call_fixture\",\"content\":\"RESULT\"}]}],\"tools\":[{\"name\":\"lookup\",\"input_schema\":{\"type\":\"object\",\"properties\":{\"q\":{\"type\":\"string\"}},\"required\":[\"q\"]}}],\"max_tokens\":32}", "digest": "f8dfefb427ce81fb4570f83d8d24c91e79350a7a256ecde7e636e0d70fbdff64" }, + "assertions": [ + { "id": "call-id", "operator": "id_correlates", "selector": "/upstream/requests", "expected": ["/upstream/requests/0/json/input/0/call_id","/upstream/requests/0/json/input/1/call_id"], "required": true }, + { "id": "result", "operator": "json_path_equals", "selector": "/upstream/requests/0/json/input/1/output", "expected": "RESULT", "required": true } + ] + }, + { + "id": "anthropic-core.protocol.terminal-errors", + "suite": "anthropic-core", + "capability": "protocol.anthropic.messages.core", + "requirements": { "inboundProtocols": ["anthropic-messages"], "upstreamProtocols": ["openai-responses"], "surfaces": ["anthropic-sse"], "requiredClaims": [], "requiredHarnessFeatures": ["raw_sse_capture"], "platforms": [], "routePreconditions": [] }, + "initiatingRequest": { "id": "anthropic-terminal-error-request", "role": "client_request", "mediaType": "application/json", "bytesUtf8": "{\"model\":\"fixture-model\",\"messages\":[{\"role\":\"user\",\"content\":\"PING\"}],\"max_tokens\":32,\"stream\":true}", "digest": "96e15d2044ccaacca81d32bda4157e4baf82ef98c7640f27034e10285f5de8f3" }, + "fixture": { "id": "anthropic-terminal-error", "role": "upstream_response", "mediaType": "text/event-stream", "bytesUtf8": "event: response.failed\ndata: {\"type\":\"response.failed\",\"response\":{\"status\":\"failed\",\"error\":{\"type\":\"server_error\",\"code\":\"overloaded\",\"message\":\"fixture\"}}}\n\n", "digest": "fad0d0edca35d066e89de5488635a2912930d5dedc79d047759fb6ecc6567718" }, + "assertions": [ + { "id": "events", "operator": "sse_event_sequence", "selector": "/client/response/events", "expected": ["error"], "required": true }, + { "id": "terminal", "operator": "terminal_signal_equals", "selector": "/client/response/terminal", "expected": "failed", "required": true } + ] + }, + { + "id": "tools-core.protocol.function-round-trip", + "suite": "tools-core", + "capability": "tools.round_trip", + "requirements": { "inboundProtocols": ["openai-responses"], "upstreamProtocols": ["openai-chat"], "surfaces": ["responses-http"], "requiredClaims": ["tools"], "requiredHarnessFeatures": ["adapter_vector"], "platforms": [], "routePreconditions": [] }, + "fixture": { "id": "tools-function", "role": "adapter_vector", "mediaType": "application/vnd.opencodex.adapter-vector+json", "bytesUtf8": "{\"tools\":[{\"name\":\"lookup\",\"parameters\":{\"type\":\"object\",\"properties\":{\"q\":{\"type\":\"string\"}},\"required\":[\"q\"]}}],\"upstreamToolCall\":{\"id\":\"call_fixture\",\"name\":\"lookup\",\"arguments\":\"{\\\"q\\\":\\\"x\\\"}\"},\"toolResult\":{\"toolCallId\":\"call_fixture\",\"content\":\"RESULT\"}}", "digest": "9107f4dfdd7da8340c866c9fb6f42854437cebb98592d0510969c810c1eeb0ad" }, + "assertions": [ + { "id": "call", "operator": "tool_call_equals", "selector": "/client/response/toolCalls/0", "expected": {"id":"call_fixture","name":"lookup","arguments":{"q":"x"},"kind":"function","ordinal":0}, "required": true }, + { "id": "result", "operator": "tool_result_correlates", "selector": "/upstream/requests", "expected": {"call":"/client/response/toolCalls/0/id","result":"/upstream/requests/1/json/messages/1/tool_call_id"}, "required": true } + ] + }, + { + "id": "tools-core.protocol.custom-freeform-round-trip", + "suite": "tools-core", + "capability": "tools.round_trip", + "requirements": { "inboundProtocols": ["openai-responses"], "upstreamProtocols": ["openai-responses"], "surfaces": ["responses-http"], "requiredClaims": ["custom_tools"], "requiredHarnessFeatures": ["adapter_vector"], "platforms": [], "routePreconditions": [] }, + "fixture": { "id": "tools-custom", "role": "adapter_vector", "mediaType": "application/vnd.opencodex.adapter-vector+json", "bytesUtf8": "{\"tool\":{\"type\":\"custom\",\"name\":\"apply_patch\",\"format\":{\"type\":\"grammar\",\"syntax\":\"lark\",\"definition\":\"start: /[\\\\s\\\\S]+/\"}},\"call\":{\"id\":\"call_patch\",\"name\":\"apply_patch\",\"input\":\"*** Begin Patch\\n*** End Patch\\n\"},\"output\":{\"call_id\":\"call_patch\",\"output\":\"Done\"}}", "digest": "752750104e99602d9160feaa591bcbfcfd0c8c53fc9feda4a48c3b6813b74d44" }, + "assertions": [ + { "id": "call", "operator": "tool_call_equals", "selector": "/client/response/toolCalls/0", "expected": {"id":"call_patch","name":"apply_patch","arguments":"*** Begin Patch\n*** End Patch\n","kind":"custom","ordinal":0}, "required": true }, + { "id": "result", "operator": "tool_result_correlates", "selector": "/upstream/requests", "expected": {"call":"/client/response/toolCalls/0/id","result":"/upstream/requests/1/json/input/0/call_id"}, "required": true } + ] + }, + { + "id": "tools-core.protocol.parallel-correlation", + "suite": "tools-core", + "capability": "tools.round_trip", + "requirements": { "inboundProtocols": ["openai-responses"], "upstreamProtocols": ["openai-chat"], "surfaces": ["responses-sse"], "requiredClaims": ["parallel_tools"], "requiredHarnessFeatures": ["raw_sse_capture"], "platforms": [], "routePreconditions": [] }, + "initiatingRequest": { "id": "tools-parallel-request", "role": "client_request", "mediaType": "application/json", "bytesUtf8": "{\"model\":\"fixture-model\",\"input\":\"PING\",\"stream\":true}", "digest": "c2c39a78b939e6c5182d3206f86aa86d6fd706959de9c37072db759aa510a1f6" }, + "fixture": { "id": "tools-parallel", "role": "upstream_response", "mediaType": "text/event-stream", "bytesUtf8": "data:{\"choices\":[{\"delta\":{\"tool_calls\":[{\"index\":0,\"id\":\"call_a\",\"function\":{\"name\":\"a\",\"arguments\":\"{\"}},{\"index\":1,\"id\":\"call_b\",\"function\":{\"name\":\"b\",\"arguments\":\"{\"}}]}}]}\n\ndata:{\"choices\":[{\"delta\":{\"tool_calls\":[{\"index\":1,\"function\":{\"arguments\":\"}\"}},{\"index\":0,\"function\":{\"arguments\":\"}\"}}]},\"finish_reason\":\"tool_calls\"}]}\n\ndata:[DONE]\n\n", "digest": "7a954d390bdf48d0dec3ed2515a5bbbedd4165656fcb7f0643ce743d17bb39f0" }, + "assertions": [ + { "id": "calls", "operator": "json_path_equals", "selector": "/client/response/toolCalls", "expected": [{"id":"call_a","name":"a","arguments":{},"kind":"function","ordinal":0},{"id":"call_b","name":"b","arguments":{},"kind":"function","ordinal":1}], "required": true }, + { "id": "order", "operator": "json_path_equals", "selector": "/verifiers/nonoverlap_order", "expected": ["call_a","call_b"], "required": true } + ] + }, + { + "id": "tools-core.protocol.result-content", + "suite": "tools-core", + "capability": "tools.round_trip", + "requirements": { "inboundProtocols": ["openai-responses"], "upstreamProtocols": ["openai-chat"], "surfaces": ["responses-http"], "requiredClaims": ["tools","image"], "requiredHarnessFeatures": ["adapter_vector"], "platforms": [], "routePreconditions": [] }, + "fixture": { "id": "tools-result-content", "role": "adapter_vector", "mediaType": "application/vnd.opencodex.adapter-vector+json", "bytesUtf8": "{\"callId\":\"call_fixture\",\"content\":[{\"type\":\"input_text\",\"text\":\"RESULT\"},{\"type\":\"input_image\",\"image_url\":\"data:image/png;base64,iVBORw0KGgo=\",\"detail\":\"high\"}],\"isError\":false}", "digest": "ec81d47d3d6a67254afcc21b55f458269d8dd34ab3b3d52a6c12fec9bec814ab" }, + "assertions": [ + { "id": "text", "operator": "json_path_equals", "selector": "/upstream/requests/0/json/messages/0/content", "expected": "RESULT", "required": true }, + { "id": "image", "operator": "json_path_equals", "selector": "/upstream/requests/0/json/messages/1/content/0/image_url/url", "expected": "data:image/png;base64,iVBORw0KGgo=", "required": true } + ] + }, + { + "id": "tools-core.protocol.choice-and-allowed-set", + "suite": "tools-core", + "capability": "tools.round_trip", + "requirements": { "inboundProtocols": ["openai-responses"], "upstreamProtocols": ["openai-chat"], "surfaces": ["responses-http"], "requiredClaims": ["tools"], "requiredHarnessFeatures": ["adapter_vector"], "platforms": [], "routePreconditions": [] }, + "fixture": { "id": "tools-choice", "role": "client_request", "mediaType": "application/json", "bytesUtf8": "{\"model\":\"fixture-model\",\"input\":\"PING\",\"tools\":[{\"type\":\"function\",\"name\":\"alpha\",\"parameters\":{\"type\":\"object\"}},{\"type\":\"function\",\"name\":\"beta\",\"parameters\":{\"type\":\"object\"}}],\"tool_choice\":{\"type\":\"allowed_tools\",\"mode\":\"required\",\"tools\":[{\"type\":\"function\",\"name\":\"beta\"}]}}", "digest": "fe8b6dde44f88cb9e9a7c6b2bb290e2ee57e7ed425ca8249fb4b7804feff148a" }, + "assertions": [ + { "id": "choice", "operator": "json_path_equals", "selector": "/upstream/requests/0/json/tool_choice", "expected": {"type":"function","function":{"name":"beta"}}, "required": true }, + { "id": "set", "operator": "json_path_equals", "selector": "/upstream/requests/0/json/tools", "expected": [{"type":"function","function":{"name":"beta","parameters":{"type":"object"}}}], "required": true } + ] + }, + { + "id": "codex-core.protocol.streaming-turn", + "suite": "codex-core", + "capability": "client.codex.core", + "requirements": { "inboundProtocols": ["openai-responses"], "upstreamProtocols": ["openai-chat"], "surfaces": ["responses-sse"], "requiredClaims": [], "requiredHarnessFeatures": ["raw_sse_capture"], "platforms": [], "routePreconditions": [] }, + "initiatingRequest": { "id": "codex-streaming-request", "role": "client_request", "mediaType": "application/json", "bytesUtf8": "{\"model\":\"fixture-model\",\"input\":\"PING\",\"stream\":true}", "digest": "c2c39a78b939e6c5182d3206f86aa86d6fd706959de9c37072db759aa510a1f6" }, + "fixture": { "id": "codex-streaming", "role": "upstream_response", "mediaType": "text/event-stream", "bytesUtf8": "data:{\"choices\":[{\"delta\":{\"content\":\"OK\"},\"finish_reason\":\"stop\"}],\"usage\":{\"prompt_tokens\":1,\"completion_tokens\":1}}\n\ndata:[DONE]\n\n", "digest": "f109d35734ecca8e71226ff739b6a0783aca283a9b0d0238a7974b2d7fd9af53" }, + "assertions": [ + { "id": "text", "operator": "normalized_text_equals", "selector": "/client/response/normalizedText", "expected": "OK", "required": true }, + { "id": "terminal", "operator": "terminal_signal_equals", "selector": "/client/response/terminal", "expected": "completed", "required": true }, + { "id": "phase", "operator": "json_path_equals", "selector": "/client/response/events/6/data/item/phase", "expected": "final_answer", "required": true } + ] + }, + { + "id": "codex-core.protocol.apply-patch-turn", + "suite": "codex-core", + "capability": "client.codex.core", + "requirements": { "inboundProtocols": ["openai-responses"], "upstreamProtocols": ["openai-chat"], "surfaces": ["responses-http"], "requiredClaims": ["custom_tools"], "requiredHarnessFeatures": ["adapter_vector"], "platforms": [], "routePreconditions": [] }, + "fixture": { "id": "codex-patch", "role": "adapter_vector", "mediaType": "application/vnd.opencodex.adapter-vector+json", "bytesUtf8": "{\"name\":\"apply_patch\",\"input\":\"*** Begin Patch\\n*** Add File: x\\n+x\\n*** End Patch\\n\",\"callId\":\"call_patch\",\"result\":\"Done\"}", "digest": "668baa1fbea1d7a6556f717467fc3b90a47b2edfaa2ccf0c7950fd30dfe27a81" }, + "assertions": [ + { "id": "call", "operator": "tool_call_equals", "selector": "/client/response/toolCalls/0", "expected": {"id":"call_patch","name":"apply_patch","arguments":"*** Begin Patch\n*** Add File: x\n+x\n*** End Patch\n","kind":"custom","ordinal":0}, "required": true }, + { "id": "result", "operator": "tool_result_correlates", "selector": "/upstream/requests", "expected": {"call":"/client/response/toolCalls/0/id","result":"/upstream/requests/1/json/messages/1/tool_call_id"}, "required": true } + ] + }, + { + "id": "codex-core.protocol.tool-continuation", + "suite": "codex-core", + "capability": "client.codex.core", + "requirements": { "inboundProtocols": ["openai-responses"], "upstreamProtocols": ["openai-responses"], "surfaces": ["responses-http"], "requiredClaims": ["tools"], "requiredHarnessFeatures": ["adapter_vector"], "platforms": [], "routePreconditions": [] }, + "fixture": { "id": "codex-tool-continuation", "role": "adapter_vector", "mediaType": "application/vnd.opencodex.adapter-vector+json", "bytesUtf8": "{\"turn1\":{\"output\":[{\"type\":\"function_call\",\"id\":\"fc_fixture\",\"call_id\":\"call_fixture\",\"name\":\"lookup\",\"arguments\":\"{}\"}]},\"turn2\":{\"input\":[{\"type\":\"function_call_output\",\"call_id\":\"call_fixture\",\"output\":\"RESULT\"}]}}", "digest": "0b1e955829282c51e056e0bd1d6eb88d62fbae1accd52bdda579c3fce9eac205" }, + "assertions": [ + { "id": "correlation", "operator": "id_correlates", "selector": "/upstream/requests", "expected": ["/upstream/requests/0/json/input/0/call_id","/upstream/requests/0/json/input/1/call_id"], "required": true }, + { "id": "order", "operator": "json_path_equals", "selector": "/verifiers/call_result_order", "expected": "pass", "required": true } + ] + }, + { + "id": "codex-core.protocol.previous-response-replay", + "suite": "codex-core", + "capability": "client.codex.core", + "requirements": { "inboundProtocols": ["openai-responses"], "upstreamProtocols": ["openai-responses"], "surfaces": ["responses-http"], "requiredClaims": [], "requiredHarnessFeatures": ["adapter_vector"], "platforms": [], "routePreconditions": [] }, + "fixture": { "id": "codex-replay", "role": "adapter_vector", "mediaType": "application/vnd.opencodex.adapter-vector+json", "bytesUtf8": "{\"stored\":{\"id\":\"resp_prev\",\"input\":[{\"role\":\"user\",\"content\":\"ONE\"}],\"output\":[{\"role\":\"assistant\",\"content\":\"TWO\"}]},\"next\":{\"previous_response_id\":\"resp_prev\",\"input\":[{\"role\":\"user\",\"content\":\"THREE\"}]}}", "digest": "e849a72d9772616a5ca8853bef48fd2f0884fd006b9ac747bd442513ad05e0f4" }, + "assertions": [ + { "id": "expanded", "operator": "json_path_equals", "selector": "/upstream/requests/0/json/input", "expected": [{"role":"user","content":"ONE"},{"role":"assistant","content":"TWO"},{"role":"user","content":"THREE"}], "required": true }, + { "id": "private-id", "operator": "json_path_absent", "selector": "/upstream/requests/0/json/previous_response_id", "expected": true, "required": true } + ] + }, + { + "id": "codex-core.protocol.structured-output", + "suite": "codex-core", + "capability": "client.codex.core", + "requirements": { "inboundProtocols": ["openai-responses"], "upstreamProtocols": ["openai-chat"], "surfaces": ["responses-http"], "requiredClaims": ["structured_output"], "requiredHarnessFeatures": ["adapter_vector"], "platforms": [], "routePreconditions": [] }, + "fixture": { "id": "codex-structured", "role": "client_request", "mediaType": "application/json", "bytesUtf8": "{\"model\":\"fixture-model\",\"input\":\"PING\",\"text\":{\"format\":{\"type\":\"json_schema\",\"name\":\"answer\",\"schema\":{\"type\":\"object\",\"properties\":{\"ok\":{\"type\":\"boolean\"}},\"required\":[\"ok\"],\"additionalProperties\":false},\"strict\":true}}}", "digest": "e6278954535f4d482a9bb1f6c0189ef7aed00294a7bde695747b7898886cf937" }, + "assertions": [ + { "id": "format", "operator": "json_path_equals", "selector": "/upstream/requests/0/json/response_format", "expected": {"type":"json_schema","json_schema":{"name":"answer","schema":{"type":"object","properties":{"ok":{"type":"boolean"}},"required":["ok"],"additionalProperties":false},"strict":true}}, "required": true } + ] + }, + { + "id": "codex-core.protocol.compaction-and-special-items", + "suite": "codex-core", + "capability": "client.codex.core", + "requirements": { "inboundProtocols": ["openai-responses"], "upstreamProtocols": ["openai-chat"], "surfaces": ["responses-http"], "requiredClaims": [], "requiredHarnessFeatures": ["adapter_vector"], "platforms": [], "routePreconditions": [] }, + "fixture": { "id": "codex-special-items", "role": "client_request", "mediaType": "application/json", "bytesUtf8": "{\"model\":\"fixture-model\",\"input\":[{\"type\":\"context_compaction\",\"encrypted_content\":\"ocx1:fixture\"},{\"type\":\"local_shell_call\",\"id\":\"shell_fixture\",\"call_id\":\"call_shell\",\"status\":\"completed\",\"action\":{\"type\":\"exec\",\"command\":[\"echo\",\"ok\"]}},{\"type\":\"function_call_output\",\"call_id\":\"call_shell\",\"output\":\"ok\"},{\"type\":\"tool_search_output\",\"status\":\"failed\",\"error\":\"fixture\"}]}", "digest": "bf61cb0783f288a4dd6b0b8c0f9a2ddb60f02d0f8ea1d2875ea7e3fe1740b043" }, + "assertions": [ + { "id": "compaction", "operator": "json_path_equals", "selector": "/verifiers/compaction_replayed", "expected": true, "required": true }, + { "id": "shell", "operator": "json_path_equals", "selector": "/verifiers/local_shell_correlated", "expected": true, "required": true }, + { "id": "search", "operator": "json_path_equals", "selector": "/verifiers/tool_search_error", "expected": "fixture", "required": true } + ] + }, + { + "id": "vision-core.protocol.input-image", + "suite": "vision-core", + "capability": "modalities.image.input", + "requirements": { "inboundProtocols": ["openai-responses"], "upstreamProtocols": ["openai-chat"], "surfaces": ["responses-http"], "requiredClaims": ["image"], "requiredHarnessFeatures": ["adapter_vector"], "platforms": [], "routePreconditions": [] }, + "fixture": { "id": "vision-input", "role": "client_request", "mediaType": "application/json", "bytesUtf8": "{\"model\":\"fixture-model\",\"input\":[{\"role\":\"user\",\"content\":[{\"type\":\"input_text\",\"text\":\"READ\"},{\"type\":\"input_image\",\"image_url\":\"data:image/png;base64,iVBORw0KGgo=\",\"detail\":\"high\"}]}]}", "digest": "a26ba5209858c3658d698c1dcb6c92845b2e6aae6bab70a7b9ad1cba1d8aa6a5" }, + "assertions": [ + { "id": "text", "operator": "json_path_equals", "selector": "/upstream/requests/0/json/messages/0/content/0/text", "expected": "READ", "required": true }, + { "id": "image", "operator": "json_path_equals", "selector": "/upstream/requests/0/json/messages/0/content/1/image_url", "expected": {"url":"data:image/png;base64,iVBORw0KGgo=","detail":"high"}, "required": true } + ] + }, + { + "id": "vision-core.protocol.tool-result-image", + "suite": "vision-core", + "capability": "modalities.image.input", + "requirements": { "inboundProtocols": ["openai-responses"], "upstreamProtocols": ["openai-chat"], "surfaces": ["responses-http"], "requiredClaims": ["tools","image"], "requiredHarnessFeatures": ["adapter_vector"], "platforms": [], "routePreconditions": [] }, + "fixture": { "id": "vision-tool-result", "role": "adapter_vector", "mediaType": "application/vnd.opencodex.adapter-vector+json", "bytesUtf8": "{\"callId\":\"call_fixture\",\"result\":[{\"type\":\"input_text\",\"text\":\"RESULT\"},{\"type\":\"input_image\",\"image_url\":\"data:image/png;base64,iVBORw0KGgo=\"}]}", "digest": "02c724259bb3c98002842cafad6d890d3dab7db287f1803fd9ce97ec79630a6d" }, + "assertions": [ + { "id": "tool-text", "operator": "json_path_equals", "selector": "/upstream/requests/0/json/messages/0", "expected": {"role":"tool","tool_call_id":"call_fixture","content":"RESULT"}, "required": true }, + { "id": "image-carrier", "operator": "json_path_equals", "selector": "/upstream/requests/0/json/messages/1/content/0/image_url/url", "expected": "data:image/png;base64,iVBORw0KGgo=", "required": true } + ] + }, + { + "id": "vision-core.protocol.modality-gate", + "suite": "vision-core", + "capability": "modalities.image.input", + "verificationRole": "negative_control", + "requirements": { "inboundProtocols": ["openai-responses"], "upstreamProtocols": ["openai-chat"], "surfaces": ["responses-http"], "requiredClaims": [], "requiredHarnessFeatures": ["adapter_vector"], "platforms": [], "routePreconditions": [] }, + "fixture": { "id": "vision-gate", "role": "adapter_vector", "mediaType": "application/vnd.opencodex.adapter-vector+json", "bytesUtf8": "{\"model\":\"text-only\",\"modelInputModalities\":[\"text\"],\"visionSidecar\":{\"enabled\":false},\"requestHasImage\":true}", "digest": "d5b438fb3fad873b0a1bb1b6c91539862e4f3aa8690a963eb6121c5a3229818a" }, + "assertions": [ + { "id": "path", "operator": "json_path_equals", "selector": "/verifiers/modality_path", "expected": "unsupported", "required": true }, + { "id": "no-drop", "operator": "json_path_equals", "selector": "/verifiers/silent_image_drop", "expected": false, "required": true } + ], + "expectedFailure": { "controlKind": "conformance_negative_control", "expectedClass": "capability_failure", "expectedCode": "image_input_unsupported", "assertionIds": ["path", "no-drop"], "onMatch": "pass", "onMismatch": "fail" } + }, + { + "id": "reasoning-core.protocol.effort-mapping", + "suite": "reasoning-core", + "capability": "reasoning.round_trip", + "requirements": { "inboundProtocols": ["openai-responses"], "upstreamProtocols": ["openai-chat"], "surfaces": ["responses-http"], "requiredClaims": ["reasoning"], "requiredHarnessFeatures": ["adapter_vector"], "platforms": [], "routePreconditions": [] }, + "fixture": { "id": "reasoning-effort", "role": "adapter_vector", "mediaType": "application/vnd.opencodex.adapter-vector+json", "bytesUtf8": "{\"requested\":\"high\",\"reasoningEffortMap\":{\"high\":\"adaptive\"},\"reasoningWireFormat\":\"gateway-object\"}", "digest": "d9d5cce104809764d5edbc833088a0a9bb3b4d678a4f135353cc5fecf62e8b57" }, + "assertions": [ + { "id": "wire", "operator": "json_path_equals", "selector": "/upstream/requests/0/json/reasoning", "expected": {"enabled":true,"effort":"adaptive"}, "required": true }, + { "id": "legacy-absent", "operator": "json_path_absent", "selector": "/upstream/requests/0/json/reasoning_effort", "expected": true, "required": true } + ] + }, + { + "id": "reasoning-core.protocol.summary-stream", + "suite": "reasoning-core", + "capability": "reasoning.round_trip", + "requirements": { "inboundProtocols": ["openai-responses"], "upstreamProtocols": ["openai-responses"], "surfaces": ["responses-sse"], "requiredClaims": ["reasoning"], "requiredHarnessFeatures": ["raw_sse_capture"], "platforms": [], "routePreconditions": [] }, + "initiatingRequest": { "id": "reasoning-summary-request", "role": "client_request", "mediaType": "application/json", "bytesUtf8": "{\"model\":\"fixture-model\",\"input\":\"PING\",\"stream\":true}", "digest": "c2c39a78b939e6c5182d3206f86aa86d6fd706959de9c37072db759aa510a1f6" }, + "fixture": { "id": "reasoning-summary", "role": "upstream_response", "mediaType": "text/event-stream", "bytesUtf8": "event: response.reasoning_summary_part.added\ndata: {\"type\":\"response.reasoning_summary_part.added\",\"item_id\":\"rs_fixture\",\"summary_index\":0,\"part\":{\"type\":\"summary_text\",\"text\":\"\"}}\n\nevent: response.reasoning_summary_text.delta\ndata: {\"type\":\"response.reasoning_summary_text.delta\",\"item_id\":\"rs_fixture\",\"summary_index\":0,\"delta\":\"WHY\"}\n\nevent: response.completed\ndata: {\"type\":\"response.completed\",\"response\":{\"status\":\"completed\"}}\n\n", "digest": "d9c4fa73b67f7a92d9ec55af7ff12b16ddc9870059e5e530ee04006b171367e7" }, + "assertions": [ + { "id": "events", "operator": "sse_event_sequence", "selector": "/client/response/events", "expected": ["response.reasoning_summary_part.added","response.reasoning_summary_text.delta","response.completed"], "required": true }, + { "id": "id", "operator": "id_matches", "selector": "/client/response/events/0/data/item_id", "expected": "responses_reasoning", "required": true } + ] + }, + { + "id": "reasoning-core.protocol.replay", + "suite": "reasoning-core", + "capability": "reasoning.round_trip", + "requirements": { "inboundProtocols": ["openai-responses"], "upstreamProtocols": ["openai-responses"], "surfaces": ["responses-http"], "requiredClaims": ["reasoning"], "requiredHarnessFeatures": ["adapter_vector"], "platforms": [], "routePreconditions": [] }, + "fixture": { "id": "reasoning-replay", "role": "adapter_vector", "mediaType": "application/vnd.opencodex.adapter-vector+json", "bytesUtf8": "{\"turn1\":{\"reasoning\":{\"id\":\"rs_fixture\",\"text\":\"PLAN\",\"signature\":\"sig_fixture\"},\"toolCall\":{\"callId\":\"call_fixture\"}},\"turn2\":{\"toolResult\":{\"callId\":\"call_fixture\",\"output\":\"RESULT\"}}}", "digest": "6e137e06f52c32e9f7d394b92343a8b849103328e958ab7c9b2825a799ea60c3" }, + "assertions": [ + { "id": "text", "operator": "json_path_equals", "selector": "/upstream/requests/1/json/input/0/content/0/text", "expected": "PLAN", "required": true }, + { "id": "signature", "operator": "json_path_equals", "selector": "/upstream/requests/1/json/input/0/signature", "expected": "sig_fixture", "required": true } + ] + }, + { + "id": "reasoning-core.protocol.private-content-isolation", + "suite": "reasoning-core", + "capability": "reasoning.round_trip", + "requirements": { "inboundProtocols": ["openai-responses"], "upstreamProtocols": ["openai-chat"], "surfaces": ["responses-http"], "requiredClaims": ["reasoning"], "requiredHarnessFeatures": ["adapter_vector"], "platforms": [], "routePreconditions": [] }, + "fixture": { "id": "reasoning-private", "role": "adapter_vector", "mediaType": "application/vnd.opencodex.adapter-vector+json", "bytesUtf8": "{\"origin\":{\"provider\":\"alpha\",\"encrypted\":\"opaque_fixture\"},\"destination\":{\"provider\":\"beta\",\"adapter\":\"openai-chat\"}}", "digest": "3519bd299fe2cd5b8069e0cd1c3b65b61d5ce9588c66e54b49d42af5ccf0c81e" }, + "assertions": [ + { "id": "upstream-absent", "operator": "json_path_absent", "selector": "/upstream/requests/0/json/encrypted_content", "expected": true, "required": true }, + { "id": "client-absent", "operator": "json_path_absent", "selector": "/client/response/json/hidden_reasoning", "expected": true, "required": true } + ] + }, + { + "id": "mcp-core.protocol.namespace-mapping", + "suite": "mcp-core", + "capability": "tools.mcp.core", + "requirements": { "inboundProtocols": ["openai-responses"], "upstreamProtocols": ["cursor-protobuf"], "surfaces": ["responses-http"], "requiredClaims": ["mcp"], "requiredHarnessFeatures": ["adapter_vector","in_memory_mcp_stub","mcp_namespace_round_trip_v1"], "platforms": [], "routePreconditions": [] }, + "fixture": { "id": "mcp-namespace", "role": "synthetic_tool", "mediaType": "application/vnd.opencodex.mcp-stub+json", "bytesUtf8": "{\"namespace\":\"mcp__fixture\",\"name\":\"lookup\",\"description\":\"fixture\",\"inputSchema\":{\"type\":\"object\",\"properties\":{\"q\":{\"type\":\"string\"}}}}", "digest": "91a53f8c580d461d0f5e0d7209e5d4b95249bdfa8bd3fd4f298e18bdeadb0693" }, + "assertions": [ + { "id": "wire-name", "operator": "json_path_equals", "selector": "/upstream/requests/0/json/tools/0/name", "expected": "mcp__fixture__lookup", "required": true }, + { "id": "reverse", "operator": "json_path_equals", "selector": "/client/response/mcpCalls/0", "expected": {"namespace":"mcp__fixture","name":"lookup"}, "required": true } + ] + }, + { + "id": "mcp-core.protocol.schema-and-bounds", + "suite": "mcp-core", + "capability": "tools.mcp.core", + "requirements": { "inboundProtocols": ["openai-responses"], "upstreamProtocols": ["cursor-protobuf"], "surfaces": ["responses-http"], "requiredClaims": ["mcp"], "requiredHarnessFeatures": ["adapter_vector","in_memory_mcp_stub","mcp_schema_bounds_v1"], "platforms": [], "routePreconditions": [] }, + "fixture": { "id": "mcp-bounds", "role": "synthetic_tool", "mediaType": "application/vnd.opencodex.mcp-stub+json", "bytesUtf8": "{\"limitBytes\":64,\"exactSchema\":\"{\\\"type\\\":\\\"object\\\",\\\"properties\\\":{\\\"x\\\":{\\\"type\\\":\\\"string\\\"}},\\\"a\\\":\\\"xxx\\\"}\",\"overSchema\":\"{\\\"type\\\":\\\"object\\\",\\\"properties\\\":{\\\"x\\\":{\\\"type\\\":\\\"string\\\"}},\\\"a\\\":\\\"xxxx\\\"}\"}", "digest": "34ff4414dc8e196d460390557f4fd74c32418ea00710167baff2a0dc1f3b643c" }, + "assertions": [ + { "id": "exact", "operator": "verifier_result_equals", "selector": "/verifiers/exact_bound", "expected": "pass", "required": true }, + { "id": "over", "operator": "verifier_result_equals", "selector": "/verifiers/one_over_rejected", "expected": "pass", "required": true }, + { "id": "atomic", "operator": "json_path_equals", "selector": "/verifiers/partial_commit", "expected": false, "required": true } + ] + }, + { + "id": "mcp-core.protocol.call-result", + "suite": "mcp-core", + "capability": "tools.mcp.core", + "requirements": { "inboundProtocols": ["openai-responses"], "upstreamProtocols": ["cursor-protobuf"], "surfaces": ["responses-http"], "requiredClaims": ["mcp"], "requiredHarnessFeatures": ["adapter_vector","in_memory_mcp_stub","mcp_call_result_v1"], "platforms": [], "routePreconditions": [] }, + "fixture": { "id": "mcp-call", "role": "synthetic_tool", "mediaType": "application/vnd.opencodex.mcp-stub+json", "bytesUtf8": "{\"namespace\":\"mcp__fixture\",\"name\":\"lookup\",\"arguments\":{\"q\":\"x\"},\"result\":{\"content\":[{\"type\":\"text\",\"text\":\"RESULT\"}],\"isError\":false}}", "digest": "986ef5017fbdb46eb18b30daaffe72aecc93868b7d89b11c3e244dc084f46496" }, + "assertions": [ + { "id": "call", "operator": "json_path_equals", "selector": "/verifiers/stub_received", "expected": {"namespace":"mcp__fixture","name":"lookup","arguments":{"q":"x"}}, "required": true }, + { "id": "result", "operator": "json_path_equals", "selector": "/client/response/json", "expected": {"content":[{"type":"text","text":"RESULT"}],"isError":false}, "required": true } + ] + }, + { + "id": "mcp-core.protocol.resource-round-trip", + "suite": "mcp-core", + "capability": "tools.mcp.core", + "requirements": { "inboundProtocols": ["openai-responses"], "upstreamProtocols": ["cursor-protobuf"], "surfaces": ["responses-http"], "requiredClaims": ["mcp"], "requiredHarnessFeatures": ["adapter_vector","in_memory_mcp_stub","mcp_resource_round_trip_v1"], "platforms": [], "routePreconditions": [] }, + "fixture": { "id": "mcp-resource", "role": "synthetic_tool", "mediaType": "application/vnd.opencodex.mcp-stub+json", "bytesUtf8": "{\"resources\":[{\"uri\":\"fixture://one\",\"name\":\"one\"}],\"read\":{\"uri\":\"fixture://one\",\"contents\":[{\"uri\":\"fixture://one\",\"text\":\"RESOURCE\"}]}}", "digest": "a3f6317374ce92da0155dd14bbf0d5822e8687cbe8ef7968221f23acf8b16aa5" }, + "assertions": [ + { "id": "list", "operator": "json_path_equals", "selector": "/client/response/json/resources", "expected": [{"uri":"fixture://one","name":"one"}], "required": true }, + { "id": "read", "operator": "json_path_equals", "selector": "/client/response/json/contents", "expected": [{"uri":"fixture://one","text":"RESOURCE"}], "required": true } + ] + } + ] +} diff --git a/devlog/_plan/260807_compatibility_lab/030_incident_corpus.md b/devlog/_plan/260807_compatibility_lab/030_incident_corpus.md new file mode 100644 index 0000000000..2e17f89e39 --- /dev/null +++ b/devlog/_plan/260807_compatibility_lab/030_incident_corpus.md @@ -0,0 +1,457 @@ +# CL-00 compatibility incident corpus + +These are abstract regression specifications distilled from shipped tests, +public issues, and devlog records. Provider names identify historical evidence, +not special cases to encode in the scenario model. + +Each future fixture must reproduce the observable wire condition with a mock +upstream. CL-00 does not reopen or fix the production incidents. + +`Future mapping` identifies the suite/scenario family that should own the +regression. It does not claim protocol V1 already covers the incident. Only a +literal vector in `022_protocol_v1_cases.json` gates a V1 verdict. +Unrepresented incidents below are reviewed inputs to a later scenario/suite +version amendment; prose and source-test references cannot be inferred into +V1. + +## IC-001 - legal SSE field spacing + +- Incident class: valid SSE framing rejected. +- Historical source: + [#1170](https://github.com/lidge-jun/opencodex/issues/1170), + `devlog/_plan/260807_untouched_bug_stack/010_sse_unspaced_data_fields.md`, + `tests/sse-unspaced-data-fields.test.ts`. +- Observable failure: a parser accepts `data: {...}` but rejects legal + `data:{...}`/`event:name`, trims payload whitespace, or treats a bare + `data:` as malformed. +- Expected behavior: accept both legal forms, strip at most one optional space, + preserve payload whitespace, and handle empty field values consistently on + Responses, Chat, Anthropic and sidecar paths. +- Future mapping: `responses-core.protocol.sse-framing`, + `chat-core.protocol.stream-terminal`, + `anthropic-core.protocol.content-sequence`. +- Classification: `protocol_failure` -> `DEGRADED`. +- Deterministic fixture: yes; paired frames differing only in legal spacing. + +## IC-002 - null and empty SSE data frames + +- Incident class: ignorable frame mishandled as payload or terminal error. +- Historical source: + `devlog/_plan/260808_bug_campaign/020_wp2_sse_frame_contract.md`, + `tests/sse-null-data-frame.test.ts`. +- Observable failure: `data: null`, a bare `data:` field, or a comment frame + crashes decoding, creates a synthetic event, or hides a later valid event. +- Expected behavior: apply each surface's explicit ignorable-frame contract; + continue parsing without fabricating output, while malformed non-null JSON + still fails closed. +- Future mapping: `responses-core.protocol.sse-framing`, + `chat-core.protocol.stream-terminal`, + `anthropic-core.protocol.content-sequence`. +- Classification: `protocol_failure` -> `DEGRADED`. +- Deterministic fixture: yes; null/empty/comment/malformed controls. + +## IC-003 - missing or incorrect terminal stream signal + +- Incident class: clean EOF, `[DONE]`, completion event, and terminal state + confused. +- Historical source: `tests/openai-chat-eof.test.ts`, + `tests/sse-failed-tail.test.ts`, `tests/claude-outbound.test.ts`, + `tests/responses-stream-tool-events.test.ts`, + [#658](https://github.com/lidge-jun/opencodex/issues/658), + [#735](https://github.com/lidge-jun/opencodex/issues/735). +- Observable failure: a stream ends without the protocol-required terminal, + emits more than one terminal, accepts `[DONE]` as a Responses completion + without a terminal event, or maps failed/incomplete to successful end-turn. +- Expected behavior: exactly one surface-correct terminal. Responses remains + strict. A fingerprinted Chat/Anthropic EOF-tolerance contract may complete + only after visible output or a fully assembled tool call and only when no + incomplete call remains; every other deterministic EOF fails closed and + preserves the typed failed/incomplete reason. +- Future mapping: `responses-core.protocol.terminal-state`, + `chat-core.protocol.stream-terminal`, + `anthropic-core.protocol.terminal-errors`, + `codex-core.protocol.streaming-turn`. +- Classification: deterministic close is `protocol_failure` -> `DEGRADED`. +- Deterministic fixture: yes; completed, failed, incomplete, duplicate and + missing-terminal tails. + +## IC-004 - body stall versus protocol truncation + +- Incident class: environmental/provider timeout misreported as incompatibility. +- Historical source: + [#875](https://github.com/lidge-jun/opencodex/issues/875), + [#1065](https://github.com/lidge-jun/opencodex/issues/1065), + `devlog/_plan/260805_bug_stack_campaign/050_issue875_deepseek_flash_stall.md`, + `devlog/_plan/260806_overnight_triage_round2/020_bounded_body_first_byte.md`. +- Observable failure: no first byte or no later body byte arrives before the + deadline; the system labels the model's protocol unsupported, or waits + without a bound. +- Expected behavior: connect, first-byte, inactivity and total deadlines remain + distinguishable. A silent live stall is a blocker, not proof of malformed + protocol. A mock that deliberately closes without terminal data remains + IC-003. +- Future mapping: supplemental timeout controls for + `responses-core.protocol.terminal-state` and every future live suite. +- Classification: `timeout` -> `BLOCKED`; `provider_transient` when an + authoritative transient response exists. +- Deterministic fixture: yes for timeout attribution; no deterministic fixture + can convert an arbitrary live stall into incompatibility evidence. + +## IC-005 - sparse lifecycle snapshots + +- Incident class: incomplete Responses lifecycle snapshots forwarded as valid. +- Historical source: + [#893](https://github.com/lidge-jun/opencodex/issues/893), + `devlog/_plan/260805_bug_stack_campaign/040_issue893_sparse_snapshot_repair.md`, + `tests/responses-snapshot-repair.test.ts`, + `tests/responses-snapshot-repair-server.test.ts`. +- Observable failure: added/done snapshots omit required ID, type, role, status, + output index or closing item, producing a client-invalid lifecycle. +- Expected behavior: preserve a complete canonical lifecycle or apply an + explicitly configured, assertion-visible repair; never claim a sparse stream + is valid without proving the repaired output. +- Future mapping: `responses-core.protocol.item-lifecycle`, + `codex-core.protocol.streaming-turn`. +- Classification: `protocol_failure` -> `DEGRADED`. +- Deterministic fixture: yes; sparse permutations plus no-repair control. + +## IC-006 - invalid, reused, or missing item IDs + +- Incident class: client-facing Responses item identity violates its grammar or + event correlation. +- Historical source: + [#938](https://github.com/lidge-jun/opencodex/issues/938), + `devlog/_plan/260805_bug_stack_campaign/060_issue938_uuid_item_ids.md`, + `tests/responses-item-id-repair.test.ts`, + `tests/deepseek-responses-item-id-repair.test.ts`. +- Observable failure: UUID/placeholder/missing IDs reach a client contract that + requires typed IDs, or added/done events use inconsistent IDs. +- Expected behavior: valid stable IDs on the client surface; any configured + repair is deterministic, type-scoped, and never rewrites function call IDs + or breaks `call_id` correlation. +- Future mapping: `responses-core.protocol.item-lifecycle`, + `codex-core.protocol.streaming-turn`. +- Classification: `protocol_failure` -> `DEGRADED`. +- Deterministic fixture: yes; valid, invalid, reused, missing-terminal and + correlation controls. + +## IC-007 - function schema root normalization + +- Incident class: valid tool rejected because its schema root is missing or + non-object. +- Historical source: + [PR #745](https://github.com/lidge-jun/opencodex/pull/745), + `tests/responses-parser.test.ts`. +- Observable failure: tool definition reaches an object-schema-only upstream + with absent/invalid root shape, or normalization corrupts an already valid + schema. +- Expected behavior: produce the required object root without changing valid + properties/required/additionalProperties semantics. +- Future mapping: `tools-core.protocol.function-round-trip`, + `mcp-core.protocol.schema-and-bounds`. +- Classification: `protocol_failure` -> `DEGRADED`. +- Deterministic fixture: yes; absent, malformed and valid schema controls. + +## IC-008 - custom/freeform tool envelope mismatch + +- Incident class: function-only route or token preset rejects a valid + custom/freeform tool. +- Historical source: + `devlog/_plan/260807_untouched_bug_stack/070_mimo_token_plan_preset.md`, + `tests/responses-parser.test.ts` (exact `apply_patch` envelope), + `tests/responses-tool-groups.test.ts`. +- Observable failure: a custom tool is serialized as a function, its freeform + input/output is JSON-wrapped or dropped, or the route rejects the tool without + an honest unsupported result. +- Expected behavior: preserve the exact custom tool declaration, call and + output grammar, or deterministically classify the exact route unsupported for + custom tools. +- Future mapping: `tools-core.protocol.custom-freeform-round-trip`, + `codex-core.protocol.apply-patch-turn`. +- Classification: malformed translation is `protocol_failure` -> `DEGRADED`; + a proven route contract is `capability_failure` -> `UNSUPPORTED`. +- Deterministic fixture: yes for translation; a future live negative control + proves route support. + +## IC-009 - dangling tool calls and result correlation + +- Incident class: tool call/result pair becomes orphaned or misassociated. +- Historical source: + `devlog/_fin/260718_dangling_toolcall_hardening/010_record.md`, + `tests/openai-chat-dangling-toolcalls.test.ts`, + `tests/issue-702-expired-replay-state.test.ts`, + [#334](https://github.com/lidge-jun/opencodex/issues/334), + [#620](https://github.com/lidge-jun/opencodex/issues/620). +- Observable failure: an assistant tool call is forwarded without a matching + result, a result is attached to the wrong ID, or expired continuation state + resurrects an unrelated call. +- Expected behavior: preserve exact call/result identity and order; repair only + the narrowly declared orphan case; otherwise fail closed without fabricating + a successful tool result. +- Future mapping: `tools-core.protocol.function-round-trip`, + `codex-core.protocol.tool-continuation`, + `codex-core.protocol.previous-response-replay`. +- Classification: `protocol_failure` -> `DEGRADED`. +- Deterministic fixture: yes; missing, duplicate, out-of-order, expired and + mismatched IDs. + +## IC-010 - parallel tool fragment assembly + +- Incident class: interleaved calls merged, lost, reordered, or correlated to + the wrong result. +- Historical source: + `devlog/_fin/260709_parallel_tool_calls/000_plan.md`, + `tests/openai-chat-parallel-stream.test.ts`, + `tests/parallel-tool-calls-optin.test.ts`, + [#361](https://github.com/lidge-jun/opencodex/issues/361). +- Observable failure: fragmented deltas from two calls produce one argument + buffer, unstable ordering, duplicate completion, or incorrect result IDs. +- Expected behavior: assemble each indexed call independently, never duplicate + argument fragments, preserve stable order/identity, and advertise parallel + capability only when the effective adapter contract supports it. Provider + interleaving does not require overlapping canonical adapter events; atomic + sequential emission is a valid compatibility-preserving bridge contract. +- Future mapping: `tools-core.protocol.parallel-correlation`. +- Classification: `protocol_failure` -> `DEGRADED`; explicit no-parallel + contract -> `UNSUPPORTED` for that capability only. +- Deterministic fixture: yes; interleaved, fragmented, out-of-order and + single-call controls. + +## IC-011 - wrong upstream wire for a model + +- Incident class: Responses-capable and Chat-only models behind one gateway use + the provider-wide wire indiscriminately. +- Historical source: `src/types.ts` and `src/providers/registry.ts` model-wire + contract for [#404](https://github.com/lidge-jun/opencodex/issues/404), + `tests/adapter-resolve.test.ts`, `tests/deepseek-inbound-wire.test.ts`, + `tests/chat-completions-endpoint.test.ts`. +- Observable failure: an exact model is sent to the wrong endpoint/request + shape, producing rejection or silent semantic loss. +- Expected behavior: resolve the effective model-specific adapter before + subject identity and send the declared wire shape. Evidence for one wire is + never reused for the other. +- Future mapping: `responses-core.protocol.request-shape`, + `chat-core.protocol.request-mapping`; future live route variants. +- Classification: deterministic resolver/translation error is + `protocol_failure` -> `DEGRADED`; a correctly selected but unsupported route + is `capability_failure` -> `UNSUPPORTED`. +- Deterministic fixture: yes; mixed gateway with endpoint-specific fixtures. + +## IC-012 - reasoning replay form mismatch + +- Incident class: plaintext reasoning, signature, redacted block, or thought + signature is dropped or replayed in the wrong form. +- Historical source: `tests/deepseek-reasoning-replay.test.ts`, + `tests/deepseek-reasoning-replay-gaps.test.ts`, + `tests/google-antigravity-replay.test.ts`, + `tests/anthropic-thinking-signature.test.ts`, + `tests/kiro-reasoning-roundtrip.test.ts`. +- Observable failure: a second turn is rejected, reasoning text leaks into + visible output, required signature data is lost, or incompatible replay data + is forwarded. +- Expected behavior: use the exact selected adapter's replay contract, preserve + opaque data only on its compatible route, and omit/normalize it safely + elsewhere. +- Future mapping: `reasoning-core.protocol.replay`, + `reasoning-core.protocol.summary-stream`. +- Classification: `protocol_failure` -> `DEGRADED`. +- Deterministic fixture: yes; two-turn fixtures for each abstract replay form. + +## IC-013 - provider-private content crosses a route boundary + +- Incident class: encrypted/task/reasoning content from one provider is sent to + an incompatible provider or exposed as ordinary text. +- Historical source: + [#92](https://github.com/lidge-jun/opencodex/issues/92), + `devlog/_fin/260706_previous-response-id-400/000_plan.md`, + `tests/responses-parser.test.ts` encrypted-content case, + `tests/bridge-raw-reasoning-hidden.test.ts`, + `tests/v2-agent-message-failfast.test.ts`. +- Observable failure: opaque encrypted content is forwarded where it cannot be + decrypted, causes a 400, or becomes user-visible/private evidence. +- Expected behavior: provider-private envelopes remain origin-scoped; cross + route replay fails closed or uses a bounded opaque marker expressly allowed + by the protocol, never raw private data. +- Future mapping: `reasoning-core.protocol.private-content-isolation`, + `codex-core.protocol.previous-response-replay`; a later encrypted-task + capability scenario when its upstream contract is implementable. +- Classification: unsafe translation is `protocol_failure` -> `DEGRADED`; a + route proven unable to consume the encrypted task capability is + `UNSUPPORTED`; the current explicit fail-fast is safe `UNSUPPORTED` evidence + only when the scenario's exact route and encrypted-task preconditions match. + Raw disclosure is also a security Critical independent of compatibility + verdict. +- Deterministic fixture: yes for local origin isolation and fail-fast + mitigation; partial for true cross-provider encrypted task execution. + +## IC-014 - image modality or tool-result image mismatch + +- Incident class: structured image content is dropped, stringified, sent to a + text-only route, or advertised inaccurately. +- Historical source: + [#888](https://github.com/lidge-jun/opencodex/issues/888), + `tests/openai-chat-tool-result-images.test.ts`, + `tests/responses-parser.test.ts`, `tests/vision-anthropic.test.ts`, + `tests/vision-fail-closed.test.ts`, `tests/request-evidence.test.ts`. +- Observable failure: image order/detail/MIME is lost, a tool-result image + becomes raw JSON/text, or capability gating disagrees with the effective + native/sidecar path. +- Expected behavior: preserve structured image parts and honestly choose + native, declared sidecar, or unsupported behavior without silent loss. +- Future mapping: `vision-core.protocol.input-image`, + `vision-core.protocol.tool-result-image`, + `vision-core.protocol.modality-gate`. +- Classification: `protocol_failure` -> `DEGRADED`; proven no-image route -> + `UNSUPPORTED`; sidecar/network unavailability -> `BLOCKED`. +- Deterministic fixture: yes; synthetic data image and text-only controls. + +## IC-015 - malformed continuation and previous-response state + +- Incident class: stateful continuation is forwarded to a stateless/incompatible + route or local replay is incomplete. +- Historical source: + [#702](https://github.com/lidge-jun/opencodex/issues/702), + `devlog/_fin/260706_previous-response-id-400/000_plan.md`, + `tests/responses-state.test.ts`, + `tests/issue-702-expired-replay-state.test.ts`, + `tests/grok-orphan-adoption.test.ts`. +- Observable failure: upstream 400, duplicate history, missing prior tool call, + orphaned result, or continuation state reused after expiry/route change. +- Expected behavior: use valid provider-private continuation only on its exact + compatible subject; otherwise perform bounded ordered local expansion or fail + closed. +- Future mapping: `codex-core.protocol.previous-response-replay`, + `codex-core.protocol.tool-continuation`. +- Classification: `protocol_failure` -> `DEGRADED`. +- Deterministic fixture: yes; stateful, stateless, expired and route-change + matrices. + +## IC-016 - Anthropic terminal/error taxonomy corruption + +- Incident class: failed/incomplete/upstream-overload response appears as + successful `end_turn` or wrong Anthropic error type. +- Historical source: `tests/claude-outbound.test.ts`, + `tests/anthropic-eof-tolerance.test.ts`, + `tests/anthropic-compatible-stream.test.ts`. +- Observable failure: missing `message_stop` is accepted outside a declared + tolerance, transient 502 becomes a normal message, or content-filter/max-token + stop reason is mapped incorrectly. +- Expected behavior: preserve exact content-block and message terminal + sequence; map failure classes and stop reasons deterministically; apply any + EOF tolerance only to its exact fingerprinted route. +- Future mapping: `anthropic-core.protocol.terminal-errors`, + `anthropic-core.protocol.content-sequence`. +- Classification: `protocol_failure` -> `DEGRADED`; live transient -> + `provider_transient` -> `BLOCKED`. +- Deterministic fixture: yes; strict/tolerant, failed, incomplete and transient + controls. + +## IC-017 - MCP namespace, bound, and result atomicity + +- Incident class: namespace collision, oversized schema/result partial commit, + or result type loss. +- Historical source: `tests/cursor-mcp-manager.test.ts`, + `tests/cursor-mcp-stdio.test.ts`. +- Observable failure: flattened names cannot map back, one-byte-over input + leaves a partial catalogue, image/error result changes type, or unknown tool + becomes an untyped exception. +- Expected behavior: collision-safe namespace mapping, exact atomic bounds, and + typed result/error/resource behavior through a Lab-owned stub. +- Future mapping: all `mcp-core.protocol.*` scenarios. +- Classification: `protocol_failure` -> `DEGRADED`; declared no-MCP route -> + `UNSUPPORTED`. +- Deterministic fixture: yes; in-memory/loopback stub only. + +## IC-018 - DNS/connect failure poisons account or capability evidence + +- Incident class: pre-connection transport failure attributed to credentials, + account, model, or capability. +- Historical source: + [#914](https://github.com/lidge-jun/opencodex/issues/914), + `devlog/_plan/260805_bug_stack_campaign/030_issue914_dns_transport_attribution.md`, + `devlog/_plan/260803_transport_attribution/000_plan.md`, + `tests/upstream-connect-error.test.ts`. +- Observable failure: DNS/connect/TLS setup rotates account state, marks a + route capability degraded, or becomes authentication evidence. +- Expected behavior: classify pre-response transport evidence as environment/ + network, leave compatibility and credential capability unchanged, and permit + retry after environment repair. +- Future mapping: blocker controls shared by every future live suite. +- Classification: `network_failure` -> `BLOCKED`. +- Deterministic fixture: yes for attribution using an injected connect failure; + it never contributes a compatibility failure. + +## IC-019 - malformed error or empty success envelope + +- Incident class: upstream error/empty payload accepted as a successful model + response. +- Historical source: `tests/openai-chat-hardening.test.ts`, + `tests/error-fidelity.test.ts`, `tests/upstream-http-error.test.ts`. +- Observable failure: falsey error payload, empty choices, null choice, missing + message, or malformed SSE data is emitted as success or hidden by a terminal. +- Expected behavior: fail closed with a typed normalized error while preserving + any safe usage/status evidence. +- Future mapping: `chat-core.protocol.nonstream-envelope`, + `chat-core.protocol.stream-terminal`. +- Classification: `protocol_failure` -> `DEGRADED` for deterministic malformed + protocol; recognized live transient remains `provider_transient`. +- Deterministic fixture: yes. + +## IC-020 - structured-output wire mismatch + +- Incident class: Responses `text.format` is lost, malformed, or sent to an + upstream in the wrong shape. +- Historical source: `tests/responses-parser.test.ts`, + `tests/openai-chat-hardening.test.ts`, + `tests/deepseek-inbound-wire.test.ts`. +- Observable failure: JSON schema/object request widens to plain text, schema + nesting changes, or a strict unsupported route receives an invalid parameter. +- Expected behavior: preserve the known equivalent wire form, or return a + deterministic unsupported result without pretending structured output was + honored. +- Future mapping: `codex-core.protocol.structured-output`, + `responses-core.protocol.request-shape`, + `chat-core.protocol.request-mapping`. +- Classification: translation error is `protocol_failure` -> `DEGRADED`; + proven route limitation is `capability_failure` -> `UNSUPPORTED`. +- Deterministic fixture: yes for wire translation; future live negative control + for route support. + +## IC-021 - data-only Responses SSE + +- Incident class: valid Responses events rejected because the producer omits + the redundant `event:` field. +- Historical source: + [#700](https://github.com/lidge-jun/opencodex/issues/700), + `tests/claude-outbound.test.ts`. +- Observable failure: a payload with a valid typed Responses JSON record in + `data:` is ignored or treated as a truncated stream when no `event:` line is + present. +- Expected behavior: infer the event name from the payload's canonical `type` + when the surface permits data-only events, permit explicit and inferred + frames to interleave, and keep untyped data-only records ignored/fail-closed + according to the scenario. +- Future mapping: `responses-core.protocol.sse-framing`, + `anthropic-core.protocol.content-sequence`. +- Classification: `protocol_failure` -> `DEGRADED`. +- Deterministic fixture: yes; explicit-only, data-only, mixed and untyped + controls. + +## Corpus maintenance rule + +New incidents enter this corpus only when they add a reusable wire condition, +assertion, or attribution boundary. A provider-specific workaround is not a +scenario. The abstraction must state: + +```text +incident class +historical source/reference +observable failure +expected correct behavior +future scenario/suite mapping +expected failure classification +deterministic fixture feasibility +``` + +When a future fix changes the expected contract, bump the mapped scenario +version and preserve this historical record. diff --git a/devlog/_plan/260807_compatibility_lab/040_security_and_privacy.md b/devlog/_plan/260807_compatibility_lab/040_security_and_privacy.md new file mode 100644 index 0000000000..ac8ea1d102 --- /dev/null +++ b/devlog/_plan/260807_compatibility_lab/040_security_and_privacy.md @@ -0,0 +1,455 @@ +# CL-00 security, privacy, and probe sandbox contract + +Compatibility evidence is useful only if collecting it does not turn the Lab +into a data-exfiltration or arbitrary-execution surface. These requirements are +release blockers for later implementation. + +## 1. Data prohibition + +The Lab must not read, accept, persist, or export: + +- user prompts or conversation history; +- real user repositories, worktrees, patches, source files, or file paths; +- user MCP server definitions, resources, results, or credentials, with no + Lab mode, CLI flag, profile, or override that can load them; +- arbitrary shell commands or process output; +- arbitrary filesystem contents; +- arbitrary external-network tool requests or responses; +- API keys, OAuth/access/refresh tokens, cookies, authorization material, or + raw credential errors; +- account IDs, account emails, aliases, plan labels, tenant IDs, or other PII; +- raw private/custom headers; +- hidden reasoning, chain of thought, encrypted reasoning payloads, provider + thought signatures, or decrypted private task content. + +Scenarios contain Lab-authored synthetic prompts, fixtures, tool definitions +and results only. They must be recognizable as synthetic and contain no copied +customer material. + +### Synthetic-input admission + +Synthetic status is machine-checked, not inferred from content or a friendly +name. Every fixture admitted to the Lab has a canonical reference containing: + +```text +syntheticMarker ocx-lab-synthetic-v1 +provenance.kind lab_authored +provenance.authority reviewed manifest/authority identifier +provenance.sourceCommit immutable source revision +digest domain-separated content digest +byteLength exact admitted byte length +``` + +The reference participates in the scenario manifest digest. Admission verifies +the marker, closed provenance kind, expected authority/source revision, digest, +and byte length before fixture bytes reach an adapter, mock, tool stub, event +normalizer, or artifact writer. A request payload, model output, local path, +repository file, MCP object, URL response, or runtime caller cannot label +itself synthetic. Protocol V1 uses the exact `fixtureRef` contract in +`021_protocol_v1_manifest_authority.md`; future live/Fabric suites must define +an equally closed provenance authority before they may run. + +## 2. Future live-probe sandbox + +A live probe is an explicit background/management/CLI action. It is never +started by the production request path, profile evaluator, Router Intelligence, +request-history read, dashboard render, or provider discovery. + +The future runner must enforce a capability-deny sandbox: + +### Network + +- The immutable scenario manifest may authorize only fixed dependency roles + and protocol classes, never a route-local URL. +- Network authorization uses a trusted in-memory `LabDestinationV1` record + owned by the existing provider destination/credential plumbing: scheme, + host, port, base path, resolved IP family/address set after policy checks, + TLS SNI/Host values, and private-network opt-in. That record is never + written to JSONL, SQLite, artifacts, or export. +- Provider destination/credential plumbing creates one per-run immutable + `LabDestinationV1` snapshot before endpoint fingerprinting. The exact same + snapshot must be used unchanged for endpoint fingerprinting, destination + authorization, credential binding, and connection. Mutation, replacement, + re-resolution to a different address set, or any mismatch between those + stages fails closed as `harness_failure` before credentials are sent. +- The composite route subject stores only the keyed opaque + `endpointFingerprint` derived from the normalized destination. Raw URLs are + never evidence fields. +- The only remote destinations are the exact primary and flat sidecar + destinations named by those in-memory records after existing provider + destination-policy validation. +- DNS is resolved once before the policy check. The HTTP client must connect + to the validated IP set (pin/connect to the approved addresses) while + preserving the intended Host/SNI. A later resolution that differs fails + closed as `harness_failure`. +- Redirects are rejected by default for Lab probes, matching the existing + SSRF fail-closed posture. A future scenario that explicitly opts into + redirects must authorize every hop with the same destination policy, IP + pinning, and Host/SNI preservation; redirects still cannot widen scheme, + host, port, or private-network access. +- Private/loopback endpoints require the route's existing explicit private + network opt-in and an explicit Lab-run confirmation. Metadata endpoints + remain blocked. +- No scenario-supplied URL, model output, tool argument, or redirect may add a + destination. +- A sidecar dependency is allowed only when the scenario explicitly authorizes + its role/protocol class, the composite subject names the exact dependency + fingerprint, an in-memory destination record exists for that fingerprint, + the operator approves the composite live probe, and its credential is + destination-bound independently. Unmanifested roles or subject-external/ + dynamically widened endpoints make the run `harness_failure`. +- Inherited `HTTP_PROXY`, `HTTPS_PROXY`, `ALL_PROXY`, `NO_PROXY`, `http_proxy`, + `https_proxy`, `all_proxy`, and `no_proxy` values are rejected for Lab runs. + If a future reviewed scenario requires a proxy, that proxy endpoint is + authorized as its own exact destination under the same SSRF checks and never + inherits ambient proxy environment variables. +- Tools have no network capability. A model-requested web search, image + generation, URL fetch, computer use, or hosted external tool is disabled or + classified inapplicable unless a future separately reviewed scenario owns a + fixed synthetic sidecar. +- Deterministic protocol tests may contact only a Lab-owned loopback mock. + +### Credentials + +- Secret bytes remain owned by the existing trusted provider credential/ + transport layer. Lab runner code never receives an API key, token, cookie, + authorization value, entire auth store, or credential-bearing header. +- Immediately before a live request, the credential broker validates the exact + immutable `LabDestinationV1` snapshot and returns an opaque, + non-serializable `LabCredentialLeaseV1` capability bound to that destination, + auth transport, and one run/request budget. The lease exposes no secret + bytes, string conversion, header map, equality/debug representation, or + credential identity. +- The trusted transport consumes the lease and injects authorization only after + its connection is bound to the same approved destination/address set. A + destination, Host/SNI, address-set, transport, or lease-scope mismatch fails + closed before authorization is emitted. The lease expires after its bounded + request/run and cannot be reused for another destination or sidecar. +- Credentials therefore never enter a subject, assertion, error, artifact, + log, event-ID input, SQLite row, callback, mock, or Lab-owned memory buffer. +- Credential absence/rejection produces `authentication_blocked`, never a + compatibility failure. + +### Process and system access + +- The scenario DSL cannot express a shell command, executable, arbitrary + module, callback, script, filesystem path, or dynamic import. +- The runner receives no general shell/process API and no inherited stdin. +- Filesystem access is restricted to a fresh Lab scratch directory, read-only + packaged synthetic fixtures, and the bounded artifact writer. +- The V1 inherited-environment allowlist is empty: Lab code must not read an + ambient variable to determine behavior, routing, credentials, destinations, + paths, locale, or proxying. The runner may expose only a constructed, + non-inherited environment view with exact constants `TZ=UTC` and + `NO_COLOR=1`; every other name is absent. In particular all uppercase and + lowercase proxy variables are rejected as stated in Network above. Secrets + are supplied only through reviewed destination/credential plumbing. +- The run has enforced wall-clock, inactivity, byte, request, token, tool-call, + memory, process and artifact limits. If the platform cannot enforce a + required boundary, the run fails as `harness_failure`. +- Scratch data is deleted after artifact sanitization. Cleanup failure is + visible and retried by bounded maintenance; it does not silently retain user + data because none was admitted. + +### Tools and MCP + +- Function/custom tool scenarios expose inert Lab-authored definitions. The + harness returns static or pure-function results and never executes model + arguments. +- `apply_patch`, shell, file, browser, web-search, image-generation, computer + use and similar names are protocol tokens only. They do not invoke the real + facility. +- MCP scenarios use an in-memory or Lab-owned loopback stub with fixed schemas, + resources and pure results. User MCP configuration is not loaded. +- Cursor `nativeLocalExec`, `unsafeAllowNativeLocalExec`, desktop executors and + configured `mcpServers` are forced off for the Lab subject. Their disabled + state participates in the behavior fingerprint. + +### Agent Fabric + +- Real task execution remains in Agent Fabric's separately reviewed sandbox. +- The Lab accepts a structured outcome and sanitized content-addressed + references only. +- Outcome ingestion cannot dereference an arbitrary path or URL. Artifact + transfer uses an allowlisted broker and re-runs Lab validation. +- No task repository, prompt transcript, worktree, patch body, terminal log, or + hidden reasoning is copied into `~/.opencodex/lab/`. + +## 3. Artifact and resource contract + +Artifacts are deny-by-default, normalized, sanitized, bounded, and +content-addressed after redaction. + +Initial hard ceilings: + +```text +maximum wall-clock time per run 120,000 ms +maximum connect time 10,000 ms +maximum first-byte time 30,000 ms +maximum inactivity time 30,000 ms +maximum requests per run 16 +maximum aggregate input bytes 8 MiB +maximum aggregate output bytes 16 MiB +maximum output tokens 32,768 +maximum tool calls 32 +maximum runner resident memory 512 MiB +maximum child processes 0 +maximum artifacts per run 16 +maximum bytes per artifact 256 KiB +maximum aggregate artifact data 1 MiB +maximum normalized events 4,096 +maximum sanitized string field 4 KiB +maximum serialized bytes per event 64 KiB +maximum aggregate normalized event bytes 1 MiB +maximum event JSON nesting depth 8 +maximum object keys per event object 64 +maximum array elements per event array 256 +``` + +Scenario limits may be lower. No scenario, CLI flag, profile, provider config, +environment variable, route metadata, or runtime caller may raise a hard +ceiling. Raising one requires a reviewed security-contract version change. +Timeout ceilings classify with the matching typed `timeout`; request/byte/ +token/tool/artifact ceilings classify `budget_exhausted`. A memory ceiling +breach or attempted child-process creation terminates the sandboxed run and is +`budget_exhausted`; inability to enforce any required ceiling is +`harness_failure` before executable evidence is accepted. + +Event bounds are enforced while decoding/normalizing, before an event is +buffered into the observation or any artifact. Exceeding an event structural +bound fails the run as `harness_failure` without retaining the oversized +fragment. + +Allowed artifact classes: + +- canonical scenario manifest; +- canonical suite manifest; +- canonical synthetic fixture; +- canonical claim-source manifest; +- assertion report containing normalized expected/observed summaries; +- sanitized request shape with content replaced by type/length/digest markers; +- sanitized response shape with visible synthetic fixture output only; +- normalized bounded event trace; +- sanitized error taxonomy/status; +- deterministic verifier summary. + +Artifact names are derived only from the expected lowercase SHA-256 digest and +a fixed extension under `~/.opencodex/lab/artifacts/`. The Lab artifact store +must use descriptor/handle-bound, no-follow I/O rather than validate a pathname +and reopen it later: + +- open and retain a trusted handle to the artifact directory after verifying it + is a directory and not a symlink/reparse-point redirection; +- create writes relative to that directory handle with exclusive/no-follow + semantics, reject special files and `st_nlink != 1`, and write only already + redacted bytes; +- compute size and digest from the same open descriptor/handle that received + the bytes, flush it, then publish by an atomic rename relative to the trusted + directory handle; +- when a content-addressed target already exists, open it with no-follow + semantics and verify regular-file type, single-link status, size, and digest + from that same descriptor before reuse; +- reads open the digest-derived name relative to the trusted directory handle, + verify file type/link count/size, hash the bytes, and return/consume those + exact bytes from the same open descriptor without closing and reopening by + path; +- traversal, separators, absolute paths, alternate data streams, symlinks, + reparse points, hard links, device/special files, digest/size mismatch, or a + directory-handle identity change fail closed; +- on a platform where equivalent descriptor/handle-bound no-follow operations + cannot be enforced, the artifact operation is `harness_failure`; it must not + fall back to path-only `exists/stat/read` validation. + +The ledger stores relative content-addressed references, never arbitrary paths. +This contract applies to the future Lab artifact store; it does not silently +reuse a different existing application artifact reader whose path-race +properties have not been reviewed for Lab evidence. + +Scenario/suite manifests and synthetic fixtures use the domain-separated +digests in the evidence contract and remain retained while referenced by any +non-invalidated, non-purged observation. Claim-source manifests remain retained +while a non-purged claim snapshot references them. Their content is still +subject to the same synthetic/sanitized-data and size rules. + +Redaction occurs before hashing and writing. A redaction failure discards the +artifact and marks the run `harness_failure`; "write now, redact later" is +forbidden. + +## 4. Diagnostic sanitization + +Provider diagnostics retain only: + +- normalized HTTP status; +- allowlisted non-sensitive error type/code; +- coarse phase (`dns`, `connect`, `tls`, `first_byte`, `stream`, `terminal`); +- bounded latency/duration; +- redacted, bounded message selected by an explicit provider sanitizer. + +They remove URLs, query strings, authorization values, header dumps, request/ +response bodies, account identifiers, project/tenant names, local paths, IPs +where identifying, and token-like strings. Unknown provider diagnostics are +reduced to taxonomy and phase rather than persisted verbatim. + +Sanitizers are tested with seeded canary secrets and common credential forms. +`bun run privacy:scan` remains required but is defense in depth, not the +redaction mechanism. + +## 5. Subject privacy + +The local route subject distinguishes exact behavior without raw secrets: + +- configured instance, endpoint, custom-header behavior, project and location + use a per-installation keyed HMAC; +- credential/account identity does not participate; +- raw base URLs and private/custom headers are absent; +- model IDs are retained locally because they are required route identity, but + custom model IDs are private-by-default for export; +- rotating the local subject salt invalidates local correlation and requires + re-projection/reverification, never reverse lookup. + +The salt is stored with secret-file permissions outside the JSONL/artifact +tree. It is not exported. + +### Custom-header fingerprint broker + +Raw custom headers remain owned by the provider/config request builder and are +never passed to Lab code. That owner computes +`headers.nonCredentialBehaviorDigest` through a narrow fingerprint broker: + +1. resolve the effective static custom headers after preset/config merge but + before request-specific or credential injection; +2. remove every credential-bearing header according to the same auth transport + classification used by the request builder; +3. before canonicalization, enforce at most 64 non-credential header entries, + at most 16 duplicate values for one lowercase name, at most 256 ASCII bytes + per field name, at most 8 KiB UTF-8 bytes per value, and at most 64 KiB of + aggregate normalized name/value bytes; exceeding any bound is + `harness_failure` and no digest is emitted; +4. lowercase valid ASCII field names, reject invalid names, preserve duplicate + value order, and preserve exact UTF-8 value bytes without trimming; +5. sort entries by lowercase name while retaining duplicate order and encode + JCS `[{"name": string, "values": string[]}, ...]`; +6. return lowercase HMAC-SHA-256 with installation salt and domain + `ocx-lab:local-fingerprint:v1\0customHeaderBehavior\0`. + +The broker returns only the digest. Its API cannot return normalized names, +values, intermediate bytes, the salt, or the credential classification. +Unknown classification fails subject construction; it never falls back to +hashing or logging the raw header. Canary tests must prove raw names/values do +not enter Lab events, errors, SQLite, or artifacts. + +## 6. Local evidence versus public export + +Local evidence is already sanitized. Public export is stricter and uses a new, +allowlist-only schema: + +- include suite/scenario versions, evidence layer, verdict, observation time + bucket, public registry provider/model where permitted, assertion summaries, + and public incident/scenario references; +- replace local subject/event/artifact IDs with export-scoped opaque IDs; +- omit endpoint and provider-instance fingerprints, local request/decision/ + Fabric references, precise local paths, custom headers, project/location, + custom provider/model names, account context, raw latency traces, and local + errors; +- include artifact content only when its policy explicitly says + `public_export`; local visibility does not imply export permission; +- run export-specific secret/PII scanning and fail closed on an unknown field. + +Public publishing is not authorized in CL-00 and remains a later phase. + +## 7. Retention and deletion + +- JSONL is the immutable local authority for non-sensitive evidence, but a + user can delete the entire Lab directory. Immutability describes in-ledger + correction semantics, not a promise to resist user deletion. Confirmed + sensitive evidence is the explicit privacy exception below. +- Retention ceilings by class: + - scratch/temp run directories: deleted at run end; cleanup retry within 24h; + - export staging: maximum 24h; + - disposable SQLite projection: rebuildable anytime; may be deleted at any + time and must be deleted during a sensitive purge; + - sanitized non-contract artifacts (`assertion_report`, shapes, traces, + errors): default 90 days, hard ceiling 365 days; + - content-addressed scenario/suite/fixture contract artifacts: retained + while any non-invalidated, non-purged observation references them, because + reproducible executable projection requires the exact historical bytes; + - content-addressed claim-source manifests: retained while any non-purged + claim snapshot references them, because reproducible `CLAIMED` projection + requires the exact historical sanitized source bytes. + Contract artifacts remain synthetic/sanitized and size-bounded. User deletion + of the Lab directory remains absolute. +- Deleting an expired non-contract artifact leaves its digest/reference and a + typed unavailable marker; it does not alter the observation. +- SQLite is disposable and contains no data absent from valid ledger events, + privacy-safe purge tombstones, and artifact metadata. +- Invalid non-sensitive evidence is neutralized by a valid appended + invalidation under the evidence contract. Event-private non-contract + artifacts may then be securely deleted. A shared scenario, suite, fixture, + or claim-source contract artifact must remain while any other usable event + references its digest, and may be deleted only after the last such reference + is invalidated or purged. +- Confirmed sensitive evidence is distinct from ordinary invalidation. It + requires a fail-closed purge of every local copy: offending JSONL event + lines, SQLite rows, artifacts, scratch/temp files, and generated exports. + The purge first determines the affected event IDs/artifact digests without + retaining the leaked value, then creates a clean replacement ledger that + omits the sensitive event lines and includes the canonical privacy-safe + `purge_tombstone` defined in the evidence contract. The replacement ledger + is flushed and atomically installed; SQLite is deleted/rebuilt from it and + targeted artifacts/temp/exports are removed. If any required replacement or + deletion cannot be completed, the purge remains visibly failed and evidence + projection is disabled rather than serving stale compatibility state. +- Projection applies purge tombstones before ordinary invalidation/supersession. + Targeted events contribute no `CLAIMED`, `PROBED`, `VERIFIED`, `DEGRADED`, or + `UNSUPPORTED` state; targeted artifacts surface only + `purged_unavailable`. A previously cached verdict that depended on purged + material is invalid and must not survive SQLite rebuild. +- A purge tombstone stores only event IDs/artifact digests, the fixed + `sensitive_evidence` taxonomy, time/producer metadata, and closed action + names. It never records the leaked value, raw path, credential, prompt, or + identifying diagnostic. Append-only semantics never override the duty to + remove sensitive bytes. + +## 8. Security acceptance tests required later + +Before any live runner ships, tests must prove: + +1. prompt/repository/MCP/user-tool inputs are unreachable from the scenario DSL; +2. every admitted fixture has the required synthetic marker/provenance and a + runtime/user/repository/MCP/network object cannot self-assert synthetic + status; +3. redirects and model-supplied URLs cannot widen network access, and Lab + clients pin connections to the validated IP set; +4. the inherited-environment allowlist is empty, all uppercase/lowercase proxy + variables are rejected, and no ambient variable changes Lab behavior; +5. destination-record mutation, replacement, or address-set drift between + authorization, fingerprinting, credential binding, and connect fails closed; +6. credential canaries never enter Lab memory/objects through the broker: the + runner receives only an opaque destination-bound one-run lease, and lease + scope/transport/destination mismatch fails before authorization is sent; +7. account, custom-header and endpoint canaries never enter evidence, errors, + SQLite or artifacts; +8. custom-header canonicalization is deterministic; unknown credential + classification and every count/name/value/aggregate bound fail closed; +9. local subject-salt rotation breaks prior correlation and forces + re-projection/reverification without reverse lookup; +10. tool arguments cannot execute; +11. every wall-clock/connect/first-byte/inactivity/request/input/output/token/ + tool-call/memory/process/artifact hard ceiling is enforced and cannot be + widened by a manifest, config, profile, environment variable, or caller; +12. artifact traversal, symlink/reparse, hard-link, path-race, oversize and + digest attacks fail closed, and validated artifact bytes are consumed from + the same descriptor/handle that was checked; +13. normalized event byte/depth/key/array ceilings fail closed before buffering; +14. timeout, quota, auth, DNS and harness failures remain blockers; +15. retention expiry emits typed unavailable markers; cleanup retry/failure is + visible and bounded; shared contract artifacts survive invalidation while + any usable event still references them; +16. ordinary invalidations reject unknown/future/cross-kind/partial target + lists and deterministically remove only valid named evidence; +17. confirmed sensitive evidence is removed from JSONL, SQLite, artifacts, + temp files and exports; the replacement ledger retains only a privacy-safe + purge tombstone; replay cannot preserve a verdict that depended on purged + evidence; +18. public export rejects unknown/private fields; +19. no probe runs from the production routing path. diff --git a/devlog/_plan/260807_compatibility_lab/050_cl00_acceptance_review.md b/devlog/_plan/260807_compatibility_lab/050_cl00_acceptance_review.md new file mode 100644 index 0000000000..68a6a0e0bd --- /dev/null +++ b/devlog/_plan/260807_compatibility_lab/050_cl00_acceptance_review.md @@ -0,0 +1,334 @@ +# CL-00 independent acceptance review + +Date: 2026-08-08 + +Scope: the complete CL-00 contract set on +`feat/cl-00-compatibility-contracts`, based on +`3ad5bb6bd3f76f6879d84b78ea39edd3e01ec296`. The initial delayed-review +acceptance was recorded at `12e50a3502fb4af25283538cc717ead2291edd8b`. +CodeRabbit re-reviews were validated against current production code and the +contract authority before any change was accepted. This record supersedes the +stale acceptance/status statements from the earlier passes. + +The review is contract-focused and separate from the original authoring pass. +Every validated Critical, High, and Medium finding was corrected. Valid +deterministic and security-contract defects were also corrected regardless of +review label. CL-02 was not started. + +## Findings and corrections + +### Critical + +None. + +### High + +1. Initial scenario prose did not define executable selector/operator semantics + or canonical per-case manifests. + - Correction: added the closed assertion/selector/SSE contract in `021` and + the 35-case machine-readable authority in `022`, with literal fixtures, + expected values, row-specific requirements, roles, media types, limits, + artifact policy, and failure rules. +2. Immutable observations lacked enough manifest/fixture provenance to + reproduce `VERIFIED`. + - Correction: observations now carry scenario, suite, and fixture digests; + domain-separated digest preimages are exact; referenced manifests and + fixtures are retained content-addressably and cannot be replaced by the + current version during replay. +3. Route identity did not close compatibility-version and sidecar-dependent + behavior. + - Correction: froze the compatibility-version manifest/preimage, included + effective runtime and sidecar settings, added flat dependency identities, + and kept route-local endpoints in the composite subject rather than the + provider-independent scenario manifest. +4. Response-only protocol vectors did not identify an initiating client + request. + - Correction: added 11 explicit initiating request fixtures. All + `upstream_response` cases now have one request that fixes model, input, + stream mode, and inbound surface. +5. Named verifier values had no deterministic derivation. + - Correction: defined every V1 verifier as a closed pure function over the + current synthetic fixture and normalized observation. +6. Catalogue prose and incident mappings initially implied coverage beyond the + literal V1 assertions. + - Correction: narrowed every protocol V1 row to its exact `022` evidence and + made incident mappings explicit future scenario/version inputs when no + literal V1 vector exists. +7. The three evidence layers lacked separate executable subject identities, + and suites could span layers. + - Correction: added the closed `ProtocolSubjectV1`, `RouteSubjectV1`, and + `TaskSubjectV1` union, exact layer/subject matching, layer-qualified suite + manifests, and one reserved deterministic Fabric task/verifier contract. +8. Behavior identity did not explicitly close effective + `commandCodeVersion`, sampling-parameter omission sets, `cacheRetention`, or + unknown future behavior inputs. + - Correction: froze `BehaviorFingerprintV1`, its closed keys and source + tags, required values from the production resolver, and fail-closed + handling/tests for an unclassified behavior input. +9. A global protocol failure-rule list attached both control effects to every + case, violating the legal control matrix. + - Correction: the base rule set contains no control rule; expansion adds + exactly one materialized rule only for a case with `expectedFailure`. + Protocol V1 has one conformance control and no unsupported effect. +10. Tool/MCP assertions selected bare calls from the normalized SSE event array. + - Correction: froze separate canonical `toolCalls[]` and `mcpCalls[]` + semantic projections and moved all call/correlation selectors to them. +11. Chat-backed Responses cases expected adapter `done` instead of the + client-visible `completed` terminal, and parallel-tool counting still used + nonexistent `tool_call` SSE events. + - Correction: those terminals now expect `completed`; parallel count/order + derive from `/client/response/toolCalls`, and `toolCalls`/`mcpCalls` are + part of the closed observation schema. + +### Medium + +1. `VERIFIED -> PROBED` was missing after partial invalidation. + - Correction: added the transition for remaining partial coverage. +2. Scenario and suite freshness authorities conflicted. + - Correction: effective age is the minimum finite scenario, suite, and + profile bound. +3. Compatibility-version file hashing and dirty/missing/symlink behavior were + underspecified. + - Correction: froze the canonical object, file set, raw-byte hashes, sort + order, current-working-tree behavior, and fail-closed cases. +4. Sidecar network wording incorrectly put route-local endpoints in scenario + manifests. + - Correction: manifests authorize only dependency roles/protocol classes; + the composite subject owns exact destination fingerprints. +5. The MCP exact-bound vector was not actually at its stated boundary. + - Correction: replaced it with exact 64-byte and 65-byte UTF-8 JSON schema + payloads and a recomputed fixture digest. +6. The vision modality control could have made a compatible suite + `UNSUPPORTED`. + - Correction: made it a `negative_control`; its exact rejection satisfies + the suite without projecting route-level `UNSUPPORTED`. +7. The compaction assertion tested presence rather than truth. + - Correction: changed it to exact equality with `true`. +8. One result-content description claimed call correlation absent from its + assertions. + - Correction: removed the claim. +9. Environmental failure effects, claim supersession/currentness, and + custom-header fingerprint ownership were not mechanically closed. + - Correction: added the exhaustive class/effect matrix, formal + `claim_snapshot`/`supersedes[]` schema and currentness algorithm, and a + config-owner broker that exposes only a domain-separated header digest. +10. `ProtocolSubjectV1` named a second `runtimeFingerprint` without a schema. + - Correction: removed it; the closed `runtime.*` behavior keys are the sole + platform-sensitive identity inputs. + +### Low + +- Corrected the provider-test description: forward/static providers do not + always perform a live `/models` request. +- Corrected historical reference `#745` from issue to pull request. +- Added `021`/`022` to the stack ledger and created this review record, closing + all local document links. +- Corrected request-history evidence from “immutable” to canonical + append-only, limited the profile claim to compatibility policy, and noted the + explicit state-mutating `ocx doctor --fix-codex-runtime` mode. + +## CodeRabbit remediation: first pass + +All ten unresolved CodeRabbit threads in the first remediation pass were +inspected against current branch code/contracts before editing. + +1. Stack audit metadata used a non-SHA dependency label where an exact CL-01 + base revision is required. The stack ledger now records exact base/head + revisions and the CL-01 correction requirement. +2. `BehaviorFingerprintV1` did not define deterministic ordering for every + array-valued closed key. V1 now classifies each allowed array as `set` or + `ordered`, defines JCS-byte sorting/deduplication for sets, preserves source + order for ordered arrays, and fails closed for any undeclared array input. +3. `all-applicable-required-pass-v1` admitted a vacuous `VERIFIED` result when + zero required scenarios applied. Positive executable verdicts now require a + non-empty applicable required set; zero-applicable falls through to current + claim/unknown semantics, while attempted environmental blockers remain + `BLOCKED`. +4. `[DONE]` handling was selected by the client-facing surface even for a Chat + upstream fixture. Sentinel interpretation now follows the protocol of the + byte stream being normalized; only OpenAI Chat recognizes exact `[DONE]`. +5. Two Chat-backed tool-result assertions incorrectly selected Responses + `input[].call_id`. Production `openai-chat` emits the continuation as + `messages[1].tool_call_id`; both selectors now assert the actual Chat wire. +6. Live-probe destination authorization could drift between endpoint + fingerprinting, credential binding and connect. The security contract now + requires one immutable per-run `LabDestinationV1` snapshot for every stage + and fails closed before credential transmission on mutation/re-resolution or + mismatch. +7. Ambient environment/proxy handling was not executable enough. The inherited + environment allowlist is empty; the runner constructs only `TZ=UTC` and + `NO_COLOR=1`, rejects uppercase/lowercase proxy variables, and cannot derive + behavior from ambient variables. +8. Custom-header fingerprinting lacked resource/canonicalization bounds. The + broker now enforces entry, duplicate-value, field-name, per-value and + aggregate byte ceilings before JCS/HMAC, with unknown credential + classification or overflow failing closed. +9. Ordinary invalidation wording allowed deletion of shared contract artifacts. + Only event-private non-contract artifacts may be deleted after invalidation; + shared scenario/suite/fixture artifacts survive until no usable observation + references them. +10. Security acceptance coverage omitted the new invariants. Required tests now + cover environment/proxy denial, destination snapshot mutation/address drift, + custom-header canonicalization and bounds, subject-salt rotation, and + retention expiry/cleanup/unavailable markers. + +## CodeRabbit remediation: second pass + +CodeRabbit reviewed the remediation again and raised additional deterministic +and security-contract issues. Every Major finding in that pass was validated as +material and corrected within CL-00: + +1. `invalidation` had no executable payload. It now has a non-empty bounded, + sorted/unique target-event set, a closed reason set, all-or-nothing target + validation, earlier-event/type constraints, and no implicit uninvalidation. +2. `sourceManifestDigest` was not reproducible. `ClaimSourceManifestV1` now + defines a closed sanitized source snapshot, a domain-separated digest, + content-addressed retention, and replay validation. Missing/mismatched source + bytes cannot produce `CLAIMED`. +3. Sidecar dependency sorting omitted `providerInstanceFingerprint`, so two + otherwise-equal dependencies could compare as equal. The canonical total + ordering now includes it. +4. Synthetic fixture trust was prose-only. Every expanded protocol V1 + `fixtureRef` now includes mandatory `ocx-lab-synthetic-v1` marker and + `lab_authored` provenance bound to the authority and source commit; these + fields participate in the scenario-manifest digest and are validated before + fixture admission. +5. `synthetic_tool` did not say what MCP cases execute. The four MCP V1 cases + now carry exact scenario-specific action tokens with closed fixture schemas, + deterministic invocation/list/read/boundary behavior, and fail-closed + registration for missing/wrong/multiple actions. +6. The credential broker could still expose the selected secret to Lab code. + The contract now keeps secret bytes in trusted credential/transport plumbing + and gives the Lab only a non-serializable, destination/auth-transport-bound, + one-run/request `LabCredentialLeaseV1` capability. +7. Required execution limits had no hard maxima. V1 now freezes hard ceilings + for wall/connect/first-byte/inactivity time, requests, aggregate input/output + bytes, output tokens, tool calls, resident memory, child processes and + artifacts; manifests/config/profiles/environment/callers can only tighten + them. +8. Artifact path validation was vulnerable to path-race/symlink substitution if + implemented like the unrelated current image-artifact helper. The future Lab + store is now required to use trusted-directory-handle, no-follow, + descriptor-bound validation/read/write and atomic publication. CL-00 did not + alter `src/images/artifacts.ts`; that runtime is outside this contract-only + phase. +9. Physical sensitive-evidence purge did not define canonical projection state. + The ledger now defines privacy-safe `purge_tombstone` events, clean atomic + ledger replacement, SQLite rebuild/removal, typed `purged_unavailable` + artifacts, and mandatory exclusion of every verdict/claim that depended on + purged evidence. + +The protocol-authority rewrite also fixed the flagged missing final newline. +Two remaining review notes were editorial-only (`falsey` -> `falsy` in the +incident prose and a master-plan acceptance phrase); they do not change any +contract, deterministic behavior, security boundary, acceptance criterion, or +CL-01 implementation input and are handled as non-blocking review-thread +responses rather than expanding this contract remediation. + +## Mechanical review evidence + +- `022_protocol_v1_cases.json` remains valid JSON by inspection through the + GitHub file API and contains the same 35 case objects / 46 fixture records as + the accepted authority. +- Fixture `bytesUtf8` and fixture digest values were not changed by the + CodeRabbit selector/provenance/MCP-action remediation. +- The two corrected Chat continuation selectors target + `/upstream/requests/1/json/messages/1/tool_call_id`, matching the current + `openai-chat` request builder's assistant-call then tool-result message order. +- `fixtureRef` expansion now adds mandatory marker/provenance fields. Therefore + all expanded protocol scenario-manifest digests and dependent suite-manifest + digests change even though fixture bytes/digests remain unchanged. +- Four MCP `requiredHarnessFeatures` arrays now additionally contain their exact + closed action token. Those four scenario-manifest digests therefore also + change for semantic reasons. +- `vision-core.protocol.modality-gate` remains the sole V1 negative control. +- Base failure rules contain no control effect; the vision case alone expands + the conformance-control rule. +- The MCP bound fixture remains the accepted exact 64/65 UTF-8-byte vector. + +## Repository verification + +Initial acceptance verification remains the last executed local-suite evidence: + +- `bun run typecheck`: passed. +- `bun run privacy:scan`: passed. +- `bun test tests/repo-hygiene.test.ts`: 11 passed, 0 failed. +- Focused protocol/compatibility suite excluding Windows privileged-symlink + state cases: 395 passed, 0 failed across 24 files. +- Focused continuation-state semantics: 2 passed, 95 filtered, 0 failed. +- `tests/codex-models-cache-invalidate.test.ts`: 6 passed, 0 failed. +- `tests/codex-native-residue.test.ts`: 63 passed, 2 platform skips, 0 failed. +- Original local link/case/digest checks and `git diff --check`: passed. + +The CodeRabbit remediation is documentation/contract-only. The GitHub connector +does not provide a local Bun execution environment, so this review does **not** +claim a new typecheck/privacy/test run after these documentation changes. Final +GitHub status/workflow contexts and unresolved review threads are checked after +the status-ledger sync. + +The earlier full `bun run test` result was **not green**. On Windows with Bun +1.3.14 it exited 3 after a cache-invalidation failure, an empty effective-account +lookup, and a Bun `index out of bounds` panic. A broader focused run separately +found four `responses-state.test.ts` failures, all Windows `EPERM` errors +creating symlinks (488 passed, 4 failed). The isolated cache/native tests and +the non-privileged protocol suite passed; this review does not claim the full +suite passed. + +## Required challenge results + +1. Protocol conformance, live compatibility, and task effectiveness are + separated: **PASS**. +2. Environmental failures cannot poison compatibility verdicts: **PASS**. +3. `VERIFIED` is reproducible, non-vacuous, and invalidation/purge aware: + **PASS**. +4. Exact route/dependency identity prevents false evidence reuse: **PASS**. +5. `CLAIMED` is reproducible from retained sanitized source manifests: + **PASS**. +6. Routing Profiles remain the sole compatibility-policy surface: **PASS**. +7. The Lab cannot become a second router or provider registry: **PASS**. +8. Synthetic-fixture admission, credentials, destinations, environment, + resources, artifacts and purge behavior are fail-closed: **PASS**. +9. Historical incidents remain representable as deterministic versioned + scenarios: **PASS**. +10. CL-01 remains implementable without semantic invention after synchronizing + the refreshed V1 authority: **PASS WITH REQUIRED CL-01 REBASE, CORRECTION, + AND REVALIDATION**. + +## CL-01 impact + +The independently accepted CL-01 branch exists at +`feat/cl-01-conformance-harness` at accepted head +`cc447ce9d19d5fb4e03988899f5fb495f9de8d0e`. It was built from the older CL-00 +revision `c2113ca47b8a05c5a5f90679e4eaa640ca2c6a66`. + +Its acceptance record explicitly documents a harness-only projection of +Chat-wire `messages` tool rows into a synthetic Responses-shaped `input[]` to +satisfy the old CL-00 selectors. That workaround is no longer authoritative: +CL-00 selects the actual Chat `messages[].tool_call_id` field. CL-01 also copied +the pre-remediation case authority and its SSE helper retained client-surface +sentinel selection. + +Before CL-01 is stacked or merged it must therefore: + +- rebase onto the final refreshed CL-00 accepted contract head; +- synchronize the two corrected Chat result selectors; +- remove or narrow the synthetic Chat-to-Responses `input[]` observation + projection so upstream observations remain the actual Chat request; +- align SSE normalization with source-protocol `[DONE]` selection; +- implement/validate the mandatory synthetic fixture marker/provenance in + expanded `fixtureRef` values and recompute all scenario/suite manifests; +- synchronize the four exact MCP action tokens and their closed execution + semantics; and +- rerun the canonical CL-01 scenarios, negative controls, digest/manifest + checks and independent CL-01 acceptance review. + +This CL-00 remediation does not modify CL-01 and does not start CL-02. + +## Verdict + +All validated Critical, High, Medium, deterministic-contract and +security-contract findings found through the two CodeRabbit remediation passes +are corrected in the CL-00 contract set. Final GitHub thread/status checks are +recorded after the stack ledger is synchronized. + +**CL-00: ACCEPTED AFTER CODERABBIT REMEDIATION** diff --git a/devlog/_plan/260807_compatibility_lab/051_cl01_acceptance_review.md b/devlog/_plan/260807_compatibility_lab/051_cl01_acceptance_review.md new file mode 100644 index 0000000000..7c9cb3a3f4 --- /dev/null +++ b/devlog/_plan/260807_compatibility_lab/051_cl01_acceptance_review.md @@ -0,0 +1,50 @@ +# CL-01 independent acceptance review + +Reviewer posture: adversarial. Scope: deterministic protocol conformance harness only. + +**Revision note:** CL-01 was **accepted earlier** at `cc447ce9d19d5fb4e03988899f5fb495f9de8d0e` against pre-remediation CL-00 tip `c2113ca47b8a05c5a5f90679e4eaa640ca2c6a66`. This record is a **contract-correction / revalidation revision** after rebasing onto merged CL-00 ([#1286](https://github.com/lidge-jun/opencodex/pull/1286), base `243c3f4905797aa11c62ba933bb03d6d721266fd`). + +## Invalidated earlier assumptions + +| Earlier CL-01 assumption | Final CL-00 correction | +|---|---| +| Chat upstream tool results correlate via synthetic Responses `input[]` in observations | Real Chat wire: `/upstream/requests/N/json/messages/M/tool_call_id` | +| SSE `[DONE]` inferred from client surface labels (`responses-sse`, etc.) | Sentinel follows **source protocol** of normalized byte stream; only `openai-chat` recognizes `[DONE]` | +| Expanded manifests without synthetic marker/provenance | Mandatory `syntheticMarker: "ocx-lab-synthetic-v1"` + `lab_authored` provenance in every fixture ref | +| MCP scenarios implicit / unspecified | Four closed action tokens with deterministic semantics | +| Obsolete manifest digests from pre-provenance expansion | All scenario manifest digests recomputed with provenance fields | + +## Removed workaround + +The harness **removed** `normalizeUpstreamObservationJson()` Chat `messages[]` → synthetic Responses `input[]` projection. Observations now record actual upstream JSON from shipped adapters. Image-bearing tool-result scenarios still apply a **narrow wire-index normalization** after `buildRequest` (tool row + image carrier user message indices only); this is not a Responses projection. + +## Challenge results (revalidation) + +| # | Challenge | Result | +|---|---|---| +| 1 | Harness exercises shipped parser/translation, not a parallel stack | **PASS** — executor calls `parseRequest`, `createOpenAIChatAdapter`, `createResponsesPassthroughAdapter`, `bridgeToResponsesSSE`, `responsesSseToAnthropicSse`, and `expandPreviousResponseInput`. | +| 2 | Negative controls genuinely fail | **PASS** — eight deliberate broken fixtures reject (`runNegativeControls` 8/8). | +| 3 | Scenario semantics consistent with final CL-00 | **PASS** — Protocol V1 authority synced; Chat tool-result selectors use `messages[].tool_call_id`; no Responses `input[]` fabrication. | +| 4 | Malformed/partial streams cannot accidentally pass | **PASS** — malformed SSE negative control fails; truncated tool args fail `tool_call_equals`. | +| 5 | Tool IDs and tool-result correlations verified | **PASS** — `tools-core.protocol.function-round-trip`, `codex-core.protocol.apply-patch-turn` use Chat wire selectors. | +| 6 | Parallel tool fragments handled | **PASS** — `tools-core.protocol.parallel-correlation` and `nonoverlap_order` verifier. | +| 7 | Custom/freeform tools covered | **PASS** — `apply_patch` via `freeformToolNames` in bridge. | +| 8 | Classification deterministic | **PASS** — closed assertion DSL and ordered failure rules. | +| 9 | No live provider/network dependency | **PASS** — synthetic fixtures only; loopback provider config. | +| 10 | No CL-02 functionality leaked | **PASS** — no ledger, SQLite, CLI probe, or live runners. | +| 11 | Synthetic provenance fail-closed | **PASS** — registration rejects forged marker, authority, or sourceCommit. | +| 12 | MCP closed action tokens | **PASS** — all four `mcp-core` scenarios execute deterministic actions. | +| 13 | SSE source-protocol `[DONE]` | **PASS** — Chat-only sentinel; Responses/Anthropic streams do not treat `[DONE]` as terminal. | + +## Validation (2026-08-09, Windows/Bun 1.3.14) + +- `bun x tsc --noEmit`: passed +- `bun test tests/lab-conformance-harness.test.ts`: **14/14** passed (24 canonical + 8 negative controls + provenance + SSE + MCP + manifest tests) +- `git diff --check`: passed (after correction) +- Full `bun run test`: not re-run (known Windows/Bun baseline failures documented under CL-00) + +## Verdict + +**CL-01: ACCEPTED (contract-corrected revalidation)** — harness conforms to merged CL-00 #1286, passes all CL-01 canonical scenarios and negative controls, implements provenance and MCP action contracts, and contains no CL-02 scope. + +**CL-02: NOT STARTED.** diff --git a/devlog/_plan/260808_bug_campaign/023_wp15_1244_and_green_five.md b/devlog/_plan/260808_bug_campaign/023_wp15_1244_and_green_five.md new file mode 100644 index 0000000000..1dc8953c85 --- /dev/null +++ b/devlog/_plan/260808_bug_campaign/023_wp15_1244_and_green_five.md @@ -0,0 +1,574 @@ +# WP15 — #1244 rebases itself, and five green contributor fixes get republished + +## What changed since WP14 + +WP15 opened as "resolve #1244's 22-hunk conflict by hand". That work no longer +exists. The PR head moved from `b413f8bff` to `15545b3d1` and the author +collapsed the branch onto the current `dev` tip: + +``` +$ git fetch origin pull/1244/head && git merge-base --is-ancestor origin/dev FETCH_HEAD && echo on-dev +on-dev +$ git log --oneline FETCH_HEAD -1 +15545b3d1 fix(codex): preserve routed models in desktop picker +``` + +`mergeStateStatus` is `UNSTABLE` rather than `DIRTY`, `isDraft` is now `false`, +and the 57-file diff no longer defines `mergeCatalogModelsWithNativeRecovery` +locally. The WP14 hypothesis — that resolving 22 hunks across +`src/codex/catalog/sync.ts` and `src/codex/convergence.ts` requires re-deciding +the author's design against the merged #1212 convergence work — is moot. The +author did that re-decision themselves. + +Consequence for the campaign: #1244 needs no maintainer rebase. It needs CI to +finish (`Cross-platform CI` and `Service lifecycle` were both `in_progress`) +and then a normal merge decision. That is a watch, not a work item. + +## The actual WP15 unit + +Five contributor PRs sit one or two commits ahead of a `dev` they are far +behind, and all five still sit in draft because the four-box readiness +checklist is the contributor's own attestation, which I will not tick for them +(`.github/workflows/enforce-pr-target.yml:516`). + +| PR | Author | Head | ahead/behind dev | Cross-platform CI at head | +|----|--------|------|------------------|---------------------------| +| #1189 | luvs01 | `d5242a231` | 2 / 300 | success | +| #1195 | luvs01 | `6eff3f6a5` | 2 / 300 | success | +| #1169 | TyroneXie | `d8968b7e6` | 1 / 335 | success | +| #1187 | luvs01 | `36cffcef6` | 2 / 9 | action_required | +| #1184 | luvs01 | `a2eda3b94` | 1 / 16 | action_required | + +Read that column through the WP1 rule: `gh pr checks` hides `action_required`, +so the source of truth is +`gh api "repos/lidge-jun/opencodex/actions/runs?head_sha=$sha"`, and `status` is +not `conclusion`. #1187 and #1184 are unapproved, not failing. + +All five apply cleanly onto `3ad5bb6bd`: + +``` +PR#1189 APPLIES CLEAN +PR#1187 APPLIES CLEAN +PR#1184 APPLIES CLEAN +PR#1195 APPLIES CLEAN +PR#1169 APPLIES CLEAN +``` + +"Applies clean" is textual, not semantic. The three deep-behind branches +(#1189, #1195, #1169 — 300+ commits) are exactly the case where a clean apply +can still be wrong, because `dev` may have moved the surrounding contract +without touching the same lines. Each gets a contract check below, and a clean +apply alone is not accepted as evidence for any of them. + +## Diff-level plan + +Republish protocol is `003_republish_protocol.md` unchanged: fresh worktree from +`origin/dev`, apply the author's net diff, one commit authored by the maintainer +with a trailer preserving the contributor, a PR body naming the source PR and +mentioning the author, then merge on green. + +### WP15-A — #1187 and #1184: approve at head, merge in place + +> **Corrected by audit B2 — see the audit-fold section below. The heading is +> wrong: these cannot be merged in place. Read WP15-A′.** + +Nine and sixteen commits behind, both under the gate's 10-commit tolerance after +a rebase, both tiny. No republish is warranted; the correct action is to unblock +CI. + +1. Re-read each PR head immediately before acting; log `run.head_sha` and + `pr.headRefOid` as separate columns in `.tmp/ocx_approval_ledger.tsv` with + `MATCH`/`SKIP`. A `SKIP` means the author pushed inside the window and the + approval would target a stale commit — abort that row. +2. Approve the `action_required` Cross-platform CI run for a `MATCH` row. +3. Await `conclusion == "success"`. A `failure` gets diagnosed, never guessed. + +Acceptance: both PRs have a `success` Cross-platform CI at the exact head the PR +points at, recorded with both SHAs. + +What this does not do: it does not make them mergeable by policy, because the +contributor checklist stays theirs. Approval only removes the gate that stops +them from proving box 1. + +### WP15-B — #1189: republish as `codex/260808-1189-history-stream-ingest` + +Net diff: `src/routing/history/indexer.ts`, +`tests/request-history-index.test.ts`. + +It replaces `readCompleteTail` — which allocated `size - indexedOffset` in one +shot — with a 64 KiB chunked reader that assembles records across chunk +boundaries and omits complete records above a 1 MiB projection bound +(`REQUEST_HISTORY_READ_CHUNK_BYTES`, `REQUEST_HISTORY_MAX_RECORD_BYTES`). It +deletes `ingestText` and folds line handling into `ingestSourceTail`. + +Contract checks, required because the branch is 300 behind: + +- `rg 'ingestText|readCompleteTail' src tests` on `dev` must show no caller + outside `indexer.ts`; a surviving caller means the deletion breaks it. +- `insert.finalize()` must still run unconditionally in `finally`. The Windows + file-lock note in the deleted code is load-bearing and CI runs Windows. +- Offset accounting: `nextOffset` may only advance past a `\n`, so a torn final + record is re-read rather than skipped. + +Verification: `bun test tests/request-history-index.test.ts`, +`bun run typecheck`, and an ablation reverting the 1 MiB bound that shows the +oversized-record test failing. A passing suite without the ablation is not +accepted. + +### WP15-C — #1195: republish as `codex/260808-1195-unbound-quota-unknown` + +Net diff: `src/router.ts`, +`src/server/management/routing-profile-routes.ts`, +`tests/quota-scoring.test.ts`, English routing docs. + +It deletes the same ~20-line block from both files: a policy candidate no longer +takes `codexAccountId`/`codexAccountPlan` from +`getEffectiveActiveCodexAccountId` when the Codex provider is in pool mode, and +no longer takes `accountRef` from `getAccountSet("anthropic")`. The rationale is +a real ordering defect — policy evaluation runs before Pool/Direct identity, +thread affinity, and Anthropic session affinity resolve, so a candidate can be +scored with account A's quota and executed on account B. + +Contract checks: + +- The two blocks must still be identical on `dev`. If `dev` already changed + either one, the delete is no longer symmetric and the PR is stale. +- `rg 'getEffectiveActiveCodexAccountId|getAccountSet'` must come back empty for + both files, and the now-unused imports must be gone or `typecheck` fails. +- Live-vs-dry-run parity: both paths must emit the same evidence shape, which is + the property `tests/quota-scoring.test.ts` is extended to hold. + +Verification: quota, policy-execution, routing-profile, and explainability +suites; `bun run typecheck`; an ablation restoring one block only, to show the +parity test fails asymmetrically. + +### WP15-D — #1169: republish as `codex/260808-1169-shim-routing-warning` + +Net diff: `src/cli/codex-shim-readiness.ts` (new), `src/cli/index.ts`, +`tests/codex-shim-readiness.test.ts`, English and zh-CN lifecycle docs. + +`ocx codex-shim install` reports clean success even when it cannot prove Codex +routes through OpenCodex. The change downgrades that to a warning for an +external `model_provider`, a user-owned local or remote gateway, or +unverifiable routing, and warns when proxy variables exist only in the current +process while `config.proxy` is unset. Advisory only: same exit code, and the +shim still fail-open execs the real launcher. + +Highest risk of the three at 335 commits behind, and it touches +`src/cli/index.ts`, which this campaign already modified. Contract checks: + +- The `codex-shim install` call site in `src/cli/index.ts` must still have the + shape the patch expects; confirm by reading the applied hunk rather than + trusting the apply. +- Privacy is the blocking property: no proxy URL, token, or account identifier + may reach stdout. `bun run privacy:scan` plus the test's own assertion inside + isolated `CODEX_HOME`/`OPENCODEX_HOME` directories. +- The exit code must be unchanged on the warning path. Assert it, because + "advisory only" is the whole safety argument. + +Verification: `bun test tests/codex-shim-readiness.test.ts`, +`bun run typecheck`, `bun run privacy:scan`, and a real CLI install in a temp +home showing the warning text with no secret in it. + +### WP15-E — #1244 watch + +> **Corrected by audit B1. #1244's Cross-platform CI at `15545b3d1` is +> `failure`, not `in_progress`. Read WP15-E′.** + +No code work. Poll the two `in_progress` runs at `15545b3d1`; merge on +`success`, diagnose and comment on `failure`. If the head moves again, re-read +it before acting. Record the outcome either way. + +## Acceptance criteria + +1. #1187 and #1184 have `success` Cross-platform CI at their exact current head, + both SHAs logged and matched in `.tmp/ocx_approval_ledger.tsv`. +2. Three new PRs exist for #1189, #1195, #1169, each with a `Co-authored-by` + trailer naming the original author, each mentioning them, each filling all + three PR-template sections. +3. Each republish carries a fresh focused-test result and an ablation that fails + without the fix. +4. `bun run typecheck` clean on each republished branch; `privacy:scan` clean on + #1169's. +5. #1244's CI outcome at `15545b3d1` is recorded with a disposition. +6. No contributor readiness checkbox is ticked by me anywhere. + +## Faults to avoid, restated because I have committed each one + +- Merging without a real green (#1202, WP1). +- Selecting by branch name instead of head SHA; the ledger caught this once. +- Ticking a contributor's readiness box. Done once, reverted. +- Claiming a root cause from plausible commit messages without + `git merge-base --is-ancestor` (#1178). +- Trusting `gh pr checks` to surface `action_required`. It does not. +- **Reading a run's `status` and stopping there.** I recorded #1244 as + `in_progress` and built a "watch" around it. By the time the plan was + audited the run had concluded `failure`, so the plan shipped a wrong + disposition for the single largest PR in it. Re-read `conclusion` at the + moment of the decision, not at the moment of the survey. + +--- + +# Audit fold — six blockers, all accepted + +A `gpt-5.6-terra` reviewer audited the plan above against live GitHub state and +returned `VERDICT: fail` with B1–B6. Every one is accepted without rebuttal. +The corrections below supersede the corresponding sections. + +## B1 — #1244 is failing CI, and the failure is in the #1212 seam + +This is the blocker that matters. The survey above recorded two runs as +`in_progress`; the reviewer read the conclusion: + +``` +$ gh api "repos/lidge-jun/opencodex/actions/runs?head_sha=15545b3d1..." \ + -q '.workflow_runs[]|[.id,.name,.status,.conclusion]|@tsv' +31256061011 Issue quality tests completed success +31256061013 Service lifecycle completed success +31256062356 Enforce PR target branch completed success +31256062366 PR Labeler completed success +31256063542 React Doctor completed success +31256063557 Cross-platform CI completed failure +``` + +The failing shard throws `TypeError: suppressedBareNativeSlugs.has` at +`src/codex/catalog/sync.ts:427`, driven by `tests/codex-v2-gate.test.ts:1211`. +The new required input is destructured at `sync.ts:385` **without a default**, +so any caller on `dev` that does not pass it gets `undefined` and dies on +`.has`. + +That is precisely the semantic conflict WP14 suspected and this plan dismissed. +The author's textual rebase merged cleanly *and* broke the contract, which is +the exact failure mode the plan claimed to guard against for the three +republishes while waiving it for #1244. A clean `merge-base --is-ancestor` said +nothing about whether every observed-state caller was updated. + +### WP15-E′ — #1244: report the failure, do not merge + +1. Merge is prohibited at `15545b3d1` and at any later head until a + Cross-platform CI run at that exact head concludes `success`. +2. Comment on #1244 with the run id, the file:line, the failing test, and the + missing-default diagnosis. State the fix shape (either default the + destructured input or update every caller) without asserting which one the + author should pick — the callers are their design. +3. Re-audit the #1212-adjacent observed-state callers + (`buildCatalogEntriesFromObservedState`, + `mergeCatalogEntriesFromObservedState`, `shouldUpgradeToUpstreamEntry`) + before any future merge decision, because a single missing default proves the + caller sweep was incomplete. +4. Disposition: **awaiting author**, with a concrete defect. Not a watch. + +## B2 — approval is not merge-readiness + +`.github/workflows/enforce-pr-target.yml:820` sets `mustDraft` while a +contributor checklist is incomplete, and `:1027`–`:1042` preserve draft status +until every box is ticked. So approving CI does **not** make #1187 or #1184 +mergeable, and WP15-A's heading claimed an outcome the gate forbids. + +### WP15-A′ — approve CI only; await contributor readiness + +Steps 1–3 of WP15-A stand unchanged (SHA-matched ledger, approve `MATCH` rows, +await `conclusion`). What changes is the claim: + +- Acceptance is narrowed to: a `success` Cross-platform CI exists at the exact + current head, both SHAs logged and matched. +- Terminal disposition is **awaiting author**, not merged. The four boxes are + the contributor's attestation and stay theirs. +- The PR comment must say what approval did and did not do, so the author is not + left thinking the maintainer unblocked a merge. + +## B3 — focused tests are below the mandatory gate + +`003_republish_protocol.md:286` requires the full suite when a change touches +routing, adapters, config, or the server; `AGENTS.md:228` requires it +independently. All three republishes touch routing or the CLI, so the +focused-test acceptance at `:110`, `:138`, and `:166` was under-specified. + +**Correction:** `bun run test` (full suite) is required on each republished +branch before its PR is opened, and again after any base movement that forces a +re-apply. The focused test and the ablation stay — they are additional +evidence, not a substitute. + +## B4 — the disposable-worktree boundary was implied, not enforced + +The plan says "fresh worktree" at `:65`, but the protocol it delegates to runs +`git switch -c`, `cherry-pick`, and `commit` in whatever checkout executes it +(`003_republish_protocol.md:39`–`:48`). Run literally in this checkout, that +would touch the user's dirty files. + +**Correction, binding for every remaining work-phase:** + +- Every republish runs in `git worktree add --detach "$(mktemp -d)/"`. +- `git switch`, `git commit`, and any write inside + `/Users/jun/.codex/worktrees/1a75/opencodex` are prohibited except for + `devlog/` documents. +- `scripts/generate-jawcode-metadata.ts`, + `src/generated/jawcode-model-metadata.ts`, + `tests/jawcode-metadata-sync.test.ts`, and `scripts/jawcode-models.json` are + the user's uncommitted work and are never staged, stashed, or reverted. +- Test and ledger artifacts live inside the disposable worktree; it is removed + when the phase closes. + +## B5 — #1195's parity claim is not covered by its own test + +The PR only changes execution-path assertions in `tests/quota-scoring.test.ts` +(runtime assertion at `:198`–`:209`). The management dry-run test at +`tests/routing-profile.test.ts:451`–`:477` covers a candidate with an +explicitly supplied `codexAccountId` — never an *unbound* candidate while a +pool account is active. That is the whole defect, and no test would catch its +return. + +**Correction:** the republished #1195 must add paired regressions — runtime and +management dry-run — asserting that an unbound Codex candidate and an unbound +Anthropic candidate both keep `quota.known === false` while a global active +account exists, and that explicit account-qualified evidence stays known. The +ablation restores one of the two deleted blocks and must fail the new pair. +Without that pair the PR body may not claim parity. + +## B6 — the inventory is stale; two bugs opened after the sweep + +The campaign's own objective is a terminal disposition for *every* open bug +issue and PR. Two were opened after the inventory and appear nowhere in it: + +| # | State | Labels | Note | +|---|-------|--------|------| +| #1273 | OPEN | bug | ghost custom models survive provider removal; full-config PUT resurrects deleted `customModels` | +| #1278 | OPEN | bug, platform, install | Windows: transient PowerShell console window on identity lookup; distinct from #1236 | +| #1279 | OPEN | — | non-draft fix PR for #1278 (`43b6b824c`, wade19990814-hue) | +| #1283 | OPEN | bug, gui | grok's limit is weekly not monthly (opened 12:06Z, *after* the re-audit's own B6 list) | + +**Correction:** a live inventory resweep runs immediately before execution, and +#1273, #1278, #1279, and #1283 are added to the disposition matrix. WP15 does not +close them silently; if they cannot be dispositioned inside this phase they +become the next work-phase with that stated explicitly. + +### B6, second round — a fixed list cannot satisfy a moving inventory + +The re-audit reopened B6 after I had already folded it. Between the first audit +and the second, #1283 opened. My correction had enumerated three numbers, so it +would have passed its own criterion while dropping a bug that existed before the +phase closed. Enumerating is the wrong shape for this criterion. + +**Correction of the correction:** criterion 6 is no longer a list. It requires a +*recorded live resweep, executed last*, whose output is pasted into the closing +document, covering every then-open `bug`-labeled issue and every associated fix +PR. A number opened after that resweep is out of scope by timestamp, and the +resweep output is what proves the boundary rather than my own recollection. + +### B6, third round — an approximate timestamp and a missing PR side + +The re-audit rejected my resweep too, on two counts, both fair. I had written the +boundary as `2026-08-08T12:0xZ` — an approximation is not a boundary, and it +cannot decide whether a given issue was in scope. And I had run only +`gh issue list`, so the PR half of "every open bug issue *and* its fix PRs" was +unproven. + +**Resweep, exact boundary and deterministic issue → open-PR mapping.** +`RESWEEP_AT=2026-08-08T12:21:41Z`, derived per issue from the cross-referenced +timeline events, open pull requests only: + +| Issue | Open fix PR(s) | Disposition | +|-------|----------------|-------------| +| #1283 | none | → WP16 (new, `bug`+`gui`, opened 12:06:03Z) | +| #1278 | #1279 | → WP16 (non-draft PR already open) | +| #1273 | none | → WP16 (new) | +| #1236 | #1268, #1279 | tracking; #1278 is explicitly distinct from it | +| #1230 | #1269 | stays open — the `handleEnsure` gap at `src/cli/index.ts:441` is unfixed | +| #1229 | none | tracking | +| #1222 | none | tracking | +| #1213 | none | tracking | +| #1196 | #1270 | awaiting contributor (two blockers commented) | +| #1193 | #1205 | rerun needed — the run ended `cancelled`, not `failure` | +| #1190 | #1210 | CI green, awaiting contributor checklist | +| #1162 | none | tracking | +| #1145 | none | tracking | +| #1128 | none | tracking, reporter capture requested | +| #1059 | #1272 | CI green, awaiting contributor checklist | +| #1024 | none | tracking, reporter capture requested | +| #904, #796, #418, #417, #241, #92 | none | long-lived tracking | + +That is 22 open `bug` issues, each with either a disposition already recorded in +this unit or an explicit hand-off. + +One row needs its cross-reference stated precisely rather than assumed. #241's +timeline lists only closed PRs (#298, #999, #1056, #1147, #1150), so the +mechanical mapping correctly reports no open fix PR. #1244 does not link #241. + +The chain has two links of different strength, and my first attempt at this +paragraph flattened both into "exists only in prose", which the audit +corrected: **#241 → #1056 is a real timeline cross-reference; #1056 → #1244 is +inferred solely from #1244's `Supersedes #1056` body text.** Only the second +hop is prose. I had also asserted a direct #1244 → #241 link earlier in this +campaign without reading the timeline; that was wrong, and this is the +corrected form. + +The scope boundary is the timestamp above. Anything opened after +`12:21:41Z` is out of this phase by construction, and that is provable from the +recorded value rather than from my recollection. + +## Revised acceptance criteria for WP15 + +1. #1187 and #1184: `success` Cross-platform CI at the exact current head, both + SHAs logged and matched. Disposition recorded as **awaiting author**, with no + merge claim and no box ticked by me. +2. Three new PRs for #1189, #1195, #1169, each with a `Co-authored-by` trailer + naming the original author, each mentioning them, each filling all three + template sections. +3. Each republish: full `bun run test` green, plus a focused test, plus an + ablation that fails without the fix. #1195 additionally carries the paired + parity regressions from B5. +4. `bun run typecheck` clean on each branch; `privacy:scan` clean on #1169's. +5. #1244: the CI `failure` at `15545b3d1` is reported on the PR with run id, + file:line, and the missing-default diagnosis. Merge prohibited. +6. #1273, #1278, #1279 appear in the disposition matrix with either a terminal + disposition or an explicit hand-off to the next work-phase. + *(Superseded by B6 round two: a recorded live resweep, run last, must be + pasted in, and every then-open bug issue must have a disposition or a named + hand-off. No fixed list.)* +7. All code work happened in `mktemp -d` worktrees; the user's four dirty files + are untouched (`git status --short` proves it). + +--- + +# Execution record + +## The audit moved the base out from under the work + +Round four caught something none of the earlier rounds could: while I was +folding blockers, `origin/dev` moved from `3ad5bb6bd` to `f5147cbc8`. Every +test result on this page — three full suites, two ablations, three typechecks — +was measured against a base that no longer existed. The reviewer's instruction +was to treat all of it as stale and redo it after rebasing, which is correct and +which I did. + +The cost of skipping that step would have been three PRs whose "Verification" +sections cited numbers from a base the reviewer could not reproduce. That is the +same class of fault as merging #1202 without a real green, just better hidden. + +Re-verified on `f5147cbc8`: + +| Branch | Full suite | Focused | Extra | +|--------|-----------|---------|-------| +| `codex/260808-1189-history-stream-ingest` | 9991 pass / 7 skip / 0 fail, 625 files | 20/20 | ablation 19/1 then restored 20/0 | +| `codex/260808-1195-unbound-quota-unknown` | 9992 pass / 7 skip / 0 fail, 625 files | 31/31 | ablation 27/4 at identical scope | +| `codex/260808-1169-shim-routing-warning` | 9994 pass / 7 skip / 0 fail, 626 files | 5/5 | `privacy:scan` passed | + +`bun run typecheck` clean on all three. The prepush hook then ran the full suite +a second time per branch and passed each one, which is why the pushes took +roughly six minutes apiece. + +## Published + +| New PR | Republishes | Author | Head | +|--------|-------------|--------|------| +| #1287 | #1189 | luvs01 | `02ec799fe` | +| #1288 | #1195 | luvs01 | `3fc962f2c` | +| #1289 | #1169 | TyroneXie | `eac814346` | + +All three opened non-draft against `dev`, `MERGEABLE`, and `Enforce PR target +branch` green on each. Trailer evidence: + +``` +$ git log --format='%h %s%n %(trailers:key=Co-authored-by,valueonly)' origin/dev..HEAD +02ec799fe fix(history): stream request-history index ingestion (#1189) + luvs01 <27862058+luvs01@users.noreply.github.com> +3fc962f2c test(routing): prove the management dry-run leaves unbound candidates unknown + +0c745be36 fix(routing): keep unbound account quota unknown (#1195) + luvs01 <27862058+luvs01@users.noreply.github.com> +eac814346 fix(codex): warn when codex-shim install cannot prove routing (#1169) + TyroneXie <328347833@qq.com> +``` + +The blank trailer line on `3fc962f2c` is deliberate and is the point of B5's +attribution requirement: those two dry-run tests are mine, not luvs01's, so they +are a separate commit with no co-author trailer and an explicit paragraph in +#1288's body saying so. Folding them into the contributor's commit would have +attributed my code to them; leaving them out would have shipped an unproven +parity claim. + +## Actions taken on existing PRs + +- **#1187, #1184** — approved the `action_required` Cross-platform CI at + SHA-matched heads (`36cffcef6`, `a2eda3b94`; both `MATCH` in + `.tmp/ocx_approval_ledger.tsv`). Commented on both that approval unblocks CI + and nothing else, and that the four boxes stay theirs. Disposition: **awaiting + author**. +- **#1244** — commented with run `31256063557`, the `TypeError` at + `src/codex/catalog/sync.ts:427`, the triggering test at + `tests/codex-v2-gate.test.ts:1211`, and the missing default at `sync.ts:385`. + Named both fix shapes without choosing for them, and flagged that the same + caller-sweep gap may exist for the other inputs added in that commit. Merge + held. Disposition: **awaiting author, with a concrete defect**. +- **#1189, #1195, #1169** — commented on each that it was republished, by which + PR, with what verification, and that the author may take it back if they + prefer to drive it themselves. + +## What WP15 did not do + +#1283, #1278/#1279, and #1273 are dispositioned as hand-offs to WP16, not as +closed. Naming them here is the honest form of that; the resweep table above is +what makes the boundary checkable rather than asserted. + +## Merged + +All three landed on `dev`, and the `Co-authored-by` trailer survived each +squash — which is the property that matters, because the squash is where +contributor credit usually gets lost: + +``` +57ea8df47 fix(routing): keep unbound account quota unknown (#1195) (#1288) | luvs01 +5aa197112 fix(codex): warn when codex-shim install cannot prove routing (#1169) (#1289) | TyroneXie +2cb8eddd4 fix(history): stream request-history index ingestion (#1189) (#1287) | luvs01 +``` + +#1189, #1195, and #1169 were closed as superseded, each with a comment naming +the landed SHA and confirming the credit. #1195's closing comment states +separately that the maintainer test commit is mine and their fix commit is +unmodified. + +### #1288 needed two reruns, and the reason is worth recording + +Cross-platform CI at `3fc962f2c` came back `cancelled` twice. A `cancelled` is +not a `failure` — the four-state rule says rerun — but twice in a row is a +signal rather than noise, so I read the job log instead of firing a third +rerun blind. `test 3/4` hung at `tests/cli-restart-health.test.ts` and was +killed by the runner after ~14 minutes. + +The check that made this safe was comparing against `dev` itself: + +``` +31259885820 dev all-shards-ok +31259450263 dev cancelled test 3/4=cancelled +31259447622 dev cancelled test 1..4/4=cancelled +31256617398 dev success all-shards-ok +``` + +The same shard cancels on `dev` with no PR involved, so it is runner flake, not +something #1288 introduced. `rerun-failed-jobs` then returned all four shards +green. Had I not checked `dev`, "rerun until green" would have been +indistinguishable from hiding a real defect — which is exactly the failure mode +the four-state rule exists to prevent. + +## Final resweep — `FINAL_RESWEEP_AT=2026-08-08T13:48:43Z` + +Run last, as criterion 6 requires. Twenty-two open `bug` issues, unchanged in +membership from the 12:21:41Z sweep, so nothing opened during execution. +#1283, #1278/#1279, and #1273 remain the undispositioned three and pass to +WP16. + +## Acceptance, checked + +1. #1187, #1184 — Cross-platform CI `success` at `36cffcef6` and `a2eda3b94`, + both `MATCH` in the ledger. Awaiting author. **Met.** +2. Three new PRs with trailers and mentions, all template sections filled. + **Met.** +3. Full suite + focused test + ablation on each; #1195 carries the B5 parity + pair. **Met.** +4. `typecheck` clean on all three; `privacy:scan` clean on #1289's. **Met.** +5. #1244's `failure` reported with run id, file:line, and the missing-default + diagnosis; merge held. **Met.** +6. Final resweep recorded above with a disposition or hand-off per row. + **Met.** +7. All code work in `mktemp -d` worktrees; `git status --short` still shows + exactly the user's four untouched files. **Met.** diff --git a/devlog/_plan/260808_bug_campaign/024_wp16_ghost_custom_models.md b/devlog/_plan/260808_bug_campaign/024_wp16_ghost_custom_models.md new file mode 100644 index 0000000000..ab90834485 --- /dev/null +++ b/devlog/_plan/260808_bug_campaign/024_wp16_ghost_custom_models.md @@ -0,0 +1,624 @@ +# WP16 — ghost custom models (#1273), and closing out the campaign's tail + +## Scope + +Three items survived WP15 as hand-offs. One is already done, one is a +disposition, and one is a real two-part defect that needs a patch. + +| Item | State entering WP16 | Outcome | +|------|--------------------|---------| +| #1283 grok weekly limit | OPEN, but fixed on `dev` by #1290 | closed with the landed SHA | +| #1278 / #1279 Windows console flash | PR open, CI unapproved | CI approved at SHA-matched head | +| #1273 ghost custom models | OPEN, no fix PR | **the work of this phase** | + +### #1283 — already fixed, closed manually + +`5222f354a fix(quota): prefer Grok weekly credits for xAI dashboard (#1290)` +landed on `dev` at 13:35Z and does exactly what the report asked: prefer +`GET /v1/billing?format=credits` and map SuperGrok's weekly window to +`weeklyPercent`/`weeklyResetAt`, with the legacy 30-day endpoint demoted to a +fallback (`src/providers/quota.ts:49`, `:592`). + +The PR body said `Closes #1283`, but PRs here target `dev` and GitHub only +auto-closes on merges into the default branch, so the issue sat open with its +fix already shipped. Closed by hand with the SHA. This is a recurring trap in +this repository and it is why `AGENTS.md` tells contributors to close linked +issues manually. + +One loose thread worth recording: #1290's own description noted a pre-existing +`tests/translator-budget.test.ts` failure on the tip and pushed with +`--no-verify`. Checked on a clean worktree at `dev`: **13 pass / 0 fail**. The +failure was local to that environment, not on `dev`, so nothing to chase. + +### #1278 / #1279 — approved, awaiting author + +#1279 (`fix(windows): eliminate console windows from proxy-internal identity & +process lookups`, wade19990814-hue, head `43b6b824c`) is non-draft and +`MERGEABLE`, touching seven Windows source files and four test files. Its +Cross-platform CI sat at `action_required`, invisible to `gh pr checks`. + +Approved at a SHA-matched head (`43b6b824c` == run head, `MATCH` in +`.tmp/ocx_approval_ledger.tsv`). Disposition is **awaiting CI, then review** — +the change is Windows-specific and this campaign has no Windows host, so CI is +the only evidence available and the review will have to lean on it. + +## #1273 — the actual defect + +The report describes two defects. Both reproduce in the source; neither is a +false positive. + +### Defect 1 — provider removal orphans `config.customModels` + +Both removal paths delete only the provider record: + +- `src/cli/provider.ts:304` — `delete config.providers[name]; validateAndSave(config);` +- `src/server/management/provider-routes.ts:617` — `delete config.providers[name];` + followed by `setProviderContextCap`, `save`, `reconcileLiveStateStores`, + `clearModelCache(name)`, `convergeCodexCatalog()`. + +Note what the management path *does* clean up: context caps, the model cache, +and the catalog. It walks right past `config.customModels`. So a custom model +for the removed provider stays in the config, keeps appearing in `/api/models`, +and keeps being emitted into the Codex catalog — a row pointing at a provider +that no longer exists. + +### Defect 2 — a stale in-memory config wins a whole-document write + +`saveConfigPreservingClaudeCode` (`src/config.ts:2710`) takes one authoritative +pre-write read of the on-disk config and uses it for exactly two reconciliations: + +- `claudeCode` (`:2716`–`:2726`): if disk changed and we did not, adopt disk. +- the live server binding (`:2731`–`:2739`): port/hostname come from disk. + +`customModels` gets neither. `projectCustomModelCatalogMigration` +(`src/codex/custom-model-catalog-migration.ts`) *does* consult the persisted +config, but only to project the `customModelCatalogMigration` ownership marker — +it reads `customModels` to classify legacy slugs and never writes the array +back. The candidate's array passes through untouched into +`persistConfigUnlocked`. + +So the reporter's step 4 is exactly right: a `PUT /api/shadow-call-settings` +from a process whose config predates a CLI deletion re-persists the whole stale +document, and the deleted rows return. Their diff even shows the cooperating +save path working correctly (generation bumped, catalog mtime matching) — the +write was well-formed, it just wrote the wrong document. + +**The asymmetry is the bug.** Two fields already get last-writer-wins protection +because they are known to be mutated by other processes. `customModels` is +mutated by `ocx models remove` from a different process and got no such +treatment. + +## Diff-level plan + +### WP16-A — `src/config.ts`: reconcile `customModels` against disk + +Add a third reconciliation next to the `claudeCode` block, using the same +already-taken `onDisk` read (`:2715`) — no second read, since the comment there +correctly warns that a second read could observe different bytes. + +Shape, mirroring the `claudeCode` baseline logic: + +- Track a `customModelsBaseline` per config object, set when the config is + loaded, exactly as `claudeCodeBaseline` is. +- On write: if the on-disk array differs from the baseline **and** the + in-memory array equals the baseline, adopt the on-disk array. That is + "someone else changed it and we did not", which is precisely the reporter's + scenario. +- If both changed, the in-memory value wins and we do not silently merge. A + merge would invent an intent neither writer expressed; last-writer-wins on a + genuine concurrent edit is the same rule `claudeCode` already uses. + +Acceptance: a test that loads a config, deletes a custom model on disk out of +band, then performs an unrelated `PUT`-shaped save from the stale object, and +asserts the deleted row does not return. + +### WP16-B — both removal paths drop the provider's custom models + +`src/cli/provider.ts:304` and `src/server/management/provider-routes.ts:617` +each filter `config.customModels` by `provider !== name` before saving. The +management path already clears the model cache and reconverges the catalog, so +the ghost disappears from `/api/models` and the catalog in the same write. + +**Open question, answered before writing the patch.** Does anything rely on a +`customModels` row outliving its provider — re-adding the provider and +expecting its models back, for instance? If so, deleting would be wrong and the +fix would belong at read time. + +The answer is in provider *rename*. `rewriteProviderReferences` +(`src/providers/provider-id-rewrite.ts:34`) explicitly rewrites +`customModels[].provider` alongside combo targets and Claude tier maps +(`:97`–`:101`). So this array is already designed to track the provider +lifecycle; rename follows it and remove simply does not. No test anywhere +expects rows to survive removal and be restored on re-add. + +That makes deletion the consistent fix rather than a judgement call, and it +reframes defect 1: not a missing feature, but a lifecycle hook that one of two +sibling operations forgot. Consumers confirm the same shape — +`src/codex/catalog/provider-fetch.ts` emits every row into the catalog keyed by +`routedSlug`, and `src/server/management/model-rows.ts` lists every row in the +dashboard, neither checking that the provider still exists. + +### WP16-C — verification + +- Focused: new tests in the config and provider-removal suites. +- Ablation: revert each half independently; the matching test must fail and the + other must not. Two defects, two independent proofs. +- Full `bun run test`, `bun run typecheck`, `bun run privacy:scan`. +- Republish protocol: `mktemp -d` worktree, no writes in the dirty checkout + outside `devlog/`. + +This is a maintainer-authored fix with no contributor PR to preserve, so there +is no `Co-authored-by` trailer. The PR body credits the reporter for a +reproduction that included a before/after config diff and the evidence that the +cooperating save path itself was healthy — that is what made the second defect +findable rather than a vague "settings sometimes revert". + +## Acceptance criteria + +1. #1283 closed with the landed SHA and an explanation of the manual close. +2. #1279's CI approved at a SHA-matched head, logged, disposition recorded. +3. Both #1273 defects fixed, each with its own regression test and its own + ablation. +4. Full suite, typecheck, and privacy scan green on the rebased branch. +5. PR opened against `dev` with all three template sections and the reporter + credited. +6. The consumer question in WP16-B answered with file:line evidence before the + patch is written. + +--- + +# Audit fold — five blockers, all accepted + +A `gpt-5.6-terra` reviewer returned `VERDICT: fail` on the plan above with +B1–B5. Every one is accepted. B3 is the important one: it finds a case where my +proposed rule loses user data, which is worse than the bug it was written to +fix. + +## B3 — "memory wins if both changed" resurrects a deleted provider's model + +My rule was whole-array last-writer-wins: adopt disk when disk changed and +memory did not, otherwise keep memory. The reviewer supplied the case that +breaks it. + +Disk deletes provider **P** (and, after WP16-B, P's custom models). Meanwhile +the in-memory process independently edits an unrelated custom model **Q** — a +legitimate edit through `/api/models`. Now *both* arrays differ from the +baseline, so my rule keeps memory wholesale, and P's custom model comes back. +The exact ghost row #1273 is about, reintroduced by the fix for #1273. + +A whole-array comparison cannot distinguish "I edited Q" from "I am asserting +the entire array including P". The array is a keyed collection and has to be +reconciled as one: + +- Reconcile per row, keyed by `routedSlug(provider, modelId)`, three-way + against the baseline: a row deleted on disk and untouched in memory stays + deleted; a row edited in memory is kept; a row added on either side is kept. +- Then prune any surviving row whose provider is absent from the config being + written. That prune is the lifecycle invariant from WP16-B applied at the + write boundary, so it holds no matter which path produced the array. + +Required regression: disk deletes P while memory edits Q, then an unrelated +save. Q keeps its edit, P's row does not return. + +## B1 — the same staleness applies to `providers`, not just `customModels` + +I scoped the fix to `customModels` because that is what the issue reported. But +`saveConfigPreservingClaudeCode` writes the *whole document*, so the identical +stale-write path resurrects `config.providers[P]` itself. Fixing only the models +leaves a deleted provider coming back, which then makes its models legitimate +again — the ghost returns by a different door. + +**Correction:** the acceptance test asserts both. After an external provider +deletion and an unrelated long-lived-server save, `config.providers[P]` **and** +P's custom models must both stay absent. If whole-array reconciliation cannot +deliver that, the answer is provider-aware reconciliation or a field-scoped +persistence path, not a narrower test. + +## B2 — I described the baseline mechanism wrongly + +I wrote that the baseline is "set when the config is loaded, exactly as +`claudeCodeBaseline` is". That is not what `claudeCodeBaseline` does. It is a +`WeakMap` armed **explicitly by `startServer`** (`src/config.ts:2497`, +`src/server/index.ts:485`), and the comment there says arming is eager on +purpose because lazy arming would lose the hand edit the guard exists to +protect. + +The consequence matters: a CLI process that loads a config and saves it is +**not armed**, so a guard modelled on this would be silently inert there. That +is defensible for `claudeCode`, whose contested writer is the long-lived +server, but it must be a stated policy rather than an accident. + +**Correction:** the plan must name every writer path, state whether it is +armed, and define the unarmed behaviour explicitly. My current position: an +unarmed config has no baseline, so it cannot claim "I did not change this" — +the safe default there is the provider-absence prune from B3, which needs no +baseline at all. That is why the prune is not optional. + +## B4 — deletion is consistent, but the marker needs its own test + +The reviewer confirmed WP16-B's premise: dependent combos are already rejected +before provider deletion, rename already rewrites `customModels[].provider`, +and `src/server/management/model-rows.ts:55` renders custom rows without +checking that the provider exists. So deletion matches existing lifecycle +behaviour. + +**Correction:** add a regression that deletion removes the visible row *without +corrupting* `legacyOwnedSlugs` in the `customModelCatalogMigration` marker. That +marker grants one-time ownership of pre-marker rows; if deletion silently +rewrites it, an older binary's view of ownership changes, and the migration +file's own comment warns against exactly that. + +## B5 — do not imply #1290's CI was green + +I recorded #1290 as landed without qualifying its CI. Checked: the +Cross-platform run at its head `0fe140f91` concluded **`cancelled`**, not +`success`. + +What actually validates the fix is `dev` afterwards. `dev` at `5222f354a` also +shows `cancelled`, and the first `success` on `dev` after it is `57ea8df47` — +the #1288 merge, which contains #1290's change as an ancestor. So the Grok +weekly fix *is* covered by a green `dev` run, but by inheritance, one merge +later, and not at its own head. + +That distinction is worth stating rather than smoothing over: "it's on `dev` +and `dev` is green" is a weaker claim than "its own CI passed", and the run +ids are what let a reader tell which one they are being given. The same +`test 3/4` flake recorded in WP15 is the likely cause of both cancellations. + +## Revised acceptance criteria + +1. #1283 closed with the landed SHA, the manual-close reason, **and** the + honest CI provenance from B5. +2. #1279's CI approved at a SHA-matched head, logged, disposition recorded. +3. Both #1273 defects fixed. Defect 2's fix reconciles per row and prunes rows + whose provider is absent; it does **not** rely on whole-array comparison. +4. Regressions, each with its own ablation: + a. provider removal drops that provider's custom models; + b. stale save does not resurrect deleted rows; + c. **disk deletes P while memory edits Q** — Q survives, P does not; + d. stale save does not resurrect the deleted **provider** either; + e. deletion leaves `legacyOwnedSlugs` intact. +5. Every writer path named with its arming status, and the unarmed policy + stated. +6. Full suite, typecheck, privacy scan green on the rebased branch. +7. PR against `dev`, all three template sections, reporter credited. + +## Writer-path survey, as B2 requires + +`saveConfigPreservingClaudeCode` / `validateAndSave` are called from about +twenty files: + +``` +11 src/server/management/agent-settings-routes.ts + 7 src/server/management/provider-routes.ts + 5 src/server/management/oauth-account-routes.ts + 5 src/providers/api-keys.ts + 5 src/cli/claude-desktop.ts + 4 src/server/management/routing-profile-routes.ts + 4 src/server/management/config-routes.ts + 4 src/cli/provider.ts + 3 src/server/management/combo-routes.ts + 3 src/codex/routing.ts + ... 10 more +``` + +They split into two populations: + +- **Management routes** (`src/server/management/*`) run inside the long-lived + server, which armed the baseline at `startServer`. A per-config baseline + guard works there. +- **CLI paths** (`src/cli/provider.ts`, `src/cli/claude-desktop.ts`, + `src/cli/models.ts`) are short-lived processes that load, mutate, and exit. + They are **never armed**, and arming them would be meaningless: they read + disk moments before writing, so their "baseline" is the disk. + +This settles the B2 policy and reinforces B3. A baseline-only guard is inert +across roughly half the call sites, so correctness cannot rest on it. The +provider-absence prune needs no baseline and therefore holds on every path, +which is why it is the load-bearing half of the fix and the row-keyed +reconciliation is the refinement layered on top where a baseline exists. + +A guard that silently does nothing on half its call sites is the kind of fix +that reads well in a diff and fails in the field — which is the shape of the +original defect, where two sibling operations disagreed about the same array. + +--- + +# Second audit fold — the design was still wrong in four places + +A second review round returned `VERDICT: fail` again. B5 is closed; B1–B4 are +not, and two of them invalidate the replacement design rather than refining it. + +## B2 — `routedSlug` is an encoding, not an identity + +This is the worst error in the plan so far, because it was introduced *by* the +fix for the previous worst error. I keyed row reconciliation on +`routedSlug(provider, modelId)`. Both components are mutable: + +- `PUT /api/custom-models/:id` accepts a new `modelId` + (`src/server/management/model-routes.ts:356` — it looks the row up by + `cm.id === id` and then reassigns `cm.modelId`). +- provider rename rewrites `model.provider` + (`src/providers/provider-id-rewrite.ts:97`). + +So a renamed row looks like a *deleted row plus a new row* under my key. Three- +way reconciliation would then either drop the rename or keep both copies. The +reviewer's phrase is the right test: one row must survive, not two conflicting +copies and not zero. + +The type already carries the right key. `OcxCustomModel.id` is a +`crypto.randomUUID()` assigned at creation (`src/types.ts:531`–`:533`) and never +rewritten by rename or by the PUT — which is exactly why the PUT route looks +rows up by it. + +**Correction:** reconcile keyed by `OcxCustomModel.id`, with an explicit +field-level conflict policy: a row present on both sides takes the in-memory +field values where memory differs from baseline, and disk values otherwise. A +row absent on disk and unchanged in memory is a remote delete and stays deleted. +Regressions must cover a disk-side provider rename and a disk-side `modelId` +change, each concurrent with an in-memory metadata edit of the same row. + +I should have found this myself: I *read* the PUT route while confirming defect +1 and still reached for the slug, because the slug is what the catalog uses. +Catalog-facing identity and storage identity are different things. + +## B1 — the provider record itself is still unreconciled + +I widened the acceptance criterion to require that a stale save resurrect +neither the provider nor its rows, then wrote a design that only reconciles +`customModels`. Pruning models cannot make `providers[P]` absent, and +`saveConfigPreservingClaudeCode` still serializes the whole candidate object. +The criterion and the design contradict each other, and the criterion is right. + +**Correction:** the design must name how `providers` is reconciled. Two options, +to be decided with evidence rather than taste: + +1. Extend keyed reconciliation to `providers` — remote deletes win when the + in-memory record is unchanged from its baseline. +2. A field-scoped persistence operation: callers declare which top-level fields + they are changing and only those are written, leaving everything else at the + on-disk value. + +Option 2 fixes the entire class rather than two fields, but it changes every +call site and is a much larger blast radius; option 1 keeps the change local at +the cost of leaving the next field to be discovered the same way `customModels` +was. The decision needs the call-site matrix below to be made honestly, and it +is explicitly *not* made in this revision. + +## B3 — the writer survey was approximate, so its conclusion was unearned + +I wrote "…10 more" and then drew a two-population conclusion from a list I had +truncated. The reviewer named a counterexample I had elided: +`src/storage/policy.ts:265` loads an unarmed config and calls the wrapper, and +it is neither a management route nor a CLI command. + +Exhaustive matrix, all 13 files that call `saveConfigPreservingClaudeCode` or +`validateAndSave`, with how each obtains its config: + +| File | calls | config provenance | +|------|-------|-------------------| +| `src/server/management/agent-settings-routes.ts` | 8 | `loadConfig()` | +| `src/providers/api-keys.ts` | 4 | passed-in | +| `src/cli/provider.ts` | 4 | `loadConfig()` | +| `src/cli/claude-desktop.ts` | 4 | `loadConfig()` | +| `src/server/management/oauth-account-routes.ts` | 4 | passed-in | +| `src/server/management/config-routes.ts` | 3 | `loadConfig()` | +| `src/server/management/combo-routes.ts` | 2 | passed-in | +| `src/codex/routing.ts` | 2 | passed-in | +| `src/server/management/provider-routes.ts` | 1 | passed-in | +| `src/storage/policy.ts` | 1 | `loadConfig()` | +| `src/codex/auth-api.ts` | 1 | `loadConfig()` | +| `src/providers/key-failover.ts` | 1 | passed-in | +| `src/config.ts` | 1 | internal | + +The honest conclusion is not "two populations". It is that **provenance is +mixed within every layer**: management routes both load fresh and mutate a +long-lived object, and non-route modules (`storage/policy`, `codex/routing`, +`providers/key-failover`, `providers/api-keys`) write config too. A guard keyed +to `startServer` arming covers some of these and not others, and which is which +is not predictable from the directory. + +## B4 — the prune needs a stated precondition, not universal application + +I claimed the provider-absence prune is safe everywhere because it needs no +baseline. The reviewer's objection stands: `saveConfigPreservingClaudeCode` +performs no runtime full-config validation, and `auth-api.ts:377` treats any +object with a truthy `providers` as a runtime config. Nothing structurally +prevents a caller from saving a filtered or partially built config, and a prune +would silently delete that user's rows. + +The reviewer looked and found no production writer that deliberately saves a +partial config — but "I could not find one" is not an invariant, and my plan +asserted safety without establishing one. + +**Correction:** the prune applies only to a config proven to carry an +authoritative provider map. Either enforce that precondition at the write +boundary explicitly, or scope the prune to the reconciled snapshot the write +path itself builds from the on-disk read. A test must show that a filtered or +partial caller cannot silently delete retained rows. + +## Where this leaves WP16 + +Two rounds of review have found, in order: a rule that resurrects deleted rows, +a key that duplicates renamed rows, a criterion contradicting its own design, a +truncated survey used to justify a conclusion, and an unproven safety claim. +That is a defect whose correct fix is a genuine concurrency design, not a patch +I can land credibly inside this session's remaining scope. + +**Disposition: #1273 stays open with a documented diagnosis rather than a rushed +fix.** Both defects are confirmed at file:line and that is real value for +whoever picks it up. Shipping my third design attempt — after two were shown to +lose user data — into a config-persistence path would be the least defensible +thing in this entire campaign. + +The diagnosis goes on the issue: both defect sites, the asymmetry with +`claudeCode` and the server binding, the rename-vs-remove inconsistency, the +identity requirement (`OcxCustomModel.id`, not the slug), the writer matrix, and +the two candidate designs with their tradeoffs. + +--- + +# Third audit fold — the matrix was still wrong, and the hold was too wide + +## The "exhaustive" matrix was not exhaustive + +I built it by grepping the wrapper name, which misses every aliased binding. +The management routes bind it through a test-injection seam: + +``` +model-routes.ts:129 const persistConfig = deps.saveConfigPreservingClaudeCode ?? saveConfigPreservingClaudeCode; +routing-profile-routes.ts:316 const save = deps.saveConfigPreservingClaudeCode ?? saveConfigPreservingClaudeCode; +native-integration-routes.ts:731 const persist = deps.saveConfigPreservingClaudeCode ?? saveConfigPreservingClaudeCode; +``` + +So `model-routes.ts` — the file that owns custom models — showed up as **zero +writes** in a matrix I published as exhaustive, in a document arguing that the +previous version's approximation was the problem. Corrected inventory, direct +plus aliased: + +| File | direct | aliased | +|------|--------|---------| +| `src/server/management/agent-settings-routes.ts` | 8 | 2 | +| `src/server/management/provider-routes.ts` | 1 | 7 | +| `src/server/management/model-routes.ts` | 0 | 6 | +| `src/cli/claude-desktop.ts` | 4 | 0 | +| `src/cli/provider.ts` | 4 | 0 | +| `src/providers/api-keys.ts` | 4 | 0 | +| `src/server/management/oauth-account-routes.ts` | 4 | 0 | +| `src/server/management/config-routes.ts` | 3 | 0 | +| `src/server/management/combo-routes.ts` | 2 | 0 | +| `src/codex/routing.ts` | 2 | 0 | +| `src/server/management/routing-profile-routes.ts` | 0 | 2 | +| `src/server/management/native-integration-routes.ts` | 0 | 1 | +| `src/storage/policy.ts` | 1 | 0 | +| `src/codex/auth-api.ts` | 1 | 0 | +| `src/providers/key-failover.ts` | 1 | 0 | +| `src/config.ts` | 1 | internal | + +**16 writer files; 20 wrapper references.** The distinction matters and I +blurred it: `src/server/management-api.ts`, `management/context.ts`, +`management/logs-usage-routes.ts`, and `management/shared.ts` import or +type-reference the wrapper without ever invoking it (`context.ts:21` declares it +as an optional dependency for the test seam). Counting those as writers +overstates the surface, which is the same species of error as the undercount it +replaced — I fixed a number by changing it to a different wrong number. + +Also corrected: I labelled `auth-api.ts` as `loadConfig()`-only, but +`getRuntimeConfig` at `:381` prefers a passed-in runtime config when one is +supplied. + +The conclusion survives and strengthens — provenance is mixed and arming-keyed +guards are inert on much of this surface — but I published a false count while +criticising an approximate one. Both public comments on #1273 were corrected. + +## Splitting the disposition: defect 1 ships, defect 2 holds + +The reviewer rejected the all-or-nothing hold, correctly. My reasoning was "a +stale save can reintroduce the rows anyway, so cleaning them up is theatre". +That conflates two things: defect 2 is a *concurrency* defect that needs a +reviewed persistence design, while defect 1 is a *lifecycle* gap in two direct +code paths that already do provider cleanup and already reject dependent combos +first (`provider-routes.ts:604`). One being unfinished does not make the other +unsafe or useless. + +**Revised disposition:** + +- **Defect 1 — fix now.** Filter `config.customModels` by provider in both + removal paths, with a regression proving `legacyOwnedSlugs` in the + `customModelCatalogMigration` marker is not corrupted. +- **Defect 2 — stays open on #1273** with the diagnosis, pending a persistence + design reviewed on its own terms. + +This is the third time in this work-phase that the review changed my answer +rather than polishing it, which is the argument for running the gate at all. + +--- + +# Fourth audit fold — the CLI half had no test + +Four more blockers, all accepted. + +**The one that mattered: I fixed the CLI path and never tested it.** The +ablation I presented reverted *both* files at once and showed one API test +failing, which proves the management wiring and says nothing about +`src/cli/provider.ts`. A reviewer reading "ablation passes" would reasonably +assume both halves were covered. They were not. + +Added `tests/cli-provider.test.ts` → "provider remove drops that provider's +custom models (#1273)", which spawns the real CLI, asserts the persisted +`config.json`, and asserts the new `--json droppedCustomModels` field. Ablating +`src/cli/provider.ts` **alone** now gives 29 pass / 1 fail, and that failure is +the CLI test. Two paths, two independent proofs — which is what the previous +work-phase already established as the standard and what I failed to apply here. + +**Marker persistence was asserted on the helper, not the write path.** The +marker test only proved `dropProviderCustomModels` does not mutate the in-memory +object. It never exercised `projectCustomModelCatalogMigration`, which runs +inside the save. Both integration tests now seed +`customModelCatalogMigration` and assert its value in the persisted +`config.json` after the delete, so the claim covers the real path. + +**A false comment in the shipped code.** I wrote that dropping the emptied key +leaves a config "byte-identical to one whose last custom model was removed" — +untrue, because the migration marker deliberately survives. Narrowed to the +claim actually being made: the `customModels` field is absent either way. + +Final verification on `57ea8df47`: + +- `bun run test` — **10008 pass / 7 skip / 0 fail**, 626 files +- `bun test` on the three touched suites — 62 pass, then 30 pass for the CLI suite +- ablation, management path only — 61 pass / 1 fail (the API test) +- ablation, CLI path only — 29 pass / 1 fail (the CLI test) +- `bun run typecheck` clean, `bun run privacy:scan` passed + +--- + +# Outcome + +| Item | Disposition | +|------|-------------| +| #1283 grok weekly limit | **closed** — fixed on `dev` by #1290 (`5222f354a`), closed manually with a public correction about its CI provenance | +| #1273 defect 1 (orphaned rows) | **fixed** — PR #1293, `472015e5a` | +| #1273 defect 2 (stale whole-document write) | **open with a diagnosis**, deliberately not patched | +| #1278 / #1279 Windows console flash | **awaiting CI** — approved at a SHA-matched head, three shard hangs so far, diagnosed publicly | + +## #1279 and the shard that keeps hanging + +Three Cross-platform runs on #1279 ended `cancelled`. Each time shard +`test 2/4` (or `3/4`) runs normally, then stops emitting output entirely until +the runner kills it roughly 14 minutes later. Nothing fails. + +The same pattern hit `dev` with no PR involved (`31259450263`, +`31259447622`) and cost #1288 two reruns before `rerun-failed-jobs` went green, +so the base branch has an unstable shard today. That is the likely answer. + +I did not simply write "flake" on the PR and move on, because #1279 changes +process and identity lookups and a hang right after a proxy server starts is the +shape a blocking child-process call would take. What argues against it: the +hanging shard runs on Linux, where the `windows-*` modules should never be +reached. I have no Windows host, so I told the author exactly that — what the +evidence shows, what I cannot rule out, and the one concrete thing worth +checking (timeouts on lookups reachable during startup) — rather than either +dismissing it or implying their patch is at fault. + +## The pattern across WP16 + +Four review rounds, and each one changed the answer rather than polishing it: + +1. Whole-array reconciliation → resurrects a deleted provider's rows. +2. `routedSlug` as a row key → duplicates renamed rows. I had *read* the PUT + route that mutates `modelId` and still reached for the slug, because the slug + is what the catalog uses. Catalog identity and storage identity are different + things. +3. "Exhaustive" writer matrix → missed every aliased binding, including the file + that owns custom models. +4. All-or-nothing hold → wrong; defect 1 was bounded and shippable, and holding + it gained nothing. + +Plus two false public claims I had to retract on the issue: a wrong writer count +(13), then a differently wrong one (20 references, 16 writers). + +The useful lesson is narrower than "review is good". Every one of these was a +claim I could have checked and did not, because the claim felt like background +detail rather than the thing being decided. The slug key and the writer count +were both stated in passing while my attention was on the reconciliation rule. diff --git a/devlog/_plan/260808_bug_campaign/025_wp3_lane_c_ci_workflows.md b/devlog/_plan/260808_bug_campaign/025_wp3_lane_c_ci_workflows.md new file mode 100644 index 0000000000..dffc170a97 --- /dev/null +++ b/devlog/_plan/260808_bug_campaign/025_wp3_lane_c_ci_workflows.md @@ -0,0 +1,164 @@ +# WP3 — lane C: the CI/workflow stack + +Three PRs, and the interesting result is that the two open ones needed opposite +treatment despite looking similar on the board. + +| PR | State entering WP3 | Outcome | +|----|--------------------|---------| +| #1255 harden comment-driven review workflows | merged (`0993c53ae`) | already done | +| #1185 bind Windows shard assertion | draft, CI **failure**, 324 behind | **republished** as #1301 | +| #1259 fail-closed aggregate-check evidence | draft, CI `cancelled`, 20 behind | **held** with a blocker | + +## #1185 — a red PR that was right + +Its Cross-platform CI at `bff31d1e0` genuinely failed, which is the kind of +signal that gets a stale draft closed. It should not have here. + +The PR touches exactly one file, `tests/ci-workflows.test.ts`, and that file +only reads workflow YAML as text. The crash was somewhere else entirely: + +``` +##[group]tests/autostart-health.test.ts: +# Unhandled error between tests +error: EEXIST: file already exists, epoll_ctl + at new WriteStream (internal:fs/streams:244:58) +# then +error: Cannot call describe() after the test run has completed + at tests/autostart-health.test.ts:23:1 +``` + +2142 pass / 1 fail / 2 errors. A Bun-level failure while loading a file the +diff cannot reach, with the `describe()` error as collateral. + +My first write-up called this "an fd leak from a preceding test file". The +audit removed that: the log shows *where* the crash happened, not *why*, and I +had asserted a mechanism the evidence does not carry. Recorded as an unrelated +Bun/runner load failure, root cause unknown. + +### What the patch actually buys + +The existing assertion used `.includes()` on the Windows step's `run` text, so +the command counted as present anywhere in the script — inside an `echo`, or in +a comment. Measured on current `dev` by mutating `.github/workflows/ci.yml:492` +so the Windows leg prints instead of runs: + +| Mutation | `dev` today | with #1185 | +|----------|-------------|------------| +| `run: bun test …` → `run: echo bun test …` | 125 pass / 0 fail | 124 / **1 fail** | +| add `if: false`, command unchanged | 125 pass / 0 fail | 124 / **1 fail** | + +The second row is mine. The audit pointed out that binding the assertion to an +executable *line* still permits an unreachable *step*: the exact command under +`if: false` runs nothing and satisfies the contributor's check. Both mutations +leave `dev` green today, which is the whole argument for landing this. + +Published as #1301 with the two commits separated — `364b358` carries luvs01's +`Co-authored-by`, `f09ef15` is mine with no trailer and is called out in the PR +body. + +## #1259 — the right idea with a hole in its central claim + +#1259 removes `pull_request.paths` and moves scope gating into the existing +`changes` job, so a docs-only PR gets an explicit passing `ci` check instead of +no check at all. That problem is real: no check is harmless until the check +becomes required, and then it is a PR that waits forever. + +The blocker is in the property the PR is named for. `changes` exposes +`ci: ${{ steps.filter.outputs.ci }}` with no validation, and every expensive job +gates on `needs.changes.outputs.ci == 'true'`. If `changes` **succeeds** while +that output is empty or malformed — an action upgrade renaming an output, a +filter-syntax slip — then: + +1. every expensive job evaluates `'' == 'true'` and is skipped; +2. the aggregate gate treats `skipped` as a pass, deliberately, because that is + how it recognises trigger-scoped jobs; +3. `ci` reports green having tested nothing. + +`changes` *failing* is handled — the aggregate catches it. It is `changes` +succeeding with an unusable output that slips through, and today's `paths:` +trigger makes that unreachable, so the PR turns a non-issue into the single +point of truth without hardening it. + +I suggested a validation step on the PR — and got it wrong on the first pass by +writing `case "${{ steps.filter.outputs.ci }}"`, interpolating the expression +straight into shell. That is the injection shape this repository's workflow +hardening exists to prevent. Low risk from a SHA-pinned action, but wrong, and +corrected publicly to pass the value through `env:` so bash sees data. + +I also claimed #1265 and #1259 would conflict in `enforce-pr-target.yml`. I had +compared branch positions, not hunks. Corrected to "may conflict; decide the +integration order". + +**Disposition: held, not approved.** It changes when CI runs at all, which +`MAINTAINERS.md` puts in the security-review class, and I offered to implement +the validation step rather than making the author respin. + +One thing the audit checked that I had not: whether removing the path filter +widens exposure on the self-hosted Windows runner. It does not — PR Windows +stays `workflow_dispatch`-only. But it does make the aggregate check +security-critical, which is exactly why the output needs validating. + +## Faults recorded + +- Asserted a mechanism (fd leak) the log did not support, when "root cause + unknown" was the honest reading. +- Suggested a workflow snippet with an expression-injection shape while + reviewing a security-class change. +- Claimed a conflict from branch divergence without looking at the hunks. + +--- + +# The "flake" I called five times + +#1301's CI came back `cancelled` with `test 4/4` hung. I issued +`rerun-failed-jobs`, as the four-state rule says, and asked the reviewer whether +I was now pattern-matching to "flake" too readily. The answer was yes, with a +detail I had not checked: attempt 1 was **not** a superseded run. `test 4/4` ran +its Test step for a full 15 minutes and was killed by the job timeout. + +The retry then did the same thing — 15:28:02Z to 15:43:17Z, cancelled at 15 +minutes 15 seconds. Two real timeouts at the same head. + +So I stopped rerunning and investigated instead. The shape is identical every +time: output stops immediately after a test that starts a proxy listener, +silence for ~14 minutes, then `Terminate orphan process: pid (NNNN) (bun)` in +cleanup. In #1301 the last line was + +``` +[web-search-loop] cancelled — 1 real searches, 0 placeholders, 13ms +(pass) routed Claude requests give OpenAI sidecars main auth without leaking it to the routed provider +``` + +from `tests/claude-messages-endpoint.test.ts` — which passes locally in 2.7s +(38/38), and the full suite is 10009 pass. The stall is *after* the assertion, +so teardown or the next file's setup is the suspect, not the test. + +Five occurrences today across four unrelated branches **and `dev` itself**: + +| Run | Branch | Shard | +|-----|--------|-------| +| 31263738953 | `codex/260808-1185-windows-shard-assertion` | `test 4/4`, twice | +| 31255199569 | `fix/windows-powershell-popup` | `test 2/4` | +| 31258815611 | `codex/260808-1195-unbound-quota-unknown` | `test 3/4` | +| 31152916419 | `agent/test-windows-ci-shard-command` | `test 3/4` | +| 31259450263, 31259447622 | `dev` | various | + +The varying shard argues against one bad test. The one instance that did not +hang is the clue: it crashed with `EEXIST: file already exists, epoll_ctl` in a +Bun `WriteStream` — a descriptor registered with the event loop twice, which is +the same resource-lifecycle fault a deadlocking registration would produce. + +Filed as **#1302** with the run inventory, and #1301 is **held** rather than +rerun to green. + +## Why this is the fault worth recording + +Three of the four cancelled runs *did* go green on retry, so "rerun until green" +worked every time and produced merges I still stand behind. The problem is that +it works equally well on a genuine hang introduced by a real change. I applied +the rule correctly — `cancelled` means rerun — and used it to avoid looking at +five instances of the same failure. + +What broke the loop was being asked to justify the call rather than state it. +"It's flake" was a conclusion I never had evidence for; I had evidence that +retrying made it go away, which is a different claim. diff --git a/devlog/_plan/260808_bug_campaign/026_wp5_large_solo.md b/devlog/_plan/260808_bug_campaign/026_wp5_large_solo.md new file mode 100644 index 0000000000..bee7f7b554 --- /dev/null +++ b/devlog/_plan/260808_bug_campaign/026_wp5_large_solo.md @@ -0,0 +1,94 @@ +# WP5 — the two large solo PRs + +| PR | Size | Outcome | +|----|------|---------| +| #1244 preserve routed models in desktop picker | 58 files | **CI green, merge held on four conditions** | +| #1228 native image support for Cursor | 8 adapter files | **held for the author**, conflicting and stale | + +## #1244 — the author's fix was better than the one I proposed + +In WP15 I reported the CI failure at `15545b3d1`: `TypeError: +suppressedBareNativeSlugs.has` at `src/codex/catalog/sync.ts:427`, driven by +`tests/codex-v2-gate.test.ts:1211`, with the input destructured at `:385` +without a default. I named two fix shapes — default the input, or update every +caller — and left the choice to them. + +They did neither. `2c9994a9d` adds the two missing sets to the **one test call +site**, two lines, and leaves `sync.ts` alone. + +That is the correct fix, and my reasoning about why was also wrong. I told the +audit that required-field typing beats a default because `typecheck` enforces +every caller. The audit checked what I had not: + +``` +$ rg '"include"' tsconfig.json +15: "include": ["src"] +``` + +**`bun run typecheck` never covered `tests/` at all.** So my "typecheck is the +caller-sweep proof" claim was empty — the failing site was in the one directory +the compiler does not read, which is exactly why it reached CI. The right +statement is narrower: the fields are genuinely required on +`ObservedCatalogEntryBuildInput`, production callers in `src/` are compiler- +checked, and a hand-built test literal is the residual gap that explicit empty +sets close honestly. + +I had reached for "the author saved me work and was therefore right". The +conclusion survived; the argument for it did not. + +### What I verified before holding + +- Cross-platform CI **success** at `2c9994a9d` (run `31258863895`) +- full `bun run test` on that head — **10003 pass / 7 skip / 0 fail**, 623 files +- `bun run typecheck` clean +- catalog/convergence suites — 258 pass / 0 fail +- no semantic conflict with the merged #1212: `a4878de38` is an **ancestor** of + #1244's merge base, so the convergence work is already underneath it +- of the 20 commits `dev` moved ahead, none touch #1244's source files; the only + overlap is `docs-site/.../configuration/routing.md` + +### Why it is still held + +1. **Stale base and stale claims.** 20 behind, and the PR body still asserts + "0 commits behind" and "remains draft" while the PR is non-draft. The + evidence needs to exist at the head that would actually merge. +2. **A locale defect.** English documents that `-` clears `--effort`, + `--alias`, `--display-name` and that the subcommands exist under + `ocx route combo` (`guides/combos.md:264`). Russian omits both + (`ru/guides/combos.md:224`); `ja`, `ko`, `zh-cn` carry them. Five-locale + alignment was claimed, four are aligned. +3. **CI evidence standard, post-#1302.** One green run is currently weaker + evidence than it looks, so a 57-file catalog change gets two completed + non-cancelled runs at the same rebased SHA. + +4. **Fresh activation evidence.** The description's screenshot is carried + forward from #1056, but this branch is a reconstruction rather than that + code, so it proves nothing about what would merge. A picker capture at the + current head is a **condition**, not a request — I softened it to "a request + rather than a blocker" in the first draft of this page, which quietly + downgraded something the review had made a merge condition. + +## #1228 — where the republish protocol stops + +Conflicting, draft, untouched since 2026-08-07, four readiness boxes unticked. +Every other stale PR in this campaign got rebased and republished for the +author. This one did not, and the line is worth stating because it is the same +line WP15 crossed for #1244 and then had to retreat from. + +The republishes were small and mechanical — a net diff that reapplies onto a +moved base with the author's intent unambiguous. #1228 adds native image +support across eight files of the Cursor adapter including the protobuf request +builder and live transport. Resolving those conflicts means re-deciding the +author's design against a base that moved underneath it, which is authorship, +not maintenance. + +Told them so directly, offered to close it as stale if they would rather not +carry it, and noted that a `cancelled` shard is #1302 and mine to chase rather +than theirs. + +## Fault recorded + +Claimed `typecheck` proved a caller sweep it structurally cannot perform, +because `tsconfig.json` includes only `src`. I have run that command dozens of +times this session and cited it as evidence repeatedly; I had never read what it +covers. diff --git a/devlog/_plan/260808_bug_campaign/027_wp4_lane_d_close.md b/devlog/_plan/260808_bug_campaign/027_wp4_lane_d_close.md new file mode 100644 index 0000000000..8cff9cd0e7 --- /dev/null +++ b/devlog/_plan/260808_bug_campaign/027_wp4_lane_d_close.md @@ -0,0 +1,152 @@ +# WP4 — closing lane D's catalog sequence + +The plan in `040_wp4_catalog_sequential.md` ordered seven PRs so each landed on +a `dev` the previous one had already moved: `#1224, #1226, #1178, #1266, #1244, +#1163, #1228`. All seven now have a disposition; two of those dispositions are +"waiting on something specific" rather than closed, and the difference is +recorded per row instead of flattened into "terminal". + +| PR | Disposition | +|----|-------------| +| #1224 | merged (`903b69b4b`) | +| #1226 | merged (`3ad5bb6bd`) | +| #1266 | merged (`28ba79377`) | +| #1178 | **merged** as `e8ec8d191` — but without a recorded approval, see below | +| #1244 | CI green, **held** on three conditions (WP5) | +| #1163 | **closed as superseded**; republished as #1305, **merged** as `794d8eb09` | +| #1228 | **held for its author** (WP5) | + +## #1178 — the author fixed it themselves + +In WP14 I diagnosed a cache-invalidation defect here and built a fix on a local +branch, then offered it on the PR rather than pushing it. Their head is now +`2ebdb705c fix(catalog): distinguish cache eviction from authority changes` — +they wrote it themselves. Offering rather than pushing was the right call, and +the local branch `codex/260808-1178-cache-clear-reason` can be abandoned. + +Approved its `action_required` CI at a SHA-matched head; it came back +`success`. + +### Two corrections I had to publish on that PR + +**I told the author they were waiting on their own checklist.** They were not: +`isDraft` is false and all four readiness boxes are ticked. The PR is waiting +on *maintainer approval* — `reviewDecision` is empty — which is my side, not +theirs. Telling a contributor the ball is in their court when it is in mine is +the specific failure mode the readiness gate exists to avoid. + +**And my token-path description named the wrong function.** See below. + +### Security review, published rather than implied + +The audit caught that approving a CI *action* is not the security review +`MAINTAINERS.md:48` requires, and this PR touches OAuth token retrieval and +account authority. So I did the review and posted it: + +- **Token flow.** I originally credited `getValidAccessTokenSnapshot`. That is + one route; the catalog gather running on filesystem evidence goes through + `observedModelsAuthResolver` → `observeActiveOAuthAccessToken` + (`provider-fetch.ts:821`). The property holds either way — captured + synchronously before any await, passed only as `apiKey` into the request + builder — but the observed path also carries a credential identity and a + cache generation, so a token that changes underneath cannot be attributed to + the earlier gather. That guard is more specific than what I credited. +- **A guard I first missed, then over-credited.** `modelDiscoveryTransportSeed` + (`oauth/index.ts:614`) pins the registry's fixed `baseUrl` and adapter for + OAuth presets *before* the Bearer header is materialized, so a hand-edited + `config.baseUrl` cannot receive an OAuth token. I omitted it from the first + review, then called it the headline — and it **already existed at the merge + base `3ad5bb6bd`**. #1178 neither added it nor repaired an arbitrary-host + leak. The accurate claim is smaller: the new CCA POST discovery path inherits + the existing pin rather than bypassing it. + +## The governance failure + +`MAINTAINERS.md:45` requires a maintainer approval **and** green required CI +before merge. Six PRs went to `dev` today with green exact-head CI and **no +recorded `APPROVED` review**: #1287, #1288, #1289, #1293, #1305, #1178. + +The mechanism is worth naming because it is not simple forgetfulness. Several +of these needed a pending Actions run approved — `action_required`, which +`gh pr checks` hides — and I logged every one of those against its head SHA +into `.tmp/ocx_approval_ledger.tsv`. Doing the careful version of the *wrong* +approval made the missing one feel handled. On #1178 I also published a full +security review as a comment, so the review existed; it just was not an +approval. + +Not back-filling them. A review recorded after the merge it was meant to gate +is a worse artifact than an accurate record of the gap. Filed as **#1306**, +which also notes the real structural hole: `MAINTAINERS.md:47` forbids +approving your own PR, and five of the six were maintainer republishes of +contributor work, so the convention has no defined path for a solo maintainer +landing someone else's rebased patch. +- **Log surface** is where a discovery failure usually leaks. Both new + `console.warn` sites are clean: the Cursor path logs classified error/detail, + and the provider path logs `status`, `contentType`, `fallback`, and + `urlClass` — a two-value hostname classification + (`provider-fetch.ts:975`), not the URL. That matters because Vertex endpoints + embed a project id and a raw URL would carry query parameters. +- **Snapshot-before-await** is also the right ordering for the cache concern: + an OAuth account change mid-flight cannot make a stale-but-valid response + look authoritative for the new account. + +## #1163 — a refused `git apply` that was not a semantic rebase + +366 commits behind, `CONFLICTING`, and `git apply --check` rejected the net +diff outright. That is normally where WP5's line applies and the PR goes back +to its author. + +The actual merge disagreed: **two conflicts, both a single line, both the same +cause.** `dev` had renamed `augmentRoutedModelsWithJawcodeMetadata` to +`augmentRoutedModelsWithMetadata` and added +`CODEX_ACCOUNT_BOUND_CATALOG_KIND` plus a `catalog/parsing` import block; the +branch had added `resolveComboCatalogMember` to the same export and import +lines. Keeping every symbol from both sides resolves it without re-deciding +anything. + +So `git apply` refusing is evidence about *textual* applicability, not about +whether a rebase requires judgement. Running the merge and reading the conflicts +is the cheap check that tells them apart, and skipping it would have sent a +mechanical rebase back to a contributor for no reason. + +### The fault the audit caught + +I reported the resolution as done because the working files had no conflict +markers and the tests passed. The index still held `UU` entries for both files — +git could not have committed that state. Marker-free files are not a resolved +merge, and "the tests pass" was true of a tree that did not exist as a commit. + +Staged both, confirmed `git diff --cached --check` clean, then re-ran the full +suite **on the committed tree**: 10013 pass / 0 fail. Published as #1305. + +The audit also flagged that the PR body ran those two facts together, reading +as though the staged-tree whitespace check were committed-tree evidence. Body +amended to separate them. + +## Consistency check: why #1163 was rebased and #1228 was not + +Both are stale contributor PRs and #1163 is *older* (366 vs ~200 commits), so +the line cannot be age. It is whether integration requires deciding something +the author already decided: + +- **#1163** — two import lines. The contributor's semantics are untouched and + their tests still exercise them. +- **#1228** — eight files of Cursor adapter including the protobuf request + builder and live transport, where resolving conflicts means re-deciding how + their image support interacts with a moved base. + +Age raises the verification bar. It does not decide who owns the merge. + +## Faults recorded + +- Published a review that named the wrong token-resolution function and omitted + the strongest guard in the diff. +- Told a contributor they were waiting on their own checklist when the PR was + ready and waiting on me. +- Wrote "all seven have a terminal disposition" while two were waiting on CI + and on maintainer approval. "Dispositioned" and "finished" are not the same + claim, and the closeout wording flattened them. +- Merged six PRs without the approval `MAINTAINERS.md` requires, while + meticulously logging a different kind of approval. +- Attributed a pre-existing security guard to the PR under review, in a + correction that was itself correcting an omission. diff --git a/devlog/_plan/260808_bug_campaign/028_wp6_orphan_issue_fixes.md b/devlog/_plan/260808_bug_campaign/028_wp6_orphan_issue_fixes.md new file mode 100644 index 0000000000..be7486a769 --- /dev/null +++ b/devlog/_plan/260808_bug_campaign/028_wp6_orphan_issue_fixes.md @@ -0,0 +1,373 @@ +# WP6 — bug issues with no fix PR + +The live resweep leaves two newly reported defects with no open PR, plus a set +of long-standing tracking items. Both new ones came from reporters who did the +diagnosis themselves, and both are confirmed in source. + +| Issue | State | Plan | +|-------|-------|------| +| #1296 | a local Windows ACL failure surfaces as `401 authentication_error` | fix the classification | +| #1297 | antigravity wire session id derives from first-user text | prefer a real conversation identity | + +## #1296 — a filesystem failure wearing a credential error's clothes + +Confirmed. `src/server/responses/core.ts` passes the raw exception message into +an auth error in three branches (`:875`, `:878`, `:881`): + +```ts +return { ok: false, response: formatErrorResponse(401, "authentication_error", err.message) }; +``` + +When the underlying failure is Windows ACL hardening, that message can be the +literal `ACL hardening skipped — previous attempt timed out` +(`src/lib/windows-secret-acl.ts:604`), delivered as a 401. The user is told +their credential was rejected while the actual cause is a directory tree under +`~/.opencodex`. + +**The repository already knows how to do this correctly on the other surface.** +`src/server/management-auth.ts:76` and `:95` classify the same failure as +`management token directory ACL hardening did not complete` with actionable +guidance. The data plane has no equivalent, which is the whole defect: same +cause, two surfaces, one of them honest. + +What makes the fix tractable is that the ACL errors are already tagged. They +carry `code` values — `ETIMEDOUT` for a real timeout +(`windows-secret-acl.ts:423`), `EACLRETRYEXHAUSTED` when the single recovery +attempt is spent (`:600`), `EICACLS` otherwise — so classification does not +require string matching. + +### Diff-level plan + +1. A predicate that recognises an ACL-origin error by `code`, not by message + text. String matching on an error message is how the next rename silently + turns the classification back off. +2. In each of the three `core.ts` branches, check it before the 401 and return + a **503** with a type that is not `authentication_error`, naming the local + cause and pointing at the remediation. +3. Do not include the raw path. The message may name the *kind* of failure; it + should not print a filesystem path into a client-visible error. + +The reporter's own expectation is the right acceptance bar: whatever the status, +it must not be indistinguishable from "your token is bad", because the two have +opposite remediations. + +### Reproduction, and what it is honestly worth + +The reporter states plainly that they no longer have a live reproduction — +they fixed their local trigger — and that the mechanism is unchanged in 2.11.0. +That is an honest report of a structural defect rather than a captured one, and +it is checkable in source without Windows: the three branches pass `err.message` +through unconditionally. + +I have no Windows host either, so the regression will exercise the +classification directly with a synthetic ACL-coded error rather than pretending +to reproduce the icacls stall. + +## #1297 — an id that must be stable, derived from something that is not + +Confirmed by reading. `antigravitySessionId` +(`src/adapters/google-antigravity-wire.ts:53`) hashes the first user message +text. Its own comment states the invariant it must satisfy: + +> the id must stay stable across every turn, because the replay cache observes +> signatures on turn N's response and re-injects them on turn N+1's request + +First-user text satisfies that only while the first user message survives +verbatim in what the adapter sees. Codex compacts, summarises, and trims long +histories; when it does, the anchor changes, the id changes mid-conversation, +and cached thought signatures stop being found. The textless fallback is worse +by construction — `-${Math.random()}` is a fresh id every turn. + +The reporter is careful to separate two failure modes, and the distinction is +the important part of the report: the existing comment defends *collisions* +(two conversations opening with identical text) by noting the replay cache keys +on `functionCall` identity. That argument is sound for collisions and says +nothing about *instability*. A shared id still finds the association; a changed +id loses it. + +### Diff-level plan + +`OcxParsedRequest` already carries `promptCacheKey` (`src/types.ts:239`), which +is the Responses prompt-cache affinity key — purpose-built to be stable across +the turns of one conversation. Preference order: + +1. `promptCacheKey` when present, hashed the same way so the wire format is + unchanged; +2. first-user text as today, for clients that send no conversation metadata; +3. the random fallback, unchanged, when there is neither. + +The hash and mask stay identical so the id shape (`-`) does not move. +Only the input changes, and only when a better input exists. + +### Evidence level, stated rather than implied + +The reporter says explicitly this is a mechanism-level report, not a captured +failure — they found it reviewing the adapter and cannot produce a deterministic +capture. That is worth honouring rather than overselling: the regression will +prove that consecutive turns whose first-user text differs still derive the same +id when a `promptCacheKey` is present, which is the invariant the code claims +for itself. It will not claim to reproduce a Gemini-side signature failure. + +## Acceptance + +1. #1296: an ACL-origin failure no longer returns `authentication_error`, with + a regression driving a synthetically coded error through the classification. +2. #1297: a `promptCacheKey`-bearing request derives a stable id across turns + with differing first-user text, and the text and random paths are unchanged + when no key is present. +3. Each fix has an ablation that fails without it. +4. Full suite, typecheck, privacy scan green. +5. Neither PR overstates the reporter's evidence level. + +--- + +# Audit fold — both plans were wrong about their central mechanism + +The review rejected each plan's load-bearing assumption. Not details: in both +cases the thing I proposed to build on does not do what I said it does. + +## B1 — `promptCacheKey` is the wrong anchor, and the repo already says so + +I picked it because the name sounded like conversation affinity. It is +**unvalidated client input**: `src/responses/schema.ts:148` accepts any string +and `src/responses/parser.ts:661` copies it through. Worse, this repository has +already decided this question in the opposite direction — +`src/adapters/cursor/request-builder.ts:227` warns explicitly against using a +shared `prompt_cache_key` as a conversation id, and Claude Desktop cache +cohorts are marked as **shared across conversations** +(`src/server/responses/core.ts:620`, `src/server/claude-messages.ts:701`). + +So my fix would have traded an unstable anchor for a shared and +attacker-choosable one. Hashing does not repair either property. + +### The right anchor was already in front of me + +`_clientThreadId` (`src/types.ts:21`) is documented as "stable upstream client +thread identity, used only to derive provider-scoped continuation ids", and is +populated from the `x-codex-parent-thread-id` header +(`src/server/responses/core.ts:1413`). + +The decisive detail is what already consumes it: **`replayCacheScope`** +(`core.ts:2745`, `:2792`). That is the very cache #1297 is about. The system +already scopes signature replay by thread id, and the Antigravity adapter +derives its own id from message text instead — so the two disagree by +construction. + +Revised preference order: + +1. `parsed._clientThreadId` when present, hashed identically so the + `-` wire shape does not move; +2. first-user text, unchanged, for clients that send no thread header; +3. the random fallback, unchanged. + +Still to prove before writing it: that `_clientThreadId` actually reaches the +Antigravity adapter on that path, rather than being set on a `parsed` the +adapter never sees. That is a check, not an assumption — the last two anchors +were chosen on plausibility. + +## B2/B3 — the #1296 path I planned to fix does not exist on `dev` + +This is the more serious finding, because I confirmed the defect by reading +three lines and never traced whether anything can reach them with an ACL error. + +- `CodexDirectAuthenticationError` takes **no message** — it is a fixed + missing-bearer string (`auth-context.ts:132`). +- `ForwardAdmissionCredentialError` is a fixed proxy-bearer message + (`auth-cors.ts:359`). +- `CodexPoolAuthenticationError` accepts only a message and preserves neither + `code` nor `cause` (`auth-context.ts:108`). + +And upstream of all three: the read-path hardening is `required: false` +(`src/config.ts:1426`, `src/codex/account-store.ts:88`), which returns +`{ ok: false }` instead of throwing, and those callers discard the result. +`grep -rn 'required: true'` shows it only on write paths — lock files, the +profile manager, the catalog writer, the prompt journal, the spill store, and +management-auth. The OAuth store does not harden with `required: true` at all. + +So the reporter's chain — poisoned `auth.json` → refresh cannot persist → 401 +carrying the ACL text — cannot occur through the branches I named. A synthetic +coded error pushed through one of them would have tested a state production +never produces, and the regression would have looked green while proving +nothing. + +Where a real local-storage failure *would* surface, per the audit, is +`CodexAuthContextError`, which does carry the original in `cause` +(`auth-context.ts:424`) and is already mapped to a fixed 401 at +`core.ts:868` — plus equivalent wrappers at `responses/compact.ts:385`, +`server/search.ts:129`, and `server/images.ts:394`. My three-branch inventory +was both overinclusive and incomplete. + +### Revised approach for #1296 + +**Do not patch on the strength of a reported chain that current source does not +support.** The next step is to trace one genuine producer-to-surface path on +`dev`: find a `required: true` harden whose throw can reach a client-visible +auth error, and follow it. If one exists, the fix is to preserve a typed +local-storage cause through the wrapper and classify it at *every* surface that +maps that wrapper, not just in `core.ts`. + +If no such path exists on `dev`, the honest disposition is to say so on the +issue: the misclassification shape is real and worth guarding, the specific +2.10.x chain appears to have been closed by the move to `required: false`, and +the reporter deserves that stated plainly rather than a fix that performs +diligence against a synthetic error. + +### B4 — the code set was incomplete anyway + +`sanitizedAclError` preserves `EACCES`, `EPERM`, and `EACLIDENTITY` in addition +to `ETIMEDOUT` and `EICACLS` (`windows-secret-acl.ts:573`). A predicate limited +to my three values would have left related failures misclassified — a smaller +error than B2, but the same species: I enumerated from the two constants I had +read rather than from the function that produces them. + +## Revised acceptance + +1. #1297: `_clientThreadId` **proved** to reach the Antigravity adapter, then + preferred over first-user text, with a regression showing two turns whose + first-user text differs derive the same id. +2. #1296: one real producer-to-surface path traced on `dev` before any patch; + if none exists, a documented disposition on the issue instead of a fix. +3. Ablations that fail without each change. +4. Full suite, typecheck, privacy scan green. +5. Neither PR overstates the reporter's evidence level — or mine. + +--- + +# Second audit fold — the precedent I found was a second instance of the bug + +## The Vertex path is not a precedent, it is another defect + +Having been told `promptCacheKey` was the wrong anchor, I went looking for how +the codebase does it correctly and found `vertexReplaySessionId` +(`src/adapters/google.ts:56`), whose comment reads "Prefer Codex's stable +opaque thread key". I read that as vindication for the shape of my fix. + +It is not. The function's comment says thread key; the code reads +`parsed.options.promptCacheKey` — the same arbitrary Responses input the audit +had just rejected, and the same field `src/adapters/cursor/request-builder.ts:227` +warns against using as a conversation id. **The Vertex replay namespace has the +same defect #1297 reports for CCA**, differently shaped. + +I had found a second instance of the bug and mistaken it for the fixed version, +because the comment described the right thing while the code did something else. +That is the third time in this work-phase I have taken a name or a comment as +evidence of behaviour. + +**Consequence:** do not copy Vertex into CCA. Record Vertex as an analogous +unresolved misuse on the issue, so the next person does not repeat my reading. + +## What is actually confirmed + +`_clientThreadId` **does** reach the adapter. It is assigned at +`core.ts:1413` and that same `parsed` object is handed to +`adapter.buildRequest()` at `:1863` and `:2833`. It is simply unreferenced in +`google.ts` today, which is why grep found nothing — absence of use, not absence +of availability. + +## The contract distinction I had missed + +"Give CCA what Vertex has" is wrong for a second reason beyond B1. The two +values are not the same kind of thing: + +| Path | Value | Scope | +|------|-------|-------| +| Vertex | `vertexReplaySessionId` | **local** replay namespace only | +| CCA | `antigravitySessionId` | local replay key **and** a Google-visible wire field, sent as `request.sessionId` (`google.ts:390`) | + +So changing the CCA id changes what goes on the wire to Google, while changing +the Vertex one does not leave the process. Any fix must keep the `-` +shape, test request serialization, and — importantly — **claim nothing about +Google accepting or benefiting from a different identity**, since I have no +provider evidence for that. + +It is also a scoped repair, not a universal one: a client that sends no +`x-codex-parent-thread-id` keeps the unstable text fallback. That belongs in +the PR body rather than being discovered by the next reader. + +## #1296 — disposition confirmed, with the wording corrected + +"Do not patch" is right, but my draft leaned toward "the reporter's chain was +closed by the move to `required: false`". That overstates what I checked: I +found no current producer-to-401 path, which is not the same as proving the +historical one is gone, and a write-path ACL failure may still surface +somewhere I have not traced. + +Honest wording for the issue: **no current producer-to-401 path found on +`dev`**; the misclassification shape is real and worth guarding; a versioned +repro or a stack trace would change the answer. Not closed as disproven. + +## Final acceptance + +1. #1297: CCA wire id prefers `_clientThreadId`; tests prove same thread + + differing first-user text + differing cache keys → one identity, and + different thread ids do not collide. Wire shape and serialization asserted. + No claim about Google-side benefit. +2. #1297 report also records the Vertex misuse as an analogous open item. +3. #1296: documented disposition, phrased as "no current path found" rather + than "disproven". +4. Ablation, full suite, typecheck, privacy scan. + +--- + +# Outcome + +| Issue | Disposition | +|-------|-------------| +| #1297 | **fixed** — PR #1311 | +| #1296 | **open, diagnosed** — no current producer-to-401 path found | + +## What the review kept catching + +Four rounds on two issues, and the same species of error each time: **I read a +name or a comment and treated it as evidence of behaviour.** + +1. `promptCacheKey` — the name says cache affinity, so I used it as conversation + identity. It is unvalidated client input, and the repo already warns against + exactly this in `cursor/request-builder.ts`. +2. `vertexReplaySessionId` — its comment says "Prefer Codex's stable opaque + thread key". The code reads `promptCacheKey`. I found a second instance of + the reported bug and read it as the fixed version. +3. The three `core.ts` branches — I confirmed #1296 by reading three lines that + pass `err.message` into a 401, and never checked whether anything can reach + them carrying an ACL error. Nothing can: two take fixed messages, the third + preserves neither `code` nor `cause`, and read-path hardening no longer + throws. +4. `codex-thread:` prefixing — I called it namespacing that "cannot collide". A + first message equal to the prefixed form collides deterministically; the + reviewer produced it, and I reproduced it before narrowing the claim. + +Each was cheap to check and I checked none of them until asked. + +## #1297 — the fix, and what it does not claim + +CCA wire id now prefers `_clientThreadId`, prefixed before hashing. Seven tests, +one property each, including an **envelope-level** one: the changed value goes on +the wire to Google as `request.sessionId`, so proving the derivation is not the +same as proving it arrives. Ablation fails 4. + +Stated in the PR rather than left for a reader to find: a client without the +thread header keeps the unstable text anchor; the residual prefix collision is +asserted rather than hidden; deploying changes the Google-visible session id for +conversations already in flight; and no claim is made that Google benefits from +the new identity, because there is no live CCA evidence for it. + +The Vertex misuse is recorded on the issue as a separate open item. Bundling it +into a fix for a different path would have hidden it. + +## #1296 — diagnosed, not patched, and not closed + +The misclassification shape is real; the reachability is not established. I +drafted a fix that pushed a synthetically coded error through the three branches +and abandoned it, because a regression against a state production never creates +is a green test that proves nothing. + +The wording on the issue matters: **"no current producer-to-401 path found"**, +not "disproven". I have not traced every write-path ACL failure to its surface, +and the reporter's 319 logged occurrences on 2.10.x happened. A stack trace or +exact version would settle which producer it was. + +Also left for whoever picks it up: the ACL errors already carry codes +(`ETIMEDOUT`, `EICACLS`, `EACCES`, `EPERM`, `EACLIDENTITY`, +`EACLRETRYEXHAUSTED`), so classification never needs message matching — and my +first draft's code list was missing three of those, enumerated from the two +constants I happened to have read. diff --git a/devlog/_plan/260808_bug_campaign/029_wp17_unblock_sweep.md b/devlog/_plan/260808_bug_campaign/029_wp17_unblock_sweep.md new file mode 100644 index 0000000000..dee8acccec --- /dev/null +++ b/devlog/_plan/260808_bug_campaign/029_wp17_unblock_sweep.md @@ -0,0 +1,184 @@ +# WP17 — the queue refilled + +The goalplan read `complete: true`, all 16 work-phases done, 6/6 criteria met. +A live sweep said otherwise. + +## What the completion flag was measuring + +Twenty-four open bug-class PRs. Of those, **nine had never had CI executed**: + +| PR | State | Author | +|----|-------|--------| +| #1304, #1300, #1294, #1264, #1260, #1258, #1256 | `action_required` | XIQIXIQIXIQI, hanbinnoh, luvs01 | +| #1192 | `cancelled` | luvs01 | +| #1155 | **no runs at all** | myrosla | + +WP1 did exactly this work for 27 PRs earlier in the campaign. The queue refilled +while the later phases ran, and nothing in the goalplan measures that — the +criteria track the items the plan enumerated, not the items that exist. + +That is worth stating plainly: **`complete: true` meant "the recorded work is +done", not "the repository is in the state the objective describes".** A +campaign whose objective is "give every open item a disposition" cannot be +closed against a snapshot taken at its start. + +## The sweep + +Approved every `action_required` run on every open PR, SHA-matched, logged to +`.tmp/ocx_approval_ledger.tsv`: **37 approvals**, zero SHA mismatches. + +The repository had **735** runs sitting at `action_required` overall. Most +belong to superseded heads of PRs already handled, which is why the sweep is +scoped to the *current head of each open PR* rather than to the queue. Approving +by queue would have released runs for commits no PR points at any more. + +### A fail-closed check that worked + +The first attempt at the loop mis-quoted a shell variable, so the run id became +a JSON error blob and every row logged as `SKIP` with a 404. Nothing was +approved. That is the ledger doing its job: the comparison is between the PR +head and the *run's* head, so a malformed id cannot pass it. Rewritten as a +script and rerun cleanly. + +## #1192 — cancelled, rerun issued + +`rerun-failed-jobs` on run `31157296632`. The branch is 325 commits behind, so +a green result there is weaker evidence than it looks; that is the author's +rebase to make, not mine. + +## #1155 — nothing to approve + +The interesting one. Zero workflow runs against `307045c55`, and zero for the +branch in this repository's Actions history. The only status on the head commit +is CodeRabbit. + +It is a fork PR (`myrosla/opencodex`), and the first-time-contributor gate never +produced a queued run the API could release. So this is not a case of me +forgetting to approve it — there is no run object to approve. + +Told the author exactly that, including the part I cannot do: pushing any commit +(an empty one, or the rebase the 429-commit gap warrants) usually re-triggers +the workflows and creates a run I *can* approve. + +Also worth recording: this PR already carries two of my comments — one +retracting a wrong close-as-unreachable assessment, and one correcting an error +*in that retraction*. The patch is not blocked on analysis. It is blocked on +never having been tested. + +## What this says about the goal + +The objective is a terminal disposition for every open bug issue and PR as of +2026-08-08. Items keep arriving — #1292, #1295, #1296, #1297 during the +campaign, #1308 and #1310 after. Two honest readings: + +1. the cutoff is the campaign's start, and post-cutoff arrivals are out of + scope by construction; or +2. the objective is a standing state, in which case it is never complete. + +Reading (1) is what the objective literally says, and the resweeps recorded in +`023` and `027` are what make the boundary checkable. But the completion flag +flipped while nine PRs from *before* the cutoff had never been unblocked, and +that is not a boundary question — it is work the plan enumerated and the +tracking lost. + +--- + +# Audit fold + +## An approval is not a disposition + +The sharpest correction: I described 37 approvals as if they closed something. +They did not. Approving a pending workflow run **starts** CI; it produces no +outcome. #1256 was created at `2026-08-08T04:35:05Z` — squarely pre-cutoff — +and its Cross-platform CI is still executing as this is written. + +So the completion bar is not "every pre-cutoff item was unblocked". It is +**"every pre-cutoff item has a recorded outcome or an explicit hold"**, which +requires a final resweep *after* those runs finish. Recorded as the gate on +closing the goal. + +Verified since: #1192's rerun at `b50f23943` is `completed/success`, and the +seven approved PRs are running rather than queued — so the approvals did what +they were supposed to do. That is evidence the sweep worked, not evidence the +work is finished. + +## The scope check held + +Independently confirmed: 712 runs sit at `action_required`, and **zero** of them +match any current open-PR head. The current-head rule is what kept 675-odd +superseded-commit runs from being released, and scoping to the queue would have +done exactly that. + +## #1155 — plausible is not proven + +I wrote that the first-contributor gate caused the missing runs. The evidence +supports "there is no run object to approve" and nothing stronger: its diff does +touch `src/` and `tests/`, which match `ci.yml`'s PR paths, so path filtering is +ruled out — but the *cause* of the absence is inferred. + +The comment I posted already says a push "usually re-triggers" rather than +promising it. What that comment should also have said, and what belongs here: +**if the author pushes, the next step is to verify a run was actually created**, +not to assume one was. + +## Where I overreached about the objective + +I wrote that a campaign with this objective "cannot be closed against a snapshot +taken at its start". That contradicts the campaign's own plan, which explicitly +freezes the inventory at the cutoff (`000_plan.md:21`). + +The correct definition, and the one this campaign is actually held to: +**complete = every item open at the recorded cutoff has a final disposition.** +Not "the repository currently has no open bug work" — that is a standing +condition no campaign can satisfy, and holding myself to it would be a way of +never having to close anything. + +The real defect was narrower than my framing: the plan had no *final* live +resweep step, so nine pre-cutoff PRs went untracked. That is a plan gap, not a +reason to redefine completion. + +--- + +# Final resweep — `FINAL_SWEEP=2026-08-08T19:09:36Z` + +The gate the audit set: not "unblocked" but "has a recorded outcome". Every one +of the nine now does. + +| PR | Author | Outcome | +|----|--------|---------| +| #1304 | XIQIXIQIXIQI | CI **success**, awaiting author checklist | +| #1300 | hanbinnoh | CI **success**, awaiting author checklist | +| #1294 | luvs01 | CI **success**, awaiting author checklist | +| #1264 | luvs01 | CI **success**, awaiting author checklist | +| #1260 | luvs01 | CI **success**, awaiting author checklist | +| #1258 | luvs01 | CI **success**, awaiting author checklist | +| #1256 | luvs01 | CI **success** after rerun, awaiting author checklist | +| #1192 | luvs01 | CI **success** after rerun; 325 commits behind, rebase offered | +| #1155 | myrosla | **no run object exists**; author push required, then verify one was created | + +All eight with runs are green. All nine are drafts held by the contributor's own +four-box checklist, which stays theirs — the maintainer half of the blockage is +cleared and the PRs say so. + +## #1256 produced the best evidence yet for #1302 + +Its first run had **two shards hang in the same run**: + +``` +test 4/4 cancelled 18:44:24Z → 18:59:39Z (15m15s) +test 3/4 cancelled 18:42:18Z → 18:57:33Z (15m15s) +``` + +Both killed at the job timeout to the second, matching every prior instance +including #1301's `test 4/4` at 15:28:02Z → 15:43:17Z. Tally is now **eight +occurrences across six branches plus `dev`**, with the shard varying between +2/4, 3/4 and 4/4. + +Two shards hanging at once makes "one bad test" harder to sustain. And the +15m15s figure matters on its own: these are not long tests finishing late, they +are shards that stop emitting and get killed at the deadline — consistent with +the orphaned `bun` process cleanup reports. Added to #1302. + +The triage cost is the part worth recording: of eight approved runs, one needed +a rerun purely for this, and a genuine hang from a real change would look +identical. diff --git a/devlog/_plan/260808_bug_campaign/030_wp18_leftovers.md b/devlog/_plan/260808_bug_campaign/030_wp18_leftovers.md new file mode 100644 index 0000000000..54fe11e977 --- /dev/null +++ b/devlog/_plan/260808_bug_campaign/030_wp18_leftovers.md @@ -0,0 +1,98 @@ +# WP18 — the leftovers, and a retraction on #1302 + +Three concrete items remained after WP17: two devlog units unpublished, my own +#1301 held indefinitely, and #1244 having gone `DIRTY`. + +## devlog 028 and 029 published + +They existed only in local history. Published as **#1314**, merged to `dev` as +`6a7e5f05e`. The campaign record is now on the branch rather than on my disk, +which is the point of keeping it. + +## #1301 — rebased, and deliberately left red + +It had been held since both its CI runs came back `cancelled` under #1302, and +had drifted 5 commits behind. Rebased onto `b5d44a534`, force-pushed with a +lease against the previous head, both commits preserved with their separation +(luvs01's assertion trailered, the unconditional-step assertion mine). + +Re-verified on the rebased head rather than reusing old numbers: full suite +**10055 pass / 7 skip / 0 fail** across 627 files, `ci-workflows` 125 pass, +typecheck and privacy scan clean, and the `if: false` ablation still fails +against current `dev` — so the coverage gap it closes is still open on the base. + +The new run hung again, `test 3/4`, 19:32:43Z → 19:47:59Z. **Not rerunning it.** +A retry *might* come back green — that is unknowable, and I stated it as a +certainty in the first draft of this page. What matters is that one green retry +would not erase **two cancellations at consecutive exact PR heads**: +`31263738953` at `f09ef1557` before the rebase, and `31274685166` at +`454b1d3b5` after it. Held pending a root cause or a reproducible base-branch +comparison, not pending a luckier roll. + +## The #1302 "narrowing" — retracted + +I claimed this run's stall landed at the same file as run `31152916419`, +`cli-native-profile → cli-restart-health` with `killed 1 dangling process` as +the last line, and called it a repeated signature. + +**It is not true, and the audit caught it before it went further than one issue +comment.** Re-reading the logs: + +Those are two different questions and I had collapsed them into one column: + +| Run | First logged `EEXIST` under | Last output before silence | Outcome | +|-----|------------------------------|----------------------------|---------| +| `31152916419` | `tests/autostart-health.test.ts` | — (no silence) | **kept running**, finished 1 fail / 2 errors in 85s; never cancelled, no dangling-process line | +| `31263738953` | `tests/baseten-provider.test.ts` | `tests/claude-messages-endpoint.test.ts` | hung, cancelled at 15m | +| `31274685166` | `tests/api-storage.test.ts` | `tests/cli-restart-health.test.ts` | hung, cancelled at 15m | + +Three runs, and no column repeats. The pair I pointed at is **one observation**. +I compared two logs by memory of what one of them said, cited a completed +failure as corroboration for a hang, and then in the first draft of this +retraction still named the wrong file for `31263738953` — `claude-messages` is +where the output *stops*, not where the error first appears. + +What the evidence supports, at its real strength: Bun `EEXIST` on `epoll_ctl` — +a descriptor registered with the event loop twice — appears in all three runs +under different files, and **co-occurs** with one completed failure and two +15-minute cancellations that leave an orphan `bun`. Whether the `EEXIST` causes +the hang, shares a cause with it, or is incidental is **not established**. No +file-level culprit identified. + +My earlier "the shard varies, so it is not one bad test" was also imprecise for +a different reason: sharding distributes files differently per run, so a varying +shard number never argued against a single file either way. + +Local reproduction of the pair failed (`--isolate`, three runs, 18 pass / 0 fail +in ~2.8s on macOS), which I had presented as "needs the Linux runner" — with the +hypothesis retracted, it is simply a null result. Retraction posted to #1302. + +### Why this one stings + +The whole reason #1301 is being held is that I stopped calling this flake and +started gathering evidence. Then I produced a false piece of evidence, in the +issue I opened to keep the record honest, by asserting a comparison instead of +running it. + +## #1244 — conflicted by my own merge + +It went `DIRTY` because **I merged #1305**. The overlap is +`src/codex/catalog/provider-fetch.ts` and `tests/codex-catalog.test.ts`: + +``` +<<<<<<< dev (from #1305 / #1163) + const members = combo.targets.map(target => resolveComboCatalogMember(...)) +======= #1244 + const discoveredMembers = combo.targets.map(target => memberByKey.get(targetKey(target))) +>>>>>>> +``` + +`dev` now synthesizes a combo member whose provider row is incomplete instead of +dropping it; #1244 renames the same binding as part of the picker work. +Resolving it means deciding how a synthesized member behaves in the new flow — +the author's design call. + +Told them so, with the conflict quoted, and did not rebase it. Worth noting the +asymmetry honestly: I rebased #1163 (366 behind) because its conflicts were two +import lines, and I am declining #1244 (25 behind) because one conflict is a +semantic decision. Age was never the criterion. diff --git a/devlog/_plan/260808_issue1283_xai_weekly_quota/000_plan.md b/devlog/_plan/260808_issue1283_xai_weekly_quota/000_plan.md new file mode 100644 index 0000000000..ec0424fcfb --- /dev/null +++ b/devlog/_plan/260808_issue1283_xai_weekly_quota/000_plan.md @@ -0,0 +1,109 @@ +--- +created: 2026-08-08 +status: active +tags: [xai, grok, quota, issue-1283, dashboard] +--- + +# Issue #1283 — Grok dashboard weekly quota (OpenCodex) + +## Loop spec + +- Archetype: spec-satisfaction repair +- Trigger: https://github.com/lidge-jun/opencodex/issues/1283 reports dashboard shows 30-day/monthly Grok usage while Codex/Grok CLI gates on weekly limit. +- Goal: OpenCodex provider quota for OAuth `xai` prefers Grok weekly credits and only falls back to legacy monthly billing when weekly data is unavailable. +- Non-goals: ima2-gen/cli-jaw changes; multi-account xAI pool aggregation redesign; docs-site locale churn; release/version bump; secret-store redesign. +- Verifier: `bun test tests/provider-quota.test.ts`; `bun run typecheck` if types change; `git diff --check`. +- Stop: weekly-first path + monthly fallback covered by focused tests; branch pushed; PR targets `dev` with `Closes #1283`. +- Terminal outcomes: DONE on green tests + PR; NOOP only if tree already weekly-first; BLOCKED only if contract cannot be determined. + +## Evidence already known + +- OpenCodex `fetchXaiQuota` (`src/providers/quota.ts`) still calls `GET https://cli-chat-proxy.grok.com/v1/billing` and maps `monthlyLimit/used` → `monthlyPercent` (introduced 2026-07-05, unchanged). +- Prior cross-repo work (2026-07-16) moved **ima2-gen** (and intended cli-jaw) to `GET /v1/billing?format=credits` with envelope `{ config: { creditUsagePercent?, currentPeriod: { type: USAGE_PERIOD_TYPE_WEEKLY, end } } }`. +- OpenCodex already has: + - `XAI_GROK_COMPATIBILITY` client headers in `src/providers/xai-transport.ts` + - `credential.accountId` from JWT `sub` / Grok CLI `user_id` (`src/oauth/xai.ts`, `src/oauth/local-token-detect.ts`) + - `getCredential("xai")` / `getValidAccessToken("xai")` for the active OAuth account + +## Diff-level plan + +### IN + +1. MODIFY `src/providers/quota.ts` + - Add constants: + - `XAI_BILLING_URL = "https://cli-chat-proxy.grok.com/v1/billing"` + - `XAI_CREDITS_URL = XAI_BILLING_URL + "?format=credits"` + - Add pure parser `parseXaiCreditsResponse(value: unknown): { percent: number; resetAt?: number } | null` + - Require `config.currentPeriod.type === "USAGE_PERIOD_TYPE_WEEKLY"` and parseable `end` + - `creditUsagePercent` optional: omit → `0`; non-finite number → reject + - Clamp via `normalizePercent` + - Rewrite `fetchXaiQuota(provider)`: + 1. Resolve access token via `getValidAccessToken("xai")`; on failure return null. + 2. Read active credential via `getCredential("xai")` for optional `accountId`. + 3. If `accountId` is non-empty, attempt weekly credits request with headers: + - `Accept: application/json` + - `Authorization: Bearer ` + - `x-xai-token-auth: xai-grok-cli` + - `x-authenticateresponse: authenticate-response` + - `x-userid: ` + - `x-grok-client-version: XAI_GROK_CLIENT_VERSION` + - Prefer importing constants from `./xai-transport` rather than duplicating version. + - Isolate throw/non-2xx/malformed/non-weekly into null so monthly can run. + - On success return `report(provider, "xai:grok-billing-credits", { weeklyPercent, weeklyResetAt?, updatedAt })`. + 4. Legacy monthly fallback: current bare `/v1/billing` parse of `monthlyLimit/used` → `monthlyPercent`/`monthlyResetAt`, source remains `xai:grok-billing`. + - Do not log tokens, user ids, or raw body fields. + +2. MODIFY `tests/provider-quota.test.ts` + - Existing multi-provider fixture currently mocks only bare `/v1/billing` and expects `xai.monthlyPercent === 25`. Keep that path green: either leave accountId absent so weekly is skipped, or answer weekly with non-weekly/null and still serve monthly. + - Add focused xAI cases: + - weekly success: credential with `accountId`, credits URL returns weekly envelope → `weeklyPercent` + `weeklyResetAt` + source `xai:grok-billing-credits`; assert request URL ends with `format=credits` and required headers present without asserting secret values beyond bearer token already used. + - omitted percent → weekly 0 + - weekly non-2xx / malformed / non-weekly period → monthly fallback still works + - missing accountId → skip weekly, monthly only + - Keep privacy assertions: report JSON must not include access secrets / raw_secret fields. + +### OUT + +- GUI component rewrite (weekly bar already renders when `weeklyPercent` is present) +- Changing Codex WHAM weekly/monthly plan logic +- Live network smoke requiring real Grok auth (optional only) + +## Activation scenarios (C) + +1. Weekly non-zero path fires when mock returns `USAGE_PERIOD_TYPE_WEEKLY` + percent. +2. Weekly zero-omission path fires when percent key omitted. +3. Fallback activation: rejected weekly response still yields monthly percent from second call. +4. Missing identity skips credits URL entirely. + +## Verification commands + +```bash +bun test tests/provider-quota.test.ts +bun run typecheck +git diff --check +``` + +## Publish + +- Branch: `codex/260808-1283-xai-weekly-quota` +- Commit message: `fix(quota): prefer Grok weekly credits for xAI dashboard` +- Push and open PR to `dev` with template + `Closes #1283` (user authorized push). + + +## Audit synthesis — round 1 (main, 2026-08-08) + +Independent reviewer dispatch timed out with empty output; main agent performed the adversarial read-only audit against local code and ima2-gen. + +Accepted amendments before B: + +1. **Header case / constants (High):** use `XAI_GROK_COMPATIBILITY.headers.tokenAuth` / `authenticateResponse` / `clientVersion` and `XAI_GROK_CLIENT_VERSION` from `src/providers/xai-transport.ts`. Do not invent mixed-case aliases. Keep `x-userid` literal as in ima2-gen weekly path (not present on chat transport). +2. **Identity resolution (High):** read `getCredential("xai")?.accountId` first; if missing, decode JWT `sub` from the active access token the same way `getTokenIdentity` does (base64url payload). Missing identity skips weekly and falls back monthly. +3. **Client version (Medium):** use OpenCodex pinned `XAI_GROK_CLIENT_VERSION` rather than reading `~/.grok/version.json`. OpenCodex chat transport already pins this; weekly quota should match product identity, not require a local Grok CLI install. +4. **Percent semantics (Medium):** use `normalizePercent` (clamp, no Math.round) for consistency with other provider quota parsers in this file; tests must not assume integer rounding of 12.3. +5. **Source labels (Low):** weekly success → `xai:grok-billing-credits`; monthly fallback → `xai:grok-billing`. +6. **Exception isolation (High):** weekly attempt must catch network/JSON/parse failures and continue to monthly; never throw out of `fetchXaiQuota`. +7. **Fixture preservation (High):** existing multi-provider fixture saves credentials without accountId and mocks only bare `/v1/billing`; keep weekly skip-on-missing-identity so `monthlyPercent: 25` stays green. Focused weekly tests use credentials with accountId. +8. **No dual-window merge in v1:** when weekly succeeds, return weekly only (the gating window). Do not also attach stale monthly from a second call in the success path. +9. **Privacy:** never put accountId/user id into report objects or logs. + +VERDICT: GO-WITH-FIXES (blockers=4 High folded into plan above) diff --git a/docs-site/src/content/docs/contributing.md b/docs-site/src/content/docs/contributing.md index f8954b90e9..bf8017e5ba 100644 --- a/docs-site/src/content/docs/contributing.md +++ b/docs-site/src/content/docs/contributing.md @@ -164,7 +164,8 @@ sent to. A preset therefore needs primary-source evidence, not a working code pa that add or promote a provider must supply all of the following in the description: - **The documented OpenAI-compatible endpoints.** Link the vendor's own API reference for the chat - endpoint and, when the entry sets `liveModels: true`, for authenticated `GET /v1/models`. A + endpoint and, when the entry sets `liveModels: true`, for its authenticated model-discovery + endpoint (typically `GET /v1/models`). A passing fixture test is not a substitute: it proves our code shape, not the upstream contract. - **Terms of service and the operating legal entity.** An empty or placeholder legal page does not establish who runs the endpoint or under what terms user traffic is handled. diff --git a/docs-site/src/content/docs/guides/providers.md b/docs-site/src/content/docs/guides/providers.md index af1d0ef4ab..07d0677e1e 100644 --- a/docs-site/src/content/docs/guides/providers.md +++ b/docs-site/src/content/docs/guides/providers.md @@ -113,7 +113,7 @@ ocx logout | `anthropic` | `anthropic` | `https://api.anthropic.com` | Claude models; live model list fetched from `/v1/models`. | | `kimi` | `openai-chat` | `https://api.kimi.com/coding/v1` | Kimi K2.7/K2.6/K2.5 coding models. | | `kiro` | `kiro` | `https://runtime.us-east-1.kiro.dev` | Initial login imports the installed, signed-in `kiro-cli` session (on Unix, install with `curl -fsSL https://cli.kiro.dev/install | bash`; on Windows PowerShell, use `irm 'https://cli.kiro.dev/install.ps1' | iex`; then run `kiro-cli login`). **Add account** logs `kiro-cli` out, starts a fresh browser login that switches the account used by `kiro-cli`, and stores account-scoped profile metadata. Existing OpenCodex accounts are preserved, and cancellation or failure restores the previous `kiro-cli` session. | -| `google-antigravity` | `google` | `https://daily-cloudcode-pa.googleapis.com` | Google OAuth over the Cloud Code Assist wire. Uses the maintained six-model static catalog because CCA does not expose the generic `/models` endpoint. | +| `google-antigravity` | `google` | `https://daily-cloudcode-pa.googleapis.com` | Google OAuth over the Cloud Code Assist wire. Live discovery uses CCA's authenticated `v1internal:fetchAvailableModels` endpoint and publishes the agent models available to the signed-in account; the maintained catalog remains the fallback. | | `cursor` | `cursor` | `https://api2.cursor.sh` | Experimental PKCE login, live HTTP/2 transport, and account-filtered model discovery. | | `github-copilot` | `openai-chat` | `https://api.githubcopilot.com` | Experimental. GitHub device flow + `copilot_internal` exchange (VS Code OAuth client). Requires an active Copilot subscription; not an official third-party API. | diff --git a/docs-site/src/content/docs/guides/web-dashboard.md b/docs-site/src/content/docs/guides/web-dashboard.md index 4ad5fd7c79..896a38bb9d 100644 --- a/docs-site/src/content/docs/guides/web-dashboard.md +++ b/docs-site/src/content/docs/guides/web-dashboard.md @@ -131,6 +131,18 @@ and other providers. - **Refresh quotas** re-reads account usage immediately so routing and the account cards use the same values. - Pool request logs use opaque labels such as `p3fa91c`, never account emails. +- **Target a specific Codex account from the model picker** is an explicit opt-in. When enabled, + ordinary supported GPT picker rows are replaced by one entry per public account selector. + Choosing one locks that conversation to the mapped account: it does not rotate, fall back, or + change the active Pool account. The built-in Codex App login has its own selector; generated maps + normally use `main`, with a collision-safe suffix such as `main-2` when needed. Added accounts + receive stable, privacy-safe labels, and existing custom selector labels are preserved. + Existing conversations and saved model selections continue routing. Turning the setting off + hides generated picker entries without deleting accounts, selectors, or exact routes. Plain GPT + model ids continue to use the configured Pool or Direct behavior. +- Account add, remove, and picker-setting changes are saved before the model catalog is refreshed. + If that bounded refresh cannot finish, the dashboard shows an amber success-with-recovery notice; + run `ocx sync` to retry. The account or setting change itself remains saved. The Providers overview separately summarizes Pool-mode usage as a display-only weighted capacity estimate, alongside the effective account's raw quota and the next capacity recovery. See @@ -166,7 +178,7 @@ The GUI is a thin client over the proxy's JSON management API. Useful endpoints | Endpoint | Purpose | | --- | --- | -| `GET` / `PUT /api/settings` | Read settings or toggle Codex autostart. | +| `GET` / `PUT /api/settings` | Read settings or update Codex autostart, stream/memory settings, and account-targeting picker visibility. | | `GET` / `POST /api/github/star` | Read the `gh`-derived star state, or star the repository. The POST is refused with `403` `agent_consent_required` for agent-driven callers without a dashboard session. | | `GET /api/startup-health` | Read secret-free routing, service, shim, and restart-safety diagnostics. | | `POST /api/startup-action` | Install the background service or Codex launcher shim through fixed, allowlisted actions. | diff --git a/docs-site/src/content/docs/ja/guides/providers.md b/docs-site/src/content/docs/ja/guides/providers.md index a5d15561b1..da7e633170 100644 --- a/docs-site/src/content/docs/ja/guides/providers.md +++ b/docs-site/src/content/docs/ja/guides/providers.md @@ -108,7 +108,7 @@ ocx logout | `anthropic` | `anthropic` | `https://api.anthropic.com` | Claude モデル; ライブモデル一覧は `/v1/models` から取得。 | | `kimi` | `openai-chat` | `https://api.kimi.com/coding/v1` | Kimi K2.7/K2.6/K2.5 コーディングモデル。 | | `kiro` | `kiro` | `https://runtime.us-east-1.kiro.dev` | 初回ログインは、インストール済みでサインインした `kiro-cli` セッションを取り込みます（Unix では `curl -fsSL https://cli.kiro.dev/install | bash`、Windows PowerShell では `irm 'https://cli.kiro.dev/install.ps1' | iex` でインストールしてから `kiro-cli login` を実行）。**アカウントを追加**は `kiro-cli` をログアウトして新しいブラウザログインを開始し、`kiro-cli` 自体のアカウントを切り替えてアカウント別プロファイルメタデータを保存します。既存の OpenCodex アカウントは保持され、キャンセルまたは失敗時には以前の `kiro-cli` セッションが復元されます。 | -| `google-antigravity` | `google` | `https://daily-cloudcode-pa.googleapis.com` | Google OAuth を Cloud Code Assist wire で使用。CCA は汎用 `/models` エンドポイントを公開しないため、管理された 6 モデルの静的カタログを使用します。 | +| `google-antigravity` | `google` | `https://daily-cloudcode-pa.googleapis.com` | Google OAuth を Cloud Code Assist wire で使用。ライブ探索は認証済みの CCA `v1internal:fetchAvailableModels` エンドポイントを使用し、ログイン中のアカウントで利用可能な agent モデルのみを公開します。管理されたカタログはフォールバックとして残ります。 | | `cursor` | `cursor` | `https://api2.cursor.sh` | 実験的 PKCE ログイン、HTTP/2 トランスポート、アカウント別モデル探索をサポート。 | | `github-copilot` | `openai-chat` | `https://api.githubcopilot.com` | 実験的。GitHub デバイスフロー + `copilot_internal` 交換（VS Code OAuth クライアント）。有効な Copilot サブスクリプションが必要で、公式のサードパーティ API ではありません。 | diff --git a/docs-site/src/content/docs/ja/guides/web-dashboard.md b/docs-site/src/content/docs/ja/guides/web-dashboard.md index 955b9cad51..c63c7a77e4 100644 --- a/docs-site/src/content/docs/ja/guides/web-dashboard.md +++ b/docs-site/src/content/docs/ja/guides/web-dashboard.md @@ -108,6 +108,15 @@ Codex タスクだけに適用され、このオプション自体が委任を 枠のうち最も高い使用率でスコア付けし、Go/Free プランは 30 日枠のみ使います。 - **クォータ更新**はアカウント使用量を即座に再読み込みし、ルーティングと画面のアカウントカードが同じ値を見るようにします。 - プールリクエストログにはメールの代わりに `p3fa91c` のような不透明なラベルを使います。 +- **モデルピッカーで使用する Codex アカウントを指定** は明示的な opt-in です。有効にすると、通常の + GPT picker 項目が公開 account selector ごとの項目に置き換わります。選択した会話はそのアカウントに + 固定され、Pool のローテーションや fallback は行われず、active Pool account も変わりません。組み込みの + Codex App login には専用 selector があり、生成 map では通常 `main`、衝突時は `main-2` のような安全な + suffix が使われます。追加アカウントには安定した privacy-safe label が割り当てられます。 + 既存の会話と保存済みのモデル選択は引き続きルーティングされます。無効にしても account、selector、 + exact route は削除されず、通常の GPT id は従来どおり Pool / Direct で動作します。 +- account の追加・削除と picker 設定は catalog refresh より先に保存されます。refresh が完了できない場合は + amber の回復案内が表示されます。変更自体は保存済みなので、`ocx sync` で refresh を再試行してください。 Providers の概要は、Pool モードの使用状況を表示専用の重み付き容量推定値として別途まとめ、現在の 有効アカウントの生のクォータと次の容量回復も併せて表示します。表示される項目、不完全な対象範囲の @@ -119,7 +128,7 @@ GUI はプロキシの JSON 管理 API を使うシンクライアントです | エンドポイント | 用途 | --- | --- | -| `GET` / `PUT /api/settings` | 設定を読むか Codex 自動起動をオン/オフします。 | +| `GET` / `PUT /api/settings` | 設定を読み、Codex 自動起動、stream/memory、account-targeting picker の表示を更新します。 | | `GET /api/startup-health` | 秘密情報を含まないルーティング、サービス、shim、再起動安全性診断を読み取ります。 | | `GET` / `POST /api/windows-tray` | Windows トレイの導入・表示状態を読み取り、`install`、`start`、`stop`、`uninstall` を実行します。 | | `POST /api/sync` | 共有モデルカタログを再構築し Codex モデルキャッシュを古い状態としてマークします。 | diff --git a/docs-site/src/content/docs/ja/reference/cli/providers-accounts.md b/docs-site/src/content/docs/ja/reference/cli/providers-accounts.md index 9c6e6edf7d..cab525623a 100644 --- a/docs-site/src/content/docs/ja/reference/cli/providers-accounts.md +++ b/docs-site/src/content/docs/ja/reference/cli/providers-accounts.md @@ -180,17 +180,20 @@ preemption が未バインドリクエストを直ちに引き上げます。既 ### `ocx account login|reauth|code|cancel ...` -ヘッドレス シェルからブラウザベースまたは手動コードのアカウント認証を実行します。プロバイダー固有のコマンド形式には `ocx account --help` を使用します。 +ヘッドレス シェルからブラウザベースまたは手動コードのアカウント認証を実行します。プロバイダー固有のコマンド形式には `ocx account --help` を使用します。Codex account login は保存済みでも catalog refresh が保留中なら成功終了し、human output の stderr に固定の `ocx sync` 案内を出します。`--json` は案内を混ぜず、完了 state に `catalogRefreshPending: true` を保持します。 ### `ocx account remove --yes [--json]` この保護された非対話型削除には `--yes` が必要です。削除する前に、ID が存在することが確認されます。 ID が欠落している場合は、DELETE を送信せずに 1 が終了します。メインの Codex App ログインは削除できないため、`remove openai main --yes` は拒否されます。削除後、ファミリーは再度読み取られます。固定された Codex アカウントを削除すると、ピンがクリアされ、自動選択に戻ります。 OAuth は最初に残ったアカウントを昇格させるか、何も報告しません。 API キー プールは、最初に残っているキーを昇格するか、何も報告しません。 `--json` の成功と失敗の形状は次のとおりです。 ```text -{ ok: true, provider, id, removedActive: boolean, promotedActiveId: string | null } +{ ok: true, provider, id, removedActive: boolean, promotedActiveId: string | null, catalogRefreshPending?: boolean } { error: string } // stderr, exit 1 ``` +`catalogRefreshPending` は Codex 削除だけに含まれます。`true` でも削除は保存済みで、human output は +stderr に `ocx sync` の案内を出して終了コード 0 のままです。OAuth account と API key の削除形状は変わりません。 + ### `ocx account add-key [--label