docs(lab): define compatibility evidence and conformance contracts - #1286
Conversation
Freeze the CL-00 evidence, verdict, scenario, identity, privacy, and incident contracts so conformance implementation can proceed without duplicating routing authority.
|
Note Reviews pausedIt looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the Use the following commands to manage reviews:
Use the checkboxes below for quick actions:
📝 WalkthroughWalkthroughThis PR adds the CL-00 Compatibility Lab plan and normative contracts. It defines evidence, protocol scenarios, manifests, fixture cases, incident specifications, security controls, privacy rules, and acceptance status. CL-00 is accepted. CL-01 remains unauthorized. ChangesCompatibility Lab CL-00
Estimated code review effort: 4 (Complex) | ~60 minutes Possibly related PRs
Suggested reviewers: 🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Record the accepted implementation head and draft pull request so the programme ledger is complete without authorizing CL-01.
Remove Markdown hard-break spaces so the final branch diff passes repository whitespace checks.
There was a problem hiding this comment.
Actionable comments posted: 13
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@devlog/_plan/260807_compatibility_lab/000_master_plan.md`:
- Around line 265-269: Update the CL-01 Authorization entry in the phase table
to replace “requires accepted CL-00” with clear grammatical wording such as
“requires CL-00 acceptance” or “requires CL-00 to be accepted,” preserving the
intended acceptance gate.
In
`@devlog/_plan/260807_compatibility_lab/010_architecture_and_evidence_contract.md`:
- Around line 457-462: Update the dependency ordering contract to define a total
deterministic order for RouteDependencyV1 records. Extend the sort key after
role, provider ID, upstream model ID, and endpoint fingerprint with
clientModelId, effectiveAdapter, upstreamProtocol, and behaviorFingerprint,
using a stable documented ordering; alternatively, explicitly reject duplicate
four-field prefixes.
- Around line 14-17: Define a provider-neutral ProtocolSubjectV1 for
protocol_conformance instead of requiring RouteSubjectV1 provider fields, with
deterministic values for the OpenCodex compatibility version, adapter,
protocols, surface, and behavior fingerprint. Update subject hashing and
projection matching to use this protocol subject, and document explicit omission
or stable synthetic-value rules for route-only fields.
In `@devlog/_plan/260807_compatibility_lab/021_protocol_v1_manifest_authority.md`:
- Around line 82-121: Unify negative-control verdict semantics across all three
sites: in
devlog/_plan/260807_compatibility_lab/021_protocol_v1_manifest_authority.md
lines 82-121, include negative_control scenarios in verificationRule and define
that their expected rejection satisfies the control; in
devlog/_plan/260807_compatibility_lab/020_scenario_contract_and_catalogue.md
lines 177-192, remove or revise the rule allowing a negative control to prove
UNSUPPORTED; and in
devlog/_plan/260807_compatibility_lab/021_protocol_v1_manifest_authority.md
lines 313-320, keep the final rule explicit that expected rejection does not
produce UNSUPPORTED.
- Around line 191-208: The Anthropic SSE parser in the vision path must conform
to the V1 normalization contract: reuse or update the shared SSE normalizer to
join repeated data fields with LF, normalize bare CR, honor explicit event
fields over inferred data.type, treat malformed nonempty JSON as terminal, and
reject [DONE] as a sentinel outside Chat surfaces. Keep data-only type inference
for valid object payloads and ensure padding values emit no event.
In `@devlog/_plan/260807_compatibility_lab/022_protocol_v1_cases.json`:
- Line 225: Restore valid JSON in
devlog/_plan/260807_compatibility_lab/022_protocol_v1_cases.json by replacing
raw line feeds with escaped \n sequences in the arguments values at lines
225-225 and 283-283; preserve the represented multiline tool arguments and all
other case data.
In `@devlog/_plan/260807_compatibility_lab/040_security_and_privacy.md`:
- Line 13: Update the user MCP prohibition in the compatibility lab plan to
remove “by default” and make it an unconditional CL-00 rule: no override or CLI
mode may load user MCP server definitions, resources, results, or credentials.
Keep it consistent with the synthetic-only boundary and the existing statement
that user MCP configuration is not loaded.
- Around line 80-82: Update the environment inheritance policy to reject
HTTP_PROXY, HTTPS_PROXY, and ALL_PROXY by default. If proxy routing is needed,
require separately authorizing the exact proxy endpoint with the same SSRF
validation as other destinations, while keeping proxy credentials bound to their
intended destination.
- Around line 43-44: Update the redirect and DNS security requirements to define
one fail-closed policy, preferably rejecting redirects as established by the
SSRF tests. If redirects remain supported, require each hop to be authorized,
connect only to its validated IP, and preserve the intended Host and SNI values.
- Around line 121-126: Expand the normalized event limits in the security and
privacy plan to include maximum serialized event bytes, aggregate event bytes,
nesting depth, map keys, and array elements. Specify that these bounds must be
validated before normalization buffers each event, alongside the existing
event-count and string-size limits, so oversized or deeply nested structures
cannot consume excessive memory or CPU.
- Around line 149-152: Define explicit maximum retention durations and cleanup
deadlines for scenario/suite manifests, synthetic fixtures, JSONL, SQLite,
artifacts, temporary files, and export staging, rather than retaining referenced
data indefinitely under storage policy. Update the retention language at the
manifest/fixture rules and the storage-policy section, and add a test that
confirms each class is cleaned up after its maximum retention period.
- Around line 222-225: Revise the evidence-handling policy to distinguish
non-sensitive invalid evidence, which may be neutralized by appending an
invalidation, from confirmed sensitive evidence. Make sensitive-evidence purge
mandatory and fail closed: delete it from the local ledger, SQLite, logs,
temporary files, and generated exports, and ensure incident records never
contain the leaked value.
- Around line 50-54: Separate the sidecar’s trusted in-memory connection
destination from its persisted subject identity: represent the raw endpoint only
in the authorization/runtime destination record, and use a keyed opaque endpoint
identity in the composite subject. Update the requirements around the sidecar
authorization and the local subject sections (including lines 179–185) so
subjects never persist or export raw base URLs while the runner still has a
usable connection target.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: ASSERTIVE
Plan: Pro Plus
Run ID: 8607a26c-f137-4c47-b55f-e692eaed2452
📒 Files selected for processing (9)
devlog/_plan/260807_compatibility_lab/000_master_plan.mddevlog/_plan/260807_compatibility_lab/001_pr_stack_status.mddevlog/_plan/260807_compatibility_lab/010_architecture_and_evidence_contract.mddevlog/_plan/260807_compatibility_lab/020_scenario_contract_and_catalogue.mddevlog/_plan/260807_compatibility_lab/021_protocol_v1_manifest_authority.mddevlog/_plan/260807_compatibility_lab/022_protocol_v1_cases.jsondevlog/_plan/260807_compatibility_lab/030_incident_corpus.mddevlog/_plan/260807_compatibility_lab/040_security_and_privacy.mddevlog/_plan/260807_compatibility_lab/050_cl00_acceptance_review.md
Freeze layer-specific subjects, claim supersession, closed behavior keys, scenario-local expectedFailure expansion, and canonical toolCall projections so CL-01 needs no semantic invention.
Record the accepted delayed-review correction commit on the stack ledger and acceptance review.
Keep the stack ledger and acceptance review pointed at the delayed-review contract correction commit rather than chasing tip SHA churn.
Tighten destination/proxy sandbox wording, event bounds, retention and sensitive purge rules, dependency total order, and CL-01 gate phrasing.
There was a problem hiding this comment.
Actionable comments posted: 10
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
devlog/_plan/260807_compatibility_lab/010_architecture_and_evidence_contract.md (1)
589-596: 🗄️ Data Integrity & Integration | 🟠 Major | ⚡ Quick winInclude
providerInstanceFingerprintin the dependency sort key.Lines 590-592 make
providerInstanceFingerprintpart ofRouteDependencyV1. Lines 593-596 omit it from the ordering. Two dependencies can therefore differ only by this fingerprint while comparing equal on every listed sort field. Their input order then changesdependencies[],RouteSubjectV1.subjectId, and derived evidence keys.Add
providerInstanceFingerprintafterproviderIdand test this tie case.Proposed contract fix
- role, provider ID, upstream model ID, endpoint fingerprint, client model - ID, effective adapter, upstream protocol, then behavior fingerprint. + role, provider ID, provider-instance fingerprint, upstream model ID, + endpoint fingerprint, client model ID, effective adapter, upstream protocol, + then behavior fingerprint.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@devlog/_plan/260807_compatibility_lab/010_architecture_and_evidence_contract.md` around lines 589 - 596, Update the RouteDependencyV1 dependency ordering contract to include providerInstanceFingerprint immediately after providerId in the total UTF-8 string comparison key. Add or update coverage for dependencies differing only by providerInstanceFingerprint, ensuring their sorted order and derived subject/evidence identifiers are input-order independent.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@devlog/_plan/260807_compatibility_lab/001_pr_stack_status.md`:
- Around line 22-23: Update the CL-01 row so its Base SHA field is explicitly
“not started,” and move the “CL-00 must be accepted first” dependency into the
State or notes field. In the CL-00 row, rename the current implementation-head
field to “Accepted contract head,” or add and maintain a separate current
ledger-head field that is updated after every sync commit, including the
ledger-sync guidance in lines 70-72.
In
`@devlog/_plan/260807_compatibility_lab/010_architecture_and_evidence_contract.md`:
- Around line 697-700: Update the V1 behavior-fingerprint allowlist and
canonicalization contract to declare an ordering mode and element-comparison
rule for every array-valued key, covering both set-like and order-sensitive
arrays. Make canonicalization require one of these declarations and reject any
undeclared ordering mode, preserving deterministic ProtocolSubjectV1 and
RouteSubjectV1 identity.
In
`@devlog/_plan/260807_compatibility_lab/020_scenario_contract_and_catalogue.md`:
- Around line 197-199: Update the VERIFIED definition in the scenario contract
so it requires at least one applicable required scenario, preventing success
with zero observations. Explicitly define the outcome for required scenarios
that are inapplicable—UNKNOWN or BLOCKED—and align the rule with the
current-observation requirement established in the architecture and evidence
contract.
In `@devlog/_plan/260807_compatibility_lab/022_protocol_v1_cases.json`:
- Around line 218-219: Update the `result` assertions in
`tools-core.protocol.function-round-trip` and
`codex-core.protocol.apply-patch-turn` to target the Chat request’s `/messages`
structure produced by `messagesToChatFormat`, using the tool result’s
`tool_call_id` correlation field. Replace only the invalid `/input/.../call_id`
selectors for these `openai-chat` cases, and preserve `/input` selectors for
Responses cases.
- Around line 147-149: Clarify the sentinel source for the Chat fixtures in the
protocol manifest by either adding an explicit Chat-facing surface to the
affected responses-SSE cases or defining that upstreamProtocols ["openai-chat"]
selects Chat SSE [DONE] handling. Update the manifest authority and dependent
fixtures consistently so the terminal signal resolves to "completed" rather than
failing closed.
In `@devlog/_plan/260807_compatibility_lab/040_security_and_privacy.md`:
- Around line 72-75: Update the environment inheritance policy to enumerate the
exact variable names and permitted values, and require runners to reject every
variable outside that allowlist. Explicitly include lowercase http_proxy,
https_proxy, all_proxy, and no_proxy alongside the uppercase proxy names in the
rejection rules, preserving the requirement that proxy endpoints are separately
authorized under SSRF checks.
- Around line 285-286: Revise the invalidation and secure-deletion requirement
so shared scenario, suite, and fixture artifacts remain available while any
non-invalidated observation references their digest. Delete only non-contract
artifacts, or condition deletion on confirming no valid observation still uses
the artifact, preserving reproducible VERIFIED projections.
- Around line 294-311: Add deterministic acceptance tests covering custom-header
canonicalization and unknown-classification failures, salt-rotation correlation
invalidation and re-projection, destination-record mutation between
authorization and connection, and retention expiry with cleanup retry/failure
and unavailable markers. Extend the existing canary, purge, and network test
coverage, and require these tests to pass before live-runner authorization.
- Around line 41-45: Add an immutability contract to the LabDestinationV1
authorization description: require provider destination/credential plumbing to
create a per-run immutable destination snapshot, preserve it unchanged from
fingerprinting through credential binding and connection, and reject any
attempted mutation.
- Around line 228-237: Add explicit limits for custom-header count, field-name
length, UTF-8 value length, aggregate bytes, and duplicate-value lists before
the JCS construction described in the custom-header fingerprint flow. Reuse the
request-builder limits where applicable or define broker-specific maximums, and
return harness_failure whenever any limit is exceeded before canonicalization or
HMAC generation.
---
Outside diff comments:
In
`@devlog/_plan/260807_compatibility_lab/010_architecture_and_evidence_contract.md`:
- Around line 589-596: Update the RouteDependencyV1 dependency ordering contract
to include providerInstanceFingerprint immediately after providerId in the total
UTF-8 string comparison key. Add or update coverage for dependencies differing
only by providerInstanceFingerprint, ensuring their sorted order and derived
subject/evidence identifiers are input-order independent.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: ASSERTIVE
Plan: Pro Plus
Run ID: 91a4fb41-d1dc-4933-b791-6de65bd62fca
📒 Files selected for processing (8)
devlog/_plan/260807_compatibility_lab/000_master_plan.mddevlog/_plan/260807_compatibility_lab/001_pr_stack_status.mddevlog/_plan/260807_compatibility_lab/010_architecture_and_evidence_contract.mddevlog/_plan/260807_compatibility_lab/020_scenario_contract_and_catalogue.mddevlog/_plan/260807_compatibility_lab/021_protocol_v1_manifest_authority.mddevlog/_plan/260807_compatibility_lab/022_protocol_v1_cases.jsondevlog/_plan/260807_compatibility_lab/040_security_and_privacy.mddevlog/_plan/260807_compatibility_lab/050_cl00_acceptance_review.md
There was a problem hiding this comment.
Actionable comments posted: 2
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@devlog/_plan/260807_compatibility_lab/040_security_and_privacy.md`:
- Around line 301-305: Define the post-purge handling in the retention and
sensitive-purge rules: when evidence is removed, invalidate every affected
observation, exclude it from canonical projections, and emit typed unavailable
markers that contain no leaked value. Update the associated retention and purge
tests to cover shared contract artifacts and event lines, ensuring no dangling
references or unreproducible VERIFIED projections remain.
- Around line 243-247: Clarify the canonicalization contract before hashing to
explicitly group header entries by lowercase name into one values array per
name, preserving input order. Ensure every grouped value counts toward the
16-value duplicate limit, while retaining the existing bounds and
harness_failure/no-digest behavior.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: ASSERTIVE
Plan: Pro Plus
Run ID: 74c168a5-0c82-4b05-988f-60b1cb27cbc3
📒 Files selected for processing (1)
devlog/_plan/260807_compatibility_lab/040_security_and_privacy.md
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@devlog/_plan/260807_compatibility_lab/021_protocol_v1_manifest_authority.md`:
- Line 428: Update the Markdown file containing the CL-00 ingestion
implementation text so it ends with exactly one trailing newline, resolving the
MD047 markdownlint violation without changing its content.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: ASSERTIVE
Plan: Pro Plus
Run ID: 7792a561-0f6b-4de6-a431-5aa4699c0acf
📒 Files selected for processing (3)
devlog/_plan/260807_compatibility_lab/010_architecture_and_evidence_contract.mddevlog/_plan/260807_compatibility_lab/020_scenario_contract_and_catalogue.mddevlog/_plan/260807_compatibility_lab/021_protocol_v1_manifest_authority.md
There was a problem hiding this comment.
Actionable comments posted: 10
♻️ Duplicate comments (1)
devlog/_plan/260807_compatibility_lab/040_security_and_privacy.md (1)
250-251:⚠️ Potential issue | 🟠 MajorDefine one JCS entry per lowercase header name.
Lines 250-251 do not state whether duplicate fields are grouped into one object or emitted as separate objects. Those encodings produce different
headers.nonCredentialBehaviorDigestvalues and route subjects.Require grouping by lowercase name. Preserve input order inside each
valuesarray. Count every grouped value toward the 16-value limit.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@devlog/_plan/260807_compatibility_lab/040_security_and_privacy.md` around lines 250 - 251, Clarify the header canonicalization step to emit exactly one JCS entry per lowercase header name, grouping all duplicate fields under that entry. Preserve the original input order within each values array, and count every grouped value toward the 16-value limit.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In
`@devlog/_plan/260807_compatibility_lab/010_architecture_and_evidence_contract.md`:
- Around line 590-597: Update the dependency sort order specification for
RouteDependencyV1 to include providerInstanceFingerprint, preferably immediately
after provider ID, so dependencies differing only by provider instance receive a
deterministic order and duplicate detection remains based on the full key.
- Around line 117-129: Define the invalidation event payload with a required,
non-empty list of target event IDs and a closed set of reason codes. Add
validation that rejects missing or empty targets, duplicate target IDs, unknown
or invalid target references, and unsupported reasons, while preserving
append-only invalidation behavior.
- Around line 183-200: Extend the claim contract around sourceManifestDigest to
define a canonical ClaimSourceManifestV1 object, persist its bytes as a
content-addressed artifact, and retain it while referenced by any claim. During
projection replay, load the retained manifest and validate that its canonical
content digest matches sourceManifestDigest before accepting the historical
CLAIMED result.
In `@devlog/_plan/260807_compatibility_lab/021_protocol_v1_manifest_authority.md`:
- Around line 203-206: Define a closed, scenario-specific action schema for
synthetic_tool covering stub registration, deterministic invocation, and
response generation, including the source of client-visible tool calls. Update
the synthetic_tool harness semantics and the affected MCP cases in
022_protocol_v1_cases.json so each asserted mcpCalls entry is produced by an
explicit action sequence rather than only a tool definition.
In `@devlog/_plan/260807_compatibility_lab/030_incident_corpus.md`:
- Line 390: Update the incident corpus entry’s terminology by replacing “falsey
error payload” with “falsy error payload,” leaving the other listed failure
cases unchanged.
In `@devlog/_plan/260807_compatibility_lab/040_security_and_privacy.md`:
- Around line 306-311: Define the post-purge state for every observation and
canonical projection that references deleted evidence: invalidate affected
observations, remove them from projections, and emit typed unavailable markers
containing no leaked value. Update the sensitive-purge behavior and its tests to
cover shared artifacts and event lines, ensuring no dangling references or
retained VERIFIED projections remain.
- Around line 25-27: Define mandatory machine-checkable marker and provenance
fields for every Lab-authored synthetic fixture, then enforce them during
manifest validation so unmarked or incomplete fixtures are rejected. Update the
required acceptance tests to cover valid synthetic metadata and rejection of
missing or invalid markers, including the scenarios and packaged fixtures
governed by this contract.
- Around line 114-116: Update the V1 resource-limits statement near the run
enforcement requirements to define explicit maximum ceilings for wall-clock,
inactivity, byte, request, token, tool-call, memory, and process resources.
State that scenario-specific limits may only lower these maxima, and preserve
the requirement that inability to enforce any required limit results in
harness_failure.
- Around line 91-96: The credential broker API must expose only a
destination-bound opaque transport capability, keeping selected credential bytes
within request and credential plumbing rather than making them readable by Lab
or probe code. Strengthen the canary test near the existing Lab isolation checks
to verify probe code cannot inspect or serialize the selected credential, while
preserving the existing prohibitions on auth-store and unrelated-credential
access.
- Around line 184-187: Update artifact validation and consumption around
resolveArtifactPath() so validation uses a no-follow opened descriptor rather
than existsSync/statSync followed by a path read. Verify the digest and size
from the bytes read through that same descriptor before consuming them,
preventing symlink traversal and replacement after validation. For newly created
artifacts, use exclusive no-follow creation with flag "wx" or an equivalent
atomic temporary-file-and-rename flow.
---
Duplicate comments:
In `@devlog/_plan/260807_compatibility_lab/040_security_and_privacy.md`:
- Around line 250-251: Clarify the header canonicalization step to emit exactly
one JCS entry per lowercase header name, grouping all duplicate fields under
that entry. Preserve the original input order within each values array, and
count every grouped value toward the 16-value limit.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: ASSERTIVE
Plan: Pro Plus
Run ID: ac365095-720a-4216-8dc9-887d4e8c8dc9
📒 Files selected for processing (9)
devlog/_plan/260807_compatibility_lab/000_master_plan.mddevlog/_plan/260807_compatibility_lab/001_pr_stack_status.mddevlog/_plan/260807_compatibility_lab/010_architecture_and_evidence_contract.mddevlog/_plan/260807_compatibility_lab/020_scenario_contract_and_catalogue.mddevlog/_plan/260807_compatibility_lab/021_protocol_v1_manifest_authority.mddevlog/_plan/260807_compatibility_lab/022_protocol_v1_cases.jsondevlog/_plan/260807_compatibility_lab/030_incident_corpus.mddevlog/_plan/260807_compatibility_lab/040_security_and_privacy.mddevlog/_plan/260807_compatibility_lab/050_cl00_acceptance_review.md
Rebase onto dev CL-00 merge, remove Chat→Responses observation projection, apply source-protocol SSE [DONE] rules, synthetic provenance, MCP actions, and Chat tool_call_id selectors per final Protocol V1 authority.
Summary
Verification
bun run typecheck— passed.bun run privacy:scan— passed.bun test tests/repo-hygiene.test.ts— 11 passed, 0 failed.git diff --check— passed.bun run test— not green on this Windows/Bun 1.3.14 host. It exited 3 after a cache-invalidation failure, an empty Windows effective-account lookup, and a Bunindex out of boundspanic. Serial isolation passed: cache invalidation 6/6; native residue 63 passed with 2 platform skips. A broader focused run had 488 pass and 4 WindowsEPERMsymlink-creation failures.Checklist
Summary by CodeRabbit