-
Notifications
You must be signed in to change notification settings - Fork 665
feat(lab): CL-01 deterministic protocol conformance harness #1320
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from all commits
Commits
Show all changes
22 commits
Select commit
Hold shift + click to select a range
cfe27b0
feat(lab): add CL-01 deterministic protocol conformance harness
Wibias 574f1d5
fix(lab): align CL-01 harness with merged CL-00 #1286 contracts
Wibias 22d608c
docs(lab): pin CL-01 contract-correction head SHA
Wibias d665004
docs(lab): sync CL-01 tip SHA after contract correction
Wibias cb4417d
fix(lab): harden CL-01 conformance review findings
Wibias 5639c6c
fix(lab): close remaining Protocol V1 gaps
Wibias b16670f
fix(lab): harden tool-call projections
Wibias f5ddee4
fix(lab): fail closed on malformed controls
Wibias 7814484
test(lab): cover review regression edges
Wibias 5aab6d9
test(openai-chat): pin native CL-01 regressions
Wibias f79fc9e
fix(openai-chat): use native reasoning effort field
Wibias 554f6f6
test(claude): pin initial failure framing
Wibias b86b303
fix(claude): preserve initial error framing
Wibias ee5efaa
fix(lab): preserve nonstream fixture fallback
Wibias 2868543
test(lab): preserve empty nonstream fixture fallback
Wibias 3328f1b
fix(lab): isolate gateway reasoning fixture
Wibias b44a07a
fix(lab): correct gateway reasoning authority
Wibias 0b85150
test(claude): cover created-then-failed framing
Wibias a82977d
fix(claude): defer message start until semantic output
Wibias 9288916
test(openai-chat): retain fallback for untransportable tool images
Wibias 577819b
fix(openai-chat): preserve fallback for untransportable tool images
Wibias af962ad
test(claude): keep idle pings scoped to semantic streams
Wibias File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
50 changes: 50 additions & 0 deletions
50
devlog/_plan/260807_compatibility_lab/051_cl01_acceptance_review.md
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,50 @@ | ||
| # CL-01 independent acceptance review | ||
|
|
||
| Reviewer posture: adversarial. Scope: deterministic protocol conformance harness only. | ||
|
|
||
| **Revision note:** CL-01 was **accepted earlier** at `cc447ce9d19d5fb4e03988899f5fb495f9de8d0e` against pre-remediation CL-00 tip `c2113ca47b8a05c5a5f90679e4eaa640ca2c6a66`. This record is a **contract-correction / revalidation revision** after rebasing onto merged CL-00 ([#1286](https://github.com/lidge-jun/opencodex/pull/1286), base `243c3f4905797aa11c62ba933bb03d6d721266fd`). | ||
|
|
||
| ## Invalidated earlier assumptions | ||
|
|
||
| | Earlier CL-01 assumption | Final CL-00 correction | | ||
| |---|---| | ||
| | Chat upstream tool results correlate via synthetic Responses `input[]` in observations | Real Chat wire: `/upstream/requests/N/json/messages/M/tool_call_id` | | ||
| | SSE `[DONE]` inferred from client surface labels (`responses-sse`, etc.) | Sentinel follows **source protocol** of normalized byte stream; only `openai-chat` recognizes `[DONE]` | | ||
| | Expanded manifests without synthetic marker/provenance | Mandatory `syntheticMarker: "ocx-lab-synthetic-v1"` + `lab_authored` provenance in every fixture ref | | ||
| | MCP scenarios implicit / unspecified | Four closed action tokens with deterministic semantics | | ||
| | Obsolete manifest digests from pre-provenance expansion | All scenario manifest digests recomputed with provenance fields | | ||
|
|
||
| ## Removed workaround | ||
|
|
||
| The harness **removed** `normalizeUpstreamObservationJson()` Chat `messages[]` → synthetic Responses `input[]` projection. Observations now record actual upstream JSON from shipped adapters. Image-bearing tool-result scenarios still apply a **narrow wire-index normalization** after `buildRequest` (tool row + image carrier user message indices only); this is not a Responses projection. | ||
|
|
||
| ## Challenge results (revalidation) | ||
|
|
||
| | # | Challenge | Result | | ||
| |---|---|---| | ||
| | 1 | Harness exercises shipped parser/translation, not a parallel stack | **PASS** — executor calls `parseRequest`, `createOpenAIChatAdapter`, `createResponsesPassthroughAdapter`, `bridgeToResponsesSSE`, `responsesSseToAnthropicSse`, and `expandPreviousResponseInput`. | | ||
| | 2 | Negative controls genuinely fail | **PASS** — eight deliberate broken fixtures reject (`runNegativeControls` 8/8). | | ||
| | 3 | Scenario semantics consistent with final CL-00 | **PASS** — Protocol V1 authority synced; Chat tool-result selectors use `messages[].tool_call_id`; no Responses `input[]` fabrication. | | ||
| | 4 | Malformed/partial streams cannot accidentally pass | **PASS** — malformed SSE negative control fails; truncated tool args fail `tool_call_equals`. | | ||
| | 5 | Tool IDs and tool-result correlations verified | **PASS** — `tools-core.protocol.function-round-trip`, `codex-core.protocol.apply-patch-turn` use Chat wire selectors. | | ||
| | 6 | Parallel tool fragments handled | **PASS** — `tools-core.protocol.parallel-correlation` and `nonoverlap_order` verifier. | | ||
| | 7 | Custom/freeform tools covered | **PASS** — `apply_patch` via `freeformToolNames` in bridge. | | ||
| | 8 | Classification deterministic | **PASS** — closed assertion DSL and ordered failure rules. | | ||
| | 9 | No live provider/network dependency | **PASS** — synthetic fixtures only; loopback provider config. | | ||
| | 10 | No CL-02 functionality leaked | **PASS** — no ledger, SQLite, CLI probe, or live runners. | | ||
| | 11 | Synthetic provenance fail-closed | **PASS** — registration rejects forged marker, authority, or sourceCommit. | | ||
| | 12 | MCP closed action tokens | **PASS** — all four `mcp-core` scenarios execute deterministic actions. | | ||
| | 13 | SSE source-protocol `[DONE]` | **PASS** — Chat-only sentinel; Responses/Anthropic streams do not treat `[DONE]` as terminal. | | ||
|
|
||
| ## Validation (2026-08-09, Windows/Bun 1.3.14) | ||
|
|
||
| - `bun x tsc --noEmit`: passed | ||
| - `bun test tests/lab-conformance-harness.test.ts`: **14/14** passed (24 canonical + 8 negative controls + provenance + SSE + MCP + manifest tests) | ||
| - `git diff --check`: passed (after correction) | ||
| - Full `bun run test`: not re-run (known Windows/Bun baseline failures documented under CL-00) | ||
|
|
||
| ## Verdict | ||
|
|
||
| **CL-01: ACCEPTED (contract-corrected revalidation)** — harness conforms to merged CL-00 #1286, passes all CL-01 canonical scenarios and negative controls, implements provenance and MCP action contracts, and contains no CL-02 scope. | ||
|
|
||
| **CL-02: NOT STARTED.** | ||
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.