feat(lab): CL-03 bounded live-route probes - #1352
Merged
Merged
Conversation
Add live manifest authority, sandboxed runner with injectable transport, RouteSubjectV1 builder, live persistence seam, and projection applicability for ten frozen live_route_compatibility scenarios.
Contributor
|
Note Reviews pausedIt looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the Use the following commands to manage reviews:
Use the checkboxes below for quick actions:
📝 WalkthroughWalkthroughThis PR adds CL-03 live-route compatibility execution. It defines authoritative scenarios, immutable route identities, sandbox and transport controls, trusted evidence persistence, route applicability, projection verdicts, and validation coverage. ChangesCL-03 live route compatibility
Estimated code review effort: 4 (Complex) | ~60 minutes Sequence Diagram(s)sequenceDiagram
participant LiveRunner
participant LiveExecutor
participant PinnedTransport
participant TrustedSender
participant ObservationLedger
LiveRunner->>LiveExecutor: runLiveScenario(case, route)
LiveExecutor->>PinnedTransport: execute request with limits and lease
PinnedTransport->>TrustedSender: send pinned GET or POST
TrustedSender-->>PinnedTransport: return status, headers, and body
PinnedTransport-->>LiveExecutor: return normalized transport result
LiveExecutor-->>LiveRunner: return scenario result and classification
LiveRunner->>ObservationLedger: persist trusted live result
Possibly related PRs
Suggested reviewers: 🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Contributor
|
✅ Deterministic PR hygiene checks passed. |
Wibias
marked this pull request as ready for review
August 9, 2026 17:29
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Actionable comments posted: 16
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@devlog/_plan/260807_compatibility_lab/001_pr_stack_status.md`:
- Around line 183-184: Correct both live-test result records to use the verified
totals for the 13 top-level tests in tests/lab-live-probe.test.ts and
tests/lab-live-sandbox.test.ts, or rerun both commands and record their exact
output. Update devlog/_plan/260807_compatibility_lab/001_pr_stack_status.md
lines 183-184 and
devlog/_plan/260807_compatibility_lab/054_cl03_implementation_record.md lines
50-51 consistently.
In `@devlog/_plan/260807_compatibility_lab/024_live_v1_cases.json`:
- Around line 1-6: The live manifest currently lacks a parity check with the
runtime fixture. Update the authority-loading or conformance test flow around
loadLiveCaseAuthority() to assert that the documented manifest and
runtime-loaded cases are byte-identical, or generate the runtime copy from the
documented authority so drift is detected and prevented.
In `@src/lab/conformance/fixtures/live-v1-cases.json`:
- Around line 143-153: Update vision-core.live.synthetic-ocr in
src/lab/conformance/fixtures/live-v1-cases.json lines 143-153 and
devlog/_plan/260807_compatibility_lab/024_live_v1_cases.json lines 143-153 so
requirements.upstreamProtocols uses ["openai-responses"], matching the existing
Responses-shaped upstream_response fixture; keep both manifest copies
byte-identical and do not re-author the fixture.
- Around line 34-50: Remove the redundant maxArtifactBytes entry from
executionLimits, retaining perArtifactBytes as the single typed and observed
per-artifact limit. Apply the identical change in
src/lab/conformance/fixtures/live-v1-cases.json (lines 34-50) and
devlog/_plan/260807_compatibility_lab/024_live_v1_cases.json (lines 34-50); keep
artifactPolicy unchanged.
In `@src/lab/live/destination.ts`:
- Around line 93-95: Update the catch block around resolvePublicAddresses in the
destination resolution flow to throw LabDestinationError with a fixed, host-free
message instead of error.message. Preserve the existing "network_blocked" code
and match the fixed-string pattern used by the other LabDestinationError
instances; do not propagate external error text into diagnostics.
- Around line 117-121: Update assertDestinationAddressSet so both
destination.addresses and addresses are passed through canonicalAddresses before
mapping and joining them for comparison. Remove the direct map/sort
canonicalization on side a, preserving the existing mismatch error and canonical
representation.
In `@src/lab/live/executor.ts`:
- Around line 200-205: Update the catch path in the live executor around
classifyTransportError and complete so diagnostics contains only an allowlisted
error code, never error.name or error.message. Preserve raw errors only through
a non-exported local logging path that applies secret redaction, and ensure
complete cannot serialize the raw error or provider response details.
- Around line 159-163: Move the routePreconditionFailure check in the live-run
flow before createLabDestination and buildRouteSubjectV1. Return the existing
inconclusive complete result immediately when a precondition fails, preserving
its current fields and ensuring destination construction only occurs for
approved routes.
In `@src/lab/live/types.ts`:
- Around line 184-197: Narrow LiveScenarioRunResult.classification from string
to FailureClassification and import FailureClassification in the live
observation conversion flow. Update from-live.ts to use the typed classification
with explicit in checks, preserving the existing "protocol_failure" route
attribution and preventing environment/test-fault or inconclusive
classifications from being stored as route failures.
In `@src/lab/observe/from-live.ts`:
- Around line 31-33: Replace the caller-controlled executionAuthority check in
observationFromLiveResult with validation of a module-private trusted-result
receipt created by the executor. Bind and verify the receipt against the
canonical authority digest, scenario ID, suite ID, and routeSubject before any
artifact writes, rejecting missing or mismatched receipts and case/authority
combinations; keep the receipt inaccessible to public callers.
In `@src/lab/projection/verification.ts`:
- Around line 55-72: Update RouteSubjectV1 to persist canonical route claims,
then change routeSubjectApplicableToRequirements to derive required-claim
applicability exclusively from that subject and remove caller-controlled
routeRequiredClaims input. In the live_route_compatibility verification flow,
return canVerify: false when opts.subject is absent or has subjectKind
"protocol" instead of evaluating live-route evidence. Add coverage for omitted
claims and protocol subjects.
- Around line 102-105: Update the requiredClaims handling in the scenario
verification flow to default to an empty array only when row.requiredClaims is
undefined; when the field is present, call parseStringArray and return null if
parsing fails. Preserve the existing valid-array behavior and fail-closed
handling used by the other required arrays.
In `@src/lab/subject/installation-salt.ts`:
- Around line 10-21: Update readInstallationSalt to create the salt file
exclusively instead of using the existsSync check-then-write flow. Handle an
EEXIST failure by re-reading and validating the existing file, while preserving
the current permissions, salt length validation, and returned bytes; add a
focused regression test alongside the existing salt/subject tests covering
repeated reads for the same configDir.
In `@src/lib/lab-live-execution-authority.ts`:
- Around line 23-35: Remove the exported createTrustedLabRouteExecutor factory
as a caller-forgeable trust boundary, and move trusted-executor capability
issuance into the trusted host integration that can verify actual sandbox
enforcement. Update the TRUSTED_EXECUTORS registration flow and consumers such
as the live executor’s trusted_route handling to accept only host-issued
capabilities, while preserving boundary validation internally. Add a regression
test proving adapters or other untrusted runtime code cannot construct a trusted
capability.
In `@src/lib/lab-live-pinned-sender.ts`:
- Around line 24-27: Update the response-header handling in the pinned sender to
copy only the allowlisted headers required by live assertions, currently
`content-type`, instead of iterating over every provider header. Preserve
case-insensitive lookup by normalizing the selected header name, and ensure no
credential, cookie, account, organization, or rate-limit headers enter
`LabTransportResponse.headers`.
- Around line 14-20: Propagate the configured connect timeout through the pinned
live transport: add a connect-timeout field to PinnedHttpRequestOptions, pass
limits.connectTimeoutMs from the options created in the pinned sender, and
enforce it during DNS lookup and request connection before idleTimeoutMs begins.
Preserve the existing connect_timeout boundary advertisement only once this
deadline is actually enforced.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: ASSERTIVE
Plan: Pro Plus
Run ID: 99a0fef2-89e3-42f6-bae0-2e541b12658b
📒 Files selected for processing (31)
devlog/_plan/260807_compatibility_lab/001_pr_stack_status.mddevlog/_plan/260807_compatibility_lab/023_live_v1_manifest_authority.mddevlog/_plan/260807_compatibility_lab/024_live_v1_cases.jsondevlog/_plan/260807_compatibility_lab/054_cl03_implementation_record.mdscripts/gen-live-digests.tssrc/lab/conformance/fixtures/live-v1-cases.jsonsrc/lab/conformance/types.tssrc/lab/index.tssrc/lab/live/credential-lease.tssrc/lab/live/destination.tssrc/lab/live/executor.tssrc/lab/live/inert-tools.tssrc/lab/live/manifest.tssrc/lab/live/mcp-loopback.tssrc/lab/live/runner.tssrc/lab/live/sandbox.tssrc/lab/live/suite-manifest.tssrc/lab/live/transport.tssrc/lab/live/types.tssrc/lab/observe/from-conformance.tssrc/lab/observe/from-live.tssrc/lab/paths.tssrc/lab/projection/verification.tssrc/lab/subject/behavior-fingerprint.tssrc/lab/subject/installation-salt.tssrc/lab/subject/route-subject.tssrc/lib/lab-live-execution-authority.tssrc/lib/lab-live-pinned-sender.tstests/lab-live-probe.test.tstests/lab-live-review-regressions.test.tstests/lab-live-sandbox.test.ts
Wibias
marked this pull request as draft
August 9, 2026 17:51
Contributor
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@src/lab/subject/installation-salt.ts`:
- Around line 21-25: Update the installation-salt creation flow around
writeFileSync and readExisting so the salt is fully written to a unique
same-directory staging file before publishing the final path. Publish atomically
(for example, linkSync) and treat an existing final path as another process’s
winner, then read that completed file; ensure cleanup handles staging files. Add
a multiprocess initialization regression test verifying both first-run runners
return the same persisted 32-byte salt.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: ASSERTIVE
Plan: Pro Plus
Run ID: fa0ed991-a3ce-4f53-805c-3380184f7dda
📒 Files selected for processing (2)
src/lab/live/destination.tssrc/lab/subject/installation-salt.ts
Collaborator
Author
|
@coderabbitai review |
Wibias
marked this pull request as ready for review
August 9, 2026 19:12
Contributor
There was a problem hiding this comment.
Actionable comments posted: 4
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@src/lab/live/executor.ts`:
- Around line 38-67: The trusted receipt currently authenticates only route
metadata, allowing persisted execution outcomes to be mutated after sealing.
Update receiptFor and TrustedLiveResultReceipt to include a canonical digest
covering every persisted LiveScenarioRunResult field, recompute it in
assertTrustedLiveResultReceipt, and reject mismatches before observation
persistence; add a regression test that mutates passed or an assertion result
after sealing and verifies persistence fails.
In `@src/lab/observe/from-live.ts`:
- Around line 39-51: Update failureFromLiveResult to persist the selected rule’s
retry policy instead of always setting retryable to false. Propagate the rule’s
retry state through LiveScenarioRunResult or resolve the selected FailureRule
during persistence, mapping "bounded" to retryable true while preserving
terminal failures as false.
In `@src/lab/subject/installation-salt.ts`:
- Around line 47-49: Update the successful publication path in the
installation-salt flow after linkSync to open dirname(path), flush the directory
using the existing fsyncDir pattern, and only then return the salt. Add a
crash-consistency regression test or document the supported filesystem guarantee
as requested.
In `@src/lib/pinned-http.ts`:
- Around line 158-174: Update the connection-timeout failure path in the pinned
HTTP request flow around the socket connect timer so it produces a
TransportError with status "connect_timeout" instead of a generic Error. Ensure
this typed error is preserved through pinnedHttpPost/pinnedHttpGet and
createLabAuthorizedPinnedSender so the executor classifies the failure as
connect_timeout.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: ASSERTIVE
Plan: Pro Plus
Run ID: 6ae0fb1f-43e0-4dba-b386-39929dd778da
📒 Files selected for processing (21)
devlog/_plan/260807_compatibility_lab/001_pr_stack_status.mddevlog/_plan/260807_compatibility_lab/024_live_v1_cases.jsondevlog/_plan/260807_compatibility_lab/054_cl03_implementation_record.mdsrc/lab/conformance/fixtures/live-v1-cases.jsonsrc/lab/conformance/types.tssrc/lab/live/destination.tssrc/lab/live/executor.tssrc/lab/live/manifest.tssrc/lab/live/transport.tssrc/lab/live/types.tssrc/lab/observe/from-live.tssrc/lab/projection/verdicts.tssrc/lab/projection/verification.tssrc/lab/subject/installation-salt.tssrc/lib/lab-live-execution-authority.tssrc/lib/lab-live-host.tssrc/lib/lab-live-pinned-sender.tssrc/lib/pinned-http.tsstructure/09_compatibility-lab.mdtests/lab-live-probe.test.tstests/lab-live-sandbox.test.ts
Contributor
There was a problem hiding this comment.
Actionable comments posted: 2
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@src/lab/observe/from-live.ts`:
- Around line 39-45: Update the live execution receipt flow so the exact failure
rule selected by executor.ts is stored as a trusted rule ID or retry policy,
then have failureFromLiveResult() reuse that value during persistence. Remove
the independent selectedFailureRule() classification fallback and do not
rederive the persisted retryability from result.classification.
In `@src/lab/subject/installation-salt.ts`:
- Around line 23-30: Update fsyncDirectory to recognize and ignore
platform/filesystem errors indicating directory fsync is unsupported, treating
the already completed atomic link plus staging-inode sync as sufficient durable
publication. Preserve fail-closed behavior for unexpected I/O errors, and
document the supported-runtime/filesystem cases handled as unsupported so the
caller does not report harness_failure after publication.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: ASSERTIVE
Plan: Pro Plus
Run ID: 7b6d8d97-ccb8-4834-8204-43e640a18d02
📒 Files selected for processing (4)
src/lab/observe/from-live.tssrc/lab/subject/installation-salt.tssrc/lib/lab-live-pinned-sender.tssrc/lib/pinned-http.ts
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
live_route_compatibilityscenarios, 7-day freshness, andlive-v1-defaultfailure rules that classify auth/quota/network/transient/timeout as blockers.LabDestinationV1, opaque credential leases, proxy-env rejection, DNS/address pinning, connect deadlines, inert tools, and MCP lab stub.RouteSubjectV1identity, host-issued trusted execution authority, opaque trusted-result receipts before persistence, and fail-closed route/claim applicability without changing the frozen V1 subject schema.Architectural seams
src/lab/subject/— installation salt, behavior fingerprint, route subject freezesrc/lab/live/— destination, sandbox, transport, manifest, executor, runnersrc/lib/lab-live-host.ts/lab-live-execution-authority.ts— host-issued execution capability boundarysrc/lab/observe/from-live.ts— receipt-gated live evidence persistencesrc/lab/projection/verification.ts/verdicts.ts— fail-closed live route applicability from validated claim statesrc/lab/conformance/fixtures/live-v1-cases.json— byte-checked runtime authority copyLive scenarios (10)
Test plan
bun x tsc --noEmitbun test tests/lab-conformance-harness.test.ts(17/17)bun test tests/lab-evidence-ledger.test.ts(37/41 — 4 Windows SQLite EPERM flakes on rebuild, pre-existing)bun test tests/lab-live-probe.test.ts(19/19)bun test tests/lab-live-sandbox.test.ts(17/17)bun run privacy:scanOut of scope
Summary by CodeRabbit