diff --git a/devlog/_plan/260807_compatibility_lab/001_pr_stack_status.md b/devlog/_plan/260807_compatibility_lab/001_pr_stack_status.md index 678e023c2..a7a230879 100644 --- a/devlog/_plan/260807_compatibility_lab/001_pr_stack_status.md +++ b/devlog/_plan/260807_compatibility_lab/001_pr_stack_status.md @@ -20,11 +20,17 @@ independent review, blockers, and whether a later phase is authorized. | Phase | Branch | Starting/base SHA | Accepted head | PR | State | |---|---|---|---|---|---| | CL-00 | `feat/cl-00-compatibility-contracts` | `3ad5bb6bd3f76f6879d84b78ea39edd3e01ec296` | `c014464237fd3c95bda08bc18bfab8ba8f532308` | [#1286](https://github.com/lidge-jun/opencodex/pull/1286) | ACCEPTED AFTER CODERABBIT REMEDIATION (merged to `dev` at `243c3f4905797aa11c62ba933bb03d6d721266fd`) | -| CL-01 | `feat/cl-01-conformance-harness` | `c2113ca47b8a05c5a5f90679e4eaa640ca2c6a66` | `22d608c82d82e2746c0cef9cd761db19a8e465ee` | [draft Wibias #10](https://github.com/Wibias/opencodex/pull/10) | ACCEPTED (contract-corrected revalidation) | +| CL-01 | `feat/cl-01-conformance-harness` | `c2113ca47b8a05c5a5f90679e4eaa640ca2c6a66` | `22d608c82d82e2746c0cef9cd761db19a8e465ee` | [#1320](https://github.com/lidge-jun/opencodex/pull/1320) | MERGED TO `dev` at `4bb249b756abd468c675d2d92fffe4da95ad3e2a` | +| CL-02 | `feat/cl-02-evidence-ledger` | `4bb249b756abd468c675d2d92fffe4da95ad3e2a` | (phase-2 review fixes in progress) | [draft #1333](https://github.com/lidge-jun/opencodex/pull/1333) | IMPLEMENTATION COMPLETE — PHASE-2 REVIEW FIXES — NOT INDEPENDENTLY ACCEPTED | +| CL-03 | — | — | — | — | NOT STARTED | The CL-01 starting SHA is the exact CL-00 tip recorded when CL-01 began. Its moving base-ref name is not a substitute for that historical SHA. +CL-02 starts from the exact CL-01 merge commit on `dev` +(`4bb249b756abd468c675d2d92fffe4da95ad3e2a` / upstream #1320). It does **not** +base on the pre-merge CL-01 feature branch tip. + ## CL-00 acceptance log - Live-tree audit covered provider registry/derivation, Routing Profiles, @@ -62,6 +68,7 @@ Independent CL-00 acceptance review is frozen at - **CL-00 merge base on `dev`:** `243c3f4905797aa11c62ba933bb03d6d721266fd` - **Post-rebase harness commit:** `cfe27b0dcb26a1bf0bb56f68f952e6e4f4d80fe9` (rebase-only) - **Correction head:** `574f1d5eb93c091494549ffc0e26ea7a4879c12c` (implementation); **tip:** `22d608c82d82e2746c0cef9cd761db19a8e465ee` +- **Merged to `dev`:** `4bb249b756abd468c675d2d92fffe4da95ad3e2a` via upstream [#1320](https://github.com/lidge-jun/opencodex/pull/1320). ### Corrections applied @@ -88,8 +95,52 @@ Independent CL-00 acceptance review is frozen at - None for CL-01 correction. - Full-suite green remains unavailable on this host for documented Windows/Bun reasons. +## CL-02 implementation log + +- **Branch:** `feat/cl-02-evidence-ledger` +- **Starting/base SHA:** `4bb249b756abd468c675d2d92fffe4da95ad3e2a` (CL-01 merge via #1320) +- **Scope:** append-only JSONL evidence ledger with an explicit sensitive-purge + exception: when the `ledger` purge action is requested, targeted evidence is + physically removed by atomic ledger rewrite and a `purge_tombstone` remains as + the auditable record; SQLite is rebuilt from the rewritten ledger and retained + content-addressed artifacts. The phase also includes the content-addressed + artifact store, disposable/rebuildable SQLite projection, ClaimSourceManifestV1, + invalidation semantics, and the CL-01 → observation persistence seam. +- **Explicitly out of scope:** CL-03 live probes, CL-04 CLI/API, CL-05 UI, + CL-06 profile fields, Fabric, shadow workflows. + +### Boundary note (verdict algorithm) + +CL-02 implements frozen `all-applicable-required-pass-v1` evaluation with +subject-aware applicability (required scenarios whose manifest requirements +match the exact protocol subject, excluding live-reserved cases in fixture +mode). Positive `VERIFIED` requires a non-empty applicable required set and a +current pass for every member. Descriptor/handle-bound artifact I/O, fail-closed +sensitive purge with shared-artifact retention, recursive event admission +ceilings, and unusable-evidence exclusion from projection are implemented. +Claims cannot produce `PROBED`/`VERIFIED`. + +### CL-02 validation status (2026-08-09 phase-2 review fixes) + +- **Prior accepted review-fix head:** `cf626d14c823413fbcd6ac2625d1da16bbac714e` +- **Phase-2 scope:** eleven independent-review blockers (artifact dirfd I/O, + purge scratch/export + explicit sensitive artifacts, streaming JSONL replay, + zero-applicable UNKNOWN, `newest-required-observation-v1`, multi-surface + applicability, historical manifest no-substitution, closed event admission, + corrupt superseding claims, ArtifactStore lifecycle, frozen behaviour + fingerprint). +- **Previous local validation:** `bun x tsc --noEmit`, `bun run privacy:scan`, + `tests/lab-evidence-ledger.test.ts` (41/41), `tests/lab-conformance-harness.test.ts` + (17/17), `tests/repo-hygiene.test.ts` (11/11), `git diff --check` green on + Windows host before the current CodeRabbit remediation pass. +- **Current CodeRabbit remediation:** committed on draft PR #1333; current CI and + review reconciliation are required before this head may be recorded as accepted. +- **Independent acceptance:** not yet — draft PR #1333 remains open for review. +- **CL-03:** not started. + ## Authorization - CL-00: **ACCEPTED** (merged #1286). -- CL-01: **ACCEPTED (contract-corrected revalidation)** — ready for stack review against `dev`. -- CL-02: **NOT STARTED / NOT AUTHORIZED**. +- CL-01: **MERGED** via #1320 at `4bb249b756abd468c675d2d92fffe4da95ad3e2a`. +- CL-02: **IMPLEMENTATION COMPLETE — PHASE-2 REVIEW FIXES — NOT INDEPENDENTLY ACCEPTED** on `feat/cl-02-evidence-ledger` (draft #1333). +- CL-03: **NOT STARTED**. diff --git a/src/lab/artifacts/sanitize.ts b/src/lab/artifacts/sanitize.ts new file mode 100644 index 000000000..cd9c7e81c --- /dev/null +++ b/src/lab/artifacts/sanitize.ts @@ -0,0 +1,114 @@ +/** + * Deny-by-default sanitization before artifact hashing/writing. + * Never persists prompts, secrets, paths, account IDs, raw URLs, or provider bodies. + */ +import type { ArtifactClass } from "../constants"; +import { MAX_SANITIZED_STRING_FIELD } from "../constants"; +import { jcsStringify } from "../digest"; +import { redactSecretString } from "../../lib/redact"; + +const FORBIDDEN_KEY = /^(?:authorization|proxy-authorization|cookie|set-cookie|api[-_]?key|x-api-key|token|secret|password|email|prompt|messages|content|body|url|hostname|baseUrl|path|account|alias)$/i; +const SECRETISH = /sk-[a-z0-9]{10,}|Bearer\s+[A-Za-z0-9._\-]+|ghp_[A-Za-z0-9]{20,}|xox[baprs]-[A-Za-z0-9-]{10,}/i; +const SECRETISH_GLOBAL = /sk-[a-z0-9]{10,}|Bearer\s+[A-Za-z0-9._\-]+|ghp_[A-Za-z0-9]{20,}|xox[baprs]-[A-Za-z0-9-]{10,}/gi; + +export function redactForArtifact(artifactClass: ArtifactClass, payload: unknown): unknown { + if ( + artifactClass === "fixture" || + artifactClass === "scenario_manifest" || + artifactClass === "suite_manifest" || + artifactClass === "claim_source_manifest" + ) { + // Contract artifacts are already synthetic/canonical. Mutating them would + // invalidate content-addressed digests; reject secret-shaped material instead. + assertNoSecretMaterial(payload, 0); + return payload; + } + return scrubValue(payload, 0); +} + +const FORBIDDEN_CONTRACT_KEYS = /^(?:authorization|proxy-authorization|cookie|set-cookie|api[-_]?key|x-api-key|token|secret|password|email|prompt|messages|baseUrl|hostname|account|alias)$/i; + +function assertNoSecretMaterial(value: unknown, depth: number): void { + if (depth > 8) { + throw new Error("contract artifact exceeds sanitization inspection depth"); + } + if (typeof value === "string") { + if (SECRETISH.test(value)) { + throw new Error("contract artifact contains forbidden secret-shaped material"); + } + return; + } + if (Array.isArray(value)) { + for (const item of value) assertNoSecretMaterial(item, depth + 1); + return; + } + if (value && typeof value === "object") { + for (const [key, child] of Object.entries(value as object)) { + if (FORBIDDEN_CONTRACT_KEYS.test(key)) { + throw new Error(`contract artifact forbids key ${key}`); + } + assertNoSecretMaterial(child, depth + 1); + } + } +} + +function scrubValue(value: unknown, depth: number): unknown { + if (depth > 8) return "[truncated_depth]"; + if (value === null || typeof value === "boolean" || typeof value === "number") return value; + if (typeof value === "string") return scrubString(value); + if (value instanceof Uint8Array) { + const text = new TextDecoder().decode(value); + return new TextEncoder().encode(scrubString(text)); + } + if (Array.isArray(value)) { + if (value.length > 256) return value.slice(0, 256).map((v) => scrubValue(v, depth + 1)); + return value.map((v) => scrubValue(v, depth + 1)); + } + if (typeof value === "object") { + const out: Record = {}; + const keys = Object.keys(value as object).slice(0, 64); + for (const key of keys) { + if (FORBIDDEN_KEY.test(key)) { + out[key] = "[redacted]"; + continue; + } + out[key] = scrubValue((value as Record)[key], depth + 1); + } + return out; + } + return "[unsupported]"; +} + +function scrubString(value: string): string { + let s = redactSecretString(value); + s = s.replace(SECRETISH_GLOBAL, "[REDACTED]"); + // Strip absolute filesystem paths (coarse) + s = s.replace(/(?:[A-Za-z]:\\|\/(?:home|Users|tmp|var|etc|root|mnt)\/)[^\s"']+/g, "[path]"); + // Strip URL userinfo / private hosts roughly + s = s.replace(/https?:\/\/[^\s"']+/gi, (url) => { + try { + const u = new URL(url); + if (u.username || u.password) return "[redacted-url]"; + if (/^(localhost|127\.|10\.|192\.168\.|172\.(1[6-9]|2\d|3[0-1])\.)/i.test(u.hostname)) { + return `${u.protocol}//[private-host]${u.pathname}`; + } + return `${u.protocol}//[host]${u.pathname}`; + } catch { + return "[redacted-url]"; + } + }); + const bytes = new TextEncoder().encode(s); + if (bytes.byteLength > MAX_SANITIZED_STRING_FIELD) { + return new TextDecoder().decode(bytes.slice(0, MAX_SANITIZED_STRING_FIELD)); + } + return s; +} + +/** Stable privacy boundary for diagnostic text that may be persisted. */ +export function sanitizeDiagnostic(value: unknown): string { + return scrubString(value instanceof Error ? value.message : String(value)); +} + +export function sanitizedJsonBytes(value: unknown): Uint8Array { + return new TextEncoder().encode(jcsStringify(scrubValue(value, 0))); +} diff --git a/src/lab/artifacts/secure-fs.ts b/src/lab/artifacts/secure-fs.ts new file mode 100644 index 000000000..421d0669b --- /dev/null +++ b/src/lab/artifacts/secure-fs.ts @@ -0,0 +1,468 @@ +/** + * Descriptor/handle-bound, no-follow artifact I/O for the Compatibility Lab store. + * + * POSIX runtimes use directory-relative `dir` opens. Windows uses the same pinned + * directory identity checks as other reviewed OpenCodex bounded readers, because + * directory-relative child opens are not durable there. + */ +import { + closeSync, + constants as fsConstants, + existsSync, + fstatSync, + fsyncSync, + lstatSync, + mkdirSync, + openSync, + readSync, + renameSync, + unlinkSync, + writeSync, + type Stats, +} from "node:fs"; +import { join } from "node:path"; +import { + ARTIFACT_FILENAME_EXT, + MAX_BYTES_PER_ARTIFACT, +} from "../constants"; +import { artifactBytesDigest, isSha256Hex } from "../digest"; + +export class ArtifactFsError extends Error { + readonly code: string; + constructor(code: string, message: string) { + super(message); + this.name = "ArtifactFsError"; + this.code = code; + } +} + +export function harnessFailure(message: string, code = "harness_failure"): never { + throw new ArtifactFsError(code, message); +} + +const O_RDONLY = fsConstants.O_RDONLY; +const O_RDWR = fsConstants.O_RDWR; +const O_CREAT = fsConstants.O_CREAT; +const O_EXCL = fsConstants.O_EXCL; +const O_NOFOLLOW = (fsConstants as { O_NOFOLLOW?: number }).O_NOFOLLOW; +const O_DIRECTORY = (fsConstants as { O_DIRECTORY?: number }).O_DIRECTORY; + +type ArtifactIoMode = "dirfd" | "win32_pinned"; + +let artifactIoMode: ArtifactIoMode | null = null; + +export function assertDigestName(digest: string): string { + if (!isSha256Hex(digest)) harnessFailure("artifact digest must be lowercase sha256 hex"); + if (digest.includes("/") || digest.includes("\\") || digest.includes(":") || digest.includes("..")) { + harnessFailure("artifact digest must not contain path separators"); + } + return digest; +} + +export function digestFileName(digest: string): string { + return `${assertDigestName(digest)}${ARTIFACT_FILENAME_EXT}`; +} + +function platformSupportsNoFollow(): boolean { + return typeof O_NOFOLLOW === "number" && O_NOFOLLOW !== 0; +} + +function openFlags(base: number, noFollow: boolean): number { + if (noFollow && platformSupportsNoFollow()) return base | O_NOFOLLOW!; + return base; +} + +function assertRegularFileStats(stats: Stats, label: string): void { + if ( + !stats.isFile() || + stats.isSymbolicLink() || + stats.isDirectory() || + stats.isFIFO() || + stats.isSocket() || + stats.isCharacterDevice() || + stats.isBlockDevice() + ) { + harnessFailure(`${label}: not a regular file`, "artifact_unsafe_target"); + } + if (stats.nlink !== 1) { + harnessFailure(`${label}: hard links prohibited (nlink=${stats.nlink})`, "artifact_unsafe_target"); + } +} + +function assertDirectoryStats(stats: Stats, label: string): void { + if (!stats.isDirectory() || stats.isSymbolicLink()) { + harnessFailure(`${label}: must be a real directory (no symlink/reparse redirection)`, "artifact_unsafe_target"); + } +} + +function identityOf(stats: Stats): string { + return `${stats.dev}:${stats.ino}`; +} + +function assertRelativeName(name: string): void { + if (name.includes("..") || name.includes("/") || name.includes("\\") || name.includes("\0")) { + harnessFailure("invalid relative artifact name", "artifact_unsafe_target"); + } +} + +export interface TrustedArtifactDir { + path: string; + fd: number; + identity: string; +} + +type OpenSyncWithDir = ( + path: string, + flags: number, + mode: number, + options: { dir: number }, +) => number; + +type RenameSyncWithDir = (from: string, to: string, options: { dir: number }) => void; +type UnlinkSyncWithDir = (path: string, options: { dir: number }) => void; + +function detectArtifactIoMode(dir: TrustedArtifactDir): ArtifactIoMode { + if (artifactIoMode !== null) return artifactIoMode; + if (process.platform === "win32") { + artifactIoMode = "win32_pinned"; + return artifactIoMode; + } + const probe = `.dirfd-probe-${process.pid}`; + const finalName = `${probe}.ok`; + try { + const openWithDir = openSync as unknown as OpenSyncWithDir; + const fd = openWithDir(probe, O_CREAT | O_EXCL | O_RDWR, 0o600, { dir: dir.fd }); + closeSync(fd); + (renameSync as unknown as RenameSyncWithDir)(probe, finalName, { dir: dir.fd }); + if (!existsSync(join(dir.path, finalName))) { + artifactIoMode = "win32_pinned"; + return artifactIoMode; + } + (unlinkSync as unknown as UnlinkSyncWithDir)(finalName, { dir: dir.fd }); + artifactIoMode = "dirfd"; + } catch { + artifactIoMode = "win32_pinned"; + } + return artifactIoMode; +} + +function childPath(dir: TrustedArtifactDir, name: string): string { + revalidateDir(dir); + assertRelativeName(name); + return join(dir.path, name); +} + +function assertOpenedPathMatchesDescriptor(dir: TrustedArtifactDir, name: string, fd: number): void { + if (detectArtifactIoMode(dir) !== "win32_pinned") return; + const opened = fstatSync(fd); + assertRegularFileStats(opened, "artifact fd"); + const pathEntry = lstatSync(childPath(dir, name)); + if ( + !pathEntry.isFile() || + pathEntry.isSymbolicLink() || + pathEntry.dev !== opened.dev || + pathEntry.ino !== opened.ino + ) { + closeSync(fd); + harnessFailure("artifact path identity mismatch after open", "artifact_unsafe_target"); + } +} + +function openAtDir(dir: TrustedArtifactDir, name: string, flags: number, mode = 0): number { + revalidateDir(dir); + assertRelativeName(name); + const mode_ = detectArtifactIoMode(dir); + if (mode_ === "dirfd") { + return (openSync as unknown as OpenSyncWithDir)(name, flags, mode, { dir: dir.fd }); + } + const fd = openSync(childPath(dir, name), flags, mode); + assertOpenedPathMatchesDescriptor(dir, name, fd); + return fd; +} + +function renameAtDir(dir: TrustedArtifactDir, from: string, to: string): void { + revalidateDir(dir); + assertRelativeName(from); + assertRelativeName(to); + if (detectArtifactIoMode(dir) === "dirfd") { + (renameSync as unknown as RenameSyncWithDir)(from, to, { dir: dir.fd }); + return; + } + renameSync(childPath(dir, from), childPath(dir, to)); +} + +function unlinkAtDir(dir: TrustedArtifactDir, name: string): void { + revalidateDir(dir); + assertRelativeName(name); + if (detectArtifactIoMode(dir) === "dirfd") { + (unlinkSync as unknown as UnlinkSyncWithDir)(name, { dir: dir.fd }); + return; + } + unlinkSync(childPath(dir, name)); +} + +function revalidateDir(dir: TrustedArtifactDir): void { + const stats = fstatSync(dir.fd); + assertDirectoryStats(stats, "artifacts dir"); + if (identityOf(stats) !== dir.identity) { + harnessFailure("artifacts directory identity changed", "artifact_unsafe_target"); + } +} + +export function openTrustedArtifactDir(artifactsDir: string): TrustedArtifactDir { + const abs = artifactsDir.replace(/[\\/]+$/, ""); + if (abs.includes("\0")) harnessFailure("NUL in artifacts path", "artifact_unsafe_target"); + mkdirSync(abs, { recursive: true, mode: 0o700 }); + + let fd: number; + if (typeof O_DIRECTORY === "number") { + try { + fd = openSync(abs, openFlags(O_RDONLY | O_DIRECTORY, true)); + } catch { + harnessFailure("failed to open artifacts directory with O_DIRECTORY", "artifact_unsafe_target"); + } + } else { + fd = openSync(abs, O_RDONLY); + } + + const stats = fstatSync(fd); + assertDirectoryStats(stats, "artifacts dir"); + const trusted = { path: abs, fd, identity: identityOf(stats) }; + detectArtifactIoMode(trusted); + return trusted; +} + +export function closeTrustedArtifactDir(dir: TrustedArtifactDir): void { + try { + closeSync(dir.fd); + } catch { + /* ignore */ + } +} + +export interface StoredArtifactBytes { + digest: string; + bytes: Uint8Array; + byteCount: number; +} + +export interface ReadArtifactOptions { + expectedByteCount?: number; + contentDigest?: (bytes: Uint8Array) => string; +} + +function readAllFromFd(fd: number, size: number): Buffer { + const buf = Buffer.alloc(size); + let offset = 0; + while (offset < buf.length) { + const n = readSync(fd, buf, offset, buf.length - offset, offset); + if (n <= 0) break; + offset += n; + } + if (offset !== size) harnessFailure("short read from artifact descriptor", "artifact_mismatch"); + return buf; +} + +function isRawMissingError(err: unknown): boolean { + return !!err && typeof err === "object" && "code" in err && (err as { code: string }).code === "ENOENT"; +} + +function isMissingArtifactError(err: unknown): boolean { + return isRawMissingError(err) || (err instanceof ArtifactFsError && err.code === "artifact_missing"); +} + +function assertArtifactTargetCreatable(dir: TrustedArtifactDir, name: string): void { + revalidateDir(dir); + assertRelativeName(name); + try { + const stats = lstatSync(childPath(dir, name)); + if (stats.isSymbolicLink()) { + harnessFailure("artifact target is a symbolic link", "artifact_unsafe_target"); + } + assertRegularFileStats(stats, "artifact create target"); + harnessFailure("artifact target exists but is not reusable", "artifact_unsafe_target"); + } catch (err) { + if (isRawMissingError(err)) return; + if (err instanceof ArtifactFsError) throw err; + harnessFailure( + `artifact create target check failed: ${err instanceof Error ? err.message : String(err)}`, + "artifact_unsafe_target", + ); + } +} + +function writeAll(fd: number, bytes: Uint8Array): void { + let offset = 0; + while (offset < bytes.byteLength) { + const n = writeSync(fd, bytes, offset, bytes.byteLength - offset); + if (n <= 0) harnessFailure("artifact write made no progress", "artifact_mismatch"); + offset += n; + } +} + +function writeTempArtifact( + dir: TrustedArtifactDir, + tmpName: string, + bytes: Uint8Array, + digest: string, + contentDigest: (b: Uint8Array) => string, +): void { + let fd: number | null = null; + try { + fd = openAtDir(dir, tmpName, openFlags(O_RDWR | O_CREAT | O_EXCL, true), 0o600); + writeAll(fd, bytes); + fsyncSync(fd); + const stats = fstatSync(fd); + assertRegularFileStats(stats, "artifact temp"); + if (stats.size !== bytes.byteLength) { + harnessFailure("size mismatch after write", "artifact_mismatch"); + } + const buf = readAllFromFd(fd, bytes.byteLength); + if (contentDigest(buf) !== digest) { + harnessFailure("digest mismatch on same descriptor", "artifact_mismatch"); + } + closeSync(fd); + fd = null; + renameAtDir(dir, tmpName, digestFileName(digest)); + } catch (err) { + if (fd !== null) { + try { closeSync(fd); } catch { /* ignore */ } + } + try { + unlinkAtDir(dir, tmpName); + } catch { + /* ignore cleanup */ + } + if (err instanceof ArtifactFsError) throw err; + harnessFailure(`artifact write failed: ${err instanceof Error ? err.message : String(err)}`); + } +} + +export function readArtifactBytes( + dir: TrustedArtifactDir, + digest: string, + expectedByteCountOrOpts?: number | ReadArtifactOptions, +): StoredArtifactBytes { + const opts: ReadArtifactOptions = + typeof expectedByteCountOrOpts === "number" + ? { expectedByteCount: expectedByteCountOrOpts } + : expectedByteCountOrOpts ?? {}; + const contentDigest = opts.contentDigest ?? artifactBytesDigest; + revalidateDir(dir); + assertDigestName(digest); + const name = digestFileName(digest); + + let fd: number | null = null; + try { + fd = openAtDir(dir, name, openFlags(O_RDONLY, true)); + const stats = fstatSync(fd); + assertRegularFileStats(stats, "artifact fd"); + if (opts.expectedByteCount !== undefined && stats.size !== opts.expectedByteCount) { + harnessFailure("artifact size mismatch on descriptor", "artifact_mismatch"); + } + if (stats.size > MAX_BYTES_PER_ARTIFACT) { + harnessFailure("artifact exceeds ceiling", "artifact_mismatch"); + } + const buf = readAllFromFd(fd, stats.size); + const got = contentDigest(buf); + if (got !== digest) harnessFailure("artifact digest mismatch on descriptor", "artifact_mismatch"); + return { digest, bytes: new Uint8Array(buf), byteCount: stats.size }; + } catch (err) { + if (isRawMissingError(err)) { + harnessFailure(`artifact missing: ${digest}`, "artifact_missing"); + } + if (err instanceof ArtifactFsError) throw err; + harnessFailure(`artifact read failed: ${err instanceof Error ? err.message : String(err)}`); + } finally { + if (fd !== null) closeSync(fd); + } + harnessFailure("artifact read failed"); +} + +export function putArtifactBytes( + dir: TrustedArtifactDir, + bytes: Uint8Array, + expectedDigest?: string, +): StoredArtifactBytes { + revalidateDir(dir); + if (bytes.byteLength > MAX_BYTES_PER_ARTIFACT) { + harnessFailure(`artifact exceeds ${MAX_BYTES_PER_ARTIFACT} bytes`); + } + const digest = artifactBytesDigest(bytes); + if (expectedDigest !== undefined) { + assertDigestName(expectedDigest); + if (digest !== expectedDigest) { + harnessFailure("artifact digest mismatch before write", "artifact_mismatch"); + } + } + + try { + return readArtifactBytes(dir, digest, bytes.byteLength); + } catch (err) { + if (!isMissingArtifactError(err)) throw err; + } + + assertArtifactTargetCreatable(dir, digestFileName(digest)); + const tmpName = `.tmp-${digest}-${process.pid}-${Date.now()}.partial`; + writeTempArtifact(dir, tmpName, bytes, digest, artifactBytesDigest); + return readArtifactBytes(dir, digest, bytes.byteLength); +} + +export function putNamedDigestBytes( + dir: TrustedArtifactDir, + digest: string, + bytes: Uint8Array, + contentDigest: (b: Uint8Array) => string, +): StoredArtifactBytes { + revalidateDir(dir); + assertDigestName(digest); + if (bytes.byteLength > MAX_BYTES_PER_ARTIFACT) { + harnessFailure(`artifact exceeds ${MAX_BYTES_PER_ARTIFACT} bytes`); + } + if (contentDigest(bytes) !== digest) { + harnessFailure("named artifact content digest mismatch before write", "artifact_mismatch"); + } + + try { + return readArtifactBytes(dir, digest, { expectedByteCount: bytes.byteLength, contentDigest }); + } catch (err) { + if (!isMissingArtifactError(err)) throw err; + } + + assertArtifactTargetCreatable(dir, digestFileName(digest)); + const tmpName = `.tmp-${digest}-${process.pid}-${Date.now()}.partial`; + writeTempArtifact(dir, tmpName, bytes, digest, contentDigest); + return readArtifactBytes(dir, digest, { expectedByteCount: bytes.byteLength, contentDigest }); +} + +export function deleteArtifactBytes(dir: TrustedArtifactDir, digest: string): void { + revalidateDir(dir); + assertDigestName(digest); + const name = digestFileName(digest); + try { + unlinkAtDir(dir, name); + } catch (err) { + if (isRawMissingError(err)) return; + if (err instanceof ArtifactFsError) throw err; + harnessFailure(`artifact delete failed: ${err instanceof Error ? err.message : String(err)}`); + } +} + +export function artifactExists(dir: TrustedArtifactDir, digest: string): boolean { + revalidateDir(dir); + assertDigestName(digest); + try { + const fd = openAtDir(dir, digestFileName(digest), openFlags(O_RDONLY, true)); + try { + const stats = fstatSync(fd); + assertRegularFileStats(stats, "artifact exists"); + return true; + } finally { + closeSync(fd); + } + } catch (err) { + if (isMissingArtifactError(err)) return false; + if (err instanceof ArtifactFsError) throw err; + harnessFailure(`artifact exists check failed: ${err instanceof Error ? err.message : String(err)}`); + } +} diff --git a/src/lab/artifacts/store.ts b/src/lab/artifacts/store.ts new file mode 100644 index 000000000..83ad5e118 --- /dev/null +++ b/src/lab/artifacts/store.ts @@ -0,0 +1,282 @@ +import { MAX_BYTES_PER_ARTIFACT, MAX_AGGREGATE_ARTIFACT_BYTES, MAX_ARTIFACTS_PER_RUN, ARTIFACT_FILENAME_EXT, type ArtifactClass, type ContractArtifactClass } from "../constants"; +import { + artifactBytesDigest, + claimSourceManifestDigest, + fixtureDigest, + isSha256Hex, + jcsStringify, + scenarioManifestDigest, + suiteManifestDigest, +} from "../digest"; +import type { ArtifactRefV1, ClaimSourceManifestV1 } from "../events/types"; +import { artifactClassMediaType, validateClaimSourceManifest } from "../events/validate"; +import { + closeTrustedArtifactDir, + ArtifactFsError, + deleteArtifactBytes, + openTrustedArtifactDir, + putArtifactBytes, + putNamedDigestBytes, + readArtifactBytes, + type TrustedArtifactDir, +} from "./secure-fs"; +import { redactForArtifact, sanitizeDiagnostic } from "./sanitize"; + +export { ArtifactFsError, openTrustedArtifactDir }; +export type { TrustedArtifactDir }; + +export interface PutArtifactInput { + artifactClass: ArtifactClass; + /** Pre-redaction payload; sanitizer runs before hash/write. */ + payload: Uint8Array | string | unknown; + mediaType?: string; + redactionPolicy?: string; + /** For contract artifacts whose digest uses a domain other than artifact-bytes. */ + expectedDigest?: string; +} + +export interface ArtifactReadOptions { + expectedByteCount?: number; + artifactClass?: ArtifactClass; +} + +export interface ArtifactStore { + dir: TrustedArtifactDir; + put(input: PutArtifactInput): ArtifactRefV1; + get(digest: string, expectedByteCountOrOpts?: number | ArtifactReadOptions): Uint8Array; + getVerified( + digest: string, + expectedByteCountOrOpts?: number | ArtifactReadOptions, + ): { bytes: Uint8Array; digest: string }; + remove(digest: string): void; + close(): void; +} + +function toBytes(payload: Uint8Array | string | unknown): Uint8Array { + if (payload instanceof Uint8Array) return payload; + if (typeof payload === "string") return new TextEncoder().encode(payload); + return new TextEncoder().encode(jcsStringify(payload)); +} + +function normalizeReadOptions(value?: number | ArtifactReadOptions): ArtifactReadOptions { + return typeof value === "number" ? { expectedByteCount: value } : value ?? {}; +} + +function jsonDigest( + digest: (value: Record) => string, +): (bytes: Uint8Array) => string { + return (bytes) => digest(JSON.parse(new TextDecoder().decode(bytes)) as Record); +} + +function digestForArtifactClass(artifactClass: ArtifactClass): (bytes: Uint8Array) => string { + switch (artifactClass) { + case "fixture": + return fixtureDigest; + case "scenario_manifest": + return jsonDigest(scenarioManifestDigest); + case "suite_manifest": + return jsonDigest(suiteManifestDigest); + case "claim_source_manifest": + return (bytes) => { + const parsed = JSON.parse(new TextDecoder().decode(bytes)); + return claimSourceManifestDigest(validateClaimSourceManifest(parsed).manifest); + }; + default: + return artifactBytesDigest; + } +} + +export function createArtifactStore(artifactsDir: string): ArtifactStore { + const dir = openTrustedArtifactDir(artifactsDir); + let aggregateBytes = 0; + let putCount = 0; + + const getVerified = ( + digest: string, + expectedByteCountOrOpts?: number | ArtifactReadOptions, + ): { bytes: Uint8Array; digest: string } => { + const opts = normalizeReadOptions(expectedByteCountOrOpts); + const candidates = opts.artifactClass + ? [digestForArtifactClass(opts.artifactClass)] + : [ + artifactBytesDigest, + fixtureDigest, + jsonDigest(scenarioManifestDigest), + jsonDigest(suiteManifestDigest), + digestForArtifactClass("claim_source_manifest"), + ]; + + let lastErr: unknown; + for (const contentDigest of candidates) { + try { + const got = readArtifactBytes(dir, digest, { + expectedByteCount: opts.expectedByteCount, + contentDigest, + }); + return { bytes: got.bytes, digest: got.digest }; + } catch (err) { + lastErr = err; + if ( + err instanceof ArtifactFsError && + err.code !== "artifact_mismatch" && + !err.message.includes("mismatch") + ) { + throw err; + } + } + } + throw lastErr instanceof Error + ? lastErr + : new ArtifactFsError("artifact_mismatch", "artifact digest verification failed"); + }; + + return { + dir, + put(input: PutArtifactInput): ArtifactRefV1 { + if (putCount >= MAX_ARTIFACTS_PER_RUN) { + throw new ArtifactFsError("budget_exhausted", "maximum artifacts per run exceeded"); + } + if (input.payload instanceof Uint8Array && input.payload.byteLength > MAX_BYTES_PER_ARTIFACT) { + throw new ArtifactFsError("budget_exhausted", `artifact exceeds ${MAX_BYTES_PER_ARTIFACT} bytes`); + } + if (typeof input.payload === "string" && new TextEncoder().encode(input.payload).byteLength > MAX_BYTES_PER_ARTIFACT) { + throw new ArtifactFsError("budget_exhausted", `artifact exceeds ${MAX_BYTES_PER_ARTIFACT} bytes`); + } + const redacted = redactForArtifact(input.artifactClass, input.payload); + const bytes = toBytes(redacted); + if (bytes.byteLength > MAX_BYTES_PER_ARTIFACT) { + throw new ArtifactFsError("budget_exhausted", `artifact exceeds ${MAX_BYTES_PER_ARTIFACT} bytes`); + } + if (aggregateBytes + bytes.byteLength > MAX_AGGREGATE_ARTIFACT_BYTES) { + throw new ArtifactFsError("budget_exhausted", "aggregate artifact ceiling exceeded"); + } + + let stored; + if (isContractClass(input.artifactClass)) { + const contractClass = input.artifactClass; + const computedDigest = computeContractDigest(contractClass, bytes, redacted); + if (input.expectedDigest !== undefined && computedDigest !== input.expectedDigest) { + throw new ArtifactFsError("artifact_mismatch", "contract artifact digest mismatch"); + } + stored = putNamedDigestBytes( + dir, + computedDigest, + bytes, + digestForArtifactClass(contractClass), + ); + } else { + stored = putArtifactBytes(dir, bytes, input.expectedDigest); + } + + putCount += 1; + aggregateBytes += stored.byteCount; + return { + digest: stored.digest, + mediaType: input.mediaType ?? artifactClassMediaType(input.artifactClass), + byteCount: stored.byteCount, + redactionPolicy: input.redactionPolicy ?? defaultRedactionPolicy(input.artifactClass), + relativePath: `${stored.digest}${ARTIFACT_FILENAME_EXT}`, + artifactClass: input.artifactClass, + }; + }, + get(digest: string, expectedByteCountOrOpts?: number | ArtifactReadOptions): Uint8Array { + return getVerified(digest, expectedByteCountOrOpts).bytes; + }, + getVerified, + remove(digest: string): void { + deleteArtifactBytes(dir, digest); + }, + close(): void { + closeTrustedArtifactDir(dir); + }, + }; +} + +function isContractClass(c: ArtifactClass): c is ContractArtifactClass { + return ( + c === "scenario_manifest" || + c === "suite_manifest" || + c === "fixture" || + c === "claim_source_manifest" + ); +} + +function computeContractDigest( + artifactClass: ContractArtifactClass, + bytes: Uint8Array, + redacted: unknown, +): string { + switch (artifactClass) { + case "fixture": + return fixtureDigest(bytes); + case "scenario_manifest": + return scenarioManifestDigest( + typeof redacted === "object" && redacted ? (redacted as Record) : JSON.parse(new TextDecoder().decode(bytes)), + ); + case "suite_manifest": + return suiteManifestDigest( + typeof redacted === "object" && redacted ? (redacted as Record) : JSON.parse(new TextDecoder().decode(bytes)), + ); + case "claim_source_manifest": { + const parsed = typeof redacted === "object" && redacted + ? redacted + : JSON.parse(new TextDecoder().decode(bytes)); + return claimSourceManifestDigest(validateClaimSourceManifest(parsed).manifest); + } + default: { + const _never: never = artifactClass; + return _never; + } + } +} + +function defaultRedactionPolicy(artifactClass: ArtifactClass): string { + switch (artifactClass) { + case "scenario_manifest": + case "suite_manifest": + case "fixture": + case "claim_source_manifest": + return "contract_canonical_v1"; + default: + return "sanitized_evidence_v1"; + } +} + +export function putClaimSourceManifest( + store: ArtifactStore, + manifest: ClaimSourceManifestV1, +): ArtifactRefV1 { + const { manifest: validated, digest } = validateClaimSourceManifest(manifest); + return store.put({ + artifactClass: "claim_source_manifest", + payload: validated, + expectedDigest: digest, + }); +} + +export type LoadClaimSourceManifestResult = + | { ok: true; manifest: ClaimSourceManifestV1; corruption?: undefined } + | { ok: false; manifest: ClaimSourceManifestV1 | null; corruption: string }; + +export function loadClaimSourceManifest( + store: ArtifactStore, + digest: string, + expected: { subjectId: string; capability: string }, +): LoadClaimSourceManifestResult { + if (!isSha256Hex(digest)) return { ok: false, manifest: null, corruption: "invalid digest" }; + try { + const bytes = store.get(digest, { artifactClass: "claim_source_manifest" }); + const parsed = JSON.parse(new TextDecoder().decode(bytes)); + const { manifest, digest: recomputed } = validateClaimSourceManifest(parsed); + if (recomputed !== digest) return { ok: false, manifest, corruption: "claim-source digest mismatch" }; + if (manifest.subjectId !== expected.subjectId) return { ok: false, manifest, corruption: "claim-source subjectId mismatch" }; + if (manifest.capability !== expected.capability) return { ok: false, manifest, corruption: "claim-source capability mismatch" }; + return { ok: true, manifest }; + } catch (err) { + return { + ok: false, + manifest: null, + corruption: sanitizeDiagnostic(err), + }; + } +} diff --git a/src/lab/conformance/executor.ts b/src/lab/conformance/executor.ts index 0d84f3094..68e17b1f5 100644 --- a/src/lab/conformance/executor.ts +++ b/src/lab/conformance/executor.ts @@ -23,7 +23,19 @@ import { recordUpstreamRequest, } from "./observation"; import { normalizeSseBytes } from "./sse-normalize"; -import type { CaseRecord, NormalizedObservation, ScenarioRunResult } from "./types"; +import type { CaseRecord, NormalizedObservation, ScenarioRunResult, ProtocolExecutionContextV1 } from "./types"; + +export function resolveProtocolExecutionContext(caseRecord: CaseRecord): ProtocolExecutionContextV1 { + const inbound = caseRecord.requirements.inboundProtocols[0] ?? "openai-responses"; + const upstream = caseRecord.requirements.upstreamProtocols[0] ?? "openai-chat"; + let surface = caseRecord.requirements.surfaces[0] ?? "responses-http"; + if (caseRecord.id === "responses-core.protocol.json-sse-equivalence") { + surface = "responses-sse"; + } else if (caseRecord.requirements.surfaces.length === 1) { + surface = caseRecord.requirements.surfaces[0]!; + } + return { inboundProtocol: inbound, upstreamProtocol: upstream, surface }; +} async function collectAdapterEvents(gen: AsyncGenerator): Promise { const events: AdapterEvent[] = []; @@ -663,6 +675,7 @@ export async function executeScenario(caseRecord: CaseRecord): Promise { const diagnostics: string[] = []; + const executionContext = resolveProtocolExecutionContext(caseRecord); try { const observation = await executeScenario(caseRecord); const assertionResults = evaluateAssertions(caseRecord.assertions, observation); @@ -688,6 +701,7 @@ export async function runScenario(caseRecord: CaseRecord): Promise c.suite === suiteId); + if (cases.length === 0) { + throw new Error(`unknown suite ${suiteId}`); + } + const defaults = authority.manifestDefaults; + const capability = cases[0]!.capability; + if (cases.some((caseRecord) => caseRecord.capability !== capability)) { + throw new Error(`suite ${suiteId} declares mixed capabilities`); + } + const scenarios: SuiteScenarioRefV1[] = cases + .map((caseRecord) => suiteScenarioRef(caseRecord, authority)) + .sort((a, b) => (a.id < b.id ? -1 : a.id > b.id ? 1 : 0)); + + return { + schemaVersion: 1, + id: suiteId, + version: String(defaults.suiteVersion), + evidenceLayer: defaults.evidenceLayer, + capability, + assertionDslVersion: authority.assertionDslVersion, + evidenceSchemaVersion: authority.evidenceSchemaVersion, + freshness: defaults.freshness ?? { maxAgeMs: null }, + contradictionRule: "newest-required-observation-v1", + scenarios, + verificationRule: "all-applicable-required-pass-v1", + }; +} + +function suiteScenarioRef(caseRecord: CaseRecord, authority: CaseAuthority): SuiteScenarioRefV1 { + const expanded = expandScenario(caseRecord, authority); + return { + id: caseRecord.id, + version: String(authority.manifestDefaults.version), + role: caseRecord.verificationRole ?? authority.manifestDefaults.verificationRole, + manifestDigest: scenarioManifestDigest(expanded), + }; +} + +export function suiteManifestObjectForCase( + caseRecord: CaseRecord, + authority: CaseAuthority, +): Record { + return expandSuiteManifest(caseRecord.suite, authority) as unknown as Record; +} + +export function suiteManifestDigestForCase(caseRecord: CaseRecord, authority: CaseAuthority): string { + return suiteManifestDigest(expandSuiteManifest(caseRecord.suite, authority) as unknown as Record); +} diff --git a/src/lab/conformance/types.ts b/src/lab/conformance/types.ts index f1c61bce4..d0d7f830c 100644 --- a/src/lab/conformance/types.ts +++ b/src/lab/conformance/types.ts @@ -141,6 +141,12 @@ export interface AssertionResult { reason?: string; } +export interface ProtocolExecutionContextV1 { + inboundProtocol: string; + upstreamProtocol: string; + surface: string; +} + export interface ScenarioRunResult { scenarioId: string; suite: string; @@ -150,6 +156,7 @@ export interface ScenarioRunResult { assertionResults: AssertionResult[]; expectedFailureMatched?: boolean; diagnostics: string[]; + executionContext: ProtocolExecutionContextV1; } /** All eight protocol-conformance suites frozen by CL-00 Protocol V1. */ diff --git a/src/lab/constants.ts b/src/lab/constants.ts new file mode 100644 index 000000000..881ca9ea4 --- /dev/null +++ b/src/lab/constants.ts @@ -0,0 +1,126 @@ +/** CL-02 projection / ledger constants frozen from CL-00 contracts. */ + +export const LAB_EVENT_SCHEMA_VERSION = 1; +export const LAB_PROJECTION_SPEC_VERSION = "cl-02.v1"; +export const LAB_PRODUCER = "opencodex-lab"; +export const LAB_PRODUCER_VERSION = "2.10.2"; + +export const MAX_INVALIDATION_TARGETS = 1024; +export const MAX_BYTES_PER_ARTIFACT = 256 * 1024; +export const MAX_AGGREGATE_ARTIFACT_BYTES = 1024 * 1024; +export const MAX_ARTIFACTS_PER_RUN = 16; +export const MAX_SERIALIZED_EVENT_BYTES = 64 * 1024; +export const MAX_SANITIZED_STRING_FIELD = 4 * 1024; +export const MAX_EVENT_NESTING_DEPTH = 8; +export const MAX_OBJECT_KEYS_PER_EVENT = 64; +export const MAX_ARRAY_ELEMENTS_PER_EVENT = 256; + +export const OBSERVATION_LIMIT_NAMES = [ + "totalTimeoutMs", + "connectTimeoutMs", + "firstByteTimeoutMs", + "inactivityTimeoutMs", + "maxRequests", + "maxInputBytes", + "maxOutputBytes", + "maxOutputTokens", + "maxToolCalls", + "maxArtifactBytes", +] as const; + +export type ObservationLimitName = (typeof OBSERVATION_LIMIT_NAMES)[number]; + +export const EVENT_KINDS = [ + "observation", + "claim_snapshot", + "invalidation", + "purge_tombstone", +] as const; + +export type LabEventKind = (typeof EVENT_KINDS)[number]; + +export const EVIDENCE_LAYERS = [ + "protocol_conformance", + "live_route_compatibility", + "task_effectiveness", +] as const; + +export type EvidenceLayer = (typeof EVIDENCE_LAYERS)[number]; + +export const VERDICTS = [ + "UNKNOWN", + "CLAIMED", + "PROBED", + "VERIFIED", + "DEGRADED", + "BLOCKED", + "UNSUPPORTED", +] as const; + +export type CompatibilityVerdict = (typeof VERDICTS)[number]; + +export const INVALIDATION_REASONS = [ + "harness_defect", + "fixture_defect", + "redaction_defect", + "integrity_defect", + "contract_artifact_missing", + "manual_correction", +] as const; + +export type InvalidationReason = (typeof INVALIDATION_REASONS)[number]; + +export const PURGE_ACTIONS = [ + "ledger", + "sqlite", + "artifact", + "scratch", + "export", +] as const; + +export type PurgeAction = (typeof PURGE_ACTIONS)[number]; + +export const CLAIM_SOURCE_KINDS = [ + "provider_config", + "provider_registry", + "cached_catalog", + "native_metadata", + "adapter_inference", +] as const; + +export type ClaimSourceKind = (typeof CLAIM_SOURCE_KINDS)[number]; + +export const ARTIFACT_CLASSES = [ + "scenario_manifest", + "suite_manifest", + "fixture", + "claim_source_manifest", + "assertion_report", + "request_shape", + "response_shape", + "event_trace", + "error_taxonomy", + "verifier_summary", +] as const; + +export type ArtifactClass = (typeof ARTIFACT_CLASSES)[number]; + +export const CONTRACT_ARTIFACT_CLASSES = [ + "scenario_manifest", + "suite_manifest", + "fixture", + "claim_source_manifest", +] as const; + +export type ContractArtifactClass = (typeof CONTRACT_ARTIFACT_CLASSES)[number]; + +export const EXECUTION_MODES = ["fixture", "live", "fabric"] as const; +export type ExecutionMode = (typeof EXECUTION_MODES)[number]; + +export const OUTCOMES = ["pass", "fail", "blocked", "inconclusive"] as const; +export type ObservationOutcome = (typeof OUTCOMES)[number]; + +export const CLAIM_POLARITIES = ["supported", "not_supported", "withdrawn"] as const; +export type ClaimPolarity = (typeof CLAIM_POLARITIES)[number]; + +export const ARTIFACT_FILENAME_EXT = ".bin"; diff --git a/src/lab/digest.ts b/src/lab/digest.ts new file mode 100644 index 000000000..c54b3b443 --- /dev/null +++ b/src/lab/digest.ts @@ -0,0 +1,64 @@ +import { createHash, createHmac } from "node:crypto"; +import { jcsStringify } from "./conformance/jcs"; +import { + fixtureDigest, + scenarioManifestDigest, + suiteManifestDigest, +} from "./conformance/digest"; + +export { fixtureDigest, scenarioManifestDigest, suiteManifestDigest, jcsStringify }; +export { jcsEqual } from "./conformance/jcs"; + +/** Domain-separated SHA-256: UTF-8(domain) || NUL || payload. */ +export function domainHash(domain: string, payload: Uint8Array | string): string { + const hash = createHash("sha256"); + hash.update(new TextEncoder().encode(`${domain}\0`)); + if (typeof payload === "string") hash.update(new TextEncoder().encode(payload)); + else hash.update(payload); + return hash.digest("hex"); +} + +export const LAB_DOMAIN = { + event: "ocx-lab:event:v1", + subject: "ocx-lab:subject:v1", + claimSourceManifest: "ocx-lab:claim-source-manifest:v1", + scenarioManifest: "ocx-lab:scenario-manifest:v1", + suiteManifest: "ocx-lab:suite-manifest:v1", + fixture: "ocx-lab:fixture:v1", + localFingerprint: "ocx-lab:local-fingerprint:v1", + artifactBytes: "ocx-lab:artifact-bytes:v1", +} as const; + +export function eventIdForPayload(eventWithoutId: unknown): string { + return domainHash(LAB_DOMAIN.event, jcsStringify(eventWithoutId)); +} + +export function subjectIdForSubject(subject: unknown): string { + return domainHash(LAB_DOMAIN.subject, jcsStringify(subject)); +} + +export function claimSourceManifestDigest(manifest: unknown): string { + return domainHash(LAB_DOMAIN.claimSourceManifest, jcsStringify(manifest)); +} + +/** Content digest for already-redacted artifact bytes (not contract fixture domain). */ +export function artifactBytesDigest(bytes: Uint8Array): string { + return domainHash(LAB_DOMAIN.artifactBytes, bytes); +} + +export function localFingerprint( + fieldName: string, + value: unknown, + salt: Uint8Array | string, +): string { + const key = typeof salt === "string" ? new TextEncoder().encode(salt) : salt; + const preimage = `${LAB_DOMAIN.localFingerprint}\0${fieldName}\0`; + return createHmac("sha256", key) + .update(new TextEncoder().encode(preimage)) + .update(new TextEncoder().encode(jcsStringify(value))) + .digest("hex"); +} + +export function isSha256Hex(value: string): boolean { + return /^[0-9a-f]{64}$/.test(value); +} diff --git a/src/lab/events/errors.ts b/src/lab/events/errors.ts new file mode 100644 index 000000000..17009183a --- /dev/null +++ b/src/lab/events/errors.ts @@ -0,0 +1,9 @@ +export class LabValidationError extends Error { + readonly code: string; + + constructor(code: string, message: string) { + super(message); + this.name = "LabValidationError"; + this.code = code; + } +} diff --git a/src/lab/events/limits.ts b/src/lab/events/limits.ts new file mode 100644 index 000000000..d6cba306f --- /dev/null +++ b/src/lab/events/limits.ts @@ -0,0 +1,114 @@ +import { + MAX_ARRAY_ELEMENTS_PER_EVENT, + MAX_EVENT_NESTING_DEPTH, + MAX_OBJECT_KEYS_PER_EVENT, + MAX_SANITIZED_STRING_FIELD, +} from "../constants"; +import { LabValidationError } from "./errors"; + +const FORBIDDEN_KEY_RE = + /(?:^|_)(?:secret|token|apikey|api_key|password|credential|authorization|cookie|bearer|prompt|repository|filepath|file_path|baseurl|base_url|hostname|rawrequest|raw_request)(?:$|_)/i; + +const FORBIDDEN_EXACT_KEYS = new Set([ + "authorization", + "apiKey", + "api_key", + "password", + "secret", + "token", + "cookie", + "cookies", + "headers", + "prompt", + "path", + "filepath", + "filePath", + "repository", + "repo", + "baseUrl", + "url", + "rawRequest", + "rawBytes", +]); + +function fieldPath(base: string, key: string | number): string { + return base ? `${base}.${String(key)}` : String(key); +} + +function assertAllowedValueType(value: unknown, path: string): void { + if (value === null) return; + const t = typeof value; + if (t === "string" || t === "boolean") return; + if (t === "number") { + if (!Number.isFinite(value as number)) { + throw new LabValidationError("non_finite", `${path} must be finite number`); + } + return; + } + if (Array.isArray(value) || (t === "object" && value !== null)) return; + throw new LabValidationError("unsupported_type", `${path} has unsupported type`); +} + +function assertKeyAllowed(key: string, path: string): void { + if (FORBIDDEN_EXACT_KEYS.has(key) || FORBIDDEN_KEY_RE.test(key)) { + throw new LabValidationError("forbidden_field", `${path} forbidden`); + } + if (key.includes("\0")) { + throw new LabValidationError("nul_forbidden", `${path} contains NUL`); + } +} + +/** Recursive deterministic CL-00 structural/privacy ceilings before JSONL admission. */ +export function enforceEventStructureLimits( + value: unknown, + path = "", + depth = 0, +): void { + assertAllowedValueType(value, path || "event"); + if (value === null) return; + + if (typeof value === "string") { + const bytes = new TextEncoder().encode(value).byteLength; + if (bytes > MAX_SANITIZED_STRING_FIELD) { + throw new LabValidationError("field_too_large", `${path} exceeds ${MAX_SANITIZED_STRING_FIELD} bytes`); + } + if (/sk-[a-z0-9]{10,}/i.test(value) || /Bearer\s+\S+/i.test(value)) { + throw new LabValidationError("secret_pattern", `${path} contains secret-shaped data`); + } + if ( + /^[A-Za-z]:\\/.test(value) || + /(?:^|[\s"'([])\/(?:home|Users|tmp|var|etc|root|mnt)\//.test(value) || + value.includes("\\Users\\") + ) { + throw new LabValidationError("raw_path", `${path} contains raw filesystem path`); + } + return; + } + + if (typeof value === "number" || typeof value === "boolean") return; + + if (Array.isArray(value)) { + if (depth >= MAX_EVENT_NESTING_DEPTH) { + throw new LabValidationError("nesting_depth", `${path} exceeds nesting depth ${MAX_EVENT_NESTING_DEPTH}`); + } + if (value.length > MAX_ARRAY_ELEMENTS_PER_EVENT) { + throw new LabValidationError("array_too_large", `${path} exceeds ${MAX_ARRAY_ELEMENTS_PER_EVENT} elements`); + } + for (let i = 0; i < value.length; i++) { + enforceEventStructureLimits(value[i], fieldPath(path, i), depth + 1); + } + return; + } + + if (depth >= MAX_EVENT_NESTING_DEPTH) { + throw new LabValidationError("nesting_depth", `${path} exceeds nesting depth ${MAX_EVENT_NESTING_DEPTH}`); + } + const keys = Object.keys(value as Record); + if (keys.length > MAX_OBJECT_KEYS_PER_EVENT) { + throw new LabValidationError("too_many_keys", `${path} exceeds ${MAX_OBJECT_KEYS_PER_EVENT} keys`); + } + for (const key of keys) { + assertKeyAllowed(key, fieldPath(path, key)); + enforceEventStructureLimits((value as Record)[key], fieldPath(path, key), depth + 1); + } +} diff --git a/src/lab/events/types.ts b/src/lab/events/types.ts new file mode 100644 index 000000000..69ed120db --- /dev/null +++ b/src/lab/events/types.ts @@ -0,0 +1,228 @@ +import type { + ArtifactClass, + ClaimPolarity, + ClaimSourceKind, + EvidenceLayer, + ExecutionMode, + InvalidationReason, + LabEventKind, + ObservationOutcome, + PurgeAction, +} from "../constants"; + +export interface LabProducer { + name: string; + version: string; +} + +export interface ProtocolSubjectV1 { + subjectSchemaVersion: 1; + subjectKind: "protocol"; + opencodexCompatibilityVersion: string; + effectiveAdapter: string; + inboundProtocol: string; + upstreamProtocol: string; + surface: string; + behaviorFingerprint: string; +} + +export interface RouteDependencyV1 { + role: string; + providerId: string; + providerInstanceFingerprint: string; + clientModelId: string; + upstreamModelId: string; + effectiveAdapter: string; + upstreamProtocol: string; + endpointFingerprint: string; + behaviorFingerprint: string; +} + +export interface RouteSubjectV1 { + subjectSchemaVersion: 1; + subjectKind: "route"; + providerId: string; + providerInstanceFingerprint: string; + clientModelId: string; + upstreamModelId: string; + effectiveAdapter: string; + inboundProtocol: string; + upstreamProtocol: string; + surface: string; + opencodexCompatibilityVersion: string; + behaviorFingerprint: string; + endpointFingerprint: string; + dependencies: RouteDependencyV1[]; +} + +export interface TaskSubjectV1 { + subjectSchemaVersion: 1; + subjectKind: "task"; + routeSubject: RouteSubjectV1; + taskClassId: string; + taskClassVersion: string; + taskFixtureDigest: string; + verifierManifestDigest: string; + fabricCompatibilityVersion: string; + sandboxProfileDigest: string; +} + +export type EvidenceSubjectV1 = ProtocolSubjectV1 | RouteSubjectV1 | TaskSubjectV1; + +export interface ArtifactRefV1 { + digest: string; + mediaType: string; + byteCount: number; + redactionPolicy: string; + /** Digest-derived relative name only (e.g. `.bin`). */ + relativePath: string; + artifactClass: ArtifactClass; +} + +export interface AssertionRecordV1 { + id: string; + operator: string; + required: boolean; + passed: boolean; + expectedSummary: string; + observedSummary: string; + reason?: string; +} + +export interface FailureRecordV1 { + class: string; + code: string; + retryable: boolean; + attribution: "opencodex" | "route" | "environment" | "harness"; +} + +export interface LabEventBase { + schemaVersion: number; + eventId: string; + eventKind: LabEventKind; + recordedAt: number; + producer: string; + producerVersion: string; +} + +export interface ObservationEvent extends LabEventBase { + eventKind: "observation"; + evidenceLayer: EvidenceLayer; + scenarioId: string; + scenarioVersion: string; + scenarioManifestDigest: string; + suiteId: string; + suiteVersion: string; + suiteManifestDigest: string; + fixtureDigests: string[]; + subject: EvidenceSubjectV1; + subjectId: string; + startedAt: number; + completedAt: number; + executionMode: ExecutionMode; + attempt: number; + limits: Record; + outcome: ObservationOutcome; + assertions: AssertionRecordV1[]; + failure?: FailureRecordV1; + expectedFailure?: Record; + environment: Record; + artifactRefs: ArtifactRefV1[]; + sourceRefs?: string[]; +} + +export interface ClaimSnapshotEvent extends LabEventBase { + eventKind: "claim_snapshot"; + evidenceLayer: "live_route_compatibility"; + subject: RouteSubjectV1; + subjectId: string; + capability: string; + polarity: ClaimPolarity; + sourceManifestDigest: string; + sourceEventIds: string[]; + supersedes: string[]; + effectiveAt: number; +} + +export interface InvalidationEvent extends LabEventBase { + eventKind: "invalidation"; + targetEventIds: string[]; + reason: InvalidationReason; +} + +export interface PurgeTombstoneEvent extends LabEventBase { + eventKind: "purge_tombstone"; + targetEventIds: string[]; + targetArtifactDigests: string[]; + reason: "sensitive_evidence"; + purgeActions: PurgeAction[]; +} + +export type LabEvent = + | ObservationEvent + | ClaimSnapshotEvent + | InvalidationEvent + | PurgeTombstoneEvent; + +export type ClaimCapabilityFactsV1 = { + contextWindow?: number; + inputModalities?: string[]; + reasoningEfforts?: string[]; + catalogCapabilityNames?: string[]; + serviceTier?: string[]; + toolCapable?: boolean; + parallelToolCalls?: boolean; + endpointLocality?: "local" | "private" | "unknown"; + canonicalOpenAiForward?: boolean; +}; + +export interface ClaimSourceV1 { + kind: ClaimSourceKind; + revision: string | null; + facts: ClaimCapabilityFactsV1; +} + +export interface RouteCapabilityEvidenceV1 { + contextWindow?: number; + tools?: boolean | "unknown"; + image?: boolean | "unknown"; + structuredOutput?: boolean | "unknown"; + reasoningEfforts?: string[]; + serviceTier?: string | "unknown"; + localOnly?: boolean | "unknown"; + remoteAllowed?: boolean | "unknown"; + encryptedCodexTasks?: boolean | "unknown"; +} + +export interface ClaimSourceManifestV1 { + schemaVersion: 1; + subjectId: string; + providerId: string; + clientModelId: string; + capability: string; + sources: ClaimSourceV1[]; + resolvedEvidence: RouteCapabilityEvidenceV1; +} + +export interface LedgerCorruption { + kind: + | "malformed_line" + | "partial_line" + | "duplicate_event" + | "invalid_event" + | "invalid_reference" + | "missing_artifact" + | "artifact_mismatch" + | "claim_corruption" + | "empty_ledger"; + lineNumber?: number; + eventId?: string; + detail: string; +} + +export interface ReplayResult { + events: LabEvent[]; + corruptions: LedgerCorruption[]; + validLineCount: number; + totalLineCount: number; +} diff --git a/src/lab/events/validate.ts b/src/lab/events/validate.ts new file mode 100644 index 000000000..d101e28b0 --- /dev/null +++ b/src/lab/events/validate.ts @@ -0,0 +1,772 @@ +import { enforceEventStructureLimits } from "./limits"; +import { LabValidationError } from "./errors"; +export { LabValidationError } from "./errors"; +import { + ARTIFACT_CLASSES, + ARTIFACT_FILENAME_EXT, + CLAIM_POLARITIES, + CLAIM_SOURCE_KINDS, + EVIDENCE_LAYERS, + EVENT_KINDS, + EXECUTION_MODES, + INVALIDATION_REASONS, + LAB_EVENT_SCHEMA_VERSION, + MAX_INVALIDATION_TARGETS, + MAX_SANITIZED_STRING_FIELD, + MAX_SERIALIZED_EVENT_BYTES, + OBSERVATION_LIMIT_NAMES, + OUTCOMES, + PURGE_ACTIONS, + type ArtifactClass, + type ClaimSourceKind, + type EvidenceLayer, + type LabEventKind, +} from "../constants"; +import { + claimSourceManifestDigest, + eventIdForPayload, + isSha256Hex, + jcsStringify, + subjectIdForSubject, +} from "../digest"; +import type { + ArtifactRefV1, + ClaimCapabilityFactsV1, + ClaimSnapshotEvent, + ClaimSourceManifestV1, + ClaimSourceV1, + EvidenceSubjectV1, + InvalidationEvent, + LabEvent, + ObservationEvent, + ProtocolSubjectV1, + PurgeTombstoneEvent, + RouteCapabilityEvidenceV1, + RouteSubjectV1, + TaskSubjectV1, +} from "./types"; + +function isPlainObject(value: unknown): value is Record { + return !!value && typeof value === "object" && !Array.isArray(value); +} + +function assertString(value: unknown, field: string, max = MAX_SANITIZED_STRING_FIELD): string { + if (typeof value !== "string") throw new LabValidationError("invalid_type", `${field} must be string`); + const bytes = new TextEncoder().encode(value).byteLength; + if (bytes > max) throw new LabValidationError("field_too_large", `${field} exceeds ${max} bytes`); + if (value.includes("\0")) throw new LabValidationError("nul_forbidden", `${field} contains NUL`); + return value; +} + +function assertIntMs(value: unknown, field: string): number { + if (typeof value !== "number" || !Number.isInteger(value) || value < 0) { + throw new LabValidationError("invalid_timestamp", `${field} must be UTC epoch milliseconds`); + } + return value; +} + +function assertClosed(value: unknown, field: string, allowed: readonly T[]): T { + if (typeof value !== "string" || !(allowed as readonly string[]).includes(value)) { + throw new LabValidationError("closed_set", `${field} must be one of ${allowed.join("|")}`); + } + return value as T; +} + +function utf8LexLess(a: string, b: string): boolean { + const ae = new TextEncoder().encode(a); + const be = new TextEncoder().encode(b); + const n = Math.min(ae.length, be.length); + for (let i = 0; i < n; i++) { + if (ae[i]! < be[i]!) return true; + if (ae[i]! > be[i]!) return false; + } + return ae.length < be.length; +} + +/** Sorted unique lowercase SHA-256 hex IDs; rejects duplicates / unsorted / uppercase. */ +export function validateSortedUniqueHexIds( + ids: unknown, + field: string, + opts: { nonEmpty?: boolean; max?: number } = {}, +): string[] { + if (!Array.isArray(ids)) throw new LabValidationError("invalid_type", `${field} must be array`); + if (opts.nonEmpty && ids.length === 0) { + throw new LabValidationError("empty_targets", `${field} must be non-empty`); + } + if (opts.max !== undefined && ids.length > opts.max) { + throw new LabValidationError("too_many_targets", `${field} exceeds ${opts.max}`); + } + const out: string[] = []; + for (let i = 0; i < ids.length; i++) { + const id = ids[i]; + if (typeof id !== "string" || !isSha256Hex(id)) { + throw new LabValidationError("invalid_id", `${field}[${i}] must be lowercase sha256 hex`); + } + if (i > 0) { + const prev = out[i - 1]!; + if (id === prev) throw new LabValidationError("duplicate_id", `${field} contains duplicates`); + if (!utf8LexLess(prev, id)) { + throw new LabValidationError("unsorted_ids", `${field} must be UTF-8 lexicographically sorted`); + } + } + out.push(id); + } + return out; +} + +function validateArtifactRef(raw: unknown, index: number): ArtifactRefV1 { + if (!isPlainObject(raw)) throw new LabValidationError("invalid_artifact_ref", `artifactRefs[${index}]`); + const digest = assertString(raw.digest, `artifactRefs[${index}].digest`); + if (!isSha256Hex(digest)) throw new LabValidationError("invalid_digest", `artifactRefs[${index}].digest`); + const mediaType = assertString(raw.mediaType, `artifactRefs[${index}].mediaType`, 256); + if (typeof raw.byteCount !== "number" || !Number.isInteger(raw.byteCount) || raw.byteCount < 0) { + throw new LabValidationError("invalid_byte_count", `artifactRefs[${index}].byteCount`); + } + const redactionPolicy = assertString(raw.redactionPolicy, `artifactRefs[${index}].redactionPolicy`, 256); + const relativePath = assertString(raw.relativePath, `artifactRefs[${index}].relativePath`, 128); + if (relativePath !== `${digest}${ARTIFACT_FILENAME_EXT}`) { + throw new LabValidationError("invalid_relative_path", `artifactRefs[${index}].relativePath must be digest-derived`); + } + if (relativePath.includes("/") || relativePath.includes("\\") || relativePath.includes("..")) { + throw new LabValidationError("path_traversal", `artifactRefs[${index}].relativePath`); + } + const artifactClass = assertClosed(raw.artifactClass, `artifactRefs[${index}].artifactClass`, ARTIFACT_CLASSES); + return { digest, mediaType, byteCount: raw.byteCount, redactionPolicy, relativePath, artifactClass }; +} + +function validateProtocolSubject(raw: Record): ProtocolSubjectV1 { + if (raw.subjectSchemaVersion !== 1 || raw.subjectKind !== "protocol") { + throw new LabValidationError("invalid_subject", "protocol subject schema/kind mismatch"); + } + return { + subjectSchemaVersion: 1, + subjectKind: "protocol", + opencodexCompatibilityVersion: assertString(raw.opencodexCompatibilityVersion, "opencodexCompatibilityVersion"), + effectiveAdapter: assertString(raw.effectiveAdapter, "effectiveAdapter"), + inboundProtocol: assertString(raw.inboundProtocol, "inboundProtocol"), + upstreamProtocol: assertString(raw.upstreamProtocol, "upstreamProtocol"), + surface: assertString(raw.surface, "surface"), + behaviorFingerprint: assertString(raw.behaviorFingerprint, "behaviorFingerprint"), + }; +} + +function validateRouteSubject(raw: Record): RouteSubjectV1 { + if (raw.subjectSchemaVersion !== 1 || raw.subjectKind !== "route") { + throw new LabValidationError("invalid_subject", "route subject schema/kind mismatch"); + } + if (!Array.isArray(raw.dependencies)) { + throw new LabValidationError("invalid_subject", "dependencies must be array"); + } + const dependencies = raw.dependencies.map((dep, i) => { + if (!isPlainObject(dep)) throw new LabValidationError("invalid_dependency", `dependencies[${i}]`); + return { + role: assertString(dep.role, `dependencies[${i}].role`), + providerId: assertString(dep.providerId, `dependencies[${i}].providerId`), + providerInstanceFingerprint: assertString(dep.providerInstanceFingerprint, `dependencies[${i}].providerInstanceFingerprint`), + clientModelId: assertString(dep.clientModelId, `dependencies[${i}].clientModelId`), + upstreamModelId: assertString(dep.upstreamModelId, `dependencies[${i}].upstreamModelId`), + effectiveAdapter: assertString(dep.effectiveAdapter, `dependencies[${i}].effectiveAdapter`), + upstreamProtocol: assertString(dep.upstreamProtocol, `dependencies[${i}].upstreamProtocol`), + endpointFingerprint: assertString(dep.endpointFingerprint, `dependencies[${i}].endpointFingerprint`), + behaviorFingerprint: assertString(dep.behaviorFingerprint, `dependencies[${i}].behaviorFingerprint`), + }; + }); + return { + subjectSchemaVersion: 1, + subjectKind: "route", + providerId: assertString(raw.providerId, "providerId"), + providerInstanceFingerprint: assertString(raw.providerInstanceFingerprint, "providerInstanceFingerprint"), + clientModelId: assertString(raw.clientModelId, "clientModelId"), + upstreamModelId: assertString(raw.upstreamModelId, "upstreamModelId"), + effectiveAdapter: assertString(raw.effectiveAdapter, "effectiveAdapter"), + inboundProtocol: assertString(raw.inboundProtocol, "inboundProtocol"), + upstreamProtocol: assertString(raw.upstreamProtocol, "upstreamProtocol"), + surface: assertString(raw.surface, "surface"), + opencodexCompatibilityVersion: assertString(raw.opencodexCompatibilityVersion, "opencodexCompatibilityVersion"), + behaviorFingerprint: assertString(raw.behaviorFingerprint, "behaviorFingerprint"), + endpointFingerprint: assertString(raw.endpointFingerprint, "endpointFingerprint"), + dependencies, + }; +} + +function validateTaskSubject(raw: Record): TaskSubjectV1 { + if (raw.subjectSchemaVersion !== 1 || raw.subjectKind !== "task") { + throw new LabValidationError("invalid_subject", "task subject schema/kind mismatch"); + } + if (!isPlainObject(raw.routeSubject)) { + throw new LabValidationError("invalid_subject", "task.routeSubject required"); + } + return { + subjectSchemaVersion: 1, + subjectKind: "task", + routeSubject: validateRouteSubject(raw.routeSubject), + taskClassId: assertString(raw.taskClassId, "taskClassId"), + taskClassVersion: assertString(raw.taskClassVersion, "taskClassVersion"), + taskFixtureDigest: assertString(raw.taskFixtureDigest, "taskFixtureDigest"), + verifierManifestDigest: assertString(raw.verifierManifestDigest, "verifierManifestDigest"), + fabricCompatibilityVersion: assertString(raw.fabricCompatibilityVersion, "fabricCompatibilityVersion"), + sandboxProfileDigest: assertString(raw.sandboxProfileDigest, "sandboxProfileDigest"), + }; +} + +export function validateSubject(raw: unknown, layer: EvidenceLayer): EvidenceSubjectV1 { + if (!isPlainObject(raw)) throw new LabValidationError("invalid_subject", "subject must be object"); + const kind = raw.subjectKind; + if (layer === "protocol_conformance") { + if (kind !== "protocol") throw new LabValidationError("layer_subject_mismatch", "protocol layer requires protocol subject"); + return validateProtocolSubject(raw); + } + if (layer === "live_route_compatibility") { + if (kind !== "route") throw new LabValidationError("layer_subject_mismatch", "live layer requires route subject"); + return validateRouteSubject(raw); + } + if (layer === "task_effectiveness") { + if (kind !== "task") throw new LabValidationError("layer_subject_mismatch", "task layer requires task subject"); + return validateTaskSubject(raw); + } + const _exhaustive: never = layer; + throw new LabValidationError("unknown_layer", String(_exhaustive)); +} + +function stripEventId(event: Record): Record { + const { eventId: _omit, ...rest } = event; + return rest; +} + +function enforceEventId(event: LabEvent): void { + const recomputed = eventIdForPayload(stripEventId(event as unknown as Record)); + if (event.eventId !== recomputed) { + throw new LabValidationError("event_id_mismatch", `eventId mismatch: got ${event.eventId}, expected ${recomputed}`); + } +} + +function enforceSerializedSize(event: LabEvent): void { + const bytes = new TextEncoder().encode(jcsStringify(event)).byteLength; + if (bytes > MAX_SERIALIZED_EVENT_BYTES) { + throw new LabValidationError("event_too_large", `serialized event exceeds ${MAX_SERIALIZED_EVENT_BYTES} bytes`); + } +} + +function validateAssertionRecord(raw: unknown, index: number): ObservationEvent["assertions"][number] { + if (!isPlainObject(raw)) throw new LabValidationError("invalid_assertions", `assertions[${index}]`); + const allowed = new Set(["id", "operator", "required", "passed", "expectedSummary", "observedSummary", "reason"]); + for (const key of Object.keys(raw)) { + if (!allowed.has(key)) { + throw new LabValidationError("unknown_assertion_key", `assertions[${index}].${key}`); + } + } + const record = { + id: assertString(raw.id, `assertions[${index}].id`), + operator: assertString(raw.operator, `assertions[${index}].operator`), + required: raw.required === true || raw.required === false + ? raw.required + : (() => { throw new LabValidationError("invalid_assertions", `assertions[${index}].required`); })(), + passed: raw.passed === true || raw.passed === false + ? raw.passed + : (() => { throw new LabValidationError("invalid_assertions", `assertions[${index}].passed`); })(), + expectedSummary: assertString(raw.expectedSummary, `assertions[${index}].expectedSummary`), + observedSummary: assertString(raw.observedSummary, `assertions[${index}].observedSummary`), + ...(raw.reason !== undefined ? { reason: assertString(raw.reason, `assertions[${index}].reason`) } : {}), + }; + return record; +} + +function validateObservationLimits(raw: unknown): Record { + if (!isPlainObject(raw)) throw new LabValidationError("invalid_limits", "limits"); + const out: Record = {}; + for (const key of Object.keys(raw)) { + if (!(OBSERVATION_LIMIT_NAMES as readonly string[]).includes(key)) { + throw new LabValidationError("unknown_limit", `limits.${key}`); + } + const value = raw[key]; + if (value === null) { + out[key] = null; + continue; + } + if (typeof value !== "number" || !Number.isInteger(value) || value < 0) { + throw new LabValidationError("invalid_limits", `limits.${key}`); + } + out[key] = value; + } + return out; +} + +function validateObservationEnvironment(raw: unknown): Record { + if (!isPlainObject(raw)) throw new LabValidationError("invalid_environment", "environment"); + const topKeys = Object.keys(raw); + if (topKeys.length !== 1 || topKeys[0] !== "runtime") { + throw new LabValidationError("invalid_environment", "environment must contain only runtime"); + } + const runtime = raw.runtime; + if (!isPlainObject(runtime)) throw new LabValidationError("invalid_environment", "environment.runtime"); + const runtimeKeys = Object.keys(runtime); + const allowedRuntime = new Set(["platform", "arch", "bunVersion"]); + for (const key of runtimeKeys) { + if (!allowedRuntime.has(key)) { + throw new LabValidationError("unknown_environment_key", `environment.runtime.${key}`); + } + } + const out: Record = { + runtime: { + platform: assertString(runtime.platform, "environment.runtime.platform"), + arch: assertString(runtime.arch, "environment.runtime.arch"), + ...(runtime.bunVersion !== undefined + ? { bunVersion: assertString(runtime.bunVersion, "environment.runtime.bunVersion") } + : {}), + }, + }; + return out; +} + +function validateExpectedFailure(raw: unknown): Record { + if (!isPlainObject(raw)) throw new LabValidationError("invalid_expected_failure", "expectedFailure"); + const allowed = new Set([ + "controlKind", + "expectedClass", + "expectedCode", + "assertionIds", + "onMatch", + "onMismatch", + ]); + for (const key of Object.keys(raw)) { + if (!allowed.has(key)) { + throw new LabValidationError("unknown_expected_failure_key", `expectedFailure.${key}`); + } + } + if (!Array.isArray(raw.assertionIds) || !raw.assertionIds.every((id) => typeof id === "string")) { + throw new LabValidationError("invalid_expected_failure", "expectedFailure.assertionIds"); + } + return { + controlKind: assertClosed(raw.controlKind, "expectedFailure.controlKind", [ + "conformance_negative_control", + "capability_absence_control", + ] as const), + expectedClass: assertString(raw.expectedClass, "expectedFailure.expectedClass"), + expectedCode: assertString(raw.expectedCode, "expectedFailure.expectedCode"), + assertionIds: raw.assertionIds as string[], + onMatch: assertClosed(raw.onMatch, "expectedFailure.onMatch", ["pass", "unsupported"] as const), + onMismatch: assertClosed(raw.onMismatch, "expectedFailure.onMismatch", ["fail", "inconclusive"] as const), + }; +} + +function validateSourceRefs(raw: unknown): string[] { + if (!Array.isArray(raw)) throw new LabValidationError("invalid_source_refs", "sourceRefs"); + return raw.map((r, i) => assertString(r, `sourceRefs[${i}]`)); +} + +function validateObservation(raw: Record): ObservationEvent { + const evidenceLayer = assertClosed(raw.evidenceLayer, "evidenceLayer", EVIDENCE_LAYERS); + const subject = validateSubject(raw.subject, evidenceLayer); + const subjectId = assertString(raw.subjectId, "subjectId"); + if (subjectId !== subjectIdForSubject(subject)) { + throw new LabValidationError("subject_id_mismatch", "subjectId does not match subject"); + } + if (!Array.isArray(raw.fixtureDigests) || raw.fixtureDigests.length === 0) { + throw new LabValidationError("missing_fixture_digests", "fixtureDigests required"); + } + const fixtureDigests = raw.fixtureDigests.map((d, i) => { + const digest = assertString(d, `fixtureDigests[${i}]`); + if (!isSha256Hex(digest)) throw new LabValidationError("invalid_digest", `fixtureDigests[${i}]`); + return digest; + }); + const scenarioManifestDigest = assertString(raw.scenarioManifestDigest, "scenarioManifestDigest"); + const suiteManifestDigest = assertString(raw.suiteManifestDigest, "suiteManifestDigest"); + if (!isSha256Hex(scenarioManifestDigest) || !isSha256Hex(suiteManifestDigest)) { + throw new LabValidationError("invalid_digest", "scenario/suite manifest digest"); + } + if (!Array.isArray(raw.assertions)) throw new LabValidationError("invalid_assertions", "assertions"); + if (!Array.isArray(raw.artifactRefs)) throw new LabValidationError("invalid_artifact_refs", "artifactRefs"); + const event: ObservationEvent = { + schemaVersion: LAB_EVENT_SCHEMA_VERSION, + eventId: assertString(raw.eventId, "eventId"), + eventKind: "observation", + recordedAt: assertIntMs(raw.recordedAt, "recordedAt"), + producer: assertString(raw.producer, "producer"), + producerVersion: assertString(raw.producerVersion, "producerVersion"), + evidenceLayer, + scenarioId: assertString(raw.scenarioId, "scenarioId"), + scenarioVersion: assertString(raw.scenarioVersion, "scenarioVersion"), + scenarioManifestDigest, + suiteId: assertString(raw.suiteId, "suiteId"), + suiteVersion: assertString(raw.suiteVersion, "suiteVersion"), + suiteManifestDigest, + fixtureDigests, + subject, + subjectId, + startedAt: assertIntMs(raw.startedAt, "startedAt"), + completedAt: assertIntMs(raw.completedAt, "completedAt"), + executionMode: assertClosed(raw.executionMode, "executionMode", EXECUTION_MODES), + attempt: typeof raw.attempt === "number" && Number.isInteger(raw.attempt) && raw.attempt >= 1 + ? raw.attempt + : (() => { throw new LabValidationError("invalid_attempt", "attempt"); })(), + limits: validateObservationLimits(raw.limits), + outcome: assertClosed(raw.outcome, "outcome", OUTCOMES), + assertions: raw.assertions.map(validateAssertionRecord), + environment: validateObservationEnvironment(raw.environment), + artifactRefs: raw.artifactRefs.map(validateArtifactRef), + }; + if (raw.failure !== undefined) { + if (!isPlainObject(raw.failure)) throw new LabValidationError("invalid_failure", "failure"); + event.failure = { + class: assertString(raw.failure.class, "failure.class"), + code: assertString(raw.failure.code, "failure.code"), + retryable: raw.failure.retryable === true, + attribution: assertClosed(raw.failure.attribution, "failure.attribution", [ + "opencodex", + "route", + "environment", + "harness", + ] as const), + }; + } + if (raw.expectedFailure !== undefined) { + event.expectedFailure = validateExpectedFailure(raw.expectedFailure); + } + if (raw.sourceRefs !== undefined) { + event.sourceRefs = validateSourceRefs(raw.sourceRefs); + } + if (event.completedAt < event.startedAt) { + throw new LabValidationError("invalid_time_range", "completedAt < startedAt"); + } + return event; +} + +function validateClaimSnapshot(raw: Record): ClaimSnapshotEvent { + if (raw.evidenceLayer !== "live_route_compatibility") { + throw new LabValidationError("invalid_claim_layer", "claims exist only for live_route_compatibility"); + } + const subject = validateSubject(raw.subject, "live_route_compatibility") as RouteSubjectV1; + const subjectId = assertString(raw.subjectId, "subjectId"); + if (subjectId !== subjectIdForSubject(subject)) { + throw new LabValidationError("subject_id_mismatch", "subjectId does not match subject"); + } + const sourceManifestDigest = assertString(raw.sourceManifestDigest, "sourceManifestDigest"); + if (!isSha256Hex(sourceManifestDigest)) { + throw new LabValidationError("invalid_digest", "sourceManifestDigest"); + } + return { + schemaVersion: LAB_EVENT_SCHEMA_VERSION, + eventId: assertString(raw.eventId, "eventId"), + eventKind: "claim_snapshot", + recordedAt: assertIntMs(raw.recordedAt, "recordedAt"), + producer: assertString(raw.producer, "producer"), + producerVersion: assertString(raw.producerVersion, "producerVersion"), + evidenceLayer: "live_route_compatibility", + subject, + subjectId, + capability: assertString(raw.capability, "capability"), + polarity: assertClosed(raw.polarity, "polarity", CLAIM_POLARITIES), + sourceManifestDigest, + sourceEventIds: validateSortedUniqueHexIds(raw.sourceEventIds, "sourceEventIds", { + nonEmpty: false, + max: MAX_INVALIDATION_TARGETS, + }), + supersedes: validateSortedUniqueHexIds(raw.supersedes, "supersedes", { + nonEmpty: false, + max: MAX_INVALIDATION_TARGETS, + }), + effectiveAt: assertIntMs(raw.effectiveAt, "effectiveAt"), + }; +} + +function validateInvalidation(raw: Record): InvalidationEvent { + return { + schemaVersion: LAB_EVENT_SCHEMA_VERSION, + eventId: assertString(raw.eventId, "eventId"), + eventKind: "invalidation", + recordedAt: assertIntMs(raw.recordedAt, "recordedAt"), + producer: assertString(raw.producer, "producer"), + producerVersion: assertString(raw.producerVersion, "producerVersion"), + targetEventIds: validateSortedUniqueHexIds(raw.targetEventIds, "targetEventIds", { + nonEmpty: true, + max: MAX_INVALIDATION_TARGETS, + }), + reason: assertClosed(raw.reason, "reason", INVALIDATION_REASONS), + }; +} + +function validatePurge(raw: Record): PurgeTombstoneEvent { + const targetEventIds = validateSortedUniqueHexIds(raw.targetEventIds ?? [], "targetEventIds", { + nonEmpty: false, + max: MAX_INVALIDATION_TARGETS, + }); + const targetArtifactDigests = validateSortedUniqueHexIds( + raw.targetArtifactDigests ?? [], + "targetArtifactDigests", + { nonEmpty: false, max: MAX_INVALIDATION_TARGETS }, + ); + if (targetEventIds.length === 0 && targetArtifactDigests.length === 0) { + throw new LabValidationError("empty_purge_targets", "at least one purge target required"); + } + if (raw.reason !== "sensitive_evidence") { + throw new LabValidationError("invalid_purge_reason", "reason must be sensitive_evidence"); + } + if (!Array.isArray(raw.purgeActions) || raw.purgeActions.length === 0) { + throw new LabValidationError("empty_purge_actions", "purgeActions required"); + } + const purgeActions = raw.purgeActions.map((a, i) => + assertClosed(a, `purgeActions[${i}]`, PURGE_ACTIONS), + ); + for (let i = 0; i < purgeActions.length; i++) { + for (let j = i + 1; j < purgeActions.length; j++) { + if (purgeActions[i] === purgeActions[j]) { + throw new LabValidationError("duplicate_purge_action", "purgeActions must be unique"); + } + } + } + const sortedActions = [...purgeActions].sort((a, b) => (a < b ? -1 : a > b ? 1 : 0)); + for (let i = 0; i < purgeActions.length; i++) { + if (purgeActions[i] !== sortedActions[i]) { + throw new LabValidationError("unsorted_purge_actions", "purgeActions must be sorted"); + } + } + return { + schemaVersion: LAB_EVENT_SCHEMA_VERSION, + eventId: assertString(raw.eventId, "eventId"), + eventKind: "purge_tombstone", + recordedAt: assertIntMs(raw.recordedAt, "recordedAt"), + producer: assertString(raw.producer, "producer"), + producerVersion: assertString(raw.producerVersion, "producerVersion"), + targetEventIds, + targetArtifactDigests, + reason: "sensitive_evidence", + purgeActions, + }; +} + +/** Structural validation of a parsed JSON object into a LabEvent (fail-closed). */ +export function validateLabEvent(raw: unknown): LabEvent { + if (!isPlainObject(raw)) throw new LabValidationError("invalid_event", "event must be object"); + if (raw.schemaVersion !== LAB_EVENT_SCHEMA_VERSION) { + throw new LabValidationError("schema_version", "unsupported schemaVersion"); + } + const kind = assertClosed(raw.eventKind, "eventKind", EVENT_KINDS) as LabEventKind; + let event: LabEvent; + switch (kind) { + case "observation": + event = validateObservation(raw); + break; + case "claim_snapshot": + event = validateClaimSnapshot(raw); + break; + case "invalidation": + event = validateInvalidation(raw); + break; + case "purge_tombstone": + event = validatePurge(raw); + break; + default: { + const _never: never = kind; + throw new LabValidationError("unknown_kind", String(_never)); + } + } + enforceEventId(event); + enforceEventStructureLimits(event); + enforceSerializedSize(event); + return event; +} + +const FORBIDDEN_FACT_KEYS = new Set([ + "baseUrl", + "hostname", + "url", + "headers", + "authorization", + "apiKey", + "credential", + "account", + "email", + "path", + "prompt", + "repository", +]); + +const ALLOWED_FACT_KEYS = new Set([ + "contextWindow", + "inputModalities", + "reasoningEfforts", + "catalogCapabilityNames", + "serviceTier", + "toolCapable", + "parallelToolCalls", + "endpointLocality", + "canonicalOpenAiForward", +]); + +function validateFacts(raw: unknown, label: string): ClaimCapabilityFactsV1 { + if (!isPlainObject(raw)) throw new LabValidationError("invalid_facts", label); + for (const key of Object.keys(raw)) { + if (FORBIDDEN_FACT_KEYS.has(key) || /url|host|header|secret|token|cookie|email|path/i.test(key)) { + throw new LabValidationError("forbidden_fact", `${label}.${key} forbidden`); + } + if (!ALLOWED_FACT_KEYS.has(key)) { + throw new LabValidationError("unknown_fact", `${label}.${key} unknown`); + } + } + const facts: ClaimCapabilityFactsV1 = {}; + if (raw.contextWindow !== undefined) { + if (typeof raw.contextWindow !== "number" || !Number.isInteger(raw.contextWindow) || raw.contextWindow < 0) { + throw new LabValidationError("invalid_facts", "contextWindow"); + } + facts.contextWindow = raw.contextWindow; + } + if (raw.inputModalities !== undefined) { + if (!Array.isArray(raw.inputModalities) || !raw.inputModalities.every((x) => typeof x === "string")) { + throw new LabValidationError("invalid_facts", "inputModalities"); + } + facts.inputModalities = raw.inputModalities as string[]; + } + if (raw.reasoningEfforts !== undefined) { + if (!Array.isArray(raw.reasoningEfforts) || !raw.reasoningEfforts.every((x) => typeof x === "string")) { + throw new LabValidationError("invalid_facts", "reasoningEfforts"); + } + facts.reasoningEfforts = raw.reasoningEfforts as string[]; + } + if (raw.catalogCapabilityNames !== undefined) { + if (!Array.isArray(raw.catalogCapabilityNames) || !raw.catalogCapabilityNames.every((x) => typeof x === "string")) { + throw new LabValidationError("invalid_facts", "catalogCapabilityNames"); + } + facts.catalogCapabilityNames = raw.catalogCapabilityNames as string[]; + } + if (raw.serviceTier !== undefined) { + if (!Array.isArray(raw.serviceTier) || !raw.serviceTier.every((x) => typeof x === "string")) { + throw new LabValidationError("invalid_facts", "serviceTier"); + } + facts.serviceTier = raw.serviceTier as string[]; + } + if (raw.toolCapable !== undefined) { + if (typeof raw.toolCapable !== "boolean") throw new LabValidationError("invalid_facts", "toolCapable"); + facts.toolCapable = raw.toolCapable; + } + if (raw.parallelToolCalls !== undefined) { + if (typeof raw.parallelToolCalls !== "boolean") throw new LabValidationError("invalid_facts", "parallelToolCalls"); + facts.parallelToolCalls = raw.parallelToolCalls; + } + if (raw.endpointLocality !== undefined) { + facts.endpointLocality = assertClosed(raw.endpointLocality, "endpointLocality", [ + "local", + "private", + "unknown", + ] as const); + } + if (raw.canonicalOpenAiForward !== undefined) { + if (typeof raw.canonicalOpenAiForward !== "boolean") { + throw new LabValidationError("invalid_facts", "canonicalOpenAiForward"); + } + facts.canonicalOpenAiForward = raw.canonicalOpenAiForward; + } + return facts; +} + +function validateResolvedEvidence(raw: unknown): RouteCapabilityEvidenceV1 { + if (!isPlainObject(raw)) throw new LabValidationError("invalid_resolved", "resolvedEvidence"); + const out: RouteCapabilityEvidenceV1 = {}; + const copyBoolOrUnknown = (key: keyof RouteCapabilityEvidenceV1) => { + const v = raw[key as string]; + if (v === undefined) return; + if (v !== true && v !== false && v !== "unknown") { + throw new LabValidationError("invalid_resolved", String(key)); + } + (out as Record)[key] = v; + }; + if (raw.contextWindow !== undefined) { + if (typeof raw.contextWindow !== "number" || !Number.isInteger(raw.contextWindow)) { + throw new LabValidationError("invalid_resolved", "contextWindow"); + } + out.contextWindow = raw.contextWindow; + } + copyBoolOrUnknown("tools"); + copyBoolOrUnknown("image"); + copyBoolOrUnknown("structuredOutput"); + copyBoolOrUnknown("localOnly"); + copyBoolOrUnknown("remoteAllowed"); + copyBoolOrUnknown("encryptedCodexTasks"); + if (raw.reasoningEfforts !== undefined) { + if (!Array.isArray(raw.reasoningEfforts) || !raw.reasoningEfforts.every((x) => typeof x === "string")) { + throw new LabValidationError("invalid_resolved", "reasoningEfforts"); + } + out.reasoningEfforts = raw.reasoningEfforts as string[]; + } + if (raw.serviceTier !== undefined) { + if (typeof raw.serviceTier !== "string") throw new LabValidationError("invalid_resolved", "serviceTier"); + out.serviceTier = raw.serviceTier; + } + return out; +} + +/** Validate ClaimSourceManifestV1 and return canonical object + digest. */ +export function validateClaimSourceManifest(raw: unknown): { + manifest: ClaimSourceManifestV1; + digest: string; +} { + if (!isPlainObject(raw)) throw new LabValidationError("invalid_claim_source", "manifest must be object"); + if (raw.schemaVersion !== 1) throw new LabValidationError("schema_version", "ClaimSourceManifest schemaVersion"); + if (!Array.isArray(raw.sources)) throw new LabValidationError("invalid_sources", "sources"); + const seen = new Set(); + const sources: ClaimSourceV1[] = []; + for (let i = 0; i < raw.sources.length; i++) { + const src = raw.sources[i]; + if (!isPlainObject(src)) throw new LabValidationError("invalid_source", `sources[${i}]`); + const kind = assertClosed(src.kind, `sources[${i}].kind`, CLAIM_SOURCE_KINDS); + if (seen.has(kind)) throw new LabValidationError("duplicate_source_kind", kind); + seen.add(kind); + if (src.revision !== null && typeof src.revision !== "string") { + throw new LabValidationError("invalid_revision", `sources[${i}].revision`); + } + sources.push({ + kind, + revision: src.revision as string | null, + facts: validateFacts(src.facts, `sources[${i}].facts`), + }); + } + // Closed kind order + for (let i = 1; i < sources.length; i++) { + const prevIdx = CLAIM_SOURCE_KINDS.indexOf(sources[i - 1]!.kind); + const curIdx = CLAIM_SOURCE_KINDS.indexOf(sources[i]!.kind); + if (curIdx <= prevIdx) { + throw new LabValidationError("source_order", "sources must follow closed kind order"); + } + } + const manifest: ClaimSourceManifestV1 = { + schemaVersion: 1, + subjectId: assertString(raw.subjectId, "subjectId"), + providerId: assertString(raw.providerId, "providerId"), + clientModelId: assertString(raw.clientModelId, "clientModelId"), + capability: assertString(raw.capability, "capability"), + sources, + resolvedEvidence: validateResolvedEvidence(raw.resolvedEvidence), + }; + if (!isSha256Hex(manifest.subjectId)) { + throw new LabValidationError("invalid_subject_id", "subjectId must be sha256 hex"); + } + return { manifest, digest: claimSourceManifestDigest(manifest) }; +} + +export function assignEventId & { eventId?: string }>( + eventWithoutId: T, +): T & { eventId: string } { + const { eventId: _ignored, ...rest } = eventWithoutId as T & { eventId?: string }; + const eventId = eventIdForPayload(rest); + return { ...rest, eventId } as T & { eventId: string }; +} + +export function artifactClassMediaType(artifactClass: ArtifactClass): string { + switch (artifactClass) { + case "scenario_manifest": + case "suite_manifest": + case "claim_source_manifest": + case "assertion_report": + case "request_shape": + case "response_shape": + case "event_trace": + case "error_taxonomy": + case "verifier_summary": + return "application/json"; + case "fixture": + return "application/octet-stream"; + default: { + const _never: never = artifactClass; + return _never; + } + } +} diff --git a/src/lab/index.ts b/src/lab/index.ts new file mode 100644 index 000000000..2515f8481 --- /dev/null +++ b/src/lab/index.ts @@ -0,0 +1,19 @@ +export * from "./constants"; +export * from "./digest"; +export * from "./paths"; +export * from "./events/types"; +export * from "./events/validate"; +export * from "./artifacts/secure-fs"; +export * from "./artifacts/store"; +export * from "./artifacts/sanitize"; +export * from "./ledger/store"; +export * from "./ledger/invalidation"; +export * from "./ledger/purge"; +export * from "./projection/schema"; +export * from "./projection/verdicts"; +export * from "./projection/rebuild"; +export * from "./events/limits"; +export * from "./conformance/suite-manifest"; +export * from "./ledger/artifact-refs"; +export * from "./projection/verification"; +export * from "./observe/from-conformance"; diff --git a/src/lab/ledger/artifact-refs.ts b/src/lab/ledger/artifact-refs.ts new file mode 100644 index 000000000..0832f535e --- /dev/null +++ b/src/lab/ledger/artifact-refs.ts @@ -0,0 +1,127 @@ +import type { ClaimSnapshotEvent, LabEvent, ObservationEvent } from "../events/types"; +import type { InvalidationIndex } from "./invalidation"; + +/** Collect artifacts that the frozen CL-00 retention rules still require. */ +export function collectReferencedArtifactDigests( + events: LabEvent[], + index: InvalidationIndex, + opts: { excludeEventIds?: Set } = {}, +): Set { + const refs = new Set(); + const exclude = opts.excludeEventIds ?? new Set(); + + for (const event of events) { + if (exclude.has(event.eventId)) continue; + if (index.purgedEventIds.has(event.eventId)) continue; + + if (event.eventKind === "observation") { + // CL-00 releases observation artifacts after invalidation or purge. + if (index.invalidatedBy.has(event.eventId)) continue; + addObservationArtifacts(event, refs); + continue; + } + if (event.eventKind === "claim_snapshot") { + // CL-00 retains claim source manifests while any non-purged claim references them. + refs.add(event.sourceManifestDigest); + } + } + return refs; +} + +function addObservationArtifacts(obs: ObservationEvent, refs: Set): void { + refs.add(obs.scenarioManifestDigest); + refs.add(obs.suiteManifestDigest); + for (const digest of obs.fixtureDigests) refs.add(digest); + for (const ref of obs.artifactRefs) refs.add(ref.digest); +} + +export function eventReferencesArtifactDigest(event: LabEvent, digest: string): boolean { + if (event.eventKind === "observation") { + if (event.scenarioManifestDigest === digest) return true; + if (event.suiteManifestDigest === digest) return true; + if (event.fixtureDigests.includes(digest)) return true; + return event.artifactRefs.some((ref) => ref.digest === digest); + } + if (event.eventKind === "claim_snapshot") { + return event.sourceManifestDigest === digest; + } + return false; +} + +/** + * Expand purge targets so explicitly sensitive artifact digests cannot survive + * while any retained ledger evidence line still references them. + */ +export function expandSensitiveArtifactEventTargets( + events: LabEvent[], + index: InvalidationIndex, + targetEventIds: Set, + explicitArtifactDigests: Set, +): Set { + const expanded = new Set(targetEventIds); + if (explicitArtifactDigests.size === 0) return expanded; + + for (const event of events) { + if (expanded.has(event.eventId)) continue; + if (index.purgedEventIds.has(event.eventId)) continue; + for (const digest of explicitArtifactDigests) { + if (eventReferencesArtifactDigest(event, digest)) { + expanded.add(event.eventId); + break; + } + } + } + return expanded; +} + +/** Artifacts still required by surviving evidence after excluding purge targets. */ +export function artifactsStillRequired( + events: LabEvent[], + index: InvalidationIndex, + excludeEventIds: Set, +): Set { + return collectReferencedArtifactDigests(events, index, { excludeEventIds }); +} + +export interface ArtifactDeletionPlan { + deletable: string[]; + /** Explicitly sensitive digests that remain pinned by surviving evidence. */ + retainedExplicit: string[]; +} + +/** Plan physical artifact deletion for a purge without silently retaining explicit targets. */ +export function artifactDeletionPlan( + events: LabEvent[], + index: InvalidationIndex, + targetEventIds: Set, + explicitArtifactDigests: string[], +): ArtifactDeletionPlan { + const stillRequired = artifactsStillRequired(events, index, targetEventIds); + const candidates = new Set(explicitArtifactDigests); + + for (const event of events) { + if (!targetEventIds.has(event.eventId)) continue; + if (event.eventKind === "observation") { + const scratch = new Set(); + addObservationArtifacts(event, scratch); + for (const digest of scratch) candidates.add(digest); + } else if (event.eventKind === "claim_snapshot") { + candidates.add(event.sourceManifestDigest); + } + } + + return { + deletable: [...candidates].filter((digest) => !stillRequired.has(digest)).sort(), + retainedExplicit: explicitArtifactDigests.filter((digest) => stillRequired.has(digest)).sort(), + }; +} + +export function observationArtifactDigests(obs: ObservationEvent): string[] { + const out = new Set(); + addObservationArtifacts(obs, out); + return [...out].sort(); +} + +export function claimArtifactDigests(claim: ClaimSnapshotEvent): string[] { + return [claim.sourceManifestDigest]; +} diff --git a/src/lab/ledger/invalidation.ts b/src/lab/ledger/invalidation.ts new file mode 100644 index 000000000..5c62b8dca --- /dev/null +++ b/src/lab/ledger/invalidation.ts @@ -0,0 +1,136 @@ +import type { + ClaimSnapshotEvent, + InvalidationEvent, + LabEvent, + LedgerCorruption, + ObservationEvent, + PurgeTombstoneEvent, +} from "../events/types"; +import { LabValidationError } from "../events/validate"; + +export interface InvalidationIndex { + /** eventId -> invalidation eventIds that targeted it */ + invalidatedBy: Map; + purgedEventIds: Set; + purgedArtifactDigests: Set; + corruptions: LedgerCorruption[]; +} + +type EventPosition = { kind: LabEvent["eventKind"]; index: number }; + +/** + * Apply purge tombstones before ordinary invalidations. + * Invalidation target lists are atomic: any bad target rejects the whole event. + */ +export function buildInvalidationIndex(events: LabEvent[]): InvalidationIndex { + const purgedEventIds = new Set(); + const purgedArtifactDigests = new Set(); + const invalidatedBy = new Map(); + const corruptions: LedgerCorruption[] = []; + + const validEvidenceIds = new Map(); + const allEventIds = new Map(); + + // First pass: record event positions and evidence positions; apply purges as encountered. + for (let i = 0; i < events.length; i++) { + const event = events[i]!; + allEventIds.set(event.eventId, { kind: event.eventKind, index: i }); + if (event.eventKind === "observation" || event.eventKind === "claim_snapshot") { + validEvidenceIds.set(event.eventId, { kind: event.eventKind, index: i }); + continue; + } + if (event.eventKind === "purge_tombstone") { + applyPurge(event, purgedEventIds, purgedArtifactDigests); + } + } + + // Second pass: validate and apply invalidations against earlier evidence. + for (let i = 0; i < events.length; i++) { + const event = events[i]!; + if (event.eventKind !== "invalidation") continue; + try { + validateInvalidationTargets(event, i, validEvidenceIds, allEventIds); + for (const target of event.targetEventIds) { + const list = invalidatedBy.get(target) ?? []; + list.push(event.eventId); + invalidatedBy.set(target, list); + } + } catch (err) { + corruptions.push({ + kind: "invalid_reference", + eventId: event.eventId, + detail: err instanceof Error ? err.message : String(err), + }); + } + } + + return { invalidatedBy, purgedEventIds, purgedArtifactDigests, corruptions }; +} + +function applyPurge( + event: PurgeTombstoneEvent, + purgedEventIds: Set, + purgedArtifactDigests: Set, +): void { + for (const id of event.targetEventIds) purgedEventIds.add(id); + for (const digest of event.targetArtifactDigests) purgedArtifactDigests.add(digest); +} + +function validateInvalidationTargets( + event: InvalidationEvent, + index: number, + validEvidenceIds: Map, + allEventIds: Map, +): void { + for (const target of event.targetEventIds) { + if (target === event.eventId) { + throw new LabValidationError("self_target", "invalidation cannot target itself"); + } + const meta = validEvidenceIds.get(target); + if (!meta) { + const other = allEventIds.get(target); + if ( + other && + other.index < index && + (other.kind === "invalidation" || other.kind === "purge_tombstone") + ) { + throw new LabValidationError("bad_target_kind", `cannot invalidate ${other.kind}`); + } + throw new LabValidationError("unknown_target", `unknown target ${target}`); + } + if (meta.index >= index) { + throw new LabValidationError("future_target", `target ${target} is not earlier`); + } + // Purged targets may no longer have a ledger line. A previously valid evidence + // ID remains addressable because validEvidenceIds records its original position. + } +} + +export function isEventExcluded( + eventId: string, + index: InvalidationIndex, +): boolean { + if (index.purgedEventIds.has(eventId)) return true; + if (index.invalidatedBy.has(eventId)) return true; + return false; +} + +export function usableObservations( + events: LabEvent[], + index: InvalidationIndex, +): ObservationEvent[] { + return events.filter( + (e): e is ObservationEvent => + e.eventKind === "observation" && !isEventExcluded(e.eventId, index), + ); +} + +export function usableClaims( + events: LabEvent[], + index: InvalidationIndex, +): ClaimSnapshotEvent[] { + return events.filter( + (e): e is ClaimSnapshotEvent => + e.eventKind === "claim_snapshot" && !isEventExcluded(e.eventId, index), + ); +} diff --git a/src/lab/ledger/purge.ts b/src/lab/ledger/purge.ts new file mode 100644 index 000000000..700221b0e --- /dev/null +++ b/src/lab/ledger/purge.ts @@ -0,0 +1,226 @@ +import { + closeTrustedArtifactDir, + deleteArtifactBytes, + openTrustedArtifactDir, + type TrustedArtifactDir, +} from "../artifacts/secure-fs"; +import { ArtifactFsError } from "../artifacts/secure-fs"; +import { + LAB_EVENT_SCHEMA_VERSION, + LAB_PRODUCER, + LAB_PRODUCER_VERSION, + PURGE_ACTIONS, +} from "../constants"; +import type { LabEvent, PurgeTombstoneEvent } from "../events/types"; +import { assignEventId, validateLabEvent } from "../events/validate"; +import { + artifactDeletionPlan, + expandSensitiveArtifactEventTargets, +} from "./artifact-refs"; +import { buildInvalidationIndex } from "./invalidation"; +import { appendLabEvent, replayLabLedger } from "./store"; +import { ensureLabDirs } from "../paths"; +import { rebuildLabProjection } from "../projection/rebuild"; +import { jcsStringify } from "../digest"; +import { + closeSync, + existsSync, + fsyncSync, + openSync, + readdirSync, + renameSync, + rmSync, + unlinkSync, + writeSync, +} from "node:fs"; +import { dirname, join } from "node:path"; + +export class PurgeError extends Error { + readonly code: string; + readonly completedActions: string[]; + constructor(code: string, message: string, completedActions: string[] = []) { + super(message); + this.name = "PurgeError"; + this.code = code; + this.completedActions = [...completedActions]; + } +} + +export interface SensitivePurgeRequest { + configDir?: string; + targetEventIds?: string[]; + targetArtifactDigests?: string[]; + purgeActions?: Array<(typeof PURGE_ACTIONS)[number]>; + recordedAt?: number; + producerVersion?: string; +} + +function writeAll(fd: number, bytes: Uint8Array): void { + let offset = 0; + while (offset < bytes.byteLength) { + const n = writeSync(fd, bytes, offset, bytes.byteLength - offset); + if (n <= 0) throw new PurgeError("short_write", "ledger rewrite short write"); + offset += n; + } +} + +function atomicRewriteLedger(ledgerPath: string, events: LabEvent[]): void { + const body = events.map((e) => jcsStringify(e)).join("\n") + (events.length ? "\n" : ""); + const bytes = new TextEncoder().encode(body); + const parent = dirname(ledgerPath); + const tmpPath = join(parent, `.purge-${process.pid}-${Date.now()}.jsonl.tmp`); + try { + const fd = openSync(tmpPath, "wx", 0o600); + try { + writeAll(fd, bytes); + fsyncSync(fd); + } finally { + closeSync(fd); + } + renameSync(tmpPath, ledgerPath); + if (process.platform !== "win32") { + const dirFd = openSync(parent, "r"); + try { + fsyncSync(dirFd); + } finally { + closeSync(dirFd); + } + } + } catch (err) { + try { + unlinkSync(tmpPath); + } catch { + // Preserve the original failure. The temp file may already have been renamed. + } + throw err; + } +} + +function deleteArtifactsFailClosed(dir: TrustedArtifactDir, digests: string[]): void { + const errors: string[] = []; + for (const digest of digests) { + try { + deleteArtifactBytes(dir, digest); + } catch (err) { + if (err instanceof ArtifactFsError && err.code === "artifact_missing") continue; + errors.push(err instanceof Error ? err.message : String(err)); + } + } + if (errors.length > 0) { + throw new PurgeError("artifact_delete_failed", errors.join("; ")); + } +} + +function purgeBoundedDirectory(dirPath: string): void { + if (!existsSync(dirPath)) return; + const entries = readdirSync(dirPath, { withFileTypes: true }); + for (const entry of entries) { + const full = join(dirPath, entry.name); + try { + rmSync(full, { recursive: entry.isDirectory(), force: true }); + } catch (err) { + throw new PurgeError( + "scratch_export_delete_failed", + err instanceof Error ? err.message : String(err), + ); + } + } +} + +/** + * Exceptional sensitive-evidence purge: + * physically remove targeted JSONL lines and artifacts, append purge_tombstone, + * rebuild SQLite. Fails closed when required sensitive bytes cannot be removed. + */ +export function purgeSensitiveEvidence(req: SensitivePurgeRequest): PurgeTombstoneEvent { + const paths = ensureLabDirs(req.configDir); + const targetEventIds = [...(req.targetEventIds ?? [])].sort(); + const targetArtifactDigests = [...(req.targetArtifactDigests ?? [])].sort(); + const purgeActions = [...(req.purgeActions ?? ["ledger", "sqlite", "artifact", "scratch"])].sort(); + const explicitSensitive = new Set(targetArtifactDigests); + + const replay = replayLabLedger(paths.ledgerPath); + const index = buildInvalidationIndex(replay.events); + const removeIds = expandSensitiveArtifactEventTargets( + replay.events, + index, + new Set(targetEventIds), + explicitSensitive, + ); + + const tombstonePayload = { + schemaVersion: LAB_EVENT_SCHEMA_VERSION, + eventKind: "purge_tombstone" as const, + recordedAt: req.recordedAt ?? Date.now(), + producer: LAB_PRODUCER, + producerVersion: req.producerVersion ?? LAB_PRODUCER_VERSION, + targetEventIds: [...removeIds].sort(), + targetArtifactDigests, + reason: "sensitive_evidence" as const, + purgeActions, + }; + const tombstone = validateLabEvent(assignEventId(tombstonePayload)) as PurgeTombstoneEvent; + + const deletionPlan = purgeActions.includes("artifact") + ? artifactDeletionPlan(replay.events, index, removeIds, targetArtifactDigests) + : { deletable: [], retainedExplicit: [] }; + + if (deletionPlan.retainedExplicit.length > 0) { + throw new PurgeError( + "sensitive_bytes_retained", + `explicit sensitive artifacts remain required: ${deletionPlan.retainedExplicit.join(",")}`, + ); + } + + let dir: TrustedArtifactDir | null = null; + const completed: string[] = []; + try { + if (purgeActions.includes("scratch")) { + purgeBoundedDirectory(paths.scratchDir); + completed.push("scratch"); + } + if (purgeActions.includes("export")) { + purgeBoundedDirectory(paths.exportDir); + completed.push("export"); + } + + if (purgeActions.includes("artifact")) { + if (deletionPlan.deletable.length > 0) { + dir = openTrustedArtifactDir(paths.artifactsDir); + deleteArtifactsFailClosed(dir, deletionPlan.deletable); + } + completed.push("artifact"); + } + + if (purgeActions.includes("ledger")) { + const kept: LabEvent[] = []; + for (const event of replay.events) { + if (removeIds.has(event.eventId)) continue; + kept.push(event); + } + kept.push(tombstone); + atomicRewriteLedger(paths.ledgerPath, kept); + } else { + appendLabEvent(paths.ledgerPath, tombstone); + } + completed.push("ledger"); + + if (purgeActions.includes("sqlite")) { + rebuildLabProjection(req.configDir); + completed.push("sqlite"); + } + + return tombstone; + } catch (err) { + if (err instanceof PurgeError) { + throw new PurgeError(err.code, err.message, [...completed, ...err.completedActions]); + } + throw new PurgeError( + "purge_failed", + err instanceof Error ? err.message : String(err), + completed, + ); + } finally { + if (dir) closeTrustedArtifactDir(dir); + } +} diff --git a/src/lab/ledger/store.ts b/src/lab/ledger/store.ts new file mode 100644 index 000000000..ade8c3e3d --- /dev/null +++ b/src/lab/ledger/store.ts @@ -0,0 +1,278 @@ +import { + closeSync, + existsSync, + fsyncSync, + mkdirSync, + openSync, + readSync, + statSync, + writeSync, +} from "node:fs"; +import { dirname } from "node:path"; +import { jcsStringify } from "../digest"; +import { MAX_SERIALIZED_EVENT_BYTES } from "../constants"; +import type { LabEvent, LedgerCorruption, ReplayResult } from "../events/types"; +import { LabValidationError, validateLabEvent } from "../events/validate"; +import { ensureLabDirs, labLedgerPath } from "../paths"; + +export interface LedgerStore { + path: string; + append(event: LabEvent): void; + replay(): ReplayResult; +} + +/** Durable append of one validated event as a single JSONL line + fsync. */ +export function appendLabEvent(ledgerPath: string, event: LabEvent): void { + const validated = validateLabEvent(event); + mkdirSync(dirname(ledgerPath), { recursive: true, mode: 0o700 }); + const line = `${jcsStringify(validated)}\n`; + const bytes = new TextEncoder().encode(line); + const fd = openSync(ledgerPath, "a", 0o600); + try { + let written = 0; + while (written < bytes.byteLength) { + const n = writeSync(fd, bytes, written, bytes.byteLength - written); + if (n <= 0) { + throw new LabValidationError("short_write", "ledger append made no progress"); + } + written += n; + } + fsyncSync(fd); + } finally { + closeSync(fd); + } +} + +function processLine( + line: string, + lineNumber: number, + lineHasTrailingNewline: boolean, + events: LabEvent[], + seenIds: Set, + corruptions: LedgerCorruption[], +): void { + if (!lineHasTrailingNewline) { + corruptions.push({ + kind: "partial_line", + lineNumber, + detail: "partial final JSONL line (missing trailing newline)", + }); + return; + } + if (line.trim() === "") { + corruptions.push({ kind: "malformed_line", lineNumber, detail: "empty line" }); + return; + } + + let parsed: unknown; + try { + parsed = JSON.parse(line); + } catch { + corruptions.push({ kind: "malformed_line", lineNumber, detail: "JSON parse failed" }); + return; + } + + let event: LabEvent; + try { + event = validateLabEvent(parsed); + } catch (err) { + corruptions.push({ + kind: "invalid_event", + lineNumber, + detail: err instanceof Error ? err.message : String(err), + }); + return; + } + + if (seenIds.has(event.eventId)) { + corruptions.push({ + kind: "duplicate_event", + lineNumber, + eventId: event.eventId, + detail: "duplicate eventId", + }); + return; + } + seenIds.add(event.eventId); + events.push(event); +} + +function splitIncompleteUtf8Tail(buf: Buffer): { processable: Buffer; remainder: Buffer } { + if (buf.length === 0) return { processable: buf, remainder: Buffer.alloc(0) }; + for (let back = 1; back <= 4 && back <= buf.length; back++) { + const byte = buf[buf.length - back]!; + if ((byte & 0xc0) === 0x80) continue; + const needed = byte >= 0xf0 ? 4 : byte >= 0xe0 ? 3 : byte >= 0xc0 ? 2 : 1; + const seqStart = buf.length - back; + const available = buf.length - seqStart; + if (available < needed) { + return { processable: buf.subarray(0, seqStart), remainder: buf.subarray(seqStart) }; + } + break; + } + return { processable: buf, remainder: Buffer.alloc(0) }; +} + +function processBufferedLines( + buf: Buffer, + state: { + lineNumber: number; + totalLineCount: number; + skippingOversizedLine: boolean; + events: LabEvent[]; + seenIds: Set; + corruptions: LedgerCorruption[]; + }, +): { carry: Buffer; skippingOversizedLine: boolean } { + let start = 0; + let skipping = state.skippingOversizedLine; + + while (start < buf.length) { + const newlineIdx = buf.indexOf(0x0a, start); + if (newlineIdx < 0) { + const tail = buf.subarray(start); + if (skipping) { + return { carry: Buffer.alloc(0), skippingOversizedLine: true }; + } + if (tail.length > MAX_SERIALIZED_EVENT_BYTES) { + state.lineNumber += 1; + state.totalLineCount += 1; + state.corruptions.push({ + kind: "malformed_line", + lineNumber: state.lineNumber, + detail: `line exceeds ${MAX_SERIALIZED_EVENT_BYTES} bytes without newline`, + }); + return { carry: tail, skippingOversizedLine: true }; + } + return { carry: tail, skippingOversizedLine: false }; + } + + const lineBytes = buf.subarray(start, newlineIdx); + start = newlineIdx + 1; + + if (skipping) { + skipping = false; + continue; + } + + state.lineNumber += 1; + state.totalLineCount += 1; + processLine(lineBytes.toString("utf8"), state.lineNumber, true, state.events, state.seenIds, state.corruptions); + } + + return { carry: Buffer.alloc(0), skippingOversizedLine: skipping }; +} + +/** + * Replay the JSONL ledger using chunked reads (no whole-file string buffer). + * Malformed or partial lines contribute no evidence and are reported as corruption. + */ +export function replayLabLedger(ledgerPath: string): ReplayResult { + const corruptions: LedgerCorruption[] = []; + const events: LabEvent[] = []; + if (!existsSync(ledgerPath)) { + return { events, corruptions, validLineCount: 0, totalLineCount: 0 }; + } + const size = statSync(ledgerPath).size; + if (size === 0) { + return { + events, + corruptions: [{ kind: "empty_ledger", detail: "ledger file is empty" }], + validLineCount: 0, + totalLineCount: 0, + }; + } + + const fd = openSync(ledgerPath, "r"); + const chunkSize = 64 * 1024; + const chunk = Buffer.alloc(chunkSize); + let carry: Buffer = Buffer.alloc(0); + let lineNumber = 0; + let totalLineCount = 0; + const seenIds = new Set(); + let offset = 0; + let skippingOversizedLine = false; + + const state = { + lineNumber, + totalLineCount, + skippingOversizedLine, + events, + seenIds, + corruptions, + }; + + try { + while (offset < size) { + const toRead = Math.min(chunkSize, size - offset); + const n = readSync(fd, chunk, 0, toRead, offset); + if (n <= 0) break; + offset += n; + + const combined = carry.length > 0 + ? Buffer.concat([carry, chunk.subarray(0, n)]) + : chunk.subarray(0, n); + const { processable, remainder } = splitIncompleteUtf8Tail(combined); + carry = remainder; + + const result = processBufferedLines(processable, state); + carry = result.carry.length > 0 + ? Buffer.from(Buffer.concat([carry, result.carry])) + : carry; + skippingOversizedLine = result.skippingOversizedLine; + state.skippingOversizedLine = skippingOversizedLine; + lineNumber = state.lineNumber; + totalLineCount = state.totalLineCount; + } + + if (carry.length > 0) { + if (skippingOversizedLine) { + lineNumber += 1; + totalLineCount += 1; + corruptions.push({ + kind: "malformed_line", + lineNumber, + detail: `oversized line exceeds ${MAX_SERIALIZED_EVENT_BYTES} bytes`, + }); + } else if (carry.length > MAX_SERIALIZED_EVENT_BYTES) { + lineNumber += 1; + totalLineCount += 1; + corruptions.push({ + kind: "malformed_line", + lineNumber, + detail: `partial line exceeds ${MAX_SERIALIZED_EVENT_BYTES} bytes`, + }); + } else { + lineNumber += 1; + totalLineCount += 1; + processLine(carry.toString("utf8"), lineNumber, false, events, seenIds, corruptions); + } + } + } finally { + closeSync(fd); + } + + return { + events, + corruptions, + validLineCount: events.length, + totalLineCount, + }; +} + +export function openLedgerStore(configDir?: string): LedgerStore { + const paths = ensureLabDirs(configDir); + return { + path: paths.ledgerPath, + append(event: LabEvent) { + appendLabEvent(paths.ledgerPath, event); + }, + replay() { + return replayLabLedger(paths.ledgerPath); + }, + }; +} + +export function defaultLedgerPath(configDir?: string): string { + return labLedgerPath(configDir); +} diff --git a/src/lab/observe/from-conformance.ts b/src/lab/observe/from-conformance.ts new file mode 100644 index 000000000..5a165439c --- /dev/null +++ b/src/lab/observe/from-conformance.ts @@ -0,0 +1,343 @@ +/** + * Persistence seam: transform CL-01 conformance results into valid observation events. + * Deterministic harness execution stays separate from ledger persistence. + */ +import { createHash } from "node:crypto"; +import { createArtifactStore, type ArtifactStore } from "../artifacts/store"; +import { + LAB_EVENT_SCHEMA_VERSION, + LAB_PRODUCER, + LAB_PRODUCER_VERSION, + type ObservationOutcome, +} from "../constants"; +import { + jcsStringify, + scenarioManifestDigest, + subjectIdForSubject, + suiteManifestDigest, +} from "../digest"; +import type { ObservationEvent, ProtocolSubjectV1 } from "../events/types"; +import { assignEventId } from "../events/validate"; +import { appendLabEvent } from "../ledger/store"; +import { ensureLabDirs } from "../paths"; +import type { + CaseAuthority, + CaseRecord, + ProtocolExecutionContextV1, + ScenarioRunResult, +} from "../conformance/types"; +import { expandScenario } from "../conformance/manifest"; +import { suiteManifestObjectForCase } from "../conformance/suite-manifest"; +import { fixtureDigest } from "../conformance/digest"; +import { resolveProtocolExecutionContext } from "../conformance/executor"; + +const COMPAT_VERSION = "protocol-v1"; + +type TimedScenarioRunResult = ScenarioRunResult & { + startedAt?: number; + completedAt?: number; +}; + +export interface PersistConformanceOptions { + configDir?: string; + recordedAt?: number; + /** Actual execution timestamps from the CL-01 runner; never fabricated. */ + startedAt?: number; + completedAt?: number; + producerVersion?: string; + artifactStore?: ArtifactStore; +} + +export interface PersistedConformanceObservation { + event: ObservationEvent; + ledgerPath: string; +} + +function validateExecutionContext( + caseRecord: CaseRecord, + ctx: ProtocolExecutionContextV1, +): ProtocolExecutionContextV1 { + const checks: Array<[string, string[], string]> = [ + ["inboundProtocols", caseRecord.requirements.inboundProtocols, ctx.inboundProtocol], + ["upstreamProtocols", caseRecord.requirements.upstreamProtocols, ctx.upstreamProtocol], + ["surfaces", caseRecord.requirements.surfaces, ctx.surface], + ]; + for (const [name, declared, actual] of checks) { + if (declared.length === 0) throw new Error(`case ${caseRecord.id} has empty ${name}`); + if (!declared.includes(actual)) { + throw new Error(`case ${caseRecord.id} execution context violates ${name}`); + } + } + return ctx; +} + +function behaviorFingerprintForCase( + caseRecord: CaseRecord, + executionContext?: ProtocolExecutionContextV1, +): string { + const ctx = validateExecutionContext( + caseRecord, + executionContext ?? resolveProtocolExecutionContext(caseRecord), + ); + const adapter = upstreamAdapter(ctx.upstreamProtocol); + const values = { + schemaVersion: 1, + resolverVersion: 1, + values: { + "wire.adapter": { + source: "lab_forced", + value: adapter, + }, + "wire.upstreamProtocol": { + source: "lab_forced", + value: ctx.upstreamProtocol, + }, + "runtime.arch": { + source: "lab_forced", + value: process.arch, + }, + "runtime.bunVersion": { + source: "lab_forced", + value: process.versions.bun ?? Bun.version, + }, + "runtime.platform": { + source: "lab_forced", + value: process.platform, + }, + }, + }; + return createHash("sha256").update(jcsStringify(values)).digest("hex"); +} + +function protocolSubject(caseRecord: CaseRecord, result: ScenarioRunResult): ProtocolSubjectV1 { + const ctx = validateExecutionContext( + caseRecord, + result.executionContext ?? resolveProtocolExecutionContext(caseRecord), + ); + return { + subjectSchemaVersion: 1, + subjectKind: "protocol", + opencodexCompatibilityVersion: COMPAT_VERSION, + effectiveAdapter: upstreamAdapter(ctx.upstreamProtocol), + inboundProtocol: ctx.inboundProtocol, + upstreamProtocol: ctx.upstreamProtocol, + surface: ctx.surface, + behaviorFingerprint: behaviorFingerprintForCase(caseRecord, ctx), + }; +} + +function upstreamAdapter(protocol: string): string { + switch (protocol) { + case "openai-responses": + return "openai-responses"; + case "anthropic-messages": + return "anthropic"; + case "openai-chat": + return "openai-chat"; + default: + throw new Error(`unsupported protocol identity: ${protocol}`); + } +} + +function outcomeFromResult(result: ScenarioRunResult): ObservationOutcome { + if (result.passed) return "pass"; + switch (result.classification) { + case "timeout": + case "budget_exhausted": + return "blocked"; + case "inconclusive": + case "harness_failure": + return "inconclusive"; + case "protocol_failure": + case "capability_failure": + case "behavioral_failure": + return "fail"; + default: { + const _never: never = result.classification; + return _never; + } + } +} + +function requireExecutionTimes( + result: ScenarioRunResult, + opts: PersistConformanceOptions, +): { startedAt: number; completedAt: number } { + const timed = result as TimedScenarioRunResult; + const startedAt = opts.startedAt ?? timed.startedAt; + const completedAt = opts.completedAt ?? timed.completedAt; + if (!Number.isInteger(startedAt) || !Number.isInteger(completedAt)) { + throw new Error("real startedAt/completedAt are required for persisted conformance evidence"); + } + if (startedAt! < 0 || completedAt! < startedAt!) { + throw new Error("invalid persisted conformance execution timestamps"); + } + return { startedAt: startedAt!, completedAt: completedAt! }; +} + +/** + * Build a valid protocol_conformance observation from one CL-01 scenario result. + * Does not append; use persistConformanceResult for ledger write. + */ +export function observationFromConformanceResult( + result: ScenarioRunResult, + caseRecord: CaseRecord, + authority: CaseAuthority, + opts: PersistConformanceOptions = {}, +): { event: ObservationEvent; artifacts: ReturnType[] } { + const paths = ensureLabDirs(opts.configDir); + const ownsStore = !opts.artifactStore; + const store = opts.artifactStore ?? createArtifactStore(paths.artifactsDir); + try { + const { startedAt, completedAt } = requireExecutionTimes(result, opts); + const recordedAt = opts.recordedAt ?? completedAt; + + const expandedScenario = expandScenario(caseRecord, authority); + const scenarioDigest = scenarioManifestDigest(expandedScenario); + const suiteExpanded = suiteManifestObjectForCase(caseRecord, authority); + const suiteDigest = suiteManifestDigest(suiteExpanded); + + const fixtureDigests: string[] = []; + const artifacts: ReturnType[] = []; + + const putFixture = (fixture: CaseRecord["fixture"]) => { + const bytes = new TextEncoder().encode(fixture.bytesUtf8); + const digest = fixtureDigest(bytes); + fixtureDigests.push(digest); + artifacts.push( + store.put({ + artifactClass: "fixture", + payload: bytes, + expectedDigest: digest, + mediaType: fixture.mediaType, + }), + ); + }; + putFixture(caseRecord.fixture); + if (caseRecord.initiatingRequest) putFixture(caseRecord.initiatingRequest); + + artifacts.push( + store.put({ + artifactClass: "scenario_manifest", + payload: expandedScenario, + expectedDigest: scenarioDigest, + }), + ); + artifacts.push( + store.put({ + artifactClass: "suite_manifest", + payload: suiteExpanded, + expectedDigest: suiteDigest, + }), + ); + + const assertionReport = store.put({ + artifactClass: "assertion_report", + payload: { + scenarioId: result.scenarioId, + passed: result.passed, + classification: result.classification, + assertions: result.assertionResults.map((a) => ({ + id: a.id, + operator: a.operator, + required: a.required, + passed: a.passed, + observedSummary: a.observedSummary, + reason: a.reason, + })), + }, + }); + artifacts.push(assertionReport); + + const subject = protocolSubject(caseRecord, result); + const subjectId = subjectIdForSubject(subject); + const outcome = outcomeFromResult(result); + const authorityLimits = authority.manifestDefaults.executionLimits; + + const eventWithoutId = { + schemaVersion: LAB_EVENT_SCHEMA_VERSION, + eventKind: "observation" as const, + recordedAt, + producer: LAB_PRODUCER, + producerVersion: opts.producerVersion ?? LAB_PRODUCER_VERSION, + evidenceLayer: "protocol_conformance" as const, + scenarioId: caseRecord.id, + scenarioVersion: String(authority.manifestDefaults.version), + scenarioManifestDigest: scenarioDigest, + suiteId: caseRecord.suite, + suiteVersion: String(authority.manifestDefaults.suiteVersion), + suiteManifestDigest: suiteDigest, + fixtureDigests, + subject, + subjectId, + startedAt, + completedAt, + executionMode: "fixture" as const, + attempt: 1, + limits: { ...authorityLimits }, + outcome, + assertions: result.assertionResults.map((a) => ({ + id: a.id, + operator: a.operator, + required: a.required, + passed: a.passed, + expectedSummary: "see_assertion_report", + observedSummary: a.observedSummary.slice(0, 512), + ...(a.reason ? { reason: a.reason } : {}), + })), + ...(caseRecord.expectedFailure + ? { expectedFailure: { ...caseRecord.expectedFailure } } + : {}), + environment: { + runtime: { + platform: process.platform, + arch: process.arch, + bunVersion: process.versions.bun ?? Bun.version, + }, + }, + artifactRefs: artifacts, + ...(result.passed + ? {} + : { + failure: { + class: result.classification, + code: result.secondaryCode ?? result.classification, + retryable: false, + attribution: + result.classification === "harness_failure" + ? ("harness" as const) + : ("opencodex" as const), + }, + }), + }; + + const event = assignEventId(eventWithoutId) as ObservationEvent; + return { event, artifacts }; + } finally { + if (ownsStore) store.close(); + } +} + +/** Transform → validate → append one CL-01 result into the canonical JSONL ledger. */ +export function persistConformanceResult( + result: ScenarioRunResult, + caseRecord: CaseRecord, + authority: CaseAuthority, + opts: PersistConformanceOptions = {}, +): PersistedConformanceObservation { + const paths = ensureLabDirs(opts.configDir); + const ownsStore = !opts.artifactStore; + const store = opts.artifactStore ?? createArtifactStore(paths.artifactsDir); + try { + const { event } = observationFromConformanceResult(result, caseRecord, authority, { + ...opts, + artifactStore: store, + }); + appendLabEvent(paths.ledgerPath, event); + return { event, ledgerPath: paths.ledgerPath }; + } finally { + if (ownsStore) store.close(); + } +} + +export { behaviorFingerprintForCase }; diff --git a/src/lab/paths.ts b/src/lab/paths.ts new file mode 100644 index 000000000..d54e11f4d --- /dev/null +++ b/src/lab/paths.ts @@ -0,0 +1,60 @@ +import { chmodSync, mkdirSync, statSync } from "node:fs"; +import { join } from "node:path"; +import { getConfigDir } from "../config"; + +function ensureRestrictedDir(dir: string): void { + mkdirSync(dir, { recursive: true, mode: 0o700 }); + if (process.platform === "win32") return; + const mode = statSync(dir).mode & 0o777; + if (mode !== 0o700) chmodSync(dir, 0o700); +} + +/** Canonical Compatibility Lab state root under the OpenCodex config dir. */ +export function labRoot(configDir = getConfigDir()): string { + return join(configDir, "lab"); +} + +export function labLedgerPath(configDir = getConfigDir()): string { + return join(labRoot(configDir), "compatibility.jsonl"); +} + +export function labSqlitePath(configDir = getConfigDir()): string { + return join(labRoot(configDir), "compatibility.sqlite"); +} + +export function labArtifactsDir(configDir = getConfigDir()): string { + return join(labRoot(configDir), "artifacts"); +} + +export function labScratchDir(configDir = getConfigDir()): string { + return join(labRoot(configDir), "scratch"); +} + +export function labExportDir(configDir = getConfigDir()): string { + return join(labRoot(configDir), "export"); +} + +/** Ensure lab directories exist with restrictive permissions where the platform allows. */ +export function ensureLabDirs(configDir = getConfigDir()): { + root: string; + ledgerPath: string; + sqlitePath: string; + artifactsDir: string; + scratchDir: string; + exportDir: string; +} { + const root = labRoot(configDir); + const artifactsDir = labArtifactsDir(configDir); + const scratchDir = labScratchDir(configDir); + const exportDir = labExportDir(configDir); + ensureRestrictedDir(root); + ensureRestrictedDir(artifactsDir); + return { + root, + ledgerPath: labLedgerPath(configDir), + sqlitePath: labSqlitePath(configDir), + artifactsDir, + scratchDir, + exportDir, + }; +} diff --git a/src/lab/projection/rebuild.ts b/src/lab/projection/rebuild.ts new file mode 100644 index 000000000..bea548167 --- /dev/null +++ b/src/lab/projection/rebuild.ts @@ -0,0 +1,477 @@ +import { Database } from "bun:sqlite"; +import { existsSync, unlinkSync } from "node:fs"; +import { createArtifactStore, loadClaimSourceManifest } from "../artifacts/store"; +import { ArtifactFsError } from "../artifacts/secure-fs"; +import { sanitizeDiagnostic } from "../artifacts/sanitize"; +import { LAB_PROJECTION_SPEC_VERSION } from "../constants"; +import { expandScenario, loadCaseAuthority } from "../conformance/manifest"; +import { scenarioManifestDigest, jcsStringify } from "../digest"; +import { + parseSuiteManifestFromArtifact, + type ScenarioRequirements, +} from "./verification"; +import type { ClaimSnapshotEvent, LabEvent, LedgerCorruption } from "../events/types"; +import { buildInvalidationIndex, isEventExcluded } from "../ledger/invalidation"; +import { replayLabLedger } from "../ledger/store"; +import { ensureLabDirs } from "../paths"; +import { LAB_SQLITE_DDL, LAB_SQLITE_SCHEMA_VERSION } from "./schema"; +import { + claimKeyString, + excludeEventIds, + projectVerdicts, + projectionKeyString, + resolveClaimStates, +} from "./verdicts"; + +export interface RebuildResult { + events: number; + verdicts: number; + corruptions: LedgerCorruption[]; + sqlitePath: string; +} + +function wipeSqlite(path: string): void { + for (const candidate of [path, `${path}-wal`, `${path}-shm`]) { + let removed = false; + for (let attempt = 0; attempt < 8; attempt++) { + try { + if (existsSync(candidate)) unlinkSync(candidate); + removed = true; + break; + } catch (err) { + const code = err && typeof err === "object" && "code" in err + ? String((err as { code: unknown }).code) + : ""; + if (code !== "EBUSY" && code !== "EPERM") throw err; + if (attempt < 7) Bun.sleepSync(20 * (attempt + 1)); + } + } + if (!removed) { + throw new Error(`failed to remove stale projection file after retries: ${candidate}`); + } + } +} + +function resetProjectionSchema(db: Database): void { + db.exec(` + DROP TABLE IF EXISTS verdicts; + DROP TABLE IF EXISTS corruption; + DROP TABLE IF EXISTS artifacts; + DROP TABLE IF EXISTS purges; + DROP TABLE IF EXISTS invalidations; + DROP TABLE IF EXISTS claims; + DROP TABLE IF EXISTS observations; + DROP TABLE IF EXISTS subjects; + DROP TABLE IF EXISTS events; + DROP TABLE IF EXISTS schema_meta; + `); + db.exec(LAB_SQLITE_DDL); +} + +interface ArtifactValidationResult { + unusableObservationIds: Set; + unusableClaimEventIds: Set; +} + +/** + * Deterministic rebuild: + * delete compatibility.sqlite → replay JSONL → validate artifacts → project. + */ +export function rebuildLabProjection(configDir?: string): RebuildResult { + const paths = ensureLabDirs(configDir); + wipeSqlite(paths.sqlitePath); + + const replay = replayLabLedger(paths.ledgerPath); + const corruptions: LedgerCorruption[] = [...replay.corruptions]; + const index = buildInvalidationIndex(replay.events); + corruptions.push(...index.corruptions); + + const artifactStore = createArtifactStore(paths.artifactsDir); + const validation = validateRequiredArtifacts(replay.events, index, artifactStore, corruptions); + + const authority = loadCaseAuthority(); + const scenarioRequirementsByDigest = new Map(); + for (const caseRecord of authority.cases) { + const expanded = expandScenario(caseRecord, authority); + scenarioRequirementsByDigest.set(scenarioManifestDigest(expanded), { + inboundProtocols: [...caseRecord.requirements.inboundProtocols], + upstreamProtocols: [...caseRecord.requirements.upstreamProtocols], + surfaces: [...caseRecord.requirements.surfaces], + freshness: { ...authority.manifestDefaults.freshness }, + }); + } + + const loadSuiteManifest = (digest: string) => { + try { + const bytes = artifactStore.get(digest, { artifactClass: "suite_manifest" }); + const parsed = JSON.parse(new TextDecoder().decode(bytes)); + return parseSuiteManifestFromArtifact(parsed); + } catch { + return null; + } + }; + const loadScenarioManifest = (digest: string) => { + try { + const bytes = artifactStore.get(digest, { artifactClass: "scenario_manifest" }); + return JSON.parse(new TextDecoder().decode(bytes)) as Record; + } catch { + return null; + } + }; + const loadScenarioRequirements = (digest: string) => scenarioRequirementsByDigest.get(digest) ?? null; + + const db = new Database(paths.sqlitePath); + let transactionOpen = false; + try { + db.exec("PRAGMA journal_mode=DELETE;"); + db.exec("PRAGMA foreign_keys=OFF;"); + db.exec("BEGIN IMMEDIATE;"); + transactionOpen = true; + resetProjectionSchema(db); + + db.prepare( + "INSERT OR REPLACE INTO schema_meta(key, value) VALUES (?, ?)", + ).run("schema_version", String(LAB_SQLITE_SCHEMA_VERSION)); + db.prepare( + "INSERT OR REPLACE INTO schema_meta(key, value) VALUES (?, ?)", + ).run("projection_spec_version", LAB_PROJECTION_SPEC_VERSION); + db.prepare( + "INSERT OR REPLACE INTO schema_meta(key, value) VALUES (?, ?)", + ).run("built_at_ms", String(Date.now())); + + const insertCorruption = db.prepare( + "INSERT INTO corruption(kind, line_number, event_id, detail) VALUES (?, ?, ?, ?)", + ); + for (const c of corruptions) { + insertCorruption.run(c.kind, c.lineNumber ?? null, c.eventId ?? null, c.detail); + } + + const insertEvent = db.prepare( + `INSERT INTO events(event_id, event_kind, recorded_at, producer, producer_version, payload_json, excluded, exclusion_reason) + VALUES (?, ?, ?, ?, ?, ?, ?, ?)`, + ); + const insertSubject = db.prepare( + `INSERT OR IGNORE INTO subjects(subject_id, subject_kind, subject_json) VALUES (?, ?, ?)`, + ); + const insertObs = db.prepare( + `INSERT INTO observations( + event_id, subject_id, evidence_layer, suite_id, suite_version, suite_manifest_digest, + scenario_id, scenario_version, scenario_manifest_digest, outcome, completed_at, execution_mode + ) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)`, + ); + const insertClaim = db.prepare( + `INSERT INTO claims( + event_id, subject_id, capability, polarity, source_manifest_digest, + effective_at, recorded_at, supersedes_json, current, usable + ) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?)`, + ); + const insertInv = db.prepare( + `INSERT INTO invalidations(event_id, reason, targets_json, recorded_at, applied) VALUES (?, ?, ?, ?, ?)`, + ); + const insertPurge = db.prepare( + `INSERT INTO purges(event_id, target_event_ids_json, target_artifact_digests_json, purge_actions_json, recorded_at) + VALUES (?, ?, ?, ?, ?)`, + ); + const insertArtifact = db.prepare( + `INSERT INTO artifacts(digest, artifact_class, media_type, byte_count, status, last_error) + VALUES (?, ?, ?, ?, ?, ?) + ON CONFLICT(digest) DO UPDATE SET + artifact_class = COALESCE(excluded.artifact_class, artifacts.artifact_class), + media_type = COALESCE(excluded.media_type, artifacts.media_type), + byte_count = COALESCE(excluded.byte_count, artifacts.byte_count), + status = CASE + WHEN artifacts.status = 'purged_unavailable' OR excluded.status = 'purged_unavailable' + THEN 'purged_unavailable' + WHEN artifacts.status = 'corrupt' OR excluded.status = 'corrupt' + THEN 'corrupt' + ELSE 'present' + END, + last_error = COALESCE(excluded.last_error, artifacts.last_error)`, + ); + + const excluded = excludeEventIds(index); + const usableClaimEvents = replay.events.filter( + (e): e is ClaimSnapshotEvent => + e.eventKind === "claim_snapshot" && !isEventExcluded(e.eventId, index), + ); + const claimStates = resolveClaimStates(usableClaimEvents, { + unusableClaimEventIds: validation.unusableClaimEventIds, + purgedEventIds: index.purgedEventIds, + }); + + for (const event of replay.events) { + const isExcluded = excluded.has(event.eventId); + let exclusionReason: string | null = null; + if (index.purgedEventIds.has(event.eventId)) exclusionReason = "purged"; + else if (index.invalidatedBy.has(event.eventId)) exclusionReason = "invalidated"; + + insertEvent.run( + event.eventId, + event.eventKind, + event.recordedAt, + event.producer, + event.producerVersion, + jcsStringify(event), + isExcluded ? 1 : 0, + exclusionReason, + ); + + if (event.eventKind === "observation") { + insertSubject.run(event.subjectId, event.subject.subjectKind, jcsStringify(event.subject)); + const usable = !isExcluded && !validation.unusableObservationIds.has(event.eventId); + if (usable) { + insertObs.run( + event.eventId, + event.subjectId, + event.evidenceLayer, + event.suiteId, + event.suiteVersion, + event.suiteManifestDigest, + event.scenarioId, + event.scenarioVersion, + event.scenarioManifestDigest, + event.outcome, + event.completedAt, + event.executionMode, + ); + } + for (const ref of event.artifactRefs) { + const purged = index.purgedArtifactDigests.has(ref.digest); + const corrupt = validation.unusableObservationIds.has(event.eventId); + insertArtifact.run( + ref.digest, + ref.artifactClass, + ref.mediaType, + ref.byteCount, + purged ? "purged_unavailable" : corrupt ? "corrupt" : "present", + corrupt ? "required artifact unusable" : null, + ); + } + } else if (event.eventKind === "claim_snapshot") { + insertSubject.run(event.subjectId, event.subject.subjectKind, jcsStringify(event.subject)); + const key = claimKeyString(event.subjectId, event.capability); + const state = claimStates.states.get(key); + const current = state?.current?.eventId === event.eventId ? 1 : 0; + const claimCorruption = corruptions.find( + (c) => c.kind === "claim_corruption" && c.eventId === event.eventId, + ); + const usable = !isExcluded && !state?.corruption && + !validation.unusableClaimEventIds.has(event.eventId) ? 1 : 0; + + if (!isExcluded) { + insertArtifact.run( + event.sourceManifestDigest, + "claim_source_manifest", + "application/json", + null, + claimCorruption ? "corrupt" : "present", + claimCorruption?.detail ?? null, + ); + } + + insertClaim.run( + event.eventId, + event.subjectId, + event.capability, + event.polarity, + event.sourceManifestDigest, + event.effectiveAt, + event.recordedAt, + jcsStringify(event.supersedes), + current, + usable, + ); + } else if (event.eventKind === "invalidation") { + const applied = !corruptions.some((c) => c.eventId === event.eventId && c.kind === "invalid_reference"); + insertInv.run( + event.eventId, + event.reason, + jcsStringify(event.targetEventIds), + event.recordedAt, + applied ? 1 : 0, + ); + } else if (event.eventKind === "purge_tombstone") { + insertPurge.run( + event.eventId, + jcsStringify(event.targetEventIds), + jcsStringify(event.targetArtifactDigests), + jcsStringify(event.purgeActions), + event.recordedAt, + ); + for (const digest of event.targetArtifactDigests) { + insertArtifact.run(digest, null, null, null, "purged_unavailable", null); + } + } + } + + const { verdicts, corruptions: verdictCorruptions } = projectVerdicts(replay.events, { + index, + unusableObservationIds: validation.unusableObservationIds, + unusableClaimEventIds: validation.unusableClaimEventIds, + loadSuiteManifest, + loadScenarioManifest, + loadScenarioRequirements, + }); + for (const c of verdictCorruptions) { + if (!corruptions.some((x) => x.detail === c.detail && x.eventId === c.eventId)) { + corruptions.push(c); + insertCorruption.run(c.kind, c.lineNumber ?? null, c.eventId ?? null, c.detail); + } + } + + const insertVerdict = db.prepare( + `INSERT INTO verdicts( + projection_key, subject_id, evidence_layer, suite_id, suite_version, suite_manifest_digest, + projection_spec_version, verdict, as_of, scenario_manifest_digests_json, claim_source_digest, + contributing_event_ids_json, contradicting_event_ids_json, notes_json + ) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)`, + ); + + for (const v of verdicts) { + insertVerdict.run( + projectionKeyString(v.key), + v.key.subjectId, + v.key.evidenceLayer, + v.key.suiteId, + v.key.suiteVersion, + v.key.suiteManifestDigest, + v.key.projectionSpecVersion, + v.verdict, + v.asOf, + jcsStringify(v.scenarioManifestDigests), + v.claimSourceDigest ?? null, + jcsStringify(v.contributingEventIds), + jcsStringify(v.contradictingEventIds), + jcsStringify(v.notes), + ); + } + + db.exec("COMMIT;"); + transactionOpen = false; + return { + events: replay.events.length, + verdicts: verdicts.length, + corruptions, + sqlitePath: paths.sqlitePath, + }; + } catch (err) { + if (transactionOpen) { + try { + db.exec("ROLLBACK;"); + } catch { + // Preserve the original rebuild failure. + } + } + throw err; + } finally { + try { + db.exec("PRAGMA foreign_keys=ON;"); + } catch { + // Closing the disposable DB is still safe if pragma restoration fails. + } + db.close(); + artifactStore.close(); + } +} + +function validateRequiredArtifacts( + events: LabEvent[], + index: ReturnType, + artifactStore: ReturnType, + corruptions: LedgerCorruption[], +): ArtifactValidationResult { + const unusableObservationIds = new Set(); + const unusableClaimEventIds = new Set(); + + for (const event of events) { + if (isEventExcluded(event.eventId, index)) continue; + if (event.eventKind === "observation") { + const required: Array<{ digest: string; artifactClass: "scenario_manifest" | "suite_manifest" | "fixture" }> = [ + { digest: event.scenarioManifestDigest, artifactClass: "scenario_manifest" }, + { digest: event.suiteManifestDigest, artifactClass: "suite_manifest" }, + ...event.fixtureDigests.map((digest) => ({ digest, artifactClass: "fixture" as const })), + ]; + let unusable = false; + for (const { digest, artifactClass } of required) { + if (index.purgedArtifactDigests.has(digest)) { + corruptions.push({ + kind: "missing_artifact", + eventId: event.eventId, + detail: `required artifact purged: ${digest}`, + }); + unusable = true; + continue; + } + try { + artifactStore.get(digest, { artifactClass }); + } catch (err) { + const detail = sanitizeDiagnostic(err); + corruptions.push({ + kind: err instanceof ArtifactFsError && + (err.code === "artifact_mismatch" || err.message.includes("mismatch")) + ? "artifact_mismatch" + : "missing_artifact", + eventId: event.eventId, + detail, + }); + unusable = true; + } + } + if (unusable) unusableObservationIds.add(event.eventId); + continue; + } + + if (event.eventKind === "claim_snapshot") { + if (index.purgedArtifactDigests.has(event.sourceManifestDigest)) { + unusableClaimEventIds.add(event.eventId); + corruptions.push({ + kind: "claim_corruption", + eventId: event.eventId, + detail: `claim source artifact purged: ${event.sourceManifestDigest}`, + }); + continue; + } + const loaded = loadClaimSourceManifest(artifactStore, event.sourceManifestDigest, { + subjectId: event.subjectId, + capability: event.capability, + }); + if (!loaded.ok) { + unusableClaimEventIds.add(event.eventId); + corruptions.push({ + kind: "claim_corruption", + eventId: event.eventId, + detail: loaded.corruption, + }); + } + } + } + + return { unusableObservationIds, unusableClaimEventIds }; +} + +/** Snapshot derived verdict rows for rebuild-determinism tests (excludes asOf wall clock). */ +export function readVerdictSnapshot(sqlitePath: string): unknown[] { + const db = new Database(sqlitePath, { readonly: true }); + try { + return db + .query( + `SELECT projection_key, subject_id, evidence_layer, suite_id, suite_version, + suite_manifest_digest, projection_spec_version, verdict, + scenario_manifest_digests_json, claim_source_digest, + contributing_event_ids_json, contradicting_event_ids_json, notes_json + FROM verdicts ORDER BY projection_key`, + ) + .all(); + } finally { + db.close(); + } +} + +export function readCorruptionRows(sqlitePath: string): unknown[] { + const db = new Database(sqlitePath, { readonly: true }); + try { + return db.query(`SELECT kind, line_number, event_id, detail FROM corruption ORDER BY id`).all(); + } finally { + db.close(); + } +} diff --git a/src/lab/projection/schema.ts b/src/lab/projection/schema.ts new file mode 100644 index 000000000..a8519ad12 --- /dev/null +++ b/src/lab/projection/schema.ts @@ -0,0 +1,122 @@ +/** + * Disposable SQLite projection schema for Compatibility Lab (CL-02). + * Not canonical storage — rebuildable from JSONL + content-addressed artifacts. + */ +export const LAB_SQLITE_SCHEMA_VERSION = 2; + +export const LAB_SQLITE_DDL = ` +CREATE TABLE IF NOT EXISTS schema_meta ( + key TEXT PRIMARY KEY, + value TEXT NOT NULL +); + +CREATE TABLE IF NOT EXISTS events ( + event_id TEXT PRIMARY KEY, + event_kind TEXT NOT NULL CHECK (event_kind IN ('observation', 'claim_snapshot', 'invalidation', 'purge_tombstone')), + recorded_at INTEGER NOT NULL, + producer TEXT NOT NULL, + producer_version TEXT NOT NULL, + payload_json TEXT NOT NULL, + excluded INTEGER NOT NULL DEFAULT 0 CHECK (excluded IN (0, 1)), + exclusion_reason TEXT +); + +CREATE INDEX IF NOT EXISTS idx_events_kind ON events(event_kind, recorded_at); +CREATE INDEX IF NOT EXISTS idx_events_excluded ON events(excluded); + +CREATE TABLE IF NOT EXISTS subjects ( + subject_id TEXT PRIMARY KEY, + subject_kind TEXT NOT NULL, + subject_json TEXT NOT NULL +); + +CREATE TABLE IF NOT EXISTS observations ( + event_id TEXT PRIMARY KEY, + subject_id TEXT NOT NULL, + evidence_layer TEXT NOT NULL CHECK (evidence_layer IN ('protocol_conformance', 'live_route_compatibility', 'task_effectiveness')), + suite_id TEXT NOT NULL, + suite_version TEXT NOT NULL, + suite_manifest_digest TEXT NOT NULL, + scenario_id TEXT NOT NULL, + scenario_version TEXT NOT NULL, + scenario_manifest_digest TEXT NOT NULL, + outcome TEXT NOT NULL CHECK (outcome IN ('pass', 'fail', 'blocked', 'inconclusive')), + completed_at INTEGER NOT NULL, + execution_mode TEXT NOT NULL CHECK (execution_mode IN ('fixture', 'live', 'fabric')), + FOREIGN KEY(event_id) REFERENCES events(event_id) +); + +CREATE INDEX IF NOT EXISTS idx_observations_proj ON observations( + subject_id, evidence_layer, suite_id, suite_version, suite_manifest_digest, completed_at +); + +CREATE TABLE IF NOT EXISTS claims ( + event_id TEXT PRIMARY KEY, + subject_id TEXT NOT NULL, + capability TEXT NOT NULL, + polarity TEXT NOT NULL CHECK (polarity IN ('supported', 'not_supported', 'withdrawn')), + source_manifest_digest TEXT NOT NULL, + effective_at INTEGER NOT NULL, + recorded_at INTEGER NOT NULL, + supersedes_json TEXT NOT NULL, + current INTEGER NOT NULL DEFAULT 0 CHECK (current IN (0, 1)), + usable INTEGER NOT NULL DEFAULT 1 CHECK (usable IN (0, 1)), + FOREIGN KEY(event_id) REFERENCES events(event_id) +); + +CREATE INDEX IF NOT EXISTS idx_claims_key ON claims(subject_id, capability, effective_at, event_id); + +CREATE TABLE IF NOT EXISTS invalidations ( + event_id TEXT PRIMARY KEY, + reason TEXT NOT NULL, + targets_json TEXT NOT NULL, + recorded_at INTEGER NOT NULL, + applied INTEGER NOT NULL DEFAULT 0 CHECK (applied IN (0, 1)), + FOREIGN KEY(event_id) REFERENCES events(event_id) +); + +CREATE TABLE IF NOT EXISTS purges ( + event_id TEXT PRIMARY KEY, + target_event_ids_json TEXT NOT NULL, + target_artifact_digests_json TEXT NOT NULL, + purge_actions_json TEXT NOT NULL, + recorded_at INTEGER NOT NULL, + FOREIGN KEY(event_id) REFERENCES events(event_id) +); + +CREATE TABLE IF NOT EXISTS artifacts ( + digest TEXT PRIMARY KEY, + artifact_class TEXT, + media_type TEXT, + byte_count INTEGER, + status TEXT NOT NULL CHECK (status IN ('present', 'corrupt', 'purged_unavailable')), + last_error TEXT +); + +CREATE TABLE IF NOT EXISTS verdicts ( + projection_key TEXT PRIMARY KEY, + subject_id TEXT NOT NULL, + evidence_layer TEXT NOT NULL, + suite_id TEXT NOT NULL, + suite_version TEXT NOT NULL, + suite_manifest_digest TEXT NOT NULL, + projection_spec_version TEXT NOT NULL, + verdict TEXT NOT NULL CHECK (verdict IN ('UNKNOWN', 'CLAIMED', 'PROBED', 'VERIFIED', 'DEGRADED', 'BLOCKED', 'UNSUPPORTED')), + as_of INTEGER NOT NULL, + scenario_manifest_digests_json TEXT NOT NULL, + claim_source_digest TEXT, + contributing_event_ids_json TEXT NOT NULL, + contradicting_event_ids_json TEXT NOT NULL, + notes_json TEXT NOT NULL +); + +CREATE INDEX IF NOT EXISTS idx_verdicts_subject ON verdicts(subject_id, evidence_layer, suite_id); + +CREATE TABLE IF NOT EXISTS corruption ( + id INTEGER PRIMARY KEY AUTOINCREMENT, + kind TEXT NOT NULL, + line_number INTEGER, + event_id TEXT, + detail TEXT NOT NULL +); +`; diff --git a/src/lab/projection/verdicts.ts b/src/lab/projection/verdicts.ts new file mode 100644 index 000000000..cafe66cd6 --- /dev/null +++ b/src/lab/projection/verdicts.ts @@ -0,0 +1,396 @@ +import type { CompatibilityVerdict } from "../constants"; +import { LAB_PROJECTION_SPEC_VERSION } from "../constants"; +import type { SuiteManifestV1 } from "../conformance/suite-manifest"; +import { jcsStringify } from "../digest"; +import type { + ClaimSnapshotEvent, + LabEvent, + LedgerCorruption, + ObservationEvent, +} from "../events/types"; +import { + buildInvalidationIndex, + isEventExcluded, + usableClaims, + usableObservations, + type InvalidationIndex, +} from "../ledger/invalidation"; +import { + evaluateAllApplicableRequiredPassV1, + newestObservationByScenario, + type ScenarioRequirements, +} from "./verification"; + +export interface ProjectionKey { + subjectId: string; + evidenceLayer: string; + suiteId: string; + suiteVersion: string; + suiteManifestDigest: string; + projectionSpecVersion: string; +} + +function componentKey(parts: readonly string[]): string { + return jcsStringify([...parts]); +} + +export function projectionKeyString(key: ProjectionKey): string { + return componentKey([ + key.subjectId, + key.evidenceLayer, + key.suiteId, + key.suiteVersion, + key.suiteManifestDigest, + key.projectionSpecVersion, + ]); +} + +export function claimKeyString(subjectId: string, capability: string): string { + return componentKey([subjectId, capability]); +} + +export interface DerivedVerdict { + key: ProjectionKey; + verdict: CompatibilityVerdict; + asOf: number; + scenarioManifestDigests: string[]; + claimSourceDigest?: string; + contributingEventIds: string[]; + contradictingEventIds: string[]; + notes: string[]; +} + +export interface ClaimState { + key: string; + current: ClaimSnapshotEvent | null; + corruption?: string; + unusable?: boolean; +} + +export interface ProjectVerdictsOptions { + asOf?: number; + index?: InvalidationIndex; + unusableObservationIds?: Set; + unusableClaimEventIds?: Set; + loadSuiteManifest?: (digest: string) => SuiteManifestV1 | null; + loadScenarioManifest?: (digest: string) => Record | null; + loadScenarioRequirements?: (digest: string) => ScenarioRequirements | null; +} + +/** + * Resolve current claims after purge/invalidation and supersession. + * Multiple unsuperseded claims, missing predecessors, cross-key supersession, or cycles → UNKNOWN + corruption. + */ +export function resolveClaimStates( + claims: ClaimSnapshotEvent[], + opts: { + unusableClaimEventIds?: Set; + purgedEventIds?: ReadonlySet; + } = {}, +): { + states: Map; + corruptions: LedgerCorruption[]; +} { + const unusableClaims = opts.unusableClaimEventIds ?? new Set(); + const purgedEventIds = opts.purgedEventIds ?? new Set(); + const byKey = new Map(); + const allById = new Map(claims.map((claim) => [claim.eventId, claim])); + for (const claim of claims) { + const key = claimKeyString(claim.subjectId, claim.capability); + const list = byKey.get(key) ?? []; + list.push(claim); + byKey.set(key, list); + } + + const states = new Map(); + const corruptions: LedgerCorruption[] = []; + + for (const [key, list] of byKey) { + const sorted = [...list].sort((a, b) => { + if (a.effectiveAt !== b.effectiveAt) return a.effectiveAt - b.effectiveAt; + if (a.recordedAt !== b.recordedAt) return a.recordedAt - b.recordedAt; + return a.eventId < b.eventId ? -1 : a.eventId > b.eventId ? 1 : 0; + }); + + const superseded = new Set(); + + for (const claim of sorted) { + for (const pred of claim.supersedes) { + const prev = allById.get(pred); + if (!prev) { + if (purgedEventIds.has(pred)) { + // A purge may physically remove a superseded predecessor. The ID remains + // valid provenance but is not a live claim candidate. + continue; + } + corruptions.push({ + kind: "claim_corruption", + eventId: claim.eventId, + detail: `missing supersedes predecessor ${pred}`, + }); + states.set(key, { key, current: null, corruption: "missing predecessor" }); + continue; + } + if (prev.subjectId !== claim.subjectId || prev.capability !== claim.capability) { + corruptions.push({ + kind: "claim_corruption", + eventId: claim.eventId, + detail: "cross-key supersession", + }); + states.set(key, { key, current: null, corruption: "cross-key supersession" }); + continue; + } + superseded.add(pred); + } + } + + if (states.get(key)?.corruption) continue; + + const current = sorted.filter((c) => !superseded.has(c.eventId)); + if (current.length > 1) { + corruptions.push({ + kind: "claim_corruption", + detail: `multiple unsuperseded claims for ${key}`, + }); + states.set(key, { key, current: null, corruption: "conflicting current claims" }); + continue; + } + const currentClaim = current[0] ?? null; + states.set(key, { + key, + current: currentClaim, + unusable: currentClaim ? unusableClaims.has(currentClaim.eventId) : undefined, + }); + } + + return { states, corruptions }; +} + +/** + * CL-02 verdict projection primitives with frozen CL-00 verification semantics. + */ +export function projectVerdicts( + events: LabEvent[], + opts: ProjectVerdictsOptions = {}, +): { verdicts: DerivedVerdict[]; corruptions: LedgerCorruption[]; index: InvalidationIndex } { + const index = opts.index ?? buildInvalidationIndex(events); + const corruptions = [...index.corruptions]; + const unusableObs = opts.unusableObservationIds ?? new Set(); + const unusableClaims = opts.unusableClaimEventIds ?? new Set(); + const asOf = + opts.asOf ?? + events.reduce((max, event) => { + if (event.eventKind === "observation") return Math.max(max, event.completedAt, event.recordedAt); + if (event.eventKind === "claim_snapshot") return Math.max(max, event.effectiveAt, event.recordedAt); + return Math.max(max, event.recordedAt); + }, 0); + + const observations = usableObservations(events, index) + .filter((o) => o.completedAt <= asOf) + .filter((o) => !unusableObs.has(o.eventId)); + const claims = usableClaims(events, index).filter((c) => c.effectiveAt <= asOf); + const { states: claimStates, corruptions: claimCorruptions } = resolveClaimStates(claims, { + unusableClaimEventIds: unusableClaims, + purgedEventIds: index.purgedEventIds, + }); + corruptions.push(...claimCorruptions); + + const groups = new Map(); + for (const obs of observations) { + const key: ProjectionKey = { + subjectId: obs.subjectId, + evidenceLayer: obs.evidenceLayer, + suiteId: obs.suiteId, + suiteVersion: obs.suiteVersion, + suiteManifestDigest: obs.suiteManifestDigest, + projectionSpecVersion: LAB_PROJECTION_SPEC_VERSION, + }; + const ks = projectionKeyString(key); + const list = groups.get(ks) ?? []; + list.push(obs); + groups.set(ks, list); + } + + const verdicts: DerivedVerdict[] = []; + + for (const [, list] of groups) { + const sample = list[0]!; + const key: ProjectionKey = { + subjectId: sample.subjectId, + evidenceLayer: sample.evidenceLayer, + suiteId: sample.suiteId, + suiteVersion: sample.suiteVersion, + suiteManifestDigest: sample.suiteManifestDigest, + projectionSpecVersion: LAB_PROJECTION_SPEC_VERSION, + }; + const ordered = [...list].sort((a, b) => { + if (a.completedAt !== b.completedAt) return a.completedAt - b.completedAt; + return a.eventId < b.eventId ? -1 : a.eventId > b.eventId ? 1 : 0; + }); + const suiteManifest = opts.loadSuiteManifest?.(key.suiteManifestDigest) ?? null; + verdicts.push( + projectObservationGroup(key, ordered, asOf, suiteManifest, { + loadScenarioManifest: opts.loadScenarioManifest, + loadScenarioRequirements: opts.loadScenarioRequirements, + }), + ); + } + + for (const [, state] of claimStates) { + if (!state.current || state.current.polarity !== "supported") continue; + if (state.corruption) continue; + const claim = state.current; + const key: ProjectionKey = { + subjectId: claim.subjectId, + evidenceLayer: "live_route_compatibility", + suiteId: "_claims", + suiteVersion: "1", + suiteManifestDigest: claim.sourceManifestDigest, + projectionSpecVersion: LAB_PROJECTION_SPEC_VERSION, + }; + const ks = projectionKeyString(key); + if (verdicts.some((v) => projectionKeyString(v.key) === ks)) continue; + if (state.unusable) { + verdicts.push({ + key, + verdict: "UNKNOWN", + asOf, + scenarioManifestDigests: [], + claimSourceDigest: claim.sourceManifestDigest, + contributingEventIds: [claim.eventId], + contradictingEventIds: [], + notes: ["current_claim_unusable"], + }); + continue; + } + verdicts.push({ + key, + verdict: "CLAIMED", + asOf, + scenarioManifestDigests: [], + claimSourceDigest: claim.sourceManifestDigest, + contributingEventIds: [claim.eventId], + contradictingEventIds: [], + notes: ["claim_snapshot"], + }); + } + + return { verdicts, corruptions, index }; +} + +function isMatchedCapabilityAbsenceControl(obs: ObservationEvent): boolean { + const expected = obs.expectedFailure; + return ( + obs.outcome === "pass" && + !!expected && + expected.controlKind === "capability_absence_control" && + expected.onMatch === "unsupported" + ); +} + +function projectObservationGroup( + key: ProjectionKey, + ordered: ObservationEvent[], + asOf: number, + suiteManifest: SuiteManifestV1 | null, + opts: { + loadScenarioManifest?: (digest: string) => Record | null; + loadScenarioRequirements?: ProjectVerdictsOptions["loadScenarioRequirements"]; + } = {}, +): DerivedVerdict { + const contributing: string[] = []; + const contradicting: string[] = []; + const digests = new Set(); + const notes: string[] = []; + + for (const obs of ordered) { + digests.add(obs.scenarioManifestDigest); + contributing.push(obs.eventId); + if (obs.outcome === "fail") contradicting.push(obs.eventId); + } + + const newest = newestObservationByScenario(ordered); + const currentObservations = [...newest.values()]; + const currentFails = currentObservations.filter((o) => o.outcome === "fail"); + const currentPasses = currentObservations.filter((o) => o.outcome === "pass"); + const currentBlocked = currentObservations.some((o) => o.outcome === "blocked"); + const currentInconclusive = currentObservations.some((o) => o.outcome === "inconclusive"); + const matchedCapabilityAbsence = currentObservations.some(isMatchedCapabilityAbsenceControl); + const currentModes = new Set(currentObservations.map((o) => o.executionMode)); + const newestCurrent = [...currentObservations].sort((a, b) => { + if (a.completedAt !== b.completedAt) return a.completedAt - b.completedAt; + return a.eventId < b.eventId ? -1 : a.eventId > b.eventId ? 1 : 0; + }).at(-1); + + let verdict: CompatibilityVerdict = "UNKNOWN"; + if ( + key.evidenceLayer !== "protocol_conformance" && + key.evidenceLayer !== "live_route_compatibility" && + key.evidenceLayer !== "task_effectiveness" + ) { + verdict = "UNKNOWN"; + } else if (matchedCapabilityAbsence) { + verdict = "UNSUPPORTED"; + notes.push("capability_absence_control"); + } else if (currentFails.length > 0) { + verdict = "DEGRADED"; + } else if (currentPasses.length > 0 && !currentInconclusive && !currentBlocked) { + if (currentModes.size > 1) { + verdict = "PROBED"; + notes.push("mixed_execution_modes"); + } else if (key.evidenceLayer === "protocol_conformance" && newestCurrent?.executionMode === "fixture") { + if (!suiteManifest) { + verdict = "PROBED"; + notes.push("suite_manifest_unavailable"); + } else { + const evaluation = evaluateAllApplicableRequiredPassV1( + suiteManifest, + ordered, + newestCurrent.executionMode, + { + subject: newestCurrent.subject.subjectKind === "protocol" ? newestCurrent.subject : undefined, + loadScenarioManifest: opts.loadScenarioManifest, + loadScenarioRequirements: opts.loadScenarioRequirements, + asOf, + }, + ); + notes.push(...evaluation.notes); + if (evaluation.applicableRequiredScenarioIds.length === 0) { + verdict = "UNKNOWN"; + } else if (evaluation.canVerify) { + verdict = "VERIFIED"; + notes.push("all-applicable-required-pass-v1"); + } else { + verdict = "PROBED"; + notes.push("incomplete_required_coverage"); + } + } + } else { + verdict = "PROBED"; + } + } else if (currentPasses.length > 0) { + verdict = "PROBED"; + } else if (currentBlocked) { + verdict = "BLOCKED"; + } else { + verdict = "UNKNOWN"; + } + + return { + key, + verdict, + asOf, + scenarioManifestDigests: [...digests].sort(), + contributingEventIds: contributing, + contradictingEventIds: contradicting, + notes, + }; +} + +export function excludeEventIds(index: InvalidationIndex): Set { + const out = new Set([...index.purgedEventIds]); + for (const id of index.invalidatedBy.keys()) out.add(id); + return out; +} + +export { isEventExcluded }; diff --git a/src/lab/projection/verification.ts b/src/lab/projection/verification.ts new file mode 100644 index 000000000..806a345e0 --- /dev/null +++ b/src/lab/projection/verification.ts @@ -0,0 +1,297 @@ +import type { ObservationEvent, ProtocolSubjectV1 } from "../events/types"; +import { EVIDENCE_LAYERS, type ExecutionMode } from "../constants"; +import type { SuiteManifestV1 } from "../conformance/suite-manifest"; +import type { VerificationRole } from "../conformance/types"; +import { isSha256Hex } from "../digest"; + +export interface VerificationEvaluation { + applicableRequiredScenarioIds: string[]; + passingRequiredScenarioIds: string[]; + missingRequiredScenarioIds: string[]; + canVerify: boolean; + notes: string[]; +} + +export type LoadScenarioManifest = (digest: string) => Record | null; + +export interface ScenarioRequirements { + inboundProtocols?: string[]; + upstreamProtocols?: string[]; + surfaces?: string[]; + freshness?: { maxAgeMs: number | null }; +} + +export type LoadScenarioRequirements = (digest: string) => ScenarioRequirements | null; + +/** Live-reserved scenarios are inapplicable in fixture-mode protocol conformance. */ +export function isScenarioApplicable( + scenarioId: string, + executionMode: ExecutionMode, + evidenceLayer: string, +): boolean { + if (evidenceLayer === "protocol_conformance" && executionMode === "fixture") { + // The frozen CL-00 scenario schema has no explicit execution-mode field. + // Use an exact dot-delimited `live` segment rather than substring matching. + if (scenarioId.split(".").includes("live")) return false; + } + return true; +} + +function scenarioApplicableToRequirements( + requirements: ScenarioRequirements, + subject: ProtocolSubjectV1, +): boolean { + const inbound = requirements.inboundProtocols ?? []; + const upstream = requirements.upstreamProtocols ?? []; + const surfaces = requirements.surfaces ?? []; + return ( + inbound.includes(subject.inboundProtocol) && + upstream.includes(subject.upstreamProtocol) && + surfaces.includes(subject.surface) + ); +} + +function isNonNegativeInteger(value: unknown): value is number { + return typeof value === "number" && Number.isInteger(value) && value >= 0; +} + +function parseFreshness(value: unknown): { maxAgeMs: number | null } | null { + if (!value || typeof value !== "object" || Array.isArray(value)) return null; + const maxAgeMs = (value as { maxAgeMs?: unknown }).maxAgeMs; + if (maxAgeMs === null) return { maxAgeMs: null }; + if (!isNonNegativeInteger(maxAgeMs)) return null; + return { maxAgeMs }; +} + +function parseStringArray(value: unknown): string[] | null { + if (!Array.isArray(value) || !value.every((item) => typeof item === "string")) return null; + return value; +} + +function scenarioContractFromManifest( + scenarioManifest: Record | null, +): ScenarioRequirements | null { + if (!scenarioManifest) return null; + const req = scenarioManifest.requirements; + if (!req || typeof req !== "object" || Array.isArray(req)) return null; + const row = req as Record; + const inboundProtocols = parseStringArray(row.inboundProtocols); + const upstreamProtocols = parseStringArray(row.upstreamProtocols); + const surfaces = parseStringArray(row.surfaces); + if (!inboundProtocols || !upstreamProtocols || !surfaces) return null; + const freshness = parseFreshness(scenarioManifest.freshness); + if (!freshness) return null; + return { inboundProtocols, upstreamProtocols, surfaces, freshness }; +} + +function effectiveMaxAgeMs( + suiteMaxAgeMs: number | null, + scenarioMaxAgeMs: number | null, +): number | null { + if (suiteMaxAgeMs === null) return scenarioMaxAgeMs; + if (scenarioMaxAgeMs === null) return suiteMaxAgeMs; + return Math.min(suiteMaxAgeMs, scenarioMaxAgeMs); +} + +export function newestObservationByScenario( + observations: ObservationEvent[], +): Map { + const byScenario = new Map(); + const ordered = [...observations].sort((a, b) => { + if (a.completedAt !== b.completedAt) return a.completedAt - b.completedAt; + return a.eventId < b.eventId ? -1 : a.eventId > b.eventId ? 1 : 0; + }); + for (const obs of ordered) { + byScenario.set(obs.scenarioId, obs); + } + return byScenario; +} + +/** + * Evaluate `all-applicable-required-pass-v1` per frozen CL-00 semantics. + * Positive VERIFIED requires a non-empty applicable required/control set and a + * current, fresh pass for every applicable required scenario and negative control. + */ +export function evaluateAllApplicableRequiredPassV1( + suiteManifest: SuiteManifestV1, + observations: ObservationEvent[], + executionMode: ExecutionMode, + opts: { + subject?: ProtocolSubjectV1; + loadScenarioManifest?: LoadScenarioManifest; + loadScenarioRequirements?: LoadScenarioRequirements; + asOf?: number; + } = {}, +): VerificationEvaluation { + const notes: string[] = []; + if (suiteManifest.verificationRule !== "all-applicable-required-pass-v1") { + return { + applicableRequiredScenarioIds: [], + passingRequiredScenarioIds: [], + missingRequiredScenarioIds: [], + canVerify: false, + notes: ["unsupported_verification_rule"], + }; + } + + const requiredScenarios = suiteManifest.scenarios.filter( + (s) => s.role === "required" || s.role === "negative_control", + ); + const applicableRequired: string[] = []; + const unavailableManifests: string[] = []; + const scenarioMaxAgeById = new Map(); + + for (const s of requiredScenarios) { + if (!isScenarioApplicable(s.id, executionMode, suiteManifest.evidenceLayer)) continue; + + let requirements: ScenarioRequirements | null = null; + const scenarioManifest = opts.loadScenarioManifest?.(s.manifestDigest) ?? null; + if (scenarioManifest) { + requirements = scenarioContractFromManifest(scenarioManifest); + if (!requirements) { + unavailableManifests.push(s.id); + continue; + } + } else { + requirements = opts.loadScenarioRequirements?.(s.manifestDigest) ?? null; + if (!requirements || !requirements.freshness) { + unavailableManifests.push(s.id); + continue; + } + } + + if (suiteManifest.evidenceLayer === "protocol_conformance" && opts.subject) { + if (!scenarioApplicableToRequirements(requirements, opts.subject)) continue; + } + applicableRequired.push(s.id); + scenarioMaxAgeById.set(s.id, requirements.freshness?.maxAgeMs ?? null); + } + applicableRequired.sort(); + + if (unavailableManifests.length > 0) { + return { + applicableRequiredScenarioIds: applicableRequired, + passingRequiredScenarioIds: [], + missingRequiredScenarioIds: [...new Set([...applicableRequired, ...unavailableManifests])].sort(), + canVerify: false, + notes: unavailableManifests.map((id) => `scenario_manifest_unavailable:${id}`), + }; + } + + if (applicableRequired.length === 0) { + return { + applicableRequiredScenarioIds: [], + passingRequiredScenarioIds: [], + missingRequiredScenarioIds: [], + canVerify: false, + notes: ["empty_applicable_required_set"], + }; + } + + const newest = newestObservationByScenario(observations); + const passing: string[] = []; + const missing: string[] = []; + const asOf = opts.asOf ?? observations.reduce((max, obs) => Math.max(max, obs.completedAt), 0); + + for (const scenarioId of applicableRequired) { + const scenarioRef = requiredScenarios.find((s) => s.id === scenarioId)!; + const obs = newest.get(scenarioId); + if (!obs) { + missing.push(scenarioId); + continue; + } + if (obs.scenarioManifestDigest !== scenarioRef.manifestDigest) { + missing.push(scenarioId); + notes.push(`digest_mismatch:${scenarioId}`); + continue; + } + const maxAgeMs = effectiveMaxAgeMs( + suiteManifest.freshness.maxAgeMs, + scenarioMaxAgeById.get(scenarioId) ?? null, + ); + if (maxAgeMs !== null && asOf - obs.completedAt > maxAgeMs) { + missing.push(scenarioId); + notes.push(`stale_observation:${scenarioId}`); + continue; + } + if (obs.outcome !== "pass") { + missing.push(scenarioId); + continue; + } + passing.push(scenarioId); + } + + return { + applicableRequiredScenarioIds: applicableRequired, + passingRequiredScenarioIds: passing, + missingRequiredScenarioIds: missing, + canVerify: missing.length === 0 && passing.length === applicableRequired.length, + notes, + }; +} + +function requireNonEmptyString(value: unknown): string | null { + return typeof value === "string" && value.length > 0 ? value : null; +} + +export function parseSuiteManifestFromArtifact(parsed: unknown): SuiteManifestV1 | null { + if (!parsed || typeof parsed !== "object" || Array.isArray(parsed)) return null; + const raw = parsed as Record; + if (raw.schemaVersion !== 1) return null; + + const id = requireNonEmptyString(raw.id); + const version = requireNonEmptyString(raw.version); + const capability = requireNonEmptyString(raw.capability); + const assertionDslVersion = requireNonEmptyString(raw.assertionDslVersion); + const evidenceSchemaVersion = requireNonEmptyString(raw.evidenceSchemaVersion); + const contradictionRule = requireNonEmptyString(raw.contradictionRule); + const verificationRule = requireNonEmptyString(raw.verificationRule); + if ( + !id || !version || !capability || !assertionDslVersion || + !evidenceSchemaVersion || !contradictionRule || + verificationRule !== "all-applicable-required-pass-v1" + ) return null; + if ( + typeof raw.evidenceLayer !== "string" || + !(EVIDENCE_LAYERS as readonly string[]).includes(raw.evidenceLayer) + ) return null; + const freshness = parseFreshness(raw.freshness); + if (!freshness || !Array.isArray(raw.scenarios)) return null; + + const roles = new Set(["required", "supplemental", "negative_control"]); + const seenScenarioIds = new Set(); + const scenarios: SuiteManifestV1["scenarios"] = []; + for (const s of raw.scenarios) { + if (!s || typeof s !== "object" || Array.isArray(s)) return null; + const row = s as Record; + const scenarioId = requireNonEmptyString(row.id); + const scenarioVersion = requireNonEmptyString(row.version); + const manifestDigest = requireNonEmptyString(row.manifestDigest); + if ( + !scenarioId || !scenarioVersion || !manifestDigest || !isSha256Hex(manifestDigest) || + typeof row.role !== "string" || !roles.has(row.role as VerificationRole) || + seenScenarioIds.has(scenarioId) + ) return null; + seenScenarioIds.add(scenarioId); + scenarios.push({ + id: scenarioId, + version: scenarioVersion, + role: row.role as VerificationRole, + manifestDigest, + }); + } + + return { + schemaVersion: 1, + id, + version, + evidenceLayer: raw.evidenceLayer, + capability, + assertionDslVersion, + evidenceSchemaVersion, + freshness, + contradictionRule, + scenarios, + verificationRule, + }; +} diff --git a/tests/lab-evidence-ledger.test.ts b/tests/lab-evidence-ledger.test.ts new file mode 100644 index 000000000..9678c9dd4 --- /dev/null +++ b/tests/lab-evidence-ledger.test.ts @@ -0,0 +1,1161 @@ +import { afterEach, describe, expect, test } from "bun:test"; +import { chmodSync, existsSync, mkdirSync, writeFileSync, symlinkSync, linkSync, rmSync } from "node:fs"; +import { join } from "node:path"; +import { tmpdir } from "node:os"; +import { + appendLabEvent, + assignEventId, + buildInvalidationIndex, + claimKeyString, + claimSourceManifestDigest, + createArtifactStore, + eventIdForPayload, + isSha256Hex, + jcsStringify, + observationFromConformanceResult, + openTrustedArtifactDir, + persistConformanceResult, + projectVerdicts, + purgeSensitiveEvidence, + readVerdictSnapshot, + rebuildLabProjection, + replayLabLedger, + resolveClaimStates, + subjectIdForSubject, + validateClaimSourceManifest, + validateLabEvent, + validateSortedUniqueHexIds, + LAB_EVENT_SCHEMA_VERSION, + LAB_PRODUCER, + LAB_PROJECTION_SPEC_VERSION, + behaviorFingerprintForCase, +} from "../src/lab"; +import { ArtifactFsError, closeTrustedArtifactDir, putArtifactBytes, putNamedDigestBytes, readArtifactBytes, digestFileName } from "../src/lab/artifacts/secure-fs"; +import { expandSuiteManifest } from "../src/lab/conformance/suite-manifest"; +import { evaluateAllApplicableRequiredPassV1 } from "../src/lab/projection/verification"; +import { discoverScenarios, expandScenario, loadCaseAuthority } from "../src/lab/conformance/manifest"; +import { artifactBytesDigest, scenarioManifestDigest } from "../src/lab/digest"; +import type { CaseRecord } from "../src/lab/conformance/types"; +import { runScenario, resolveProtocolExecutionContext } from "../src/lab/conformance/executor"; +import { LabValidationError } from "../src/lab/events/validate"; +import { enforceEventStructureLimits } from "../src/lab/events/limits"; +import type { ClaimSnapshotEvent, ObservationEvent, ProtocolSubjectV1 } from "../src/lab/events/types"; + +const HOMES: string[] = []; + +function tempHome(): string { + const dir = join(tmpdir(), `ocx-lab-${process.pid}-${Math.random().toString(16).slice(2)}`); + mkdirSync(dir, { recursive: true, mode: 0o700 }); + HOMES.push(dir); + return dir; +} + +afterEach(() => { + for (const dir of HOMES.splice(0)) { + try { + rmSync(dir, { recursive: true, force: true }); + } catch { + /* ignore */ + } + } + delete process.env.OPENCODEX_HOME; +}); + +function withHome(fn: (home: string) => T): T { + const home = tempHome(); + process.env.OPENCODEX_HOME = home; + return fn(home); +} + +function syntheticPassResult(caseRecord: CaseRecord) { + return { + scenarioId: caseRecord.id, + suite: caseRecord.suite, + passed: true, + classification: "inconclusive" as const, + assertionResults: caseRecord.assertions.map((a) => ({ + id: a.id, + operator: a.operator, + required: a.required, + passed: true, + observedSummary: "ok", + })), + diagnostics: [], + executionContext: resolveProtocolExecutionContext(caseRecord), + startedAt: 999, + completedAt: 1000, + }; +} + +function persistAllSuiteScenarios(home: string, suiteId: string, authority = loadCaseAuthority()) { + const scenarios = discoverScenarios(authority, [suiteId]); + let recordedAt = 1_700_000_000_000; + for (const caseRecord of scenarios) { + const store = createArtifactStore(join(home, "lab", "artifacts")); + persistConformanceResult(syntheticPassResult(caseRecord), caseRecord, authority, { + configDir: home, + recordedAt: recordedAt++, + artifactStore: store, + }); + store.close(); + } +} + +function protocolSubject(seed = "a"): ProtocolSubjectV1 { + return { + subjectSchemaVersion: 1, + subjectKind: "protocol", + opencodexCompatibilityVersion: "protocol-v1", + effectiveAdapter: "openai-chat", + inboundProtocol: "openai-responses", + upstreamProtocol: "openai-chat", + surface: "responses-http", + behaviorFingerprint: createHashHex(seed), + }; +} + +function createHashHex(s: string): string { + return Bun.CryptoHasher.hash("sha256", s, "hex"); +} + +function baseObservation(overrides: Partial = {}): ObservationEvent { + const subject = protocolSubject(overrides.scenarioId ?? "s1"); + const subjectId = subjectIdForSubject(subject); + const fixtureDigest = createHashHex("fixture"); + const scenarioManifestDigest = createHashHex("scenario"); + const suiteManifestDigest = createHashHex("suite"); + const event = assignEventId({ + schemaVersion: LAB_EVENT_SCHEMA_VERSION, + eventKind: "observation" as const, + recordedAt: 1_700_000_000_000, + producer: LAB_PRODUCER, + producerVersion: "2.10.2", + evidenceLayer: "protocol_conformance" as const, + scenarioId: "responses-core.protocol.request-shape", + scenarioVersion: "1", + scenarioManifestDigest, + suiteId: "responses-core", + suiteVersion: "1", + suiteManifestDigest, + fixtureDigests: [fixtureDigest], + subject, + subjectId, + startedAt: 1_700_000_000_000, + completedAt: 1_700_000_000_100, + executionMode: "fixture" as const, + attempt: 1, + limits: { totalTimeoutMs: 1000 }, + outcome: "pass" as const, + assertions: [ + { + id: "a1", + operator: "equals", + required: true, + passed: true, + expectedSummary: "ok", + observedSummary: "ok", + }, + ], + environment: { runtime: { platform: "test", arch: "x64", bunVersion: "1.0.0" } }, + artifactRefs: [ + { + digest: scenarioManifestDigest, + mediaType: "application/json", + byteCount: 2, + redactionPolicy: "contract_canonical_v1", + relativePath: `${scenarioManifestDigest}.bin`, + artifactClass: "scenario_manifest", + }, + { + digest: suiteManifestDigest, + mediaType: "application/json", + byteCount: 2, + redactionPolicy: "contract_canonical_v1", + relativePath: `${suiteManifestDigest}.bin`, + artifactClass: "suite_manifest", + }, + { + digest: fixtureDigest, + mediaType: "application/json", + byteCount: 2, + redactionPolicy: "contract_canonical_v1", + relativePath: `${fixtureDigest}.bin`, + artifactClass: "fixture", + }, + ], + ...overrides, + }); + return event as ObservationEvent; +} + +describe("CL-02 JCS and IDs", () => { + test("eventId and subjectId are domain-separated deterministic digests", () => { + const subject = protocolSubject("x"); + const subjectId = subjectIdForSubject(subject); + expect(isSha256Hex(subjectId)).toBe(true); + expect(subjectId).toBe(subjectIdForSubject(subject)); + + const payload = { + schemaVersion: 1, + eventKind: "invalidation", + recordedAt: 1, + producer: "p", + producerVersion: "1", + targetEventIds: [createHashHex("t")], + reason: "harness_defect", + }; + const id1 = eventIdForPayload(payload); + const id2 = eventIdForPayload(payload); + expect(id1).toBe(id2); + expect(isSha256Hex(id1)).toBe(true); + expect(id1).not.toBe(subjectId); + }); + + test("JCS key order is deterministic", () => { + expect(jcsStringify({ b: 1, a: 2 })).toBe('{"a":2,"b":1}'); + }); +}); + +describe("CL-02 ledger append/replay", () => { + test("append and replay round-trip", () => { + withHome((home) => { + const event = baseObservation(); + const store = createArtifactStore(join(home, "lab", "artifacts")); + for (const ref of event.artifactRefs) { + store.put({ + artifactClass: "assertion_report", + payload: { ok: true, seed: ref.digest }, + expectedDigest: undefined, + }); + } + const dir = openTrustedArtifactDir(join(home, "lab", "artifacts")); + try { + for (const ref of event.artifactRefs) { + putNamedDigestBytes(dir, ref.digest, new TextEncoder().encode("{}"), () => ref.digest); + } + } finally { + closeTrustedArtifactDir(dir); + } + + appendLabEvent(join(home, "lab", "compatibility.jsonl"), event); + const replay = replayLabLedger(join(home, "lab", "compatibility.jsonl")); + expect(replay.validLineCount).toBe(1); + expect(replay.events[0]!.eventId).toBe(event.eventId); + expect(replay.corruptions).toEqual([]); + }); + }); + + test("partial final JSONL line is corruption", () => { + withHome((home) => { + const ledger = join(home, "lab", "compatibility.jsonl"); + mkdirSync(join(home, "lab"), { recursive: true }); + writeFileSync(ledger, '{"schemaVersion":1,"eventKind":"observation"'); + const replay = replayLabLedger(ledger); + expect(replay.events).toEqual([]); + expect(replay.corruptions.some((c) => c.kind === "partial_line")).toBe(true); + }); + }); + + test("malformed line is corruption and contributes no evidence", () => { + withHome((home) => { + const ledger = join(home, "lab", "compatibility.jsonl"); + mkdirSync(join(home, "lab"), { recursive: true }); + writeFileSync(ledger, "not-json\n"); + const replay = replayLabLedger(ledger); + expect(replay.events).toEqual([]); + expect(replay.corruptions.some((c) => c.kind === "malformed_line")).toBe(true); + }); + }); + + test("duplicate eventId is reported", () => { + withHome((home) => { + const event = baseObservation(); + const ledger = join(home, "lab", "compatibility.jsonl"); + mkdirSync(join(home, "lab"), { recursive: true }); + const line = `${jcsStringify(event)}\n`; + writeFileSync(ledger, line + line); + const replay = replayLabLedger(ledger); + expect(replay.validLineCount).toBe(1); + expect(replay.corruptions.some((c) => c.kind === "duplicate_event")).toBe(true); + }); + }); +}); + +describe("CL-02 invalidation validation", () => { + test("rejects unsorted, duplicate, empty, and oversize target lists", () => { + expect(() => validateSortedUniqueHexIds([], "t", { nonEmpty: true })).toThrow(); + const a = createHashHex("a"); + const b = createHashHex("b"); + const [lo, hi] = a < b ? [a, b] : [b, a]; + expect(() => validateSortedUniqueHexIds([hi, lo], "t")).toThrow(); + expect(() => validateSortedUniqueHexIds([lo, lo], "t")).toThrow(); + expect(validateSortedUniqueHexIds([lo, hi], "t")).toEqual([lo, hi]); + }); + + test("invalidation targets must be earlier observation/claim only", () => { + const obs = baseObservation(); + const inv = assignEventId({ + schemaVersion: LAB_EVENT_SCHEMA_VERSION, + eventKind: "invalidation" as const, + recordedAt: obs.recordedAt + 1, + producer: LAB_PRODUCER, + producerVersion: "2.10.2", + targetEventIds: [obs.eventId], + reason: "harness_defect" as const, + }); + const index = buildInvalidationIndex([obs, inv as never]); + expect(index.invalidatedBy.has(obs.eventId)).toBe(true); + expect(index.corruptions).toEqual([]); + + const bad = assignEventId({ + schemaVersion: LAB_EVENT_SCHEMA_VERSION, + eventKind: "invalidation" as const, + recordedAt: obs.recordedAt + 2, + producer: LAB_PRODUCER, + producerVersion: "2.10.2", + targetEventIds: [createHashHex("missing")], + reason: "fixture_defect" as const, + }); + const index2 = buildInvalidationIndex([obs, bad as never]); + expect(index2.corruptions.some((c) => c.kind === "invalid_reference")).toBe(true); + + const invalidatesInvalidation = assignEventId({ + schemaVersion: LAB_EVENT_SCHEMA_VERSION, + eventKind: "invalidation" as const, + recordedAt: obs.recordedAt + 3, + producer: LAB_PRODUCER, + producerVersion: "2.10.2", + targetEventIds: [inv.eventId], + reason: "manual_correction" as const, + }); + const index3 = buildInvalidationIndex([obs, inv as never, invalidatesInvalidation as never]); + expect(index3.corruptions.some((c) => c.kind === "invalid_reference")).toBe(true); + }); +}); + +describe("CL-02 claim supersession and conflicts", () => { + test("supersession leaves one current claim; conflicts report corruption", () => { + const subject = { + subjectSchemaVersion: 1 as const, + subjectKind: "route" as const, + providerId: "openai", + providerInstanceFingerprint: createHashHex("inst"), + clientModelId: "gpt", + upstreamModelId: "gpt", + effectiveAdapter: "openai-responses", + inboundProtocol: "openai-responses", + upstreamProtocol: "openai-responses", + surface: "responses-http", + opencodexCompatibilityVersion: "protocol-v1", + behaviorFingerprint: createHashHex("bf"), + endpointFingerprint: createHashHex("ep"), + dependencies: [], + }; + const subjectId = subjectIdForSubject(subject); + const manifest = validateClaimSourceManifest({ + schemaVersion: 1, + subjectId, + providerId: "openai", + clientModelId: "gpt", + capability: "tools", + sources: [ + { + kind: "provider_registry", + revision: "1", + facts: { toolCapable: true }, + }, + ], + resolvedEvidence: { tools: true }, + }); + + const c1 = assignEventId({ + schemaVersion: LAB_EVENT_SCHEMA_VERSION, + eventKind: "claim_snapshot" as const, + recordedAt: 10, + producer: LAB_PRODUCER, + producerVersion: "2.10.2", + evidenceLayer: "live_route_compatibility" as const, + subject, + subjectId, + capability: "tools", + polarity: "supported" as const, + sourceManifestDigest: manifest.digest, + sourceEventIds: [], + supersedes: [], + effectiveAt: 10, + }) as ClaimSnapshotEvent; + + const c2 = assignEventId({ + schemaVersion: LAB_EVENT_SCHEMA_VERSION, + eventKind: "claim_snapshot" as const, + recordedAt: 20, + producer: LAB_PRODUCER, + producerVersion: "2.10.2", + evidenceLayer: "live_route_compatibility" as const, + subject, + subjectId, + capability: "tools", + polarity: "supported" as const, + sourceManifestDigest: manifest.digest, + sourceEventIds: [], + supersedes: [c1.eventId], + effectiveAt: 20, + }) as ClaimSnapshotEvent; + + const ok = resolveClaimStates([c1, c2]); + expect(ok.states.get(claimKeyString(subjectId, "tools"))?.current?.eventId).toBe(c2.eventId); + + const c3 = assignEventId({ + ...c2, + eventId: undefined, + recordedAt: 30, + effectiveAt: 30, + supersedes: [], + }) as ClaimSnapshotEvent; + const conflict = resolveClaimStates([c1, c2, c3]); + expect(conflict.states.get(claimKeyString(subjectId, "tools"))?.corruption).toBeTruthy(); + }); + + test("ClaimSourceManifest rejects secrets and unknown facts", () => { + expect(() => + validateClaimSourceManifest({ + schemaVersion: 1, + subjectId: createHashHex("s"), + providerId: "openai", + clientModelId: "gpt", + capability: "tools", + sources: [{ kind: "provider_config", revision: null, facts: { baseUrl: "https://evil" } }], + resolvedEvidence: {}, + }), + ).toThrow(); + }); +}); + +describe("CL-02 artifacts and secure FS", () => { + test("content-addressed put/get and size ceiling", () => { + withHome((home) => { + const store = createArtifactStore(join(home, "lab", "artifacts")); + try { + const ref = store.put({ artifactClass: "assertion_report", payload: { a: 1 } }); + expect(isSha256Hex(ref.digest)).toBe(true); + const bytes = store.get(ref.digest); + expect(bytes.byteLength).toBe(ref.byteCount); + + const huge = new Uint8Array(256 * 1024 + 1); + expect(() => + store.put({ artifactClass: "assertion_report", payload: huge }), + ).toThrow(); + } finally { + store.close(); + } + }); + }); + + test("traversal and absolute digest names are rejected", () => { + withHome((home) => { + const dir = openTrustedArtifactDir(join(home, "lab", "artifacts")); + try { + expect(() => digestFileName("../passwd")).toThrow(); + expect(() => digestFileName("C:\\Windows\\x")).toThrow(); + expect(() => readArtifactBytes(dir, "../aaaa")).toThrow(); + } finally { + closeTrustedArtifactDir(dir); + } + }); + }); + + test("symlink attack fails closed where testable", () => { + withHome((home) => { + const artifacts = join(home, "lab", "artifacts"); + mkdirSync(artifacts, { recursive: true }); + const target = join(home, "outside.bin"); + writeFileSync(target, "secret-canary-symlink"); + const digest = createHashHex("symlink-target"); + const linkPath = join(artifacts, `${digest}.bin`); + try { + symlinkSync(target, linkPath); + } catch { + return; + } + const dir = openTrustedArtifactDir(artifacts); + try { + expect(() => readArtifactBytes(dir, digest, { contentDigest: () => digest })).toThrow(ArtifactFsError); + } finally { + closeTrustedArtifactDir(dir); + } + }); + }); + + test("hard-link attack fails closed where testable", () => { + withHome((home) => { + const artifacts = join(home, "lab", "artifacts"); + const dir = openTrustedArtifactDir(artifacts); + try { + const bytes = new TextEncoder().encode("hl"); + const stored = putArtifactBytes(dir, bytes); + const second = join(artifacts, `${createHashHex("other")}.bin`); + try { + linkSync(join(artifacts, `${stored.digest}.bin`), second); + } catch { + return; + } + expect(() => readArtifactBytes(dir, stored.digest)).toThrow(); + } finally { + closeTrustedArtifactDir(dir); + } + }); + }); + + test("digest mismatch fails closed", () => { + withHome((home) => { + const dir = openTrustedArtifactDir(join(home, "lab", "artifacts")); + try { + const bytes = new TextEncoder().encode("abc"); + expect(() => putArtifactBytes(dir, bytes, createHashHex("wrong"))).toThrow(); + } finally { + closeTrustedArtifactDir(dir); + } + }); + }); +}); + +describe("CL-02 projection rebuild determinism", () => { + test("partial required coverage for a subject yields PROBED not VERIFIED", () => { + withHome((home) => { + const authority = loadCaseAuthority(); + const caseRecord = discoverScenarios(authority, ["responses-core"]).find( + (c) => c.id === "responses-core.protocol.sse-framing", + )!; + persistConformanceResult(syntheticPassResult(caseRecord), caseRecord, authority, { + configDir: home, + recordedAt: 1_700_000_000_000, + }); + const rebuilt = rebuildLabProjection(home); + const snap = readVerdictSnapshot(rebuilt.sqlitePath); + expect(snap.length).toBe(1); + expect((snap[0] as { verdict: string }).verdict).toBe("PROBED"); + }); + }); + + test("rebuild twice reproduces the same non-purged derived verdict rows", () => { + withHome((home) => { + const authority = loadCaseAuthority(); + const caseRecord = discoverScenarios(authority, ["responses-core"]).find( + (c) => c.id === "responses-core.protocol.sse-framing", + )!; + persistConformanceResult(syntheticPassResult(caseRecord), caseRecord, authority, { + configDir: home, + recordedAt: 1_700_000_000_000, + }); + + const first = rebuildLabProjection(home); + const snap1 = readVerdictSnapshot(first.sqlitePath); + const second = rebuildLabProjection(home); + const snap2 = readVerdictSnapshot(second.sqlitePath); + const third = rebuildLabProjection(home); + const snap3 = readVerdictSnapshot(third.sqlitePath); + expect(snap1).toEqual(snap2); + expect(snap2).toEqual(snap3); + expect(snap1.length).toBe(1); + expect((snap1[0] as { verdict: string }).verdict).toBe("PROBED"); + expect((snap1[0] as { projection_spec_version: string }).projection_spec_version).toBe( + LAB_PROJECTION_SPEC_VERSION, + ); + }); + }); + + test("full applicable required coverage yields VERIFIED", () => { + withHome((home) => { + const authority = loadCaseAuthority(); + const scenarios = discoverScenarios(authority, ["responses-core"]).filter((c) => + c.requirements.upstreamProtocols[0] === "openai-responses" && + c.requirements.surfaces.includes("responses-sse"), + ); + let recordedAt = 1_700_000_000_000; + for (const caseRecord of scenarios) { + const store = createArtifactStore(join(home, "lab", "artifacts")); + try { + persistConformanceResult(syntheticPassResult(caseRecord), caseRecord, authority, { + configDir: home, + recordedAt: recordedAt++, + artifactStore: store, + }); + } finally { + store.close(); + } + } + const rebuilt = rebuildLabProjection(home); + const snap = readVerdictSnapshot(rebuilt.sqlitePath); + expect(snap.some((row) => (row as { verdict: string }).verdict === "VERIFIED")).toBe(true); + }); + }); + + test("projection after invalidation excludes evidence", () => { + withHome((home) => { + const authority = loadCaseAuthority(); + const caseRecord = discoverScenarios(authority, ["responses-core"])[0]!; + const { event } = persistConformanceResult( + { + scenarioId: caseRecord.id, + suite: caseRecord.suite, + passed: true, + classification: "inconclusive", + assertionResults: [], + diagnostics: [], + executionContext: resolveProtocolExecutionContext(caseRecord), + }, + caseRecord, + authority, + { configDir: home, recordedAt: 1000, startedAt: 999, completedAt: 1000 }, + ); + const inv = assignEventId({ + schemaVersion: LAB_EVENT_SCHEMA_VERSION, + eventKind: "invalidation" as const, + recordedAt: 2000, + producer: LAB_PRODUCER, + producerVersion: "2.10.2", + targetEventIds: [event.eventId], + reason: "harness_defect" as const, + }); + appendLabEvent(join(home, "lab", "compatibility.jsonl"), inv as never); + const rebuilt = rebuildLabProjection(home); + const snap = readVerdictSnapshot(rebuilt.sqlitePath); + expect(snap.length).toBe(0); + }); + }); + + test("projection after purge drops purged evidence", () => { + withHome((home) => { + const authority = loadCaseAuthority(); + const caseRecord = discoverScenarios(authority, ["chat-core"])[0]!; + const { event } = persistConformanceResult( + { + scenarioId: caseRecord.id, + suite: caseRecord.suite, + passed: true, + classification: "inconclusive", + assertionResults: [], + diagnostics: [], + executionContext: resolveProtocolExecutionContext(caseRecord), + }, + caseRecord, + authority, + { configDir: home, recordedAt: 1000, startedAt: 999, completedAt: 1000 }, + ); + purgeSensitiveEvidence({ + configDir: home, + targetEventIds: [event.eventId], + targetArtifactDigests: event.fixtureDigests.slice().sort(), + recordedAt: 2000, + }); + const rebuilt = rebuildLabProjection(home); + const snap = readVerdictSnapshot(rebuilt.sqlitePath); + expect(snap.every((row) => { + const ids = JSON.parse((row as { contributing_event_ids_json: string }).contributing_event_ids_json) as string[]; + return !ids.includes(event.eventId); + })).toBe(true); + }); + }); +}); + +describe("CL-02 CL-01 integration", () => { + test("run → observation → JSONL → replay → SQLite protocol projection", async () => { + await withHome(async (home) => { + const authority = loadCaseAuthority(); + const caseRecord = discoverScenarios(authority, ["responses-core"]).find( + (c) => c.id === "responses-core.protocol.request-shape", + )!; + const startedAt = Date.now(); + const result = await runScenario(caseRecord); + const completedAt = Date.now(); + expect(result.passed).toBe(true); + const { event } = observationFromConformanceResult(result, caseRecord, authority, { + configDir: home, + recordedAt: completedAt, + startedAt, + completedAt, + }); + expect(validateLabEvent(event).eventKind).toBe("observation"); + persistConformanceResult(result, caseRecord, authority, { + configDir: home, + recordedAt: completedAt, + startedAt, + completedAt, + }); + const replay = replayLabLedger(join(home, "lab", "compatibility.jsonl")); + expect(replay.validLineCount).toBe(1); + const rebuilt = rebuildLabProjection(home); + expect(rebuilt.events).toBe(1); + const snap = readVerdictSnapshot(rebuilt.sqlitePath); + expect(snap.length).toBe(1); + expect((snap[0] as { evidence_layer: string }).evidence_layer).toBe("protocol_conformance"); + expect((snap[0] as { verdict: string }).verdict).toBe("VERIFIED"); + }); + }); +}); + +describe("CL-02 privacy canaries", () => { + test("sanitizer strips secret-shaped and path material from evidence artifacts", () => { + withHome((home) => { + const secretCanary = "sk-" + "a".repeat(32); + const store = createArtifactStore(join(home, "lab", "artifacts")); + try { + const ref = store.put({ + artifactClass: "error_taxonomy", + payload: { + message: `failed ${secretCanary}`, + path: "C:\\Users\\victim\\secrets\\token.txt", + authorization: "Bearer SUPERSECRET", + url: "https://user:pass@example.com/v1", + }, + }); + const text = new TextDecoder().decode(store.get(ref.digest)); + expect(text).not.toContain(secretCanary); + expect(text).not.toContain("SUPERSECRET"); + expect(text).not.toContain("victim"); + expect(text).not.toContain("user:pass"); + } finally { + store.close(); + } + }); + }); +}); + +describe("CL-02 empty/corrupt ledger", () => { + test("empty ledger reports corruption and rebuilds empty projection", () => { + withHome((home) => { + mkdirSync(join(home, "lab"), { recursive: true }); + writeFileSync(join(home, "lab", "compatibility.jsonl"), ""); + const replay = replayLabLedger(join(home, "lab", "compatibility.jsonl")); + expect(replay.corruptions.some((c) => c.kind === "empty_ledger")).toBe(true); + const rebuilt = rebuildLabProjection(home); + expect(readVerdictSnapshot(rebuilt.sqlitePath)).toEqual([]); + }); + }); +}); + +void chmodSync; +void existsSync; +void claimSourceManifestDigest; +void LabValidationError; + +describe("CL-02 review regression coverage", () => { + test("multiple scenarios for same subject aggregate under one verdict group", () => { + withHome((home) => { + const authority = loadCaseAuthority(); + const scenarios = discoverScenarios(authority, ["responses-core"]).filter((c) => + c.requirements.upstreamProtocols[0] === "openai-responses" && + c.requirements.surfaces.includes("responses-sse"), + ); + expect(scenarios.length).toBeGreaterThan(1); + let t = 1_700_000_000_000; + const subjectIds = new Set(); + for (const caseRecord of scenarios.slice(0, 2)) { + const store = createArtifactStore(join(home, "lab", "artifacts")); + try { + const { event } = observationFromConformanceResult( + syntheticPassResult(caseRecord), + caseRecord, + authority, + { configDir: home, recordedAt: t++, artifactStore: store }, + ); + subjectIds.add(event.subjectId); + appendLabEvent(join(home, "lab", "compatibility.jsonl"), event); + } finally { + store.close(); + } + } + expect(subjectIds.size).toBe(1); + const replay = replayLabLedger(join(home, "lab", "compatibility.jsonl")); + const projected = projectVerdicts(replay.events); + expect(projected.verdicts.length).toBe(1); + expect(projected.verdicts[0]!.contributingEventIds.length).toBe(2); + }); + }); + + test("shared artifact survives partial purge when another observation still references it", () => { + withHome((home) => { + const authority = loadCaseAuthority(); + const scenarios = discoverScenarios(authority, ["responses-core"]).filter((c) => + c.requirements.upstreamProtocols[0] === "openai-responses" && + c.requirements.surfaces.includes("responses-sse"), + ).slice(0, 2); + let t = 1000; + const events = scenarios.map((caseRecord) => { + const store = createArtifactStore(join(home, "lab", "artifacts")); + try { + return persistConformanceResult(syntheticPassResult(caseRecord), caseRecord, authority, { + configDir: home, + recordedAt: t++, + artifactStore: store, + }).event; + } finally { + store.close(); + } + }); + const sharedDigest = events[0]!.suiteManifestDigest; + expect(events[1]!.suiteManifestDigest).toBe(sharedDigest); + purgeSensitiveEvidence({ + configDir: home, + targetEventIds: [events[0]!.eventId], + targetArtifactDigests: [], + recordedAt: 5000, + }); + const store = createArtifactStore(join(home, "lab", "artifacts")); + try { + expect(store.get(sharedDigest).byteLength).toBeGreaterThan(0); + } finally { + store.close(); + } + }); + }); + + test("missing required artifact excludes observation from positive projection", () => { + withHome((home) => { + const authority = loadCaseAuthority(); + const caseRecord = discoverScenarios(authority, ["responses-core"]).find( + (c) => c.id === "responses-core.protocol.sse-framing", + )!; + const { event } = persistConformanceResult( + syntheticPassResult(caseRecord), + caseRecord, + authority, + { configDir: home, recordedAt: 1000 }, + ); + const rebuilt = rebuildLabProjection(home); + expect((readVerdictSnapshot(rebuilt.sqlitePath)[0] as { verdict: string }).verdict).toBe("PROBED"); + const artifactPath = join(home, "lab", "artifacts", `${event.scenarioManifestDigest}.bin`); + expect(existsSync(artifactPath)).toBe(true); + rmSync(artifactPath); + const after = rebuildLabProjection(home); + expect(readVerdictSnapshot(after.sqlitePath)).toEqual([]); + expect(after.corruptions.some((c) => c.kind === "missing_artifact")).toBe(true); + }); + }); + + test("unusable claim source prevents CLAIMED projection", () => { + withHome((home) => { + const subject = { + subjectSchemaVersion: 1 as const, + subjectKind: "route" as const, + providerId: "openai", + providerInstanceFingerprint: createHashHex("inst"), + clientModelId: "gpt", + upstreamModelId: "gpt", + effectiveAdapter: "openai-responses", + inboundProtocol: "openai-responses", + upstreamProtocol: "openai-responses", + surface: "responses-http", + opencodexCompatibilityVersion: "protocol-v1", + behaviorFingerprint: createHashHex("bf"), + endpointFingerprint: createHashHex("ep"), + dependencies: [], + }; + const subjectId = subjectIdForSubject(subject); + const bogusDigest = createHashHex("missing-claim-source"); + const claim = assignEventId({ + schemaVersion: LAB_EVENT_SCHEMA_VERSION, + eventKind: "claim_snapshot" as const, + recordedAt: 10, + producer: LAB_PRODUCER, + producerVersion: "2.10.2", + evidenceLayer: "live_route_compatibility" as const, + subject, + subjectId, + capability: "tools", + polarity: "supported" as const, + sourceManifestDigest: bogusDigest, + sourceEventIds: [], + supersedes: [], + effectiveAt: 10, + }) as ClaimSnapshotEvent; + appendLabEvent(join(home, "lab", "compatibility.jsonl"), claim); + const rebuilt = rebuildLabProjection(home); + const snap = readVerdictSnapshot(rebuilt.sqlitePath); + expect(snap.every((row) => (row as { verdict: string }).verdict !== "CLAIMED")).toBe(true); + }); + }); +}); + +describe("CL-02 phase-2 review regressions", () => { + test("putArtifactBytes fails closed on pre-existing symlink target", () => { + withHome((home) => { + const artifacts = join(home, "lab", "artifacts"); + mkdirSync(artifacts, { recursive: true }); + const bytes = new TextEncoder().encode("reuse-me"); + const digest = artifactBytesDigest(bytes); + const linkPath = join(artifacts, `${digest}.bin`); + const outside = join(home, "outside.bin"); + writeFileSync(outside, "evil"); + try { + symlinkSync(outside, linkPath); + } catch { + return; + } + const dir = openTrustedArtifactDir(artifacts); + try { + expect(() => putArtifactBytes(dir, bytes)).toThrow(ArtifactFsError); + } finally { + closeTrustedArtifactDir(dir); + } + }); + }); + + test("replay rejects valid line followed by unterminated JSON line", () => { + withHome((home) => { + const ledger = join(home, "lab", "compatibility.jsonl"); + mkdirSync(join(home, "lab"), { recursive: true }); + const good = baseObservation(); + const goodLine = `${jcsStringify(good)}\n`; + const partial = jcsStringify(baseObservation({ scenarioId: "responses-core.protocol.request-shape" })); + writeFileSync(ledger, goodLine + partial, "utf8"); + const replay = replayLabLedger(ledger); + expect(replay.events).toHaveLength(1); + expect(replay.corruptions.some((c) => c.kind === "partial_line")).toBe(true); + }); + }); + + test("replay bounds oversized line without newline and continues after newline", () => { + withHome((home) => { + const ledger = join(home, "lab", "compatibility.jsonl"); + mkdirSync(join(home, "lab"), { recursive: true }); + const good = baseObservation(); + const huge = "x".repeat(70 * 1024); + writeFileSync(ledger, `${jcsStringify(good)}\n${huge}\n${jcsStringify({ ...good, scenarioId: "responses-core.protocol.sse-framing" })}\n`, "utf8"); + const replay = replayLabLedger(ledger); + expect(replay.events.length).toBeGreaterThanOrEqual(1); + expect(replay.corruptions.some((c) => c.kind === "malformed_line")).toBe(true); + }); + }); + + test("replay handles UTF-8 split across chunk boundaries", () => { + withHome((home) => { + const ledger = join(home, "lab", "compatibility.jsonl"); + mkdirSync(join(home, "lab"), { recursive: true }); + const obs = baseObservation({ + assertions: [{ + id: "utf8", + operator: "equals", + required: true, + passed: true, + expectedSummary: "café", + observedSummary: "café", + }], + }); + writeFileSync(ledger, `${jcsStringify(obs)}\n`, "utf8"); + const replay = replayLabLedger(ledger); + expect(replay.events).toHaveLength(1); + expect(replay.corruptions).toHaveLength(0); + }); + }); + + test("zero applicable required scenarios yields UNKNOWN not PROBED", () => { + const authority = loadCaseAuthority(); + const suiteManifest = expandSuiteManifest("responses-core", authority); + const subject = protocolSubject("zero-applicable"); + const loadScenarioManifest = (digest: string) => { + for (const caseRecord of authority.cases) { + const expanded = expandScenario(caseRecord, authority); + if (scenarioManifestDigest(expanded) === digest) return expanded; + } + return null; + }; + const evaluation = evaluateAllApplicableRequiredPassV1( + suiteManifest, + [], + "fixture", + { subject: { ...subject, surface: "responses-ws" }, loadScenarioManifest }, + ); + expect(evaluation.applicableRequiredScenarioIds).toEqual([]); + expect(evaluation.notes).toContain("empty_applicable_required_set"); + const projected = projectVerdicts([]); + expect(projected.verdicts).toEqual([]); + }); + + test("newer pass supersedes older fail without DEGRADED", () => { + const failEvent = assignEventId({ ...baseObservation({ completedAt: 100 }), outcome: "fail" }); + const passEvent = assignEventId({ ...baseObservation({ completedAt: 200 }), outcome: "pass" }); + const projected = projectVerdicts([failEvent, passEvent]); + expect(projected.verdicts[0]?.verdict).not.toBe("DEGRADED"); + expect(projected.verdicts[0]?.contradictingEventIds).toContain(failEvent.eventId); + }); + + test("json-sse-equivalence uses responses-sse execution context", () => { + const authority = loadCaseAuthority(); + const caseRecord = discoverScenarios(authority, ["responses-core"]).find( + (c) => c.id === "responses-core.protocol.json-sse-equivalence", + )!; + const ctx = resolveProtocolExecutionContext(caseRecord); + expect(ctx.surface).toBe("responses-sse"); + expect(caseRecord.requirements.surfaces).toContain("responses-http"); + expect(caseRecord.requirements.surfaces).toContain("responses-sse"); + }); + + test("historical manifest missing cannot recover from packaged authority", () => { + withHome((home) => { + const authority = loadCaseAuthority(); + const caseRecord = discoverScenarios(authority, ["responses-core"]).find( + (c) => c.id === "responses-core.protocol.sse-framing", + )!; + const { event } = persistConformanceResult(syntheticPassResult(caseRecord), caseRecord, authority, { + configDir: home, + recordedAt: 1000, + }); + rebuildLabProjection(home); + const artifactPath = join(home, "lab", "artifacts", `${event.scenarioManifestDigest}.bin`); + expect(existsSync(artifactPath)).toBe(true); + rmSync(artifactPath); + const after = rebuildLabProjection(home); + expect(readVerdictSnapshot(after.sqlitePath)).toEqual([]); + }); + }); + + test("nested array depth and unknown admission keys fail closed", () => { + let deep: unknown[] = [1]; + for (let i = 0; i < 10; i++) deep = [deep]; + expect(() => enforceEventStructureLimits(deep)).toThrow(); + const obs = baseObservation(); + expect(() => validateLabEvent({ ...obs, limits: { ...obs.limits, rogue: 1 } })).toThrow(); + expect(() => validateLabEvent({ ...obs, assertions: [{ ...obs.assertions[0]!, extra: "x" }] })).toThrow(); + }); + + test("corrupt superseding claim blocks CLAIMED without resurrecting C1", () => { + withHome((home) => { + const subject = { + subjectSchemaVersion: 1 as const, + subjectKind: "route" as const, + providerId: "openai", + providerInstanceFingerprint: createHashHex("inst"), + clientModelId: "gpt", + upstreamModelId: "gpt", + effectiveAdapter: "openai-responses", + inboundProtocol: "openai-responses", + upstreamProtocol: "openai-responses", + surface: "responses-http", + opencodexCompatibilityVersion: "protocol-v1", + behaviorFingerprint: createHashHex("bf"), + endpointFingerprint: createHashHex("ep"), + dependencies: [], + }; + const subjectId = subjectIdForSubject(subject); + const store = createArtifactStore(join(home, "lab", "artifacts")); + const goodDigest = store.put({ + artifactClass: "claim_source_manifest", + payload: { + schemaVersion: 1, + subjectId, + providerId: "openai", + clientModelId: "gpt", + capability: "tools", + sources: [{ kind: "provider_config", revision: null, facts: { toolCapable: true } }], + resolvedEvidence: { tools: true }, + }, + }).digest; + store.close(); + const c1 = assignEventId({ + schemaVersion: LAB_EVENT_SCHEMA_VERSION, + eventKind: "claim_snapshot" as const, + recordedAt: 1, + producer: LAB_PRODUCER, + producerVersion: "2.10.2", + evidenceLayer: "live_route_compatibility" as const, + subject, + subjectId, + capability: "tools", + polarity: "supported" as const, + sourceManifestDigest: goodDigest, + sourceEventIds: [], + supersedes: [], + effectiveAt: 1, + }) as ClaimSnapshotEvent; + const c2 = assignEventId({ + ...c1, + eventId: undefined as unknown as string, + recordedAt: 2, + effectiveAt: 2, + sourceManifestDigest: createHashHex("missing-manifest"), + supersedes: [c1.eventId], + }) as ClaimSnapshotEvent; + appendLabEvent(join(home, "lab", "compatibility.jsonl"), c1); + appendLabEvent(join(home, "lab", "compatibility.jsonl"), c2); + const rebuilt = rebuildLabProjection(home); + const snap = readVerdictSnapshot(rebuilt.sqlitePath); + expect(snap.every((row) => (row as { verdict: string }).verdict !== "CLAIMED")).toBe(true); + }); + }); + + test("explicit sensitive shared artifact purge removes bytes and dependent events", () => { + withHome((home) => { + const authority = loadCaseAuthority(); + const scenarios = discoverScenarios(authority, ["responses-core"]).slice(0, 2); + const events = scenarios.map((caseRecord, i) => + persistConformanceResult(syntheticPassResult(caseRecord), caseRecord, authority, { + configDir: home, + recordedAt: 1000 + i, + }).event, + ); + const sharedDigest = events[0]!.suiteManifestDigest; + purgeSensitiveEvidence({ + configDir: home, + targetEventIds: [events[0]!.eventId], + targetArtifactDigests: [sharedDigest], + recordedAt: 5000, + }); + const store = createArtifactStore(join(home, "lab", "artifacts")); + try { + expect(() => store.get(sharedDigest)).toThrow(); + } finally { + store.close(); + } + const replay = replayLabLedger(join(home, "lab", "compatibility.jsonl")); + expect(replay.events.some((e) => e.eventId === events[0]!.eventId)).toBe(false); + }); + }); + + test("scratch and export purge actions remove bounded lab directories", () => { + withHome((home) => { + const authority = loadCaseAuthority(); + const caseRecord = discoverScenarios(authority, ["responses-core"])[0]!; + const { event } = persistConformanceResult(syntheticPassResult(caseRecord), caseRecord, authority, { + configDir: home, + recordedAt: 1000, + }); + const scratch = join(home, "lab", "scratch"); + const exportDir = join(home, "lab", "export"); + mkdirSync(scratch, { recursive: true }); + mkdirSync(exportDir, { recursive: true }); + writeFileSync(join(scratch, "temp.txt"), "x"); + writeFileSync(join(exportDir, "bundle.json"), "{}"); + purgeSensitiveEvidence({ + configDir: home, + targetEventIds: [event.eventId], + targetArtifactDigests: [], + purgeActions: ["ledger", "sqlite", "scratch", "export"], + recordedAt: 5000, + }); + expect(existsSync(join(scratch, "temp.txt"))).toBe(false); + expect(existsSync(join(exportDir, "bundle.json"))).toBe(false); + }); + }); + + test("behavior fingerprint includes frozen runtime keys deterministically", () => { + const authority = loadCaseAuthority(); + const caseRecord = discoverScenarios(authority, ["responses-core"])[0]!; + const a = behaviorFingerprintForCase(caseRecord); + const b = behaviorFingerprintForCase(caseRecord); + expect(a).toBe(b); + expect(a).toMatch(/^[a-f0-9]{64}$/); + }); + + test("observationFromConformanceResult closes internally created artifact store", () => { + withHome((home) => { + const authority = loadCaseAuthority(); + const caseRecord = discoverScenarios(authority, ["responses-core"])[0]!; + const result = syntheticPassResult(caseRecord); + const { event } = observationFromConformanceResult(result, caseRecord, authority, { configDir: home }); + expect(event.eventKind).toBe("observation"); + const dir = openTrustedArtifactDir(join(home, "lab", "artifacts")); + closeTrustedArtifactDir(dir); + }); + }); +}); \ No newline at end of file