Skip to content

[codex] add isolated judge QA execution - #65

Merged
greenthree merged 2 commits into
mainfrom
codex/judge-qa-execution
Aug 6, 2026
Merged

[codex] add isolated judge QA execution#65
greenthree merged 2 commits into
mainfrom
codex/judge-qa-execution

Conversation

@greenthree

Copy link
Copy Markdown
Owner

What changed

  • add the lazy top-level judge_qa_problem(...) Core API and an isolated Judge QA runtime
  • run Validator, Checker/Interactor fixtures, controlled contestants, and automatic robustness probes from a byte-consistent snapshot under the existing per-problem Judge lock
  • enforce an overall deadline, default cancel-file and caller cancellation callbacks, live source/data/fixture hash fences, bounded structured results, and full process-tree cleanup
  • atomically publish judge-qa-evidence-v1.json only after a complete successful run; failures, cancellation, timeout, input drift, lock contention, cleanup failure, or publication failure preserve the previous evidence
  • share Validator policy and cancellation primitives with the formal Judge, and keep Interactor resource-failure attribution stable during simultaneous exits
  • require official Validator/Checker/Interactor programs to remain reproducible C++ sources, matching the formal Judge and DOMjudge package path; controlled Interactor contestants may use C++ or Python

Why

The P0 schema can describe adversarial Checker and Interactor fixtures, but it could not execute them. This P1 slice provides the isolated execution and evidence layer needed to actively test each problem's special Judge without mutating formal PDF, ZIP, metadata, Manifest, source, or data artifacts.

This PR intentionally does not add the judge-qa CLI, seal integration, lint/status/report evidence state, Skill/reference updates, WebUI, or Legacy support. Those remain the P2/P3 work.

Validation

  • targeted Python/unit/integration suite: 154 passed, 13 skipped
  • npm run check: 643 passed, 30 skipped
  • npm run pack:check: probhub=217, probhub-skill=5
  • Skill quick validation passed
  • git diff --check passed
  • local phtest nine-problem lint passed
  • local phtest L02 judge --no-cache passed: 33 inputs, 3 solutions, 99 case runs, all_expectations_met
  • final same-model read-only review found no blocking issues

@greenthree
greenthree marked this pull request as ready for review August 6, 2026 04:48
@greenthree
greenthree merged commit b11570b into main Aug 6, 2026
11 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant