Skip to content

fix(agent): verify exhaustive result requests#740

Merged
ericleepi314 merged 1 commit into
mainfrom
fix/exhaustive-result-audit
Jul 22, 2026
Merged

fix(agent): verify exhaustive result requests#740
ericleepi314 merged 1 commit into
mainfrom
fix/exhaustive-result-audit

Conversation

@ericleepi314

Copy link
Copy Markdown
Collaborator

Summary

  • add a completion-audit contract to the core task prompt
  • require one bounded tool-backed audit when a user explicitly requests all/multiple/exhaustive results
  • preserve the audit latch across tool rounds and add regression coverage

Root cause

On terminal-bench 2.1 chess-best-move, clawcodex found e2e4 and stopped despite the request to return all winning moves. Claude Code mechanically enumerated candidates and also found g2g4. A prose-only follow-up audit repeated the same error, so the runtime guard now requires tool-backed validation for mechanically enumerable state spaces.

Verification

  • pytest -q tests/test_query_loop_wires_round3.py tests/parity/test_query_state_parity.py tests/parity/test_error_recovery_flow.py tests/test_system_prompt_full.py tests/test_query_extended_thinking.py tests/test_effort.py — 125 passed
  • terminal-bench 2.1 terminal-bench/chess-best-move, Claude Opus 4.8, effort high — reward improved from 0.0 to 1.0
  • successful local job: eval/harbor/jobs/chess-rerun-tool-backed-audit

@ericleepi314
ericleepi314 merged commit 4632433 into main Jul 22, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant