Require the checker to complete in run_pass_at_1 (block early-exit reward hack)#253
Open
boskodev790 wants to merge 1 commit into
Open
Require the checker to complete in run_pass_at_1 (block early-exit reward hack)#253boskodev790 wants to merge 1 commit into
boskodev790 wants to merge 1 commit into
Conversation
…ward hack) The assert-based (BigCodeBench) and call-based (LiveCodeBench functional) test flavors append the verification code AFTER the untrusted candidate and judged a pass by the subprocess exit code alone. A candidate that terminates the process with a zero status before the appended checks ran — sys.exit(0), exit()/quit(), raise SystemExit(0), or a hard os._exit(0) — short-circuited the asserts and scored correct regardless of correctness. Since this reward is exactly what sep-CMA-ES optimizes, it is directly reward-hackable. Route both flavors through _run_checked_script, which appends a per-run random success sentinel after the checker and requires exit 0 AND that sentinel in stdout. Any early termination before the checker completes fails, because the sentinel is never emitted; the token is unguessable per run so it cannot be forged by printing a fixed string. The stdin/stdout flavor is unaffected (it already compared captured output). Removes the now-unused _exec_script helper. Adds regression tests for sys.exit / exit() / os._exit / raise SystemExit before both the assert and functional checkers, a forged-sentinel attempt, and a correct-code control.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
run_pass_at_1grades code benchmarks in an isolated subprocess. For theassert-based (BigCodeBench) and call-based (LiveCodeBench functional)
flavors, the verification code is concatenated after the untrusted candidate
code, and a pass was judged by the subprocess exit code alone (
returncode == 0).A candidate that terminates the process with a zero status before the appended
checks run —
sys.exit(0),exit()/quit(),raise SystemExit(0), or a hardos._exit(0)— short-circuits the asserts and is scored correct regardless ofcorrectness. Because this reward is exactly what sep-CMA-ES optimizes, it is
directly reward-hackable: a pool model that learns to append a trailing
exit()after its code would "pass" every assert/functional code task.
Repro (before the fix)
Fix
Route both append-a-checker flavors through a new
_run_checked_script, whichappends a write of a per-run random sentinel after the checker and requires
exit 0 AND the sentinel present in stdout. Any early termination before the
checker completes fails, because the sentinel is never emitted. The token is
unguessable per run, so a candidate cannot forge a pass by printing a fixed
string.
The stdin/stdout flavor is unaffected — it already compared captured output,
so an early
exit(0)with wrong/no output already failed. The now-unused_exec_scripthelper is removed.With the fix, all four early-exit variants above score
False; correct codestill passes.
Testing
tests/test_reward_sandbox.py:sys.exit,exit(),os._exit,and
raise SystemExitbefore both the assert and functional checkers; aforged-sentinel attempt; and a correct-code control.
Residual (called out honestly)
The sentinel defeats any early process termination, including hard
os._exit. Itdoes not defend against a candidate that reads its own on-disk script to recover
the random token, prints it, then exits — far outside an evolved router's
behavior and outside the reported threat, but noted for completeness. Fully
closing that would require executing the candidate and checker in separate
address spaces.
Scope
Two files (
reward.py+ tests). No behavior change for the stdin/stdout path orfor correct candidates.