Skip to content

Require the checker to complete in run_pass_at_1 (block early-exit reward hack)#253

Open
boskodev790 wants to merge 1 commit into
mini-router:mainfrom
boskodev790:sn74-boskodev790-sandbox-exit
Open

Require the checker to complete in run_pass_at_1 (block early-exit reward hack)#253
boskodev790 wants to merge 1 commit into
mini-router:mainfrom
boskodev790:sn74-boskodev790-sandbox-exit

Conversation

@boskodev790

Copy link
Copy Markdown

Problem

run_pass_at_1 grades code benchmarks in an isolated subprocess. For the
assert-based (BigCodeBench) and call-based (LiveCodeBench functional)
flavors, the verification code is concatenated after the untrusted candidate
code, and a pass was judged by the subprocess exit code alone (returncode == 0).

A candidate that terminates the process with a zero status before the appended
checks run — sys.exit(0), exit()/quit(), raise SystemExit(0), or a hard
os._exit(0) — short-circuits the asserts and is scored correct regardless of
correctness
. Because this reward is exactly what sep-CMA-ES optimizes, it is
directly reward-hackable: a pool model that learns to append a trailing exit()
after its code would "pass" every assert/functional code task.

Repro (before the fix)

import trinity.orchestration.reward as R
# wrong code that exits 0 before the appended assert runs:
print(R.run_pass_at_1("def add(a,b):\n return 999\nimport sys\nsys.exit(0)\n",
                      ["assert add(2,3)==5"], timeout_s=5))   # True  (BUG)
print(R.run_pass_at_1("def add(a,b):\n return 999\nimport os\nos._exit(0)\n",
                      ["assert add(2,3)==5"], timeout_s=5))    # True  (BUG)

Fix

Route both append-a-checker flavors through a new _run_checked_script, which
appends a write of a per-run random sentinel after the checker and requires
exit 0 AND the sentinel present in stdout. Any early termination before the
checker completes fails, because the sentinel is never emitted. The token is
unguessable per run, so a candidate cannot forge a pass by printing a fixed
string.

The stdin/stdout flavor is unaffected — it already compared captured output,
so an early exit(0) with wrong/no output already failed. The now-unused
_exec_script helper is removed.

With the fix, all four early-exit variants above score False; correct code
still passes.

Testing

  • New tests in tests/test_reward_sandbox.py: sys.exit, exit(), os._exit,
    and raise SystemExit before both the assert and functional checkers; a
    forged-sentinel attempt; and a correct-code control.
  • Existing sandbox / functional / checker tests unchanged and green.
  • Full router suite: 253 passed.

Residual (called out honestly)

The sentinel defeats any early process termination, including hard os._exit. It
does not defend against a candidate that reads its own on-disk script to recover
the random token, prints it, then exits — far outside an evolved router's
behavior and outside the reported threat, but noted for completeness. Fully
closing that would require executing the candidate and checker in separate
address spaces.

Scope

Two files (reward.py + tests). No behavior change for the stdin/stdout path or
for correct candidates.

…ward hack)

The assert-based (BigCodeBench) and call-based (LiveCodeBench functional) test
flavors append the verification code AFTER the untrusted candidate and judged a
pass by the subprocess exit code alone. A candidate that terminates the process
with a zero status before the appended checks ran — sys.exit(0), exit()/quit(),
raise SystemExit(0), or a hard os._exit(0) — short-circuited the asserts and
scored correct regardless of correctness. Since this reward is exactly what
sep-CMA-ES optimizes, it is directly reward-hackable.

Route both flavors through _run_checked_script, which appends a per-run random
success sentinel after the checker and requires exit 0 AND that sentinel in
stdout. Any early termination before the checker completes fails, because the
sentinel is never emitted; the token is unguessable per run so it cannot be
forged by printing a fixed string. The stdin/stdout flavor is unaffected (it
already compared captured output). Removes the now-unused _exec_script helper.

Adds regression tests for sys.exit / exit() / os._exit / raise SystemExit before
both the assert and functional checkers, a forged-sentinel attempt, and a
correct-code control.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant