Skip to content

fix(reward): extract \boxed{} before grading RLPR WebInstruct math (#170)#254

Open
minion1227 wants to merge 1 commit into
mini-router:mainfrom
minion1227:sn74-minion1227-170
Open

fix(reward): extract \boxed{} before grading RLPR WebInstruct math (#170)#254
minion1227 wants to merge 1 commit into
mini-router:mainfrom
minion1227:sn74-minion1227-170

Conversation

@minion1227

Copy link
Copy Markdown
Contributor

Fixes #170

Problem

_check_rlpr_webinstruct in src/trinity/orchestration/reward.py scores WebInstruct-verified RLPR items (WebInstruct-verified-val_Avg2). For a math answer it fed the raw candidate prose straight into math_equal / normalize_math_answer — unlike _check_math, which calls extract_boxed / extract_last_number first. Since normalize_math_answer doesn't strip \boxed{...}, a correctly boxed answer never matched the bare gold and scored 0.0 — even though format_hint("rlpr") explicitly instructs the worker to "use a boxed final answer for math items". The requested output format was the one graded wrong.

from trinity.orchestration import reward as R
ref = {"ground_truth": "15", "source": "WebInstruct-verified-val_Avg2"}
R.score_text("rlpr", r"The area is \boxed{15}.", ref)      # 0.0  <- correct answer, scored wrong
R.score_text("rlpr", "The answer is 15", ref)             # 1.0  (plain number happened to work)
R.score_text("math500", r"The area is \boxed{15}.", "15") # 1.0  (math path extracts the box)

Impact: every WebInstruct-verified RLPR math item answered in the requested \boxed{...} format is a false negative — understating accuracy on that benchmark and corrupting the per-benchmark reward signal.

Fix

Extract the answer from candidate (and gold) before the math compare, mirroring _check_math:

cand_math = extract_boxed(cand) or extract_last_number(cand) or cand
gold_math = extract_boxed(gold) or gold
if math_equal(cand_math, gold_math):
    return True
return normalize_math_answer(cand_math) == normalize_math_answer(gold_math)

The or cand fallback is load-bearing: when a candidate has neither a box nor a number (a free-text answer), cand_math is the whole candidate, so non-math text answers grade exactly as before — the behaviour only diverges when an answer is actually extracted.

Distinct from #116 / #122

Those cover the RLPR MMLU-Pro choice path (letters E–J). This is the WebInstruct math branch — a different code path and a different fix. The letter-choice branch here is untouched.

Tests

Adds tests/test_reward_rlpr_webinstruct_boxed.py (offline, pure stdlib — no network/GPU):

  • the boxed repro now grades 1.0; a boxed value wins over a stray earlier number; a number-in-prose without a box is recovered; a boxed LaTeX fraction matches 1/2; a boxed gold is unwrapped
  • guards: the plain-number, free-text ("Paris"), and letter-choice paths are unchanged; empty candidate/gold still False

5 of the new cases fail on the old code and pass on the fix; the 10 guard cases pass either way. Full root suite: 261 passed. ruff clean on the changed lines (the 3 pre-existing findings in reward.py are on unrelated lines).

…ini-router#170)

_check_rlpr_webinstruct fed the raw candidate prose into math_equal /
normalize_math_answer, neither of which strips \boxed{...}. The RLPR
format hint tells the worker to box math answers, so every correctly
boxed WebInstruct math item was a false negative (scored 0), understating
that benchmark's accuracy and corrupting its reward signal.

Extract the answer first, mirroring _check_math:
  cand_math = extract_boxed(cand) or extract_last_number(cand) or cand
  gold_math = extract_boxed(gold) or gold
The 'or cand' fallback preserves free-text answers unchanged; behaviour
only diverges when an answer is actually extracted. Distinct from the
MMLU-Pro choice path (mini-router#116/mini-router#122).

Adds tests/test_reward_rlpr_webinstruct_boxed.py (offline): boxed repro,
box-beats-stray-number, number-in-prose fallback, boxed fraction, boxed
gold, plus guards that the plain-number/free-text/letter paths are
unchanged. 5 cases fail on old code; full root suite 261 passed.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

reward: RLPR WebInstruct math grader ignores \boxed{...} answers, scoring correct answers 0

1 participant