fix(reward): widen multiple-choice alphabet to A-J so MMLU-Pro E-J grades (#116)

[minion1227]

Fixes #116
Fixes #122

(Both issues report the same root bug, independently — closing them together.)

Problem

RLPR routes MMLU-Pro (MMLUPro-1000_Avg2) to the choice grader, but that path is hard-capped at A–D while MMLU-Pro has up to ten options (A–J). Every item whose gold answer is E–J scores 0.0 even when the model answers correctly — a silently-wrong number, not a crash. MMLU-Pro gold answers spread across all ten letters, so a large fraction of that subset is unconditionally graded 0, deflating measured accuracy and distorting the per-benchmark routing signal the coordinator is trained and evaluated on.

for gold in ["A", "D", "E", "F", "J"]:
    ref = {"ground_truth": gold, "source": "MMLUPro-1000_Avg2"}
    print(gold, R.score_text("rlpr", f"The answer is ({gold}).", ref))
# A 1.0 / D 1.0 / E 0.0 / F 0.0 / J 0.0   <-- correct answers scored wrong

Both sides of the check were capped, so an E–J item could neither be extracted from the output nor normalized from the reference:

• the five _CHOICE_PATTERNS captured ([A-D])
• extract_choice_letter's last-line fallback matched ([A-D])
• _normalize_reference_letter accepted only {A,B,C,D} and integer index <= 3

Fix

Introduce a single _CHOICE_ALPHABET = "ABCDEFGHIJ" (plus a derived character class) and drive all four sites from it, so the alphabet cannot drift out of sync again.

Why this is safe for the 4-option benchmarks

MMLU / GPQA gold answers are always A–D, so extracting a stray higher letter still cannot match a 4-option gold — no score can flip. Widening only adds the ability to grade E–J. Verified by explicit guards below.

Tests

Adds tests/test_reward_choice_alphabet.py (41 cases, pure stdlib — no network/GPU/torch):

• every letter A–J grades end-to-end on RLPR MMLU-Pro, and a wrong letter still fails
• extract_choice_letter / _normalize_reference_letter span A–J, by letter and integer index (0→A, 4→E, 9→J), with out-of-range still None
• guards: MMLU/GPQA unchanged; a stray high letter (I/J) cannot match a 4-option gold; the prose article "A" is still not read as a choice

Full root suite: 287 passed; ruff clean on the changed lines (no new findings).

[minion1227]

CI: register_submission failure is not from this PR

The red check is repo-wide infrastructure, not this diff. actions/checkout now refuses to check out fork code under pull_request_target (pwn-request guard), so the job fails at checkout on every fork PR — currently #226, #232, #233, #240 and this one. Because pull_request_target runs the workflow definition from base main, it cannot be fixed from a PR branch. #233 already proposes the fix (dropping the unused checkout step from that job).

All substantive checks pass here: test-router, test-validator, web, and check_submission_body.

Overlap with #217 (heads-up for whoever merges second)

#217 (#124) touches the same two places in reward.py — _CHOICE_PATTERNS and extract_choice_letter — so the two branches conflict textually, though the fixes are complementary and deliberately scoped apart (that PR's own journal entry notes "letter-range A–J is tracked separately (#122 / #116)"). I verified the conflict by test-merging locally.

Worth noting for the rebase, in either order:

• fix(reward): grade committed MC letter, not first echoed option (#124) #217 removes the line-start ^\s*\(?\s*([A-D])\s*[\).:] pattern (it matches echoed option lines and returns the first one). This PR widens that same pattern to A–J. If fix(reward): grade committed MC letter, not first echoed option (#124) #217 lands first, the pattern is simply gone and there is nothing to widen — resolve by dropping it.
• fix(reward): grade committed MC letter, not first echoed option (#124) #217 adds a trailing-standalone-letter fallback, re.search(r"(?<![A-Za-z])([A-D])(?![A-Za-z])\s*$", ...), with a fresh [A-D] literal. That path needs _CHOICE_LETTER_CLASS too, or the ten-option cap is silently reintroduced on it — an MMLU-Pro answer ending in a bare "F" would go back to scoring 0.

Happy to rebase this onto #217 whenever that one lands.

[minion1227]

Thanks @tima — I dug into the classification to make sure I'm following the right lane.

#239 doesn't include a submissions/final_model/ bundle (its files are just reward.py, a test, and a JOURNAL.md entry), so pr-automation.yml labels it miner/train and it isn't queued as a submission — same shape as the issue-fix PRs that normally get merged here, like my earlier #114 (#105). I opened it as a code fix for issue #116 (the A–D multiple-choice cap that was zeroing MMLU-Pro E–J), not as a model submission.

The red register_submission check on it is the repo-wide pull_request_target checkout-guard bug — it fails at checkout on every fork PR (including #233 itself, which proposes the fix), independent of whether the PR is a submission. That's all my earlier comment meant; happy to drop that comment if it read as dismissive.

Quick check on convention so I stay in the right lane: are code/issue-fix PRs welcome on my sn74-minion1227-<topic> branches, or would you prefer I keep sn74- strictly for final-model submissions and use a different branch prefix for code fixes? Glad to follow whatever you'd like.