Skip Verifier turns when selecting the committed answer#246
Open
boskodev790 wants to merge 1 commit into
Open
Conversation
_committed_answer falls back to scanning earlier turns when the final output has no parseable answer, but it scanned every turn including Verifier turns. The Verifier is a critic, not a solver: it routinely quotes the gold answer while returning VERDICT: REVISE. Crediting that quote scores a trajectory the Verifier explicitly rejected as correct — inflating both the eval metric and the sep-CMA-ES training fitness whenever the last Worker turn lacks a parseable answer. Skip Verifier turns in the scan, mirroring _final_answer (session.py), which already falls back to the last non-Verifier output. Adds regression tests for the verifier-quote case, preserved recovery of an earlier non-Verifier answer, and the accept-control case.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
_committed_answer(src/trinity/orchestration/reward.py) chooses which text in amulti-turn trajectory gets scored. When the final output has no parseable answer,
it falls back to scanning earlier turns and returns the most recent one that has
an extractable answer — but it scans every turn, including Verifier turns.
The Verifier is a critic, not a solver, and it routinely restates the gold answer
while returning
VERDICT: REVISE. Crediting that quote scores a trajectory theVerifier explicitly rejected as correct. This inflates both the eval metric
and the sep-CMA-ES training fitness (both go through
reward.score) whenever thelast Worker turn lacks a parseable answer.
The session layer's own
_final_answer(session.py) already avoids this — itfalls back to the last non-Verifier output — so the two answer-selection
paths currently disagree.
Repro (before the fix)
Fix
Skip Verifier turns in the fallback scan, mirroring
_final_answer. A one-lineguard; the legitimate behavior (recovering an answer from an earlier Worker/Thinker
turn) is unchanged.
With the fix, the repro above scores
0.0.Testing
tests/test_reward_checkers.py:test_committed_answer_skips_verifier_that_quotes_gold— the rejected trajectory now scores0.0.test_committed_answer_still_recovers_earlier_nonverifier_answer— legitimate recovery preserved.test_committed_answer_uses_worker_answer_with_verifier_accept— accept-control still scores from the Worker.Scope
Two files (
reward.pyguard + tests). No behavior change for single-turn baselinesor for trajectories whose committed answer already comes from a Worker/Thinker turn.