Skip to content

Skip Verifier turns when selecting the committed answer#246

Open
boskodev790 wants to merge 1 commit into
mini-router:mainfrom
boskodev790:sn74-boskodev790-committed-answer
Open

Skip Verifier turns when selecting the committed answer#246
boskodev790 wants to merge 1 commit into
mini-router:mainfrom
boskodev790:sn74-boskodev790-committed-answer

Conversation

@boskodev790

Copy link
Copy Markdown

Problem

_committed_answer (src/trinity/orchestration/reward.py) chooses which text in a
multi-turn trajectory gets scored. When the final output has no parseable answer,
it falls back to scanning earlier turns and returns the most recent one that has
an extractable answer — but it scans every turn, including Verifier turns.

The Verifier is a critic, not a solver, and it routinely restates the gold answer
while returning VERDICT: REVISE. Crediting that quote scores a trajectory the
Verifier explicitly rejected as correct. This inflates both the eval metric
and the sep-CMA-ES training fitness (both go through reward.score) whenever the
last Worker turn lacks a parseable answer.

The session layer's own _final_answer (session.py) already avoids this — it
falls back to the last non-Verifier output — so the two answer-selection
paths currently disagree.

Repro (before the fix)

from trinity.orchestration import reward as R
from trinity.orchestration.session import _final_answer
from trinity.types import Trajectory, Task, TurnRecord, Role

def rec(role, out): return TurnRecord(turn=1, agent_name="m", role=role, raw_output=out, processed_output=out)
task = Task("t", "math500", "q", "42")
traj = Trajectory(task=task, turns=[
    rec(Role.WORKER,   "I am not sure how to finish this."),
    rec(Role.VERIFIER, "The correct value is 42, but the worker never computed it.\nVERDICT: REVISE"),
])
traj.final_answer = _final_answer(traj)
print(R.score(traj))   # 1.0  (BUG: the Verifier rejected it; the Worker gave no answer)

Fix

Skip Verifier turns in the fallback scan, mirroring _final_answer. A one-line
guard; the legitimate behavior (recovering an answer from an earlier Worker/Thinker
turn) is unchanged.

for tr in reversed(turns):
    if getattr(tr, "role", None) == Role.VERIFIER:
        continue
    ...

With the fix, the repro above scores 0.0.

Testing

  • New tests in tests/test_reward_checkers.py:
    • test_committed_answer_skips_verifier_that_quotes_gold — the rejected trajectory now scores 0.0.
    • test_committed_answer_still_recovers_earlier_nonverifier_answer — legitimate recovery preserved.
    • test_committed_answer_uses_worker_answer_with_verifier_accept — accept-control still scores from the Worker.
  • Full router suite green: 249 passed.

Scope

Two files (reward.py guard + tests). No behavior change for single-turn baselines
or for trajectories whose committed answer already comes from a Worker/Thinker turn.

_committed_answer falls back to scanning earlier turns when the final output
has no parseable answer, but it scanned every turn including Verifier turns.
The Verifier is a critic, not a solver: it routinely quotes the gold answer
while returning VERDICT: REVISE. Crediting that quote scores a trajectory the
Verifier explicitly rejected as correct — inflating both the eval metric and
the sep-CMA-ES training fitness whenever the last Worker turn lacks a parseable
answer.

Skip Verifier turns in the scan, mirroring _final_answer (session.py), which
already falls back to the last non-Verifier output. Adds regression tests for
the verifier-quote case, preserved recovery of an earlier non-Verifier answer,
and the accept-control case.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant