Fix IFEval loose scoring: build each verifier once, not per variant - #296
Open
abhishekraok wants to merge 3 commits into
Open
Fix IFEval loose scoring: build each verifier once, not per variant#296abhishekraok wants to merge 3 commits into
abhishekraok wants to merge 3 commits into
Conversation
IFEvalScorer rebuilt a fresh instruction instance for the strict check and again for each of the 8 loose response variants. Several ifbench verifiers (e.g. count:word_count_range) draw random parameters for any kwarg the dataset leaves unset, so rebuilding per variant scored each one against different criteria instead of the same one. Two observable symptoms: loose accuracy could fall below strict, which is impossible since loose's variant set includes the unmodified response; and scores drifted across repeated scoring of identical model output (measured spread up to 0.0084 on inst_level_strict_acc over 5 runs on 150 IFEval responses). Now each instruction is built once via _build_instruction and reused for the strict check and every loose variant, matching oe-eval's reference implementation in test_instruction_following_loose. This affects ifeval_ood, which is already live in the ifbench suite. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01B9VUctQuzfAtrdpk4wDjEx
|
You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard. |
6 tasks
Per-trial flip probability under the pre-fix code is ~0.04 (measured by simulating the verifier's random draws), so 50 trials caught the regression with only ~87% probability; 200 trials raise that above 99.9% at negligible runtime cost. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01B9VUctQuzfAtrdpk4wDjEx
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01B9VUctQuzfAtrdpk4wDjEx
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
IFEvalScorerrebuilt the instruction verifier for the strict check and again for each of the 8 loose variants. Severalifbenchverifiers (e.g.count:word_count_range) draw random parameters for kwargs the dataset leaves unset, so the nine checks could each run against different criteria. Measured on 150 identical responses:inst_level_strict_acc)Fix: build each verifier once (
_build_instruction) and reuse it for the strict check and all loose variants.Notes for review
EndCheckermutates state incheck_following, and idempotently.ifeval_ood, live in theifbenchsuite. Zero effect onallenai/IFBench_test(kwargs fully specified); ~0.8pt ongoogle/IFEval-style prompts.Test plan
count:word_count_rangewith unset kwargs, 200 trials, asserts strict ⇒ loose. Fails on pre-fix code, passes on the fix.🤖 Generated with Claude Code
https://claude.ai/code/session_01B9VUctQuzfAtrdpk4wDjEx