Generalizing Verifiable Instruction Following

This repo contains IFBench, which is a new, challenging benchmark for precise instruction following. Read the IFBench paper, accepted to NeurIPS 2025, D&B.

IFBench

IFBench consists of two parts:

• OOD Constraints: 58 new and challenging constraints, with corresponding verification functions. The constraint templates are combined with prompts from a held-out set of WildChat (Zhao et al. 2024).

• (optionally) Multiturn Constraint Isolation in 2 turns: The prompt and the constraint are separated over two turns, i.e. the first turn is the user prompt and the model's response to the prompt, and the second turn is the constraint that modifies the initial prompt.

• New IF-RLVR training constraints: 29 new and challenging constraints, with corresponding verification functions.

Installation

IFBench is pip-installable. The package ships all OOD verifiers as well as the classic Google IFEval verifiers, so it can serve as a single source of truth for both registries.

pip install ifbench
# or
uv add ifbench

To work on IFBench itself, clone and sync with uv:

git clone https://github.com/allenai/IFBench.git
cd IFBench
uv sync

The package is namespaced under ifbench — submodules are ifbench.instructions, ifbench.classic_instructions, ifbench.instructions_registry, and ifbench.instructions_util. The test jsonl is bundled inside the wheel; access it via ifbench.data_path().

Programmatic use

from ifbench import instructions_registry

checker_cls = instructions_registry.INSTRUCTION_DICT["keywords:existence"]
checker = checker_cls("keywords:existence")
checker.build_description(keywords=["cat", "dog"])
checker.check_following("I saw a cat and a dog today.")  # True

INSTRUCTION_DICT contains 83 verifiers in total: 25 classic IFEval keys (prefixed keywords:, language:, length_constraints:, detectable_content:, detectable_format:, combination:, startend:, change_case:, punctuation:) plus 58 IFBench OOD keys.

How to run the evaluation

You need two jsonl files, one the IFBench_test.jsonl file (in the data folder) and one your file with eval prompts and completions (see sample_output.jsonl as an example). Then run:

uv run python -m run_eval --input_data=IFBench_test.jsonl --input_response_data=sample_output.jsonl --output_dir=eval

Note: In the paper we generally report the prompt-level loose accuracy of IFBench. When we generate for evaluation, we use a temperature of 0 and adjust the maximum generated tokens depending on the model type, i.e. for thinking models we allow to generate more tokens and we then process the output to extract the answer without the reasoning chains.

Released Datasets

You can find our released datasets in this collection, which contains the test data, the multi-turn test data and the IF-RLVR training data.

RLVR for Precise Instruction Following

We also release our IF-RLVR code, as part of open-instruct. You can run this GRPO script, using our training data. This is an example command.

The new training constraints and verification functions are here: https://github.com/allenai/open-instruct/tree/main/open_instruct/IFEvalG

Licensing

This codebase is licensed under Apache 2.0 as given in LICENSE.

The data is licensed under ODC-BY-1.0. It is intended for research and educational use in accordance with Ai2's Responsible Use Guidelines. The dataset includes output data generated from third party models that are subject to separate terms governing their use.

Acknowledgements

Parts of IFBench are built upon and extend IFEval (Zhou et al. 2023) and we would like to thank them for their great work!

Citation

If you used this repository or our models, please cite our work:

@misc{pyatkin2025generalizing,
   title={Generalizing Verifiable Instruction Following}, 
   author={Valentina Pyatkin and Saumya Malik and Victoria Graf and Hamish Ivison and Shengyi Huang and Pradeep Dasigi and Nathan Lambert and Hannaneh Hajishirzi},
   year={2025},
  journal={Advances in Neural Information Processing Systems},
  volume={38},
  year={2025}
}