Add HELMET task suite: all seven categories, recall extended to 2m tokens - #281
Open
jopetty wants to merge 17 commits into
Open
Add HELMET task suite: all seven categories, recall extended to 2m tokens#281jopetty wants to merge 17 commits into
jopetty wants to merge 17 commits into
Conversation
Registers helmet_json_kv__{262144,524288,1048576,2097152} tasks (and
helmet_recall__*/helmet_all__* suites) that read the ai2-internal
allenai/helmet-plus dataset off the Hub, extending HELMET's json_kv
recall task past its standard 128k ceiling. Follows the same
task/suite registration pattern already used for RULER.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
helmet-plus now covers the original 4k/8k/16k/32k/64k/128k tiers in addition to the 256k/512k/1m/2m extension, making it a strict superset of standard HELMET's json_kv lengths. Task/suite generation already looped generically over CONTEXT_SIZES, so this only required adding the six shorter tiers to LENGTH_NAMES. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Carries HELMET's five ICL tasks (trec_coarse, trec_fine, banking77, clinic150, nlu) into olmo-eval, keeping them HELMET-namespaced rather than factoring them out: the length-controlled construction (shot count per length tier, shuffled integer labels) is HELMET's, not a property of the underlying datasets, so they aren't interchangeable with standalone tasks. Three of the five source datasets no longer load: datasets 4.0 dropped script-based loading, which HELMET relies on via trust_remote_code. trec and nlu_evaluation_data resolve to the Hub's auto-converted parquet branch, and banking77 to legacy-datasets/banking77, since PolyAI/banking77 hosts only the script. Split sizes and label counts verified against HELMET's hardcoded values (6/50/77/151/68). Also: - helmet_tasks.py now takes per-task context_sizes, since coverage is ragged (json_kv 4k-2m, ICL 4k-128k only). - helmet_all__* aggregates category suites via AVERAGE_OF_AVERAGES instead of flat-averaging tasks, so recall isn't outweighed 5:1 by ICL's task count, and is only registered where >1 category exists -- above 128k it would silently be a rename of helmet_recall__* whose composition differs from the same suite at shorter lengths. Verified: demo labels match each instance's own permutation and the answer field agrees (20/20); output is deterministic per seed; the answer parser is byte-identical to HELMET's parse_output across 11 cases; all five datasets produce well-formed prompts. ruff, ty, and 820 existing tests pass.
HELMET fit its ICL shot counts against a Llama-2-era tokenizer, so under Olmo 3's larger vocabulary the rendered prompts land at 0.78-0.85x their nominal length: icl_*__4096 is really ~3.2-3.5k tokens and __131072 is ~104-108k. Keeps HELMET's counts verbatim so our ICL numbers stay directly comparable to published HELMET results, and records the measured shortfall in-file so it is visible when reading a length sweep rather than being a silent property of the tasks. Note this leaves ICL inconsistent with json_kv, which is calibrated against Olmo 3 and does hit its nominal lengths -- a helmet_all__4096 average therefore mixes a 4.1k-token task with ~3.3k-token ones. Adds scripts/internal/calibrate_helmet_icl_shots.py to measure that shortfall for any tokenizer. It also prints Olmo-3-calibrated counts, but is explicitly documented as a diagnostic rather than the source of the committed values, and labels that output as a deliberate trade so it doesn't get pasted in by accident.
Adds the rougeL_f1 metric HELMET uses as the primary score for infbench_qa_eng, plus rougeL_recall (HELMET's metric for qmsum-style tasks), neither of which olmo-eval had. Backed by Google Research's rouge_score with use_stemmer=True, matching HELMET so numbers stay comparable to published results -- hand-rolling ROUGE-L is easy to get subtly wrong, since tokenization and stemming both move the score. Verified byte-identical to HELMET's calculate_metrics across five cases including multi-reference, no-overlap, and empty-prediction. Scoring takes the max over all reference answers, as HELMET does. Reads either multi-reference metadata convention already present in this repo (all_answers, used by the SQuAD scorer; all_gold_answers, used by RULER/HELMET tasks), falling back to a single gold_answer. rouge-score adds cleanly: the lockfile diff is the one new package, no version changes elsewhere, and all 820 existing tests pass.
Registers helmet_infbench_qa_eng__{4096..131072}, scored with the rougeL_f1
metric added in the previous commit -- HELMET's primary metric for this task.
Capped at standard HELMET's lengths since the context is a real book and
can't be stretched the way json_kv's synthetic context can.
Contexts are truncated with a fixed reference tokenizer (Llama 2, via an
ungated mirror so the repo needs no gated-model access) rather than the
tokenizer of the model under test. That is what makes every model see the
same text, and it keeps results comparable to published HELMET -- the same
trade already taken for the ICL shot counts.
Two deviations from a literal port, both verified safe:
- HELMET's >=65536-token filter is skipped. It is a no-op (the shortest
InfiniteBench story is ~78k reference tokens, and HELMET itself calls it
"just a sanity step"), which lets us sample first and truncate only the
rows we keep instead of tokenizing all 351 book-length contexts.
- Truncation is memoized per distinct context, since 351 questions are drawn
from just 69 stories.
Demos are still drawn from the full pool, not the sampled subset, so they
don't depend on how many instances are evaluated.
Also adds use_chat_template handling to the shared HelmetTask: HELMET runs
LongQA in chat format, where the "Answer:" prefix is not fed in as a partial
assistant turn. json_kv and ICL are unaffected and keep their prefixes.
Verified: truncation lands exactly on budget (3886/3886 at 4k, 65326/65326 at
64k), the rendered 4k prompt is 4002 reference tokens, the truncation notice
is present, gold answers reach the scorer as a list, and ROUGE gives 1.0 on
exact gold with sensible partial credit. ruff, ty, and 820 tests pass.
Registers helmet_infbench_choice_eng__{4096..131072}: same book-length story
as infbench_qa_eng, but with four options and a single-letter answer, scored
by exact match.
Reuses the InfiniteBench loader and task class, with a per-task metrics_key
so subsets sharing a loader can score differently (ROUGE for qa_eng, exact
match here) instead of both being pinned to the task kind.
Adds InfbenchChoiceScorer, which ports HELMET's choice_post_process: a
response counts as correct if the raw generation, or its "Answer:"-stripped
form, normalizes to either the bare letter or "letter. option text", or if
the latter appears anywhere in the generation -- the last rule being what
credits a model that answers in a sentence. It reads output.text rather than
only extracted_answer because that rule is defined against the untouched
generation. Verified to agree with HELMET on all 11 cases tried, including
bare/lowercase/trailing-period letters, prefixed and embedded answers, wrong
answers, and empty output.
Note the rendered prompt can run slightly over nominal at the smallest tier
(~4109 vs 4096 tokens at 4k), because the options block eats into HELMET's
fixed 200-token prompt reserve. This is faithful -- HELMET budgets both
subsets identically at size - 200 - 10 -- and is 0.3% at 4k, less above.
HELMET's LongQA category is now complete except narrativeqa, whose primary
metric is LLM-judged. ruff, ty, and 820 tests pass.
undfined
approved these changes
Aug 14, 2026
undfined
left a comment
undfined
left a comment
Collaborator
There was a problem hiding this comment.
Two requests but looks good!
- Do we want to pass a fixed revision?
- The loader decodes and processes the entire JSONL file before applying max_samples similar to RULER-plus case. We should try to stream and sample this instead of loading the entire dataset and sampling with max_samples.
Registers helmet_narrativeqa__{4096..131072}, HELMET's third LongQA task and
the first here graded by an LLM judge rather than string overlap.
Ports HELMET's LongQA rubric (scripts/eval_gpt4_longqa.py) onto olmo-eval's
existing LLMJudgeScorer, keeping upstream's judge model and temperature
(gpt-4o-2024-05-13 @ 0.1) so scores stay comparable. HELMET's gpt-4-score is
fluency (0/1) x correctness (0-3); that product is divided by 3 here so the
scorer honors the [0,1] contract its base class documents and doesn't
outweigh proportion-valued metrics in a suite average -- multiply a reported
score by 3 to recover HELMET's number. Verified the parse+score path agrees
with HELMET across 7 responses including trailing-JSON, multiple JSON
objects, missing keys, and malformed output.
The judge is given the bare question via new `judge_question` metadata, not
Instance.question -- the latter is the fully rendered prompt, so using it
would ship a book-length context to the judge on every call. Verified the
built judge prompt is ~2.2k chars and contains no story text.
Loading narrativeqa is the expensive part: HELMET keeps only documents over
131072 reference tokens, which it establishes by tokenizing all 10.5k rows of
the test split. Two shortcuts avoid that without changing which instances are
selected -- an exact character-count lower bound (a document cannot have more
tokens than characters), and stopping once enough rows are found, which is
equivalent because HELMET shuffles before filtering and then takes a prefix.
Loading 3 instances takes ~11s rather than tokenizing the whole split.
Judged tasks change how suites should be run, so following the split in
suites/science.py: per-category suites and helmet_all__* now include
narrativeqa and therefore need a judge configured, while new
helmet_nojudge__* suites are the subset that runs without one.
HELMET scores msmarco_rerank_psg with NDCG@10, which olmo-eval had no metric for -- it had no ranking metrics at all. This adds the metric ahead of the task itself, which additionally needs its pre-computed retrieval data hosted before it can run. Reimplements the formula rather than taking on pytrec_eval, a compiled C extension, as a permanent dependency. Determined trec_eval's exact convention empirically instead of assuming it -- linear gain (rel / log2(rank+1)), not the exponential 2^rel-1 variant -- then verified agreement with pytrec_eval across 900 randomized comparisons spanning partial rankings, over-long rankings containing unjudged ids, and all-zero relevance: zero mismatches. The ideal ranking is drawn from every judged document rather than only those the model returned, so omitting a relevant document costs score instead of being silently ignored. Accepts relevance judgements as a mapping or as HELMET's [[doc_id, label], ...] pair list.
Registers helmet_kilt_{nq,triviaqa,hotpotqa,popqa}__{4096..131072}: an
open-domain question plus a stack of retrieved Wikipedia passages. Data comes
from allenai/helmet-plus, whose kilt/manifest.json maps each length tier to
its pre-retrieved file, so no filenames are hardcoded. Capped at standard
HELMET's lengths since retrieval depth is baked into the files.
Adds SubstringExactMatchScorer, which HELMET's RAG metric needs and olmo-eval
lacked. The existing SubstringRecallScorer returns the *fraction* of gold
answers found, which is wrong here: these gold answers are aliases for one
another, so a model answering Natural Questions correctly with one surface
form would score 1/N rather than 1. The new scorer takes the max and applies
HELMET's normalization (lowercase, strip punctuation, drop articles, collapse
whitespace); verified identical to HELMET across 7 cases, where the recall
scorer would have given 0.33-0.67 for fully correct answers.
Two behaviors worth knowing, both faithful to HELMET:
- `limit` caps *questions*, not instances. Each question appears once per
gold-passage depth -- 6 for nq/triviaqa/popqa (the dep6 in the filenames:
0.0, 0.2, 0.4, 0.6, 0.8, 0.95) and 3 for hotpotqa -- so scores average over
where in the context the answer sits, which is the "lost in the middle"
effect these tasks measure. A limit of 100 therefore yields ~600 nq
instances. Documented in the loader; confirmed 6/3/6 empirically.
- PopQA is restricted to long-tail entities via log10(s_pop) < 3, encoded in
HELMET's task name as kilt_popqa_3. Verified this removes 50% of rows
rather than being a no-op.
RAG joins the suite as its own category, so helmet_all__4096 now spans 13
tasks across recall, rag, longqa and icl. ruff, ty, and 820 tests pass.
Registers helmet_msmarco_rerank_psg__{4096..131072}: a query plus ID-tagged
candidate passages the model must return in relevance order, scored with the
NDCG@10 metric added earlier. Data comes from allenai/helmet-plus via
msmarco/manifest.json; capped at standard HELMET's lengths since candidate
depth is fixed in the pre-retrieved files.
Ports HELMET's parse_rankings. Upstream returns a dict of id -> descending
score for pytrec_eval; this returns the equivalent ranked list, which is what
NDCGScorer consumes. Verified the two agree on 7 cases including bracketed
IDs, duplicate ids (first position wins), no-separator output, and
unparseable text.
Relevance judgements now flow from the loader into instance metadata as
`qrel`, which is where NDCGScorer reads them.
Sanity-checked scoring on real data: a gold-order ranking scores 1.0, its
reverse 0.26, and unparseable output 0.0.
Note re-ranking multiplies instances the same way RAG does -- these files
carry three gold-position variants per query, so a limit of N yields ~3N
instances.
HELMET support now covers 5 of 6 categories (88 tasks): recall, rag, rerank,
longqa, icl. Remaining are the two summarization tasks and ALCE.
Registers helmet_{infbench_sum_eng,multi_lexsum}__{4096..131072}, completing
HELMET's six task categories: 100 tasks now span recall, rag, rerank, longqa,
summ and icl.
Adds HelmetSummJudgeScorer, which unlike the LongQA judge needs three calls
per summary -- fluency, key-point recall, and sentence-level precision --
combined into HELMET's gpt-4-f1 as
fluency * 2 * rec * prec / (rec + prec). Fluency multiplies rather than
averages, so a disfluent summary scores zero however much it covers; that is
deliberate upstream, since degenerate repetition otherwise scores well on
recall. Verified the combination matches HELMET on 5 cases including the
fluency gate and both zero-denominator paths, and that missing or malformed
judge output scores 0.0 rather than raising.
The six rubrics (~55KB, each with worked examples) were extracted
programmatically from eval_gpt4_summ.py rather than retyped, and verified to
round-trip identically -- a paraphrase would silently shift scores. Book
variants grade novels, plain variants grade civil lawsuits.
multi_lexsum reads from allenai/helmet-plus instead of calling load_dataset,
which no longer works: datasets 4.0 dropped script-based loading and the Hub's
parquet conversion has no v20230518 config. The hosted multi_lexsum_val.jsonl
turned out to be the full validation split -- sources, both summary lengths,
and the pre-extracted key points -- so bypassing the broken path loses
nothing and pins the data besides.
Two details that differ from the other long-context tasks: multi_lexsum
reserves 300 tokens of each tier for prompt and buffer where the rest reserve
200, and the precision rubric grades against the expert *long* summary rather
than the short one the model is asked to match. Both follow upstream.
Verified no source text leaks into judge prompts, that the correct rubric
variant is selected per task, and that key points and expert summaries reach
the scorer through instance metadata. All three summarization tasks are
judge-gated, so they appear in helmet_all__* but not helmet_nojudge__*.
ruff, ty, and 820 tests pass.
Registers helmet_alce_{asqa,qampari,asqa_nocite,qampari_nocite}__{4096..131072}
-- 124 HELMET tasks now, covering all seven of HELMET's reported categories.
Scores ALCE's answer-correctness half only:
- str_em (ASQA): the fraction of a question's disambiguated sub-questions the
response answers, so partial coverage earns partial credit.
- qampari_rec_top5 (QAMPARI): recall over an answer set, capped at five so
questions with long answer lists don't dominate the average.
Both verified identical to ALCE's eval_alce.py across 6 cases, including
generations carrying inline citations.
Deliberately NOT scored: citation_rec / citation_prec, which measure whether
each claim is supported by the sources it cites. Those need AutoAIS, an 11B
NLI model, wired up as an auxiliary provider -- a separate change. The tasks
are useful without it, just silent on grounding, which is arguably ALCE's more
interesting half.
Scoring runs on a normalized generation (newlines collapsed, chat end-marker
dropped, inline [1] markers stripped) so a well-cited answer isn't penalized
for carrying its citations, matching upstream's preprocessing.
Unlike the RAG tasks, every ALCE tier reads one file -- a fixed 2000-document
pool per question -- and shows a prefix of it, so length is set by how many
documents are displayed. The nocite variants run zero-shot with a larger
generation budget, per HELMET's configs; verified they emit no worked
examples while still showing the full document set.
ruff, ty, and 820 tests pass.
The provider hardcoded AutoModelForCausalLM, so olmo-eval could not run any encoder-decoder model. This blocks ALCE's citation metrics, which score with AutoAIS (google/t5_xxl_true_nli_mixture, a T5) -- and vLLM is not an alternative route: its 0.19.1 model registry has no T5 entry and no true encoder-decoder support, only multimodal *ForConditionalGeneration models whose decoders are still causal. Detects encoder-decoder models from their config rather than adding a config flag, so nothing needs declaring at the call site and existing causal models are unaffected. Three touch points: - pick AutoModelForSeq2SeqLM when config.is_encoder_decoder - skip the prompt slice in generate(): a causal LM returns prompt + continuation, but an encoder-decoder returns only decoder tokens, so output_ids[prompt_len:] would eat the answer - skip the logprob path for encoder-decoder, since it reads logits at prompt-relative offsets of one concatenated sequence, which only holds for a causal LM. AutoAIS needs no logprobs; encoder-decoder support there would require a decoder-side forward pass. Verified auto-detection returns True for google/t5_xxl_true_nli_mixture and t5-small, and False for Olmo 3 and gpt2, so the causal path is untouched. Caveat: the generate path itself is NOT exercised locally -- torch is Linux/CUDA-gated and absent from the macOS dev environment -- so the seq2seq branch needs a smoke test on a GPU node before AutoAIS is wired up.
The previous commit widened the transformers surface the provider imports (AutoConfig, AutoModelForSeq2SeqLM), which broke test_huggingface_provider_passes_force_download_to_tokenizer_and_model -- it stubs sys.modules["transformers"] with only the two names the old code used. I pushed that commit before reading the test output; this repairs it. - extends the stub with the new names, and asserts a causal model is *not* routed through the seq2seq class - adds a test for the new branch: an encoder-decoder config must load via AutoModelForSeq2SeqLM and must not touch the causal class - makes the provider's log line use getattr(auto_class, "__name__", ...); a log statement should not raise, and it did against a test double Full suite: 1996 passed, 26 skipped.
…s, tests Findings from a full review of the 13-commit HELMET integration, fixed here. Scoring fidelity (would have shifted reported numbers): - ICL exact match now uses HELMET's normalization. The generic ExactMatchScorer only lowercases and strips, but HELMET's exact_match (drqa) also removes punctuation and articles -- so "label: 42." or a quoted answer scored 0 here and 1 under HELMET. Realistic outputs, since stop-at-newline sampling leaves trailing punctuation intact. Added HelmetExactMatchScorer and wired it into the ICL metric; divergence demonstrated against upstream's normalization before fixing. - ROUGE scorers now take the max over the raw and extracted generations, matching HELMET's default_post_process. Scoring only the extracted answer under-credited any multi-line generation whose answer-prefix parse picked the wrong line. Robustness: - HelmetLongQAJudgeScorer inherited the base ascore_with_context, whose provider path defaults to max_tokens=10 -- sized for letter-grade judges. This rubric asks for step-by-step reasoning before a JSON verdict, so any provider-wired run would truncate every response and silently score zero. Overridden to pass the judge budget, mirroring the summ scorer. - The KILT loader now samples by streaming over the file instead of parsing all of it: the 128k tiers are 1-3GB of JSONL that expand severalfold as dicts (upstream avoids this via arrow memory-mapping). Selection verified byte-identical to the old path on real nq and popqa files across caps of 5/50/None, including the popularity-filter interaction. Tests: the branch's verification lived in throwaway shell commands; none of it survived as regression protection. Added 44 offline tests covering the ranking/label parsers (HELMET reference cases), NDCG against pytrec_eval values, all six new scorers including the fluency gate and malformed-judge paths, the summ rubric brace-escaping, ICL demo balancing, the streaming sampler (fixture file), and the task/suite inventory -- 124 tasks, category counts, context budgets, judged-task exclusions, and the absence of helmet_all above 128k. Also refreshed module docstrings that still described the branch as json_kv-plus-ICL with narrativeqa the only judged task. Reviewed but left alone: scorer-instance handling (Scorer.__call__ returns self, so metrics accept instances -- suspected bug, disproven), the choice scorer's embedded-answer rule, popqa key handling, and the ALCE preprocessing chain, all verified against upstream during review. Full suite: 2040 passed, 26 skipped.
jopetty
changed the title
Contributor
Author
|
Heads up for anyone who reviewed earlier: this PR just grew from the recall-only extension (2 commits) to the full HELMET integration (16 commits) — the original recall commits are unchanged at the base of the branch. Title and description updated to match. The same content is also flowing to |
A demo run of the judged HELMET tasks crashed at results aggregation -- after all 1800 instances were scored and every judge call paid for -- with "Object of type function is not JSON serializable". The chain: TaskConfig.to_dict() -> Metric.to_dict() -> Scorer.to_dict(), whose dataclasses.asdict includes judge_fn, the live async closure built by build_openai_judge_fn. compute_task_hash then json.dumps's that dict. Class-valued scorers serialize as __name__ (why every non-judge task was fine), and the wildguard judge dodges it because provider_name nulls judge_fn in __post_init__ -- the HELMET judges are just the first judge_fn-configured scorers to reach aggregation. SimpleQAJudgeScorer had the same latent bug. Fixed with a to_dict override on LLMJudgeScorer that replaces the closure with a stable "<configured>" marker. That is also more correct for hashing: a function repr embeds a memory address, so even a serializable form would have made task hashes differ run to run. No deserialization path needs the function back (verified: no from_dict in scorers/metrics). Regression tests: every registered helmet task's config now round-trips through json.dumps (instantiation touches no data, so the sweep is cheap), and judge scorer serialization is asserted stable across independent constructions. Reproduced the crash via the real config path before fixing; verified the same path serializes after.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds support for HELMET (ICLR 2025), the long-context benchmark: 124 tasks across all seven categories, with the synthetic recall task regenerated as a strict superset of upstream's 4k–128k tiers plus 256k/512k/1m/2m extensions calibrated against the Olmo 3 tokenizer.
json_kvkilt_{nq,triviaqa,hotpotqa,popqa}msmarco_rerank_psginfbench_{qa,choice}_eng,narrativeqainfbench_sum_eng,multi_lexsumtrec_{coarse,fine},banking77,clinic150,nlualce_{asqa,qampari}(+_nocite)Only the synthetic recall task extends past 128k — every other category is capped by real documents, fixed retrieval depth, or finite demonstration pools. Suites:
helmet_{category}__{size},helmet_all__{size}(mean over category averages, registered only where >1 category exists), andhelmet_nojudge__{size}(excludes the three LLM-judged tasks, following thescience:judge/science:nojudgesplit).Data:
allenai/helmet-pluson the Hub hosts the regenerated json_kv tiers, HELMET's pre-retrieved KILT/MS MARCO/ALCE files (re-hosted unpacked from its 10GBdata.tar.gzso consumers fetch per-task files), the summarization keypoints, and per-directory manifests so loaders never hardcode filenames. LongQA/ICL sources load from the Hub directly; three ICL datasets are repointed to parquet mirrors becausedatasets≥4 dropped script loading (split sizes and label counts verified against HELMET's hardcoded values).Metric infrastructure added: ROUGE-L F1/recall (byte-identical to HELMET's
calculate_metrics), NDCG@10 (verified against pytrec_eval on 900 randomized cases without taking the compiled dep), HELMET's LongQA and 3-call summarization judges (rubrics extracted verbatim, ~55KB), substring/HELMET-normalized exact-match scorers, ALCE answer-correctness scorers, and encoder-decoder support in the HF provider (auto-detected; groundwork for AutoAIS citation scoring, which is a deliberate follow-up — ALCE here scores answer correctness only).Fidelity: scorers and parsers are verified against upstream implementations case-by-case (see the review commit and
tests/evals/tasks/test_helmet.py, 44 offline tests). Known, documented deviations: ICL shot counts are HELMET's own and render at 0.78–0.85× nominal under Olmo 3 (kept for comparability; measurable viascripts/internal/calibrate_helmet_icl_shots.py); the judged tasks'gpt-4-scoreis normalized to [0,1] (×3 recovers HELMET's number).Caveats for reviewers: RAG/rerank
limitcaps questions, not instances — each question repeats per gold-passage depth (HELMET's "lost in the middle" sweep), solimit=100⇒ ~600 nq instances. The HF provider's seq2seq generate path needs a GPU smoke test (torch is CUDA-gated off the dev machine).Full suite: 2029 passed, 26 skipped.
🤖 Generated with Claude Code