Skip to content

Pull requests: allenai/olmo-eval

New pull request New
29 Open 259 Closed
29 Open 259 Closed
Author
Filter by author
Loading
Label
Filter by label
Loading
Use alt + click/return to exclude labels
or ⇧ + click/return for logical OR
Projects
Filter by project
Loading
Milestones
Filter by milestone
Loading
Reviews
Filter by reviews
No reviews Review required Approved review Changes requested
Assignee
Filter by who’s assigned
Assigned to nobody Loading
Sort
Sort by
Newest Oldest Most commented Least commented Recently updated Least recently updated Best match
Most reactions
👍 👎 😄 🎉 😕 ❤️ 🚀 👀

Pull requests list

Fix judge-task crash at results aggregation (judge_fn not JSON serializable)
#305 opened Aug 19, 2026 by jopetty Contributor Loading…
Fix ladder dataset resolution and vLLM runtime installs
#303 opened Aug 19, 2026 by AkshitaB Contributor Loading…
1
Fail fast when provider has a terminal failure
#300 opened Aug 17, 2026 by undfined Collaborator Loading…
Add single-turn IFEval task
#297 opened Aug 17, 2026 by abhishekraok • Draft
4 of 6 tasks
Fix IFEval loose scoring: build each verifier once, not per variant
#296 opened Aug 17, 2026 by abhishekraok Loading…
2
Adding CTC Suite (upcoming release) to olmo eval
#295 opened Aug 17, 2026 by PrasannS • Draft
Fix Hugging Face stop-sequence decoding
#294 opened Aug 16, 2026 by sylvesterkaczmarek Loading…
1
Add FIRMCore evaluation task implementation
#292 opened Aug 15, 2026 by pardiszahraei Loading…
6 tasks done
feat: support prompt truncation across inference providers
#291 opened Aug 14, 2026 by sylvesterkaczmarek Loading…
6 of 13 tasks
1
feat: add weighted suite aggregation
#290 opened Aug 14, 2026 by sylvesterkaczmarek Loading…
6 of 13 tasks
1
Prasann/ctc suite
#289 opened Aug 14, 2026 by PrasannS Loading…
Add HELMET task suite: all seven categories, recall extended to 2m tokens
#281 opened Aug 14, 2026 by jopetty Contributor Loading…
2
Add multimodal evaluation enhancement New feature or request
#278 opened Aug 11, 2026 by jason718 Loading…
11 tasks done
@jason718 @uwGZQ
1
Add ruler-plus extending RULER evals to 2m tokens
#276 opened Aug 10, 2026 by jopetty Contributor Loading…
7
[Draft] Add StrongReject
#275 opened Aug 6, 2026 by mgmorgan23 Contributor Loading…
1 of 13 tasks
Add FrontierScience benchmark
#267 opened Jul 29, 2026 by donovanr Contributor Loading…
8 of 13 tasks
1
7
add support for multi-agent systems
#257 opened Jul 16, 2026 by aetting • Draft
13 tasks
Deepscholar-Bench
#240 opened Jul 2, 2026 by donovanr Contributor Loading…
1 of 5 tasks
7
Use encrypted Modal sandbox tunnels
#237 opened Jul 1, 2026 by abhinav-bellapu Loading…
8 of 13 tasks
1
Fix vLLM harness provider kind precedence
#236 opened Jul 1, 2026 by abhinav-bellapu Loading…
6 of 13 tasks
1
Fix Hugging Face dataset paths and worker startup diagnostics
#235 opened Jun 30, 2026 by TianhuaTao • Draft
Add general:posttrain:dev suite
#193 opened May 26, 2026 by finbarrtimbers Contributor Loading…
2 tasks done
2
Report cumulative vLLM generation progress across batches
#179 opened May 15, 2026 by finbarrtimbers Contributor Loading…
1 task
Bake SciCode sandbox image once; pre-build from launch host
#159 opened Apr 27, 2026 by finbarrtimbers Contributor • Draft
3
Finbarr/prebuild sandbox
#158 opened Apr 27, 2026 by finbarrtimbers Contributor • Draft
13 tasks
ProTip! Type g i on any issue or pull request to go back to the issue listing page.