Skip to content

feat(m3): add curated consistent_pass eval-key subsets (#102)#104

Open
haroldship wants to merge 6 commits into
mainfrom
feat/m3-eval-key-consistent-pass
Open

feat(m3): add curated consistent_pass eval-key subsets (#102)#104
haroldship wants to merge 6 commits into
mainfrom
feat/m3-eval-key-consistent-pass

Conversation

@haroldship

@haroldship haroldship commented Jul 6, 2026 •
edited by coderabbitai Bot
Loading

Copy link
Copy Markdown
Collaborator

Feature Pull Request

Related Issue

Closes #102

Description

Adds curated --eval-key subsets for M3 capability 2/3 tasks that reliably pass across repeated runs, plus _all variants for full coverage.

Three commits:

  • Initial consistent_pass eval-key ladder from 3-run bundle analysis
  • Refresh keys against updated 3-run bundle data
  • Add _all variants for each consistent_pass tier

Only benchmarks/m3/eval_config.toml is changed (+119 lines).

Type of Changes

  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to not work as expected)

Testing

  • I have tested this feature locally
  • I have added tests that prove my feature works
  • All new and existing tests passed (321 passed, 2 skipped via pytest -m "sanity or regression")

Documentation

  • I have updated the documentation accordingly
  • I have added docstrings to new functions/classes

Checklist

  • My code follows the code style of this project
  • I have performed a self-review of my own code
  • I have made corresponding changes to the documentation

Made with Cursor

Summary by CodeRabbit

  • New Features
    • Added new M3 evaluation presets for “consistent passing” selection.
    • Expanded available task sets with capability-capped and combined variants, including “all” listings for every qualifying UUID in the reference bundle.

haroldship added 3 commits July 2, 2026 18:15
@haroldship
feat(m3): add curated eval-key subsets for reliably-passing capabilit…
409df45 
…y 2/3 tasks

- consistent_pass_cap2: 10 capability-2 tasks (one per domain, 10 domains)
- consistent_pass_cap3: 10 capability-3 tasks (across 8 domains)
- consistent_pass_combined: union of the two, for quick regression smoke-tests

Each task was selected because it passes with Groundedness = 1.00 in both full
200-task --m3-data sweeps that scored the groundedness judge
(evaluation_bundles/20260628_162414_default and
evaluation_bundles/m3-toolcalling-20260625). Rows with 0 tokens/LLM calls in
either bundle (cached/reused results) were excluded from consideration.

Fixes #102
@haroldship
fix(m3): refresh consistent_pass eval-keys against 3-run bundle data
c2f5b50 
- Rebuild consistent_pass_cap2/cap3/combined from evaluation_bundles/20260705_031554_compare_gpt-oss_no-policies (200 tasks x 3 runs), replacing the prior groundedness-based picks, several of which no longer pass even 2/3 against this data.
- cap2: one task per domain (10 domains), preferring 3/3, falling back to the cheapest/fastest 2/3 for professional_basketball and soccer_2016, which had no 3/3 task.
- cap3: capped to the 6 domains that reached >=2/3 (beer_factory, books, disney, mondial_geo, soccer_2016, world_development_indicators); computer_student, trains, and university top out at 1/3 and college_completion never passed once, so they're excluded rather than padded in with unreliable picks.
@haroldship
feat(m3): add _all variants of consistent_pass eval-keys
8bf1ec9 
Add consistent_pass_cap2_all/cap3_all/combined_all: every task that reached
>=2/3 in the 20260705_031554_compare_gpt-oss_no-policies bundle per domain,
not just the cheapest one-per-domain pick in consistent_pass_cap2/cap3.
cap3_all is identical to consistent_pass_cap3 since no cap3 domain had extra
candidates beyond what's already curated there.
@coderabbitai

coderabbitai Bot commented Jul 6, 2026 •
edited
Loading

Copy link
Copy Markdown

Review Change Stack

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: 56773bfc-8d33-4fee-963b-de47241bc64f

📥 Commits

Reviewing files that changed from the base of the PR and between a8699a3 and fef3261.

📒 Files selected for processing (1)
  • benchmarks/m3/eval_config.toml
📝 Walkthrough

Walkthrough

Adds six new hand-curated --eval-key entries to benchmarks/m3/eval_config.toml: consistent_pass_cap2, consistent_pass_cap3, consistent_pass_combined, and their _all variants, enumerating task UUIDs selected from a 20260705 comparison bundle.

Changes

Curated eval-key subsets

Layer / File(s) Summary
Consistent pass eval-key subsets
benchmarks/m3/eval_config.toml		 Adds new eval-key arrays (consistent_pass_cap2, consistent_pass_cap3, consistent_pass_combined, and their _all variants) with comments describing the selection basis and the enumerated UUIDs for each list.

Estimated code review effort: 1 (Trivial) | ~5 minutes

Possibly related PRs

  • cuga-project/cuga-eval#76: Both PRs extend --eval-key support in benchmarks/m3/eval_config.toml by adding new named eval-key task UUID lists used by the eval-key filtering logic.

Suggested reviewers: sami-marreed, Sergey-Zeltyn

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Linked Issues check ⚠️ Warning The PR is related, but consistent_pass_cap3 and the selection provenance do not match the issue's required 10-task, 8-domain groundedness-based set. Rebuild the three eval-keys from the specified 20260628 and 20260625 sweeps, exclude zero-token/LLM-call rows, and add the requested provenance note.
✅ Passed checks (4 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly summarizes the main change: adding curated consistent_pass eval-key subsets for M3.
Out of Scope Changes check ✅ Passed The extra _all variants are closely related to the requested M3 subsets and remain within the same config scope.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch feat/m3-eval-key-consistent-pass

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@haroldship haroldship requested a review from AnkitaNaik July 6, 2026 17:59
@haroldship

haroldship commented Jul 8, 2026

Copy link
Copy Markdown
Collaborator Author

@CodeRabbit full review

@coderabbitai

coderabbitai Bot commented Jul 8, 2026 •
edited
Loading

Copy link
Copy Markdown
✅ Action performed

Full review finished.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@AnkitaNaik AnkitaNaik Awaiting requested review from AnkitaNaik

At least 1 approving review is required to merge this pull request.

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

[Feature]: curated --eval-key subsets for reliably-passing M3 capability 2/3 tasks

1 participant

@haroldship