Feature Request
Add three named --eval-key entries to benchmarks/m3/eval_config.toml:
consistent_pass_cap2 — 10 capability-2 tasks (one from each of 10 domains)consistent_pass_cap3 — 10 capability-3 tasks (across 8 domains)consistent_pass_combined — the union of the two (20 tasks)
Each task was selected because it passes with Groundedness = 1.00 in both of the current full 200-task --m3-data sweeps that scored the groundedness judge (evaluation_bundles/20260628_162414_default and evaluation_bundles/m3-toolcalling-20260625). Rows with 0 tokens/LLM calls in either bundle (a sign the result was cached/reused rather than freshly evaluated) were excluded from consideration.
Motivation / Problem
There's no existing "known-good" subset of M3 tasks to smoke-test against. Every existing --eval-key split (train/test, schema_bug_*) is either a random stratified split or a curated failure slice for debugging a specific regression. When iterating on the agent, there's no fast way to check "did I just break something that used to reliably work" without running the full 200-task sweep (or waiting on a much larger multi-run comparison) and manually cross-referencing report.md rows.
Use Case
As someone iterating on cuga-agent changes and periodically re-running M3 comparisons, I want a small, high-confidence "known-good" task subset so I can quickly catch regressions (in tool use, dialogue scoring, or groundedness judging) without running the full 200-task corpus every time, and without the run being dominated by tasks that are already flaky/failing for unrelated reasons.
Proposed Solution
Hand-curate the three eval-keys directly in benchmarks/m3/eval_config.toml (following the existing schema_bug_* precedent for hand-curated, non-generated entries), with a comment block documenting the selection methodology and provenance bundles. Each UUID is resolved from capability/domain/# (as printed in report.md) via M3DataLoader, and commented inline with its domain/# for readability.
Usage once merged:
./compare.sh --runs 5 --m3-data --eval-key consistent_pass_combined
Alternatives Considered
- Re-run
generate_eval_split.py with a filter for "always passing" — rejected because that script derives splits from raw sample lists, not from evaluation results, so it can't select on pass/fail or groundedness outcomes.
- Leave this as an ad-hoc list outside the repo (e.g. a scratch file) — rejected since it isn't reusable or shareable via
--eval-key.
Priority
Low — this is a convenience/tooling addition, not a bug fix or blocking issue.
Additional Context
Selection cross-referenced against two older full-200-task sweeps that predate groundedness scoring (20260603_220001_default, cuga-gpt-oss-m3-training-baseline-small-train-20260428) as a secondary confidence signal, though those aren't required for inclusion since they don't score groundedness at all.
Feature Request
Add three named
--eval-keyentries tobenchmarks/m3/eval_config.toml:consistent_pass_cap2— 10 capability-2 tasks (one from each of 10 domains)consistent_pass_cap3— 10 capability-3 tasks (across 8 domains)consistent_pass_combined— the union of the two (20 tasks)Each task was selected because it passes with Groundedness = 1.00 in both of the current full 200-task
--m3-datasweeps that scored the groundedness judge (evaluation_bundles/20260628_162414_defaultandevaluation_bundles/m3-toolcalling-20260625). Rows with 0 tokens/LLM calls in either bundle (a sign the result was cached/reused rather than freshly evaluated) were excluded from consideration.Motivation / Problem
There's no existing "known-good" subset of M3 tasks to smoke-test against. Every existing
--eval-keysplit (train/test,schema_bug_*) is either a random stratified split or a curated failure slice for debugging a specific regression. When iterating on the agent, there's no fast way to check "did I just break something that used to reliably work" without running the full 200-task sweep (or waiting on a much larger multi-run comparison) and manually cross-referencing report.md rows.Use Case
As someone iterating on
cuga-agentchanges and periodically re-running M3 comparisons, I want a small, high-confidence "known-good" task subset so I can quickly catch regressions (in tool use, dialogue scoring, or groundedness judging) without running the full 200-task corpus every time, and without the run being dominated by tasks that are already flaky/failing for unrelated reasons.Proposed Solution
Hand-curate the three eval-keys directly in
benchmarks/m3/eval_config.toml(following the existingschema_bug_*precedent for hand-curated, non-generated entries), with a comment block documenting the selection methodology and provenance bundles. Each UUID is resolved fromcapability/domain/#(as printed inreport.md) viaM3DataLoader, and commented inline with its domain/# for readability.Usage once merged:
Alternatives Considered
generate_eval_split.pywith a filter for "always passing" — rejected because that script derives splits from raw sample lists, not from evaluation results, so it can't select on pass/fail or groundedness outcomes.--eval-key.Priority
Low — this is a convenience/tooling addition, not a bug fix or blocking issue.
Additional Context
Selection cross-referenced against two older full-200-task sweeps that predate groundedness scoring (
20260603_220001_default,cuga-gpt-oss-m3-training-baseline-small-train-20260428) as a secondary confidence signal, though those aren't required for inclusion since they don't score groundedness at all.