Skip to content

feat(experimentalist): hold out half the Insight suite for independent scoring - #1086

Draft
aleckhoury wants to merge 3 commits into
mainfrom
insight-suite-train-validation-split/akhoury
Draft

feat(experimentalist): hold out half the Insight suite for independent scoring#1086
aleckhoury wants to merge 3 commits into
mainfrom
insight-suite-train-validation-split/akhoury

Conversation

@aleckhoury

Copy link
Copy Markdown
Contributor

Summary

The Eval Author produced a single Insight suite that served as both the optimizer's development feedback and its scoring evidence, so nothing measured whether an agent generalized to production failures it had not already been tuned against. This splits the suite into held-out halves and wires the validation half into selection.

  • Split and holdout. The finalized suite is halved (odd task to train) and each half is materialized with its own content provenance, so a candidate records which suite it was scored against. The validation half is hidden via the existing path-based holdout, so the coder cannot read it.
  • Analyzer gets the train half. Insight-train trials reach trial-level diagnosis alongside train trials, so Insight failures get trace-level root-causing.
  • Validation half enters Pareto selection as insight/-prefixed dimensions, with union-based zero-fill so a candidate missing an Insight score isn't silently undominated. Survivor selection, convergence, and winner choice all rank on the same merged axes, so a candidate whose only gain is on the held-out half moves the front instead of looking stagnant.
  • One shared metric key set across the Insight suite and the user's train and validation datasets, because comparing aggregates across splits requires identical keys.
  • Two guards fail early. A verifier content-hash comparison rejects authoring that left a task untouched; a baseline key-uniformity check fails at round 0 rather than crashing aggregation mid-run.

Notable fixes found along the way

  • _select_survivors ranked on slim() copies and returned them, so survivors carried into later rounds lost their per-trial detail — starving the analyzer from round 2 on and blanking metadata.json. It now ranks on slimmed copies but hands back the full objects.
  • The terminator's _has_converged and EvolutionTree.get_best() ranked on val_reward alone, so a run improving only on the held-out half was declared converged.

Verification

  • plugins/nemo-eval-author: 67 passed, 2 skipped
  • plugins/nemo-experimentalist: 608 passed
  • ruff check and ruff format --check clean on both plugins
  • ty check: zero diagnostics in nemo-eval-author

Live e2e on the tau3-nooa-agent example confirmed on-disk artifacts: 7/7 tasks authored, one shared key set across all four splits, both halves persisted on the candidate with distinct suite identities and populated trial detail, and round-0 analysis citing insight-train trials with the two halves correctly labeled by role.

Test plan

  • Rebase onto latest main — this branch was cut from a tree predating the merge of refactor: make analyst/experimentalist/eval-author CLI agents-only (ASE-702) #1052, so the diff currently carries ~9 files of already-merged ASE-702 changes
  • Complete an e2e run with non-degenerate scores (the validating run was blocked by an unrelated TAU2_USER_MODEL misconfiguration defaulting to an unqualified gpt-5.2)
  • Confirm round-2 survivor selection discriminates on insight/ dimensions once real signal is present
  • Decide whether the authoring hash check should also cover the user's train/validation datasets on fresh runs

Drop legacy top-level aliases and document only `nemo agents …` paths so the optimizer plugins match the platform agent CLI naming.

Signed-off-by: Alec Khoury <akhoury@nvidia.com>
Drop entry-point and fake AgentsCLI mount coverage; keep verb placeholders and a single ctx.command_path check.

Signed-off-by: Alec Khoury <akhoury@nvidia.com>
…t scoring

The Eval Author produced a single Insight suite that served as both the
optimizer's development feedback and its scoring evidence, so nothing measured
whether an agent generalized to the production failures it had not already been
tuned against.

Split the finalized suite down the middle, giving the odd task to train, and
materialize each half with its own content provenance so a candidate records
which suite it was scored against. The validation half is hidden through the
existing path-based holdout, so the coder cannot read it.

Feed the train half's trials to the analyzer for trace-level diagnosis, and
merge the validation half into Pareto selection as insight/-prefixed
dimensions. Survivor selection, convergence, and winner choice now rank on the
same merged axes, so a candidate whose only gain is on the held-out half still
moves the front instead of looking stagnant.

Author one shared metric key set across the Insight suite and the user's train
and validation datasets, because comparing aggregates across splits requires
identical keys. Two guards catch violations early: a verifier content-hash
comparison fails authoring that left a task untouched, and a baseline key
uniformity check fails at round 0 rather than crashing aggregation mid-run.

Signed-off-by: Alec Khoury <akhoury@nvidia.com>
@github-actions github-actions Bot added the feat label Aug 4, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant