feat(experimentalist): hold out half the Insight suite for independent scoring - #1086
Draft
aleckhoury wants to merge 3 commits into
Draft
feat(experimentalist): hold out half the Insight suite for independent scoring#1086aleckhoury wants to merge 3 commits into
aleckhoury wants to merge 3 commits into
Conversation
Drop legacy top-level aliases and document only `nemo agents …` paths so the optimizer plugins match the platform agent CLI naming. Signed-off-by: Alec Khoury <akhoury@nvidia.com>
Drop entry-point and fake AgentsCLI mount coverage; keep verb placeholders and a single ctx.command_path check. Signed-off-by: Alec Khoury <akhoury@nvidia.com>
…t scoring The Eval Author produced a single Insight suite that served as both the optimizer's development feedback and its scoring evidence, so nothing measured whether an agent generalized to the production failures it had not already been tuned against. Split the finalized suite down the middle, giving the odd task to train, and materialize each half with its own content provenance so a candidate records which suite it was scored against. The validation half is hidden through the existing path-based holdout, so the coder cannot read it. Feed the train half's trials to the analyzer for trace-level diagnosis, and merge the validation half into Pareto selection as insight/-prefixed dimensions. Survivor selection, convergence, and winner choice now rank on the same merged axes, so a candidate whose only gain is on the held-out half still moves the front instead of looking stagnant. Author one shared metric key set across the Insight suite and the user's train and validation datasets, because comparing aggregates across splits requires identical keys. Two guards catch violations early: a verifier content-hash comparison fails authoring that left a task untouched, and a baseline key uniformity check fails at round 0 rather than crashing aggregation mid-run. Signed-off-by: Alec Khoury <akhoury@nvidia.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The Eval Author produced a single Insight suite that served as both the optimizer's development feedback and its scoring evidence, so nothing measured whether an agent generalized to production failures it had not already been tuned against. This splits the suite into held-out halves and wires the validation half into selection.
insight/-prefixed dimensions, with union-based zero-fill so a candidate missing an Insight score isn't silently undominated. Survivor selection, convergence, and winner choice all rank on the same merged axes, so a candidate whose only gain is on the held-out half moves the front instead of looking stagnant.Notable fixes found along the way
_select_survivorsranked onslim()copies and returned them, so survivors carried into later rounds lost their per-trial detail — starving the analyzer from round 2 on and blankingmetadata.json. It now ranks on slimmed copies but hands back the full objects._has_convergedandEvolutionTree.get_best()ranked onval_rewardalone, so a run improving only on the held-out half was declared converged.Verification
plugins/nemo-eval-author: 67 passed, 2 skippedplugins/nemo-experimentalist: 608 passedruff checkandruff format --checkclean on both pluginsty check: zero diagnostics innemo-eval-authorLive e2e on the
tau3-nooa-agentexample confirmed on-disk artifacts: 7/7 tasks authored, one shared key set across all four splits, both halves persisted on the candidate with distinct suite identities and populated trial detail, and round-0 analysis citing insight-train trials with the two halves correctly labeled by role.Test plan
main— this branch was cut from a tree predating the merge of refactor: make analyst/experimentalist/eval-author CLI agents-only (ASE-702) #1052, so the diff currently carries ~9 files of already-merged ASE-702 changesTAU2_USER_MODELmisconfiguration defaulting to an unqualifiedgpt-5.2)insight/dimensions once real signal is present