feat(cli): add --no-eval to turn the accuracy eval off for a whole run - #1105
Merged
Conversation
A CI/CD tuning run that cares only about throughput has no way to skip the eval today: RUN_EVAL=false leaves the baseline without an accuracy reference, and the baseline guard rejects that, so the run stops before it optimizes anything. --no-eval writes RUN_EVAL=false into the baseline's materialized YAML and exempts the baseline guard. Everything downstream follows for free: candidates template from that YAML so they inherit the disable, and baseline_accuracy stays 0, which accuracy_keep_block and explore's inline gate already treat as "grade on throughput only". So no candidate-side gate needs touching. The eval-concurrency preflight hard-fail also self-disables, since it keys off the materialized RUN_EVAL. A measured regression still blocks: the scriptable image quality_gate is computed by the benchmark run itself and never consulted RUN_EVAL. Closes #1089 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Conflict in baseline.py::_run_once: main added the defer_accuracy_until_after_measure local right where this branch added --no-eval to the RUN_EVAL force-off condition. Both kept. Main's staged accuracy round is a new eval participation point that --no-eval has to cover: it re-runs the benchmark on the hot server with run_eval=True hard-coded, so the flag would have leaked one eval per baseline. The deferral is now cancelled when the flag is set. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
_eval_disabled read `ctx.extra["shared_state"] or self.shared_state`, but the kernel integrate lane constructs BaselineExecutor with neither, so the flag resolved False there. That left defer_accuracy_until_after_measure (set unconditionally by that lane) uncancelled and _write_lifecycle_config re-wrote RUN_EVAL=true, i.e. a full lm_eval per re-baseline. Route through the existing _resolve_shared_state, which falls back to the session dir, and take the sibling _eval_enablement_active with it so both read the same source. Refs #1089 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Co-authored-by: Cursor <cursoragent@cursor.com>
_enqueue_framework_agent_task is the only bench lane without a config_path, so FrameworkAgentExecutor templates from the shipped default YAML, which carries no RUN_EVAL and is therefore materialized with the "true" default. Under --no-eval every framework candidate still paid a full GSM8K run. Injecting baseline_config_path the way the other lanes do is not an option here: materialize_config_with_envs is not idempotent for sglang, so re-templating from the baseline's materialized YAML re-prepends $INFERENCE_OPTIMIZER_SERVER_ARGS and, since sglang is last-wins, resurrects the operator value over a baseline that deliberately overrode it. Carry the decision as a disable_run_eval param instead, which is all the lane needs. Refs #1089 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Co-authored-by: Cursor <cursoragent@cursor.com>
The eval-origin enablement lane exists to repair a broken accuracy eval, so it is incoherent with --no-eval and the two disagreed in opposite directions: integrate_patch forced RUN_EVAL=true for an enablement candidate, re-enabling the eval the flag had turned off, while the revalidation baseline's disable_run_eval=False was overridden by the flag, leaving a revalidation that could never revalidate anything. Gate the lane itself rather than patching either side. eval_enablement_allowed already takes shared_state and is the single admission point for all three callers (baseline routing, writeback budget suppression, specialist dispatch), so closing it there makes both branches unreachable instead of contradictory. launch_enablement_allowed is untouched: a baseline that cannot boot is orthogonal to the eval. Refs #1089 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Co-authored-by: Cursor <cursoragent@cursor.com>
Both enablement RUN_EVAL force-ons keyed on params["enablement"] alone, but only the eval-origin gate requires a score: _runnable_decision fails closed on a missing accuracy for eval-origin and leaves a boot-origin candidate provisional. The launch lane stays open under --no-eval by design, so its integrate candidate was re-enabling an eval the session had turned off and that nothing would have rejected it for lacking. Scope both sites to the eval origin through one predicate, which the runnable gate already keyed on. Refs #1089 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Co-authored-by: Cursor <cursoragent@cursor.com>
_run_eval_enabled had the precedence inverted against every other eval site and against its own neighbours (MAGPIE_EVAL_TASKS / _LIMIT), so an exported RUN_EVAL=true could resurrect an eval the session had turned off. Refs #1089 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Co-authored-by: Cursor <cursoragent@cursor.com>
Two eval costs that survived --no-eval because neither reads the session state. Multi-node preflight installed lm_eval off the process RUN_EVAL alone, which a CLI flag never sets. The install runs with check=True, so a run that will never load the harness could still fail to start on it. Preflight runs before the --resume block reads state.json, so the flag is resolved from both the args and the resumed session rather than the args alone. The GEAK GEMM shape-capture wrapper defaulted RUN_EVAL to true from the inherited env, so every capture paid a GSM8K run. Nothing grades it -- the payload asks for E2E_METRIC=output and consumes baseline_tput only -- so this is disabled outright rather than gated on the flag, matching the sibling capture path that already hard-sets RUN_EVAL=false. Refs #1089 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Co-authored-by: Cursor <cursoragent@cursor.com>
The other run-scoped toggles (kernel, framework phase, explore) each print their persisted state on resume and refuse to apply retroactively once the phase they gate has been entered. eval_disabled had neither: a resume silently inherited the persisted value with no line in the summary, and --no-eval against a normal session was dropped without a word. The cutoff is the anchored accuracy rather than a phase, because the baseline runs inside PRELUDE and the phase alone cannot say whether a reference exists. Once it does, every KEEP so far was graded against it, so the flag is refused with a warning naming the score. Refs #1089 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Co-authored-by: Cursor <cursoragent@cursor.com>
ZhengGong-amd
force-pushed
the
feat/zgong/eval-policy
branch
from
August 6, 2026 11:15
5e1577a to
ebcacdc
Compare
CI E2E report — ❌ Timeout
|
Textual auto-merge was clean across all ten files both sides touched. The semantic review that matters for this branch is whether main opened a lane the --no-eval feature does not cover: it did not. main adds no RUN_EVAL write site outside tests, no new accuracy-gated enqueue, and leaves the single require_accuracy_for_keep site the branch already carries disable_run_eval through. eval_disabled is likewise not surfaced in the prompt render, matching main's treatment of the sibling kernel_enabled toggle, so nothing needs mirroring there either. Co-authored-by: Cursor <cursoragent@cursor.com>
The three flags retired with the kernel LLM role were re-registered as no-ops so a launcher or operator template that still passes them would not exit 2 in argparse. --kernel-prompt took a path, and registering it as store_true meant the path was not consumed: it fell through as a stray positional and argparse exited 2 anyway, reproducing the exact failure the shim exists to prevent. Register it separately with type=str so the value is swallowed. Restore the launcher guard the shim silently disarmed. Once the parser accepts the retired names they are "known" flags, so the name-membership check can no longer notice a launcher reintroducing one; assert their absence directly, as before, and add a parse-level regression test for the no-op contract itself. The membership check also never modelled arity, which is why it would have reported --kernel-prompt as accepted while a real run died. Check that a flag a launcher hands a value to is one argparse takes a value for. Both directions were confirmed by mutation: injecting a store_true flag with a value fails the new arity case while the name case still passes. Co-authored-by: Cursor <cursoragent@cursor.com>
The bug this test guards was a shared-state dump assembled before the gap was recomputed, so the prompt carried both the stale 0.00 and the fresh value. The positive assertion alone cannot see that shape: it passes as soon as 15.00 appears anywhere, whether or not 0.00 is still sitting above it. Restore the negative half. Co-authored-by: Cursor <cursoragent@cursor.com>
The Removed entry listed the three kernel CLI flags as removed, and marked the whole entry breaking, while the parser accepts them. Split the two cases: the env vars really are gone, the flags parse as hidden no-ops until the callers that pass them are updated. Note that --kernel-prompt still takes its path, so the distinction is not merely cosmetic. Add the missing Unreleased entry for --no-eval: what it changes about the baseline anchor and the candidate gates, that a measured regression still blocks, that it is session state rather than a parsed arg so the lanes that template their own bench config are covered, and that it is refused once an accuracy has been anchored. Co-authored-by: Cursor <cursoragent@cursor.com>
xiaofei-zheng
approved these changes
Aug 7, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Branch:
feat/zgong/eval-policy, 1 commit on top of789436d1e. Closes #1089RUN_EVAL=falseleaves the baseline without an accuracy reference, and since08da16dthe baseline guard rejects exactly that — so the run stops before itoptimizes anything.
--no-evalis onestore_trueflag; default off istoday's behaviour unchanged.
The change
Two edits, both joining a condition that already exists:
baseline.py::_run_once— thedisable_run_evalforce-off alreadywrites
RUN_EVAL=false; the flag joins it.baseline.py::_maybe_stop_on_missing_baseline_accuracy— an early returnbeside the
_should_establish_quality_refone. The guard rejects a referencethat was supposed to be measured; under
--no-evalnone was asked for.Same expression also absorbs the flag in
eval_already_off: with eval off, aneval-failure marker cannot be about an eval that ran.
What follows for free
No candidate-side gate is touched:
RUN_EVALaccuracy_keep_blockbaseline_accuracystays0, sobase > 0is false and it already returns(False, "", True)baseline_accuracy > 0, never armsaccuracy_baselinefromstate.baseline_accuracy(phases/framework.py:3485)_materialized_run_eval_disabled, nowTrueRUN_EVAL=falseThe inheritance was verified, not assumed: materializing the baseline with
RUN_EVAL=falsethen re-materializing it as a candidate yields'false'onboth. That is why the change stays at two sites — an earlier draft that pushed
the decision into
materialize_config_with_envsfought the same inheritance.A measured regression still blocks: the scriptable (xDiT) image
quality_gateis computed by the benchmark run itself and never consulted
RUN_EVAL.Files
cli/parser.py--no-eval,store_true, following--no-explorecli/bootstrap.pySharedStateconstructionstate/shared_state.pyeval_disabled: bool = Falseexecutors/baseline.py_eval_disabledhelper + 3 call sites, docstrings refresheddocs/reference/environment-variables.mdNet ~30 source lines.
Tests
test_no_eval_forces_run_eval_false— end to end: materialized YAML carriesRUN_EVAL=false,stop_reasonempty.test_no_stop_when_eval_is_disabled— the guard exemption. Its siblings(
test_stop_serving_operator_disabled_via_config,test_stop_serving_eval_failure_fallback) are untouched and still stop,which is what proves the exemption is narrow.
test_cli_no_explore.py— flag defaultsFalse, setsTrue, state fielddefaults
False.343 passed across the 12 affected files;
ruff checkclean.pylint --errors-onlyshows only the three pre-existing__dataclass_fields__findings in
shared_state.py(confirmed by stashing).Compatibility
Default-off is byte-for-byte today's behaviour.
SharedState.from_dictfiltersto known
__dataclass_fields__, so an oldstate.jsonloads and readsFalse; the field persists across a resume.Not in this branch
HYPERLOOM_EVAL_POLICY(
full/baseline_only/off). One flag covers the ask, and a tri-statecosts a policy check at every candidate gate — all of which the
baseline_accuracy == 0path already handles.(specialist, kernel-agent, multi-node) to survive a subprocess.
reports/final.{json,md}. Under--no-evalabaseline_accuracyof0.0reads like a collapsed model, so an annotationon
report.py::_format_completeness_annotationsis worth adding — but it isreporting, not this flag's logic.