fix(cli): synthesis skip + zero-cost recording on single-model ensemble path, help/output polish#580
Conversation
…le path, help/output polish - panel run --models <one-model> (weight-free, single entry) was classified as an ensemble and took the ensemble branch: no synthesis, no cost summary, exit 0 with a bare "Ensemble complete". Single-entry specs now demote to the standard single-model path (synthesis included) with a stderr note; ensembles require >= 2 models. - resolve_cost trusted a provider-reported cost of exactly $0 verbatim (OpenRouter returns usage.cost: 0 for BYOK keys), recording token-consuming runs as $0.0000 and starving CostGate so --max-cost could never trip. A zero provider cost with nonzero tokens now falls back to the local pricing-table estimate; genuinely free models (local table also $0) still record $0. - Ensemble text stdout now states models/personas/questions, incident count, recorded cost, "Synthesis: skipped" plus the exact `panel synthesize <result-id>` follow-up, and where results live; JSON carries synthesis_status: "skipped" explicitly. Standard text runs end with a RUN SUMMARY block (size, errors, cost, synthesis status, results path or --save hint). - --personas/--instrument help now documents bundled pack/instrument names (pack list / instruments list) alongside YAML paths. - New `cost show <result-id>`: per-run cost breakdown as a thin alias over panel inspect's data; saved-result hint updated. - Regression tests in tests/test_r4_cli_fixes.py. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 1c94327eb4
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| # Fall back to the local pricing table whenever it prices the same | ||
| # usage as nonzero; genuinely free models (local table also prices $0) | ||
| # still record $0. | ||
| if provider_total == 0.0 and usage.total_tokens > 0 and local_total > 0: |
There was a problem hiding this comment.
When a provider genuinely bills $0—such as an OpenRouter :free route or a promotion—this branch replaces the authoritative zero with paid local pricing. lookup_pricing has no special handling for :free, so such slugs either match the paid model family or fall back to nonzero DEFAULT_PRICING; the run is then misrecorded as paid and CostGate can halt a workload that has accrued no charge. A zero alone cannot distinguish BYOK from a genuinely free request, so the fallback needs an explicit BYOK/provider signal rather than overriding every zero.
Useful? React with 👍 / 👎.
| if synthesis_dict: | ||
| synthesis_status = f"completed (cost: {synthesis_dict['cost']})" |
There was a problem hiding this comment.
When synthesis returns a fallback result, the earlier is_fallback branch explicitly marks the run invalid and populates synthesis_error_payload, but the fallback still produces a truthy synthesis_dict. This condition therefore prints Synthesis: completed in the new run summary immediately after stderr reported that synthesis was incomplete, even though the command exits 2. Check the synthesis error/fallback state before treating any dictionary as a successful completion.
Useful? React with 👍 / 👎.
| "total_cost": report.total_cost, | ||
| "panelist_cost": report.panelist_cost, | ||
| "total_tokens": report.total_tokens, |
There was a problem hiding this comment.
For a normal result created by save_panel_result, panelist_cost is not persisted at the top level, so build_inspect_report leaves this field None. Consequently the new cost show <result-id> command returns panelist_cost: null in JSON and silently omits the panelist-cost line in text for its primary input type; the test only passes because its hand-written fixture includes a field real saves lack. Derive the panelist amount from persisted metadata/synthesis cost or persist it when saving.
Useful? React with 👍 / 👎.
…tion CI typecheck flagged commands.py:1731 — the demotion branch nulls model_spec together with has_models, but mypy cannot see the invariant across the two blocks. Assert it explicitly. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Deploying synthpanel with
|
| Latest commit: |
16d8c1a
|
| Status: | ✅ Deploy successful! |
| Preview URL: | https://06c54a54.synthpanel.pages.dev |
| Branch Preview URL: | https://fix-r4-cli-bugs.synthpanel.pages.dev |
2 participants
What / why
A naive-user test of the shipped 1.5.7 package surfaced six CLI/runtime issues on a default single-model
panel run(bundled packgeneral-consumer+ instrumentgeneral-survey,openrouter/google/gemini-2.5-flash-lite). All were verified against currentmain(1cf0e3b) before fixing.Repro + root causes
1. HIGH — synthesis silently skipped, stdout just "Ensemble complete", exit 0
Repro on main:
panel run --models openrouter/google/gemini-2.5-flash-lite ...(single model, weight-free spec).--model(singular) does not reproduce — it takes the standard path with synthesis. The naive-user run had to have gone through--models.Root cause:
is_ensemble_spec()classifies any colon-free--modelsspec as an ensemble, including a one-entry spec.handle_panel_runthen takes the ensemble branch, which by design never runs synthesis, returns before the standard-path cost summary / cost gate / synthesis machinery, and emitted the single line "Ensemble complete" with exit 0 — no signal that synthesis was skipped.Fix: a one-entry weight-free spec is demoted to the standard single-model path (identical to
--model X, synthesis included) with a stderr note; ensemble comparison requires >= 2 models. Genuine (>=2 model) ensemble runs now print a terse summary on stdout — models x personas x questions, incident count, recorded cost,Synthesis: skippedwith the exactsynthpanel panel synthesize <result-id>follow-up (or the--saveprerequisite), and where the results live. JSON output carriessynthesis: null+synthesis_status: "skipped"explicitly.2. HIGH — run cost recorded as $0.0000 despite 33k in / 7.6k out tokens
Root cause:
resolve_cost()prefers the provider-reported cost over the local pricing table and trusted a reported value of exactly $0 verbatim. OpenRouter returnsusage.cost: 0for BYOK keys (and some upstreams omit native cost); one $0-reporting response also poisons the aggregate (TokenUsage.__add__treats None+0 as 0, not None). The dry-run estimate and synthesis cost priced correctly because both use the local table directly (estimate_cost) — that asymmetry was the tell. Because the orchestrator's cost gate accrues via the sameresolve_costpath,--max-costcould never trip on such runs.Note: the ensemble path itself does accumulate per-model costs correctly (verified with a mocked provider — nonzero
metadata.cost); the "ensemble path drops constituent costs" hypothesis was not reproducible on main. The zero came from trusting the provider's $0.Fix: a provider-reported $0 for usage the local table prices as nonzero now falls back to the local estimate, with a logged warning. Genuinely free models (local table also $0) still record $0. Regression tests assert nonzero recorded cost with token usage and that
CostGateaccrues + trips.3. MED — help text said "Path to a YAML file" for
--personas/--instrumentBundled names are the primary happy path. Both flags now read "Name of a bundled/installed pack/instrument (see
pack list/instruments list) or a path to a YAML file", metavarNAME_OR_PATH.4. MED — default run stdout nearly empty
Text-mode output for a completed run now ends with a RUN SUMMARY block: personas, questions, error count, recorded cost, synthesis status, and results path (or the exact
--savenext step). Ensemble runs get the equivalent summary (see #1).--output-format json/ndjsonpurity untouched — additions there are JSON fields only.5. LOW — "Ensemble" terminology for single-model runs
Resolved by the routing fix: single-model runs never enter the ensemble branch, so they never log/print "Ensemble" anymore.
6. LOW — no per-run cost command
New
synthpanel cost show <result-id>: thin alias surfacing the per-run cost breakdown (total/panelist cost, per-model token+USD rollup, synthesis cost) from the sameInspectReportdatapanel inspectuses. The "Result saved" hint now lists it.Testing
tests/test_r4_cli_fixes.py(12 tests): routing demotion (text + JSON), ensemble synthesis-status stdout/JSON, zero-provider-cost fallback (incl. CostGate accrual/trip throughrun_panel_parallel),cost show(text/JSON/missing), help text.test_mcp_stdio_sampling.pyx4,test_seed.pyx2,test_structured_output.pyx1; provider-resolution/env-sensitive, pre-existing, unrelated to this diff).CostGate's public interface untouched (concurrent MCPmax_costwork depends on it); the only shared-internal change is insideresolve_cost.🤖 Generated with Claude Code