feat(mcp): expose max_cost budget ceiling on panel-running tools#579
Conversation
The CLI's --max-cost projected-total spend gate had no MCP analog, so agents could only budget-gate by piloting small and scaling manually. run_panel / run_quick_poll / extend_panel now accept max_cost (USD), wired into the existing CostGate machinery (cost.py) with the CLI's soft-halt semantics: in-flight panelists finish, no new panelists start, synthesis is skipped, and the response is a valid partial envelope with run_invalid, cost_exceeded, abort_reason: "cost_exceeded", halted_at_panelist, the cost_gate snapshot, and an agent-legible resume block (persisted partial result_id, completed panelists, remaining personas). Unsupported combinations refuse loudly with typed INVALID_TOOL_ARG before any spend (parity with the CLI support matrix): sampling mode, models ensembles, variants, and instrument/instrument_pack inputs. Closes #576 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: ab23f76980
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| synthesis_temperature=synthesis_temperature, | ||
| variants=variants, | ||
| sessions_out=run_sessions, | ||
| cost_gate=cost_gate, |
There was a problem hiding this comment.
When a default run_panel or run_quick_poll call supplies max_cost, this passes the gate to run_panel_parallel without limiting max_workers; that function defaults to len(personas) and submits every panelist before processing the first completion (orchestrator.py:1551,1594-1634). Consequently, by the time the gate trips, all panelists are normally already running, their futures cannot be cancelled, and executor shutdown waits for them to spend tokens even though only the first processed results and costs appear in the partial envelope. The advertised ceiling therefore neither prevents the remaining spend nor reports it; dispatch must be incremental or concurrency must be constrained while the gate is armed.
Useful? React with 👍 / 👎.
| cost_gate_halted = cost_gate is not None and cost_gate.should_halt() | ||
| if synthesis and cost_gate_halted: | ||
| logger.warning("cost gate halted the panel; skipping synthesis on the partial result") | ||
| synthesis = False | ||
| if synthesis: |
There was a problem hiding this comment.
When all panelist completions leave the projection at or below max_cost, this condition permits synthesis even though CostGate.record() is only called for panelist costs in orchestrator.py:1693-1698. For example, a panel costing $1 under a $2 cap can still issue a $5 synthesis request and return a $6 total without any cost_exceeded marker. Thus any capped call with synthesis enabled can exceed the documented hard total-spend ceiling; synthesis must be budgeted or refused when the remaining ceiling cannot cover it.
Useful? React with 👍 / 👎.
2 participants
Closes #576.
What
Adds an optional
max_cost(USD, float > 0) argument to the three panel-running MCP tools —run_panel,run_quick_poll, andextend_panel— wired into the existingCostGatemachinery (src/synth_panel/cost.py) the CLI's--max-costuses. No parallel implementation: the gate is constructed at the tool boundary, threaded through_runners.run_panel_syncintoorchestrator.run_panel_parallel, which already knew how to soft-halt on it.Behavior on a gate trip
Same soft-halt semantics as the CLI: the in-flight panelist(s) complete, no new panelists start, and synthesis is skipped (synthesizing a deliberately-truncated panel would spend past the cap for an untrustworthy summary). The tool response is a valid partial envelope that composes with
detail:"summary"and thepanel_verdictcontract:run_invalid: true,cost_exceeded: true,abort_reason: "cost_exceeded",halted_at_panelistcost_gatesnapshot — spend so far, the cap, and the projection that tripped itresumeblock — persisted partialresult_id(the MCP analog of the CLI's checkpoint),completed_panelists,remaining_personas, and ahow_to_resumesentencecost_exceeded: ...line inwarnings[]The completed prefix is persisted as usual, so the full partial transcript stays retrievable via
get_panel_result(result_id).Loud refusals (typed
INVALID_TOOL_ARGonmax_cost, before any spend)Mirrors the CLI's
--max-costsupport matrix (_multi_round_flag_errors):max_cost <= 0modelsensembles — the CLI doesn't gate ensembles either; run each model separately with its own capvariants— the gate projection is sized to the base panel; variant expansion would skew itinstrument/instrument_pack— on the MCP surface these always dispatch through the multi-round engine, which the cost gate does not cover (the CLI refuses--max-costthere too)Docs
docs/mcp.md— new "max_cost: hard spend ceiling" section with call + trip-envelope examples; tools table rows updated (checked by thetest_doc_example_params.py/test_skill_tool_conformance.pyCI guards)site/mcp/index.html— "Budget-gated iteration" no longer claims max_cost is CLI-only; "Cost ceiling hit" runbook row covers both surfaces;site/mcp/index.mdregenerated viascripts/render_site_markdown.pydocs/production-operations.md— MCPmax_costbullet under Cost controls + support-matrix noteTests
New
tests/mcp/test_max_cost.py(21 tests): param plumbing (a realCostGatewith the right ceiling/panel size reachesrun_panel_parallel; no gate whenmax_costomitted), gate-trip envelope shape for all three tools (partial markers, snapshot math, resume block, synthesis skipped,detailcomposition, verdict retained), and every refusal path. Only the LLM boundary is stubbed; the tool functions,run_panel_sync, andCostGaterun for real.How verified
ruff check/ruff format --check— cleanmypy src/synth_panel/— cleantest_seed,test_structured_outputescalation,test_mcp_stdio_sampling) also fail on a clean checkout ofmainin this environment (missingOPENROUTER_API_KEYetc.) — pre-existing, unrelated to this changetest_doc_example_params,test_skill_tool_conformance,test_site_markdown) — passDesign decisions worth reviewing
max_costeven for single-round instruments. The CLI supports--max-coston single-round instruments because it flattens them onto the single-round orchestrator; MCP instruments always run throughrun_multi_round_panel, which has no gate wiring. Wiring the gate through the multi-round engine (per-roundrun_panel_parallel+ inter-round halt check) is a natural follow-up if single-round-instrument coverage matters.extend_panelpersists the partial extension round before attaching the halt markers (the pre-extend snapshot still allows rollback); itsresumehint is honest that re-callingextend_panelre-asks every saved panelist.CostGate.recordpath) — the MCP layer adds zero new cost math.🤖 Generated with Claude Code