diff --git a/docs/mcp.md b/docs/mcp.md index 8e91647..ba2f0b5 100644 --- a/docs/mcp.md +++ b/docs/mcp.md @@ -227,9 +227,9 @@ and log/stream discipline — is documented in | Tool | Description | |------|-------------| | `run_prompt` | Send a single prompt to an LLM. No personas required. The simplest tool — ask a quick research question. **Does not require `decision_being_informed`.** | -| `run_panel` | Run a full synthetic focus group panel. Each persona answers all questions independently in parallel, followed by synthesis. Accepts inline `questions`, an inline `instrument` dict (v1/v2/v3), or an `instrument_pack` name. Template placeholders in the instrument (e.g. the bundled packs' `{problem}` / `{candidates}`) are filled via the `vars` argument — the MCP equivalent of the CLI's `--var`. **Requires `decision_being_informed`.** | -| `run_quick_poll` | Quick single-question poll across personas. A simplified `run_panel` for one question with synthesis. Accepts inline `personas` and/or a saved `pack_id` (merged; falls back to a built-in diverse set when both are omitted). **Requires `decision_being_informed`.** | -| `extend_panel` | Append a single ad-hoc round to a saved panel result. Reuses each panelist's saved session for conversational context. **Not** a re-entry into the v3 DAG — use for human-in-the-loop follow-ups. **Requires `decision_being_informed`.** | +| `run_panel` | Run a full synthetic focus group panel. Each persona answers all questions independently in parallel, followed by synthesis. Accepts inline `questions`, an inline `instrument` dict (v1/v2/v3), or an `instrument_pack` name. Template placeholders in the instrument (e.g. the bundled packs' `{problem}` / `{candidates}`) are filled via the `vars` argument — the MCP equivalent of the CLI's `--var`. Optional `max_cost` (USD) arms the mid-run cost gate (see [`max_cost`](#max_cost-hard-spend-ceiling)). **Requires `decision_being_informed`.** | +| `run_quick_poll` | Quick single-question poll across personas. A simplified `run_panel` for one question with synthesis. Accepts inline `personas` and/or a saved `pack_id` (merged; falls back to a built-in diverse set when both are omitted). Optional `max_cost` (USD) spend ceiling. **Requires `decision_being_informed`.** | +| `extend_panel` | Append a single ad-hoc round to a saved panel result. Reuses each panelist's saved session for conversational context. **Not** a re-entry into the v3 DAG — use for human-in-the-loop follow-ups. Optional `max_cost` (USD) spend ceiling for the extension round. **Requires `decision_being_informed`.** | ### Tool-call examples @@ -407,6 +407,72 @@ saved panel. is not retrievable later — sampling always returns full transcripts regardless of `detail`. `extend_panel` returns its single appended round in full. +### `max_cost`: hard spend ceiling + +`run_panel`, `run_quick_poll`, and `extend_panel` accept an optional +**`max_cost`** argument — a hard ceiling on the run's total spend, in USD. It +is the MCP analog of the CLI's `--max-cost` and wires into the same `CostGate` +machinery (`src/synth_panel/cost.py`): after each panelist completes, +`running_cost / completed_n * total_n` is compared against the ceiling. + +```jsonc +// run_panel with a $2 ceiling +{ + "tool": "run_panel", + "arguments": { + "pack_id": "general-consumer", + "questions": [{ "text": "What would you pay for this?" }], + "max_cost": 2.0, + "decision_being_informed": "choosing launch tier price" + } +} +``` + +When the projection exceeds the ceiling the run **soft-halts**: the current +panelist(s) finish, no new panelists start, and synthesis is skipped (spending +more to summarize a deliberately-truncated panel would produce an +untrustworthy result). The response is still a **valid partial envelope** — +the completed prefix is persisted under `result_id` as usual — carrying: + +```jsonc +// response (truncated) after a cost-gate trip +{ + "run_invalid": true, + "cost_exceeded": true, + "abort_reason": "cost_exceeded", + "halted_at_panelist": 4, + "cost_gate": { + "max_cost_usd": 2.0, + "running_cost_usd": 1.62, // spend so far + "projected_total_usd": 4.05, // what tripped the gate + "completed": 4, + "total_panelists": 10, + "halted": true, + "halted_projection_usd": 4.05 + }, + "resume": { + "partial_result_id": "result-20260716-...", + "completed_panelists": ["..."], + "remaining_personas": ["..."], + "how_to_resume": "..." // fetch partial via get_panel_result; + // re-run remaining_personas with a raised cap + }, + "synthesis": null // skipped on the partial +} +``` + +The partial envelope composes with `detail: "summary"` and still carries the +`panel_verdict` contract fields; the full partial transcript stays retrievable +via `get_panel_result(result_id)`. + +Support matrix (mirrors the CLI's `--max-cost`, which the multi-round engine +refuses): `max_cost` applies to **BYOK** runs with inline `questions` (and +`extend_panel`'s single extension round). Combining it with sampling mode (the +host agent pays; the server sees no per-panelist costs), `models` ensembles, +`variants`, or `instrument` / `instrument_pack` inputs returns a typed +`INVALID_TOOL_ARG` on `max_cost` — a loud refusal before any spend, never a +silently unenforced ceiling. `max_cost` must be > 0. + ### Typed error envelopes Boundary errors return a typed `INVALID_TOOL_ARG` envelope diff --git a/docs/production-operations.md b/docs/production-operations.md index 0a10588..f23b892 100644 --- a/docs/production-operations.md +++ b/docs/production-operations.md @@ -158,6 +158,18 @@ Two overflow safeguards on the synthesis step itself: `halted_at_panelist`, and a `cost_gate` snapshot. Exit code `2`. (`CostGate` in `src/synth_panel/cost.py`; wiring in `src/synth_panel/cli/commands.py`.) +- **MCP `max_cost` (GH#576)** — the same gate on the MCP surface: the + `run_panel`, `run_quick_poll`, and `extend_panel` tools accept a + `max_cost` (USD) argument wired into the identical `CostGate` machinery + with the same soft-halt semantics. On a trip, the tool response is a + valid partial envelope with `run_invalid: true`, `cost_exceeded: true`, + `abort_reason: "cost_exceeded"`, `halted_at_panelist`, the `cost_gate` + snapshot, and an agent-legible `resume` block (persisted partial + `result_id`, completed panelists, remaining personas). Synthesis is + skipped on the partial. BYOK inline-`questions` runs only — sampling + mode, `models` ensembles, `variants`, and instrument inputs refuse + `max_cost` with a typed `INVALID_TOOL_ARG` (parity with the CLI's + multi-round refusal). See [docs/mcp.md](mcp.md#max_cost-hard-spend-ceiling). - **Per-turn telemetry** — token usage is tracked per turn in four buckets (input / output / cache-write / cache-read; `TokenUsage` and `UsageTracker` in `src/synth_panel/cost.py`). Every panelist row and the @@ -286,4 +298,7 @@ Kept here so this page stays trustworthy: `--question-failure-budget` apply to single-round runs; multi-round (branching) instruments refuse these flags loudly up front rather than degrading silently (`_multi_round_flag_errors`, - `src/synth_panel/cli/commands.py`). + `src/synth_panel/cli/commands.py`). The MCP `max_cost` argument mirrors + the same matrix: instrument inputs (which always dispatch through the + multi-round engine on the MCP surface), ensembles, variants, and + sampling mode refuse it with a typed `INVALID_TOOL_ARG`. diff --git a/site/mcp/index.html b/site/mcp/index.html index 91d5984..32113d8 100644 --- a/site/mcp/index.html +++ b/site/mcp/index.html @@ -1133,12 +1133,23 @@

cost, so the loop is: pilot with run_quick_poll, gate on the returned cost, then scale to a full - run_panel. Note the hard - mid-run spend ceiling - (--max-cost) is a CLI flag — - there is no max_cost tool - argument on the MCP surface; unattended hard ceilings mean - driving the CLI. + run_panel. For unattended + hard ceilings, pass + max_cost (USD) to + run_panel / + run_quick_poll / + extend_panel — the MCP + analog of the CLI's + --max-cost, wired into the + same mid-run cost gate. On a trip the response is a valid + partial envelope with + cost_exceeded: true, + abort_reason: + "cost_exceeded", a + cost_gate snapshot (spend + so far, cap, projection), and a + resume block naming the + persisted partial and the remaining personas. @@ -1222,11 +1233,13 @@

Cost ceiling hit - --max-cost halts gracefully mid-run: pending - panelists cancelled, valid partial JSON with - run_invalid: true, + --max-cost (CLI) or the max_cost + tool argument (MCP) halts gracefully mid-run: pending + panelists cancelled, synthesis skipped, valid partial JSON + with run_invalid: true, cost_exceeded: true, - halted_at_panelist. Exit 2. + halted_at_panelist, and (MCP) a + resume block. CLI exit 2. diff --git a/site/mcp/index.md b/site/mcp/index.md index 02e3fcd..ccac8a4 100644 --- a/site/mcp/index.md +++ b/site/mcp/index.md @@ -289,7 +289,7 @@ Summary first, transcripts on demand `run_panel` and `run_quick_poll` default to Structured polling — no prose parsing Both run tools accept a `response_schema` argument (JSON Schema) that forces structured output at generation time — forced-choice, Likert, tagged themes, ranking — so downstream code branches on fields, not on regexes over prose. Pattern catalogue: [docs/structured-polling.md](https://github.com/DataViking-Tech/SynthPanel/blob/main/docs/structured-polling.md). -Budget-gated iteration MCP mode defaults to `haiku` and caps panels at 100 personas × 50 questions. Every response carries in-band `total_cost` and per-panelist `usage` / `cost`, so the loop is: pilot with `run_quick_poll`, gate on the returned cost, then scale to a full `run_panel`. Note the hard mid-run spend ceiling (`--max-cost`) is a CLI flag — there is no `max_cost` tool argument on the MCP surface; unattended hard ceilings mean driving the CLI. +Budget-gated iteration MCP mode defaults to `haiku` and caps panels at 100 personas × 50 questions. Every response carries in-band `total_cost` and per-panelist `usage` / `cost`, so the loop is: pilot with `run_quick_poll`, gate on the returned cost, then scale to a full `run_panel`. For unattended hard ceilings, pass `max_cost` (USD) to `run_panel` / `run_quick_poll` / `extend_panel` — the MCP analog of the CLI's `--max-cost`, wired into the same mid-run cost gate. On a trip the response is a valid partial envelope with `cost_exceeded: true`, `abort_reason: "cost_exceeded"`, a `cost_gate` snapshot (spend so far, cap, projection), and a `resume` block naming the persisted partial and the remaining personas. Full tool semantics: [docs/mcp.md](https://github.com/DataViking-Tech/SynthPanel/blob/main/docs/mcp.md) · operational contract: [docs/production-operations.md](https://github.com/DataViking-Tech/SynthPanel/blob/main/docs/production-operations.md). @@ -302,7 +302,7 @@ What actually happens when a panel run goes sideways. Failures return typed enve | K of N panelists fail | Errors are recorded per persona and per question (`error` field per row), the panel continues, and `failure_stats` reports the rate. The run is invalid only past `--failure-threshold` (default 0.5) — then synthesis is auto-disabled and the CLI exits `2`. `--strict` = zero tolerance. | | Synthesis fails | Never shaped like success: the envelope carries `synthesis_error` and the run is marked invalid — but panelist data is saved. Recover post-hoc without re-running the panel: `synthpanel panel synthesize --synthesis-model sonnet`. A synthesis-model outage costs one cheap retry, not the panel spend. | | Rate limits | `--max-concurrent` caps in-flight requests; `--rate-limit-rps` adds a token-bucket cap (fractional values accepted). Rate-limit exhaustion is classified distinctly in `abort_reason` — the signal is "raise the throttle," not "chase a model bug." | -| Cost ceiling hit | `--max-cost` halts gracefully mid-run: pending panelists cancelled, valid partial JSON with `run_invalid: true`, `cost_exceeded: true`, `halted_at_panelist`. Exit `2`. | +| Cost ceiling hit | `--max-cost` (CLI) or the `max_cost` tool argument (MCP) halts gracefully mid-run: pending panelists cancelled, synthesis skipped, valid partial JSON with `run_invalid: true`, `cost_exceeded: true`, `halted_at_panelist`, and (MCP) a `resume` block. CLI exit `2`. | | SIGINT / SIGTERM | With checkpointing active, a final checkpoint is flushed and the run marked aborted; stdout still emits a complete, parseable envelope with `abort_reason: "sigint"`. Resume picks up where it stopped. | | Every panelist fails | Loud, not shaped like success: a structured total-failure diagnostic names the upstream cause (bad model id, provider 400). MCP serializes `{"error", "run_invalid": true, "total_failure"}`; the CLI exits `2`. | diff --git a/src/synth_panel/_runners.py b/src/synth_panel/_runners.py index acb764c..0864cb3 100644 --- a/src/synth_panel/_runners.py +++ b/src/synth_panel/_runners.py @@ -17,7 +17,7 @@ from pydantic import BaseModel -from synth_panel.cost import ZERO_USAGE, resolve_cost +from synth_panel.cost import ZERO_USAGE, CostGate, resolve_cost from synth_panel.cost import TokenUsage as CostTokenUsage from synth_panel.instrument import Instrument from synth_panel.llm.client import LLMClient @@ -691,6 +691,7 @@ def run_panel_sync( attachment_bank: dict[str, dict[str, Any]] | None = None, allow_empty_attachments: bool = False, sessions_out: dict[str, Any] | None = None, + cost_gate: CostGate | None = None, ) -> tuple[ list[PanelistResult], list[dict[str, Any]], @@ -711,6 +712,16 @@ def run_panel_sync( keyed by persona name. Kept as an out-param rather than a seventh tuple element so existing unpack sites stay valid. The MCP server uses this to persist AC-7 decision-stamped transcripts. + + ``cost_gate``: optional :class:`~synth_panel.cost.CostGate` (sp-utnk / + GH#576). Threaded to :func:`run_panel_parallel`, which records each + completing panelist's priced cost and soft-halts dispatch when the + projected run total exceeds the ceiling. When the gate has halted, + synthesis is skipped (same rationale as the CLI: synthesizing a + deliberately-truncated panel burns more budget without producing a + trustworthy result). The caller keeps the gate reference and is + expected to inspect ``cost_gate.should_halt()`` on return to shape + its partial-result envelope. """ all_personas = list(personas) variant_names: set[str] = set() @@ -728,6 +739,9 @@ def run_panel_sync( variant_count += 1 logger.info("Running panel with %d base + %d variant personas", len(personas), variant_count) + # GH#576: pass the gate only when armed so existing monkeypatched + # test doubles / caller shims with explicit signatures keep working. + _gate_kwargs: dict[str, Any] = {"cost_gate": cost_gate} if cost_gate is not None else {} panelist_results, _registry, _sessions = run_panel_parallel( client=client, personas=all_personas, @@ -743,6 +757,7 @@ def run_panel_sync( extract_schema=extract_schema, attachment_bank=attachment_bank, allow_empty_attachments=allow_empty_attachments, + **_gate_kwargs, ) if sessions_out is not None: sessions_out.update(_sessions) @@ -773,6 +788,13 @@ def run_panel_sync( # ``synthesis_dict``. Callers (MCP server / SDK) detect this and lift # it to ``run_invalid=True`` + top-level ``synthesis_error`` on the # result envelope. + # GH#576: mirror the CLI's cost-gate discipline — a halted run is a + # deliberately-truncated partial, so spending more budget synthesizing + # it would produce an untrustworthy result. Skip synthesis entirely. + cost_gate_halted = cost_gate is not None and cost_gate.should_halt() + if synthesis and cost_gate_halted: + logger.warning("cost gate halted the panel; skipping synthesis on the partial result") + synthesis = False if synthesis: synth_model_for_check = synthesis_model or model overflow = detect_synthesis_context_overflow( diff --git a/src/synth_panel/mcp/server.py b/src/synth_panel/mcp/server.py index 1200331..cd0f577 100644 --- a/src/synth_panel/mcp/server.py +++ b/src/synth_panel/mcp/server.py @@ -66,6 +66,7 @@ from synth_panel.cost import ( ZERO_USAGE, CostEstimate, + CostGate, aggregate_per_model, build_cost_fallback_warnings, estimate_cost, @@ -347,6 +348,93 @@ def _invalid_tool_arg(message: str, *, field_path: str | None = None) -> str: return json.dumps(env) +def _validate_max_cost(max_cost: float | None) -> str | None: + """Boundary validation for the ``max_cost`` tool argument (GH#576). + + Returns a typed ``INVALID_TOOL_ARG`` envelope string when the value is + unusable, or ``None`` when it is absent or valid. Mirrors the CLI's + ``--max-cost must be > 0`` check so both surfaces refuse the same way. + """ + if max_cost is None: + return None + try: + value = float(max_cost) + except (TypeError, ValueError): + return _invalid_tool_arg( + f"'max_cost' must be a number (USD ceiling), got {max_cost!r}.", + field_path="max_cost", + ) + if value <= 0: + return _invalid_tool_arg( + f"'max_cost' must be > 0 (a positive USD ceiling), got {max_cost!r}.", + field_path="max_cost", + ) + return None + + +def _attach_cost_gate_halt( + result: dict[str, Any], + *, + cost_gate: CostGate, + personas: list[dict[str, Any]], + completed_names: list[str], + result_id: str | None, + tool: str, + resume_hint: str | None = None, +) -> None: + """Mark a response envelope as a cost-gate partial (GH#576). + + Mirrors the CLI's ``--max-cost`` trip envelope (sp-utnk): the run is + flagged ``run_invalid`` with ``cost_exceeded: true``, a specific + ``abort_reason``, the panelist index at halt, and the machine-readable + ``cost_gate`` snapshot (spend so far, ceiling, projection). On top of + the CLI shape, an agent-legible ``resume`` block states exactly how to + continue: the completed prefix is already persisted under + ``result_id`` (the MCP analog of the CLI's checkpoint file), and the + remaining personas can be re-run in a follow-up call. + """ + snap = cost_gate.snapshot() + completed_set = set(completed_names) + remaining = [p.get("name", "Anonymous") for p in personas if p.get("name", "Anonymous") not in completed_set] + result["run_invalid"] = True + result["cost_exceeded"] = True + result["abort_reason"] = "cost_exceeded" + result["halted_at_panelist"] = cost_gate.completed + result["cost_gate"] = snap + if resume_hint is None: + resume_hint = ( + f"To finish the panel, call {tool} again with only remaining_personas " + "and a raised (or omitted) max_cost, then combine with the persisted partial. " + "Synthesis was skipped on this partial to avoid spending past the cap." + ) + resume: dict[str, Any] = { + "completed_panelists": list(completed_names), + "remaining_personas": remaining, + "how_to_resume": ( + f"The completed prefix ({len(completed_names)} panelist(s), " + f"${snap['running_cost_usd']:.4f} spent of the ${snap['max_cost_usd']:.4f} cap) " + + ( + f"is persisted under result_id {result_id!r} — fetch it with get_panel_result({result_id!r}). " + if result_id + else "is returned in this envelope. " + ) + + resume_hint + ), + } + if result_id: + resume["partial_result_id"] = result_id + result["resume"] = resume + warnings_list = result.setdefault("warnings", []) + if isinstance(warnings_list, list): + warnings_list.append( + "cost_exceeded: projected total " + f"${snap['projected_total_usd']:.4f} > max ${snap['max_cost_usd']:.4f} after " + f"{snap['completed']}/{snap['total_panelists']} panelists " + f"(running ${snap['running_cost_usd']:.4f}); pending panelists were cancelled " + "and synthesis was skipped" + ) + + def _panel_timeout_envelope( *, personas: int, @@ -946,6 +1034,7 @@ def _server_run_panel_sync( synthesis_temperature: float | None = None, variants: int = 0, sessions_out: dict[str, Any] | None = None, + cost_gate: CostGate | None = None, ) -> tuple[ list[PanelistResult], list[dict[str, Any]], CostTokenUsage, Any, dict[str, Any] | None, dict[str, Any] | None ]: @@ -966,6 +1055,7 @@ def _server_run_panel_sync( synthesis_temperature=synthesis_temperature, variants=variants, sessions_out=sessions_out, + cost_gate=cost_gate, ) @@ -1285,6 +1375,7 @@ async def _run_panel_async( decision_being_informed: str | None = None, decision_warnings: list[str] | tuple[str, ...] = (), detail: str = "summary", + max_cost: float | None = None, ) -> dict[str, Any]: """Run panel via asyncio.to_thread with progress notifications. @@ -1298,11 +1389,24 @@ async def _run_panel_async( panelist row; ``"summary"`` (the default) drops the top-level ``results`` mirror and ``rounds[].results`` via :func:`_apply_detail` so a large BYOK panel doesn't flood the caller's context. + + ``max_cost`` (GH#576) arms a :class:`~synth_panel.cost.CostGate` sized + to the dispatched personas — the exact machinery behind the CLI's + ``--max-cost``. On a trip, the returned envelope is a valid partial + with ``run_invalid``, ``cost_exceeded``, ``abort_reason: + "cost_exceeded"``, the ``cost_gate`` snapshot, and a ``resume`` block + (see :func:`_attach_cost_gate_halt`). """ total = len(personas) timer = PanelTimer() await ctx.report_progress(0, total) + # GH#576: arm the projected-total cost gate before dispatch. Sized to + # the personas actually being dispatched, same as the CLI wiring. + cost_gate: CostGate | None = None + if max_cost is not None: + cost_gate = CostGate(max_cost_usd=max_cost, total_panelists=total) + # Run the blocking panel execution in a thread run_sessions: dict[str, Any] = {} ( @@ -1329,6 +1433,7 @@ async def _run_panel_async( synthesis_temperature=synthesis_temperature, variants=variants, sessions_out=run_sessions, + cost_gate=cost_gate, ), timeout=PANELIST_TIMEOUT * total * (1 + variants), ) @@ -1469,6 +1574,20 @@ async def _run_panel_async( result["run_invalid"] = True result["synthesis_error"] = synthesis_dict["synthesis_error"] + # GH#576: cost gate tripped mid-run — the orchestrator cancelled the + # pending panelists and ``result_dicts`` is a valid completed prefix. + # Surface the same typed partial-result markers the CLI emits, plus an + # agent-legible resume block. + if cost_gate is not None and cost_gate.should_halt(): + _attach_cost_gate_halt( + result, + cost_gate=cost_gate, + personas=personas, + completed_names=[pr.persona_name for pr in panelist_results_full], + result_id=result_id, + tool="run_panel", + ) + if variant_data: result["robustness_scores"] = variant_data["robustness_scores"] result["per_persona_robustness"] = variant_data["per_persona_robustness"] @@ -1624,6 +1743,7 @@ async def run_panel( models: list[str] | None = None, synthesis_temperature: float | None = None, variants: int | None = None, + max_cost: float | None = None, use_sampling: bool | None = None, accept_multimodal_sampling: bool = False, decision_being_informed: str | None = None, @@ -1784,6 +1904,27 @@ async def run_panel( robustness analysis. When > 0, each persona is perturbed K times and all variants run through the same questions. Results include robustness_scores and per_persona_robustness. Default: no variants. + max_cost: Hard ceiling on total panel spend, in USD (the MCP + analog of the CLI's ``--max-cost``, GH#576). After each + panelist completes, ``running_cost / completed_n * total_n`` + is compared against the ceiling; if the projected total + exceeds it the run soft-halts — the in-flight panelists + finish, no new panelists start, synthesis is skipped — and + the response is a valid *partial* envelope with + ``run_invalid: true``, ``cost_exceeded: true``, + ``abort_reason: "cost_exceeded"``, ``halted_at_panelist``, a + machine-readable ``cost_gate`` snapshot (spend so far, cap, + projection), and a ``resume`` block naming the persisted + partial ``result_id``, the completed panelists, and the + remaining personas to re-run. Composes with + ``detail="summary"`` and the ``panel_verdict`` envelope. + Must be > 0. BYOK inline-``questions`` runs only: typed + ``INVALID_TOOL_ARG`` when combined with sampling mode (the + host agent pays; no server-side cost accounting), ``models`` + ensembles, ``variants``, or ``instrument`` / + ``instrument_pack`` inputs (parity with the CLI, which + refuses ``--max-cost`` on the multi-round engine). Default: + no ceiling. use_sampling: Explicit mode override. ``True`` forces sampling (error if unsupported or if limits exceeded), ``False`` forces BYOK. ``None`` auto-picks based on creds + client capability. @@ -1845,6 +1986,37 @@ async def run_panel( if variants_k < 0 or variants_k > 20: return json.dumps({"error": "variants must be between 0 and 20."}) + # ── GH#576: max_cost boundary checks ───────────────────────────────── + # The gate wires into the single-round BYOK orchestrator only (exact + # parity with the CLI's --max-cost support matrix); every unsupported + # combination refuses loudly before any LLM spend. + max_cost_error = _validate_max_cost(max_cost) + if max_cost_error is not None: + return max_cost_error + if max_cost is not None: + if models and len(models) >= 2: + return _invalid_tool_arg( + "max_cost is not supported with multi-model ensembles (models=[...]); " + "run each model separately with its own max_cost, or drop max_cost.", + field_path="max_cost", + ) + if instrument is not None or instrument_pack is not None: + return _invalid_tool_arg( + "max_cost is not supported with instrument / instrument_pack inputs " + "(they dispatch through the multi-round engine, which the cost gate " + "does not cover — same refusal as the CLI's --max-cost). Pass the " + "instrument's questions inline via 'questions', or drop max_cost.", + field_path="max_cost", + ) + if variants_k > 0: + return _invalid_tool_arg( + "max_cost is not supported together with variants (robustness " + "perturbation): the gate's projection is sized to the base panel " + "and variant expansion would skew it. Run the variant sweep " + "ungated, or pilot ungated variants on a smaller panel first.", + field_path="max_cost", + ) + # Resolve extract_schema name → dict before threading to orchestrator. try: resolved_extract_schema = _resolve_extract_schema(extract_schema) @@ -1930,6 +2102,19 @@ async def run_panel( if decision.mode == "error": return json.dumps({"error": decision.error}) if decision.mode == "sampling": + # GH#576: sampling mode has no server-side cost accounting + # (tokens bill to the host agent's subscription), so a USD + # ceiling cannot be enforced. Refuse loudly rather than + # silently running unbounded. + if max_cost is not None: + return _invalid_tool_arg( + "max_cost requires BYOK cost accounting; this call resolved to " + "sampling mode, where token spend bills to the host agent and " + "the server sees no per-panelist costs. Set a provider API key " + "(e.g. ANTHROPIC_API_KEY) / pass use_sampling=false, or drop " + "max_cost.", + field_path="max_cost", + ) # Resolve question stream for sampling — no v3 branching. # The instrument itself was resolved + vars-substituted above. sampling_questions: list[dict[str, Any]] @@ -2107,6 +2292,7 @@ async def run_panel( decision_being_informed=decision_being_informed, decision_warnings=decision_warnings, detail=detail, + max_cost=max_cost, ) except PanelTotalFailureError as exc: logger.error("run_panel: total failure: %s", exc) @@ -2147,6 +2333,7 @@ async def run_quick_poll( synthesis_prompt: str | None = None, temperature: float | None = None, top_p: float | None = None, + max_cost: float | None = None, use_sampling: bool | None = None, accept_multimodal_sampling: bool = False, decision_being_informed: str | None = None, @@ -2219,6 +2406,18 @@ async def run_quick_poll( synthesis_prompt: Custom synthesis prompt. Replaces the default. temperature: Sampling temperature (0.0-1.0). Controls randomness. top_p: Nucleus sampling threshold (0.0-1.0). Alternative to temperature. + max_cost: Hard ceiling on total poll spend, in USD (the MCP + analog of the CLI's ``--max-cost``, GH#576). Same semantics + as ``run_panel``'s ``max_cost``: soft-halt when the + projected total exceeds the ceiling — the in-flight + panelists finish, no new panelists start, synthesis is + skipped — and the response is a valid partial envelope + carrying ``run_invalid: true``, ``cost_exceeded: true``, + ``abort_reason: "cost_exceeded"``, ``halted_at_panelist``, + the ``cost_gate`` snapshot, and a ``resume`` block. Must be + > 0. BYOK only — typed ``INVALID_TOOL_ARG`` when the call + resolves to sampling mode (the host agent pays; no + server-side cost accounting). Default: no ceiling. use_sampling: Explicit mode override. ``True`` forces sampling (error if unsupported), ``False`` forces BYOK. ``None`` auto-picks based on creds + client capability. @@ -2254,6 +2453,11 @@ async def run_quick_poll( return spec_error model_was_explicit = model is not None + # GH#576: max_cost boundary check (same rule as run_panel / the CLI). + max_cost_error = _validate_max_cost(max_cost) + if max_cost_error is not None: + return max_cost_error + if not question or not question.strip(): return json.dumps({"error": "Question text must be a non-empty string."}) @@ -2292,6 +2496,17 @@ async def run_quick_poll( return json.dumps({"error": decision.error}) if decision.mode == "sampling": + # GH#576: no server-side cost accounting in sampling mode — a USD + # ceiling cannot be enforced. Refuse loudly (parity with run_panel). + if max_cost is not None: + return _invalid_tool_arg( + "max_cost requires BYOK cost accounting; this call resolved to " + "sampling mode, where token spend bills to the host agent and " + "the server sees no per-panelist costs. Set a provider API key " + "(e.g. ANTHROPIC_API_KEY) / pass use_sampling=false, or drop " + "max_cost.", + field_path="max_cost", + ) if len(personas) > SAMPLING_MAX_PERSONAS: return json.dumps( { @@ -2333,6 +2548,7 @@ async def run_quick_poll( decision_being_informed=decision_being_informed, decision_warnings=decision_warnings, detail=detail, + max_cost=max_cost, ) except PanelTotalFailureError as exc: # Same typed envelope run_panel returns — a knowingly-bad model @@ -2681,6 +2897,7 @@ async def extend_panel( synthesis: bool = True, synthesis_model: str | None = None, synthesis_prompt: str | None = None, + max_cost: float | None = None, accept_multimodal_sampling: bool = False, decision_being_informed: str | None = None, ctx: Context = None, @@ -2710,6 +2927,18 @@ async def extend_panel( synthesis: Whether to synthesize the new round. synthesis_model: Synthesis model. Defaults to panelist model. synthesis_prompt: Custom synthesis prompt for the new round. + max_cost: Hard ceiling on the extension round's spend, in USD + (the MCP analog of the CLI's ``--max-cost``, GH#576). Same + soft-halt semantics as ``run_panel``: the in-flight + panelists finish, no new panelists start, the round's + synthesis is skipped, and the response is a valid partial + envelope with ``run_invalid: true``, ``cost_exceeded: + true``, ``abort_reason: "cost_exceeded"``, + ``halted_at_panelist``, the ``cost_gate`` snapshot, and a + ``resume`` block naming the panelists still to be asked. + The partial extension round IS persisted onto the result + (and the pre-extend snapshot remains for rollback). Must + be > 0. Default: no ceiling. decision_being_informed: Required v1.0.0 contract field — the decision this extension informs, in 12-280 characters (trimmed), single line, UTF-8. Validation failures return a @@ -2729,6 +2958,10 @@ async def extend_panel( ) if spec_error is not None: return spec_error + # GH#576: max_cost boundary check (same rule as run_panel / the CLI). + max_cost_error = _validate_max_cost(max_cost) + if max_cost_error is not None: + return max_cost_error model = model or _resolve_mcp_default_model() logger.info("extend_panel: result_id=%s questions=%d model=%s", result_id, len(questions), model) existing, result_error = _resolve_panel_result_or_error(result_id) @@ -2754,6 +2987,15 @@ async def extend_panel( if ctx is not None: await ctx.report_progress(0, len(personas)) + # GH#576: arm the projected-total cost gate for the extension round. + # Sized to the panelists being re-asked (one per saved session), same + # machinery as the CLI's --max-cost. Passed conditionally so existing + # monkeypatched doubles with explicit signatures keep working. + cost_gate: CostGate | None = None + if max_cost is not None: + cost_gate = CostGate(max_cost_usd=max_cost, total_panelists=len(personas)) + _gate_kwargs: dict[str, Any] = {"cost_gate": cost_gate} if cost_gate is not None else {} + def _go() -> tuple[list[PanelistResult], dict[str, Any], Any, dict[str, Any] | None]: client = _get_shared_client() results, _registry, out_sessions = run_panel_parallel( @@ -2764,10 +3006,15 @@ def _go() -> tuple[list[PanelistResult], dict[str, Any], Any, dict[str, Any] | N system_prompt_fn=persona_system_prompt, question_prompt_fn=build_question_prompt, sessions=sessions, + **_gate_kwargs, ) synth = None synth_error: dict[str, Any] | None = None - if synthesis: + # GH#576: a cost-halted extension is a deliberately-truncated + # partial; synthesizing it would spend past the cap for an + # untrustworthy summary (same rationale as the CLI / run_panel). + run_synthesis = synthesis and not (cost_gate is not None and cost_gate.should_halt()) + if run_synthesis: try: synth = synthesize_panel( client, @@ -2883,6 +3130,28 @@ def _go() -> tuple[list[PanelistResult], dict[str, Any], Any, dict[str, Any] | N # envelope shape without inspecting the appended round. response["synthesis_error"] = synthesis_error + # GH#576: cost gate tripped mid-extension. The appended round is a + # valid completed prefix (already persisted above); surface the same + # typed partial markers run_panel emits plus the resume block. + if cost_gate is not None and cost_gate.should_halt(): + _attach_cost_gate_halt( + response, + cost_gate=cost_gate, + personas=personas, + completed_names=[pr.persona_name for pr in panelist_results], + result_id=result_id, + tool="extend_panel", + resume_hint=( + "The partial extension round is already appended to the result " + "(the pre-extend snapshot remains for rollback). extend_panel " + "re-asks every saved panelist, so to cover remaining_personas " + "either call extend_panel again with a raised (or omitted) " + "max_cost — which appends a fresh full round — or accept the " + "partial. The round's synthesis was skipped to avoid spending " + "past the cap." + ), + ) + # v1.0.0 contract fields for the extension round. response = _finalize_contract_response( response, diff --git a/tests/mcp/test_max_cost.py b/tests/mcp/test_max_cost.py new file mode 100644 index 0000000..13dec03 --- /dev/null +++ b/tests/mcp/test_max_cost.py @@ -0,0 +1,500 @@ +"""GH#576: the ``max_cost`` budget ceiling on the panel-running MCP tools. + +The CLI's ``--max-cost`` projected-total spend gate previously had no MCP +analog — agents could only budget-gate by piloting small, reading in-band +``total_cost``, and scaling manually. ``run_panel`` / ``run_quick_poll`` / +``extend_panel`` now accept ``max_cost`` (USD) and wire it into the same +:class:`~synth_panel.cost.CostGate` machinery the CLI uses, with the same +soft-halt semantics: in-flight panelists finish, no new panelists start, +synthesis is skipped, and the response is a valid *partial* envelope with +``run_invalid: true``, ``cost_exceeded: true``, ``abort_reason: +"cost_exceeded"``, ``halted_at_panelist``, the ``cost_gate`` snapshot, and +an agent-legible ``resume`` block. + +Only the LLM boundary (``run_panel_parallel`` / ``synthesize_panel``) is +stubbed; the MCP tool functions, ``run_panel_sync``, and the CostGate run +for real. +""" + +from __future__ import annotations + +import json +from unittest.mock import MagicMock, patch + +import pytest + +pytest.importorskip("mcp") + + +@pytest.fixture(autouse=True) +def _data_dir(tmp_path, monkeypatch): + monkeypatch.setenv("SYNTH_PANEL_DATA_DIR", str(tmp_path)) + monkeypatch.setenv("ANTHROPIC_API_KEY", "test-key-placeholder") + monkeypatch.delenv("SYNTHPANEL_SCHEMA_MIN", raising=False) + monkeypatch.delenv("SYNTHPANEL_DRIFT_DEGRADE", raising=False) + + +from .test_decision_wiring import _stub_synthesize_panel, _StubMcpContext + +_VALID_DECISION = "Should we ship the new pricing tier next quarter?" +_PERSONAS = [{"name": "Alice"}, {"name": "Bob"}, {"name": "Cara"}] +_QUESTIONS = [{"text": "What would you pay?"}] + + +def _gate_aware_run_panel_parallel(per_panelist_cost: float, captured: dict): + """Stub for ``run_panel_parallel`` that honors the cost gate for real. + + Emulates the orchestrator's soft-halt loop: each persona "completes" + in order, its cost is recorded against the gate, and once the gate + halts no further personas are dispatched — exactly the contract + ``run_panel_parallel`` documents. The real :class:`CostGate` does the + projection math; nothing about the trip is faked. + """ + from synth_panel.cost import TokenUsage + from synth_panel.orchestrator import PanelistResult + from synth_panel.persistence import ConversationMessage, Session + + def _fake(client=None, personas=None, questions=None, model=None, sessions=None, cost_gate=None, **kwargs): + captured["cost_gate"] = cost_gate + captured["personas"] = list(personas or []) + results = [] + out_sessions = dict(sessions or {}) + for p in personas or []: + if cost_gate is not None and cost_gate.should_halt(): + break + name = p.get("name", "anon") + results.append( + PanelistResult( + persona_name=name, + responses=[{"question": (questions or [{}])[0].get("text", ""), "response": "I would pay $49."}], + usage=TokenUsage(input_tokens=5, output_tokens=3), + model=model, + ) + ) + sess = out_sessions.get(name) + if not isinstance(sess, Session): + sess = Session() + sess.push_message(ConversationMessage(role="user", content=[{"type": "text", "text": "q"}])) + sess.push_message(ConversationMessage(role="assistant", content=[{"type": "text", "text": "a"}])) + out_sessions[name] = sess + if cost_gate is not None: + # Record a fixed priced cost per panelist (the real path + # prices pr.usage; the gate math under test is identical). + cost_gate.record(per_panelist_cost) + return results, {}, out_sessions + + return _fake + + +async def _run_panel_tool(max_cost, per_panelist_cost: float = 10.0, captured: dict | None = None, **extra): + from synth_panel.mcp import server as _server + + captured = captured if captured is not None else {} + synth_mock = MagicMock(side_effect=_stub_synthesize_panel) + with ( + patch( + "synth_panel._runners.run_panel_parallel", + side_effect=_gate_aware_run_panel_parallel(per_panelist_cost, captured), + ), + patch("synth_panel._runners.synthesize_panel", synth_mock), + patch("synth_panel.mcp.server._shared_client", None), + patch("synth_panel.mcp.server.LLMClient", MagicMock()), + ): + raw = await _server.run_panel( + personas=list(_PERSONAS), + questions=list(_QUESTIONS), + model="haiku", + max_cost=max_cost, + decision_being_informed=_VALID_DECISION, + ctx=_StubMcpContext(), + **extra, + ) + return json.loads(raw), captured, synth_mock + + +# --------------------------------------------------------------------------- +# Param plumbing +# --------------------------------------------------------------------------- + + +class TestMaxCostPlumbing: + @pytest.mark.asyncio + async def test_run_panel_builds_gate_from_max_cost(self): + """max_cost reaches run_panel_parallel as a real CostGate sized to the panel.""" + from synth_panel.cost import CostGate + + data, captured, _synth = await _run_panel_tool(max_cost=50.0, per_panelist_cost=0.01) + gate = captured["cost_gate"] + assert isinstance(gate, CostGate) + assert gate.max_cost_usd == 50.0 + assert gate.total_panelists == len(_PERSONAS) + # Generous ceiling: run completes normally, no partial markers. + assert "cost_exceeded" not in data + assert data.get("run_invalid") is not True + assert data.get("synthesis") is not None + + @pytest.mark.asyncio + async def test_run_panel_without_max_cost_passes_no_gate(self): + data, captured, _synth = await _run_panel_tool(max_cost=None) + assert captured["cost_gate"] is None + assert "cost_exceeded" not in data + assert "cost_gate" not in data + + @pytest.mark.asyncio + async def test_run_quick_poll_builds_gate(self): + from synth_panel.cost import CostGate + from synth_panel.mcp import server as _server + + captured: dict = {} + with ( + patch( + "synth_panel._runners.run_panel_parallel", + side_effect=_gate_aware_run_panel_parallel(0.01, captured), + ), + patch("synth_panel._runners.synthesize_panel", side_effect=_stub_synthesize_panel), + patch("synth_panel.mcp.server._shared_client", None), + patch("synth_panel.mcp.server.LLMClient", MagicMock()), + ): + raw = await _server.run_quick_poll( + question="What would you pay?", + personas=list(_PERSONAS), + model="haiku", + max_cost=25.0, + decision_being_informed=_VALID_DECISION, + ctx=_StubMcpContext(), + ) + data = json.loads(raw) + gate = captured["cost_gate"] + assert isinstance(gate, CostGate) + assert gate.max_cost_usd == 25.0 + assert gate.total_panelists == len(_PERSONAS) + assert "cost_exceeded" not in data + + +# --------------------------------------------------------------------------- +# Gate-trip envelope shape +# --------------------------------------------------------------------------- + + +class TestGateTripEnvelope: + @pytest.mark.asyncio + async def test_run_panel_trip_is_typed_partial(self): + # $10/panelist against a $5 cap for 3 personas: the first completion + # projects $30 > $5 and halts — 1 completed, 2 never dispatched. + data, _captured, _synth = await _run_panel_tool(max_cost=5.0, per_panelist_cost=10.0) + + assert data["run_invalid"] is True + assert data["cost_exceeded"] is True + assert data["abort_reason"] == "cost_exceeded" + assert data["halted_at_panelist"] == 1 + + snap = data["cost_gate"] + assert snap["halted"] is True + assert snap["max_cost_usd"] == 5.0 + assert snap["running_cost_usd"] == pytest.approx(10.0) + assert snap["projected_total_usd"] == pytest.approx(30.0) + assert snap["completed"] == 1 + assert snap["total_panelists"] == 3 + + # Spend so far, the cap, and how to resume — all agent-legible. + resume = data["resume"] + assert resume["completed_panelists"] == ["Alice"] + assert resume["remaining_personas"] == ["Bob", "Cara"] + assert resume["partial_result_id"] == data["result_id"] + assert "get_panel_result" in resume["how_to_resume"] + assert any("cost_exceeded" in w for w in data.get("warnings", [])) + + @pytest.mark.asyncio + async def test_trip_skips_synthesis(self): + data, _captured, synth_mock = await _run_panel_tool(max_cost=5.0, per_panelist_cost=10.0) + synth_mock.assert_not_called() + assert data.get("synthesis") is None + + @pytest.mark.asyncio + async def test_trip_composes_with_detail_summary_and_verdict(self): + """The partial envelope still carries panel_verdict and honors detail.""" + data, _captured, _synth = await _run_panel_tool(max_cost=5.0, per_panelist_cost=10.0, detail="summary") + assert data.get("detail") == "summary" + assert "results" not in data # transcript dropped under summary + assert isinstance(data.get("panel_verdict"), dict) + assert data.get("schema_version") == "1.0.0" + # The persisted partial is retrievable. + assert data.get("transcript_uri") == f"panel-result://{data['result_id']}" + + @pytest.mark.asyncio + async def test_trip_full_detail_returns_partial_transcript(self): + data, _captured, _synth = await _run_panel_tool(max_cost=5.0, per_panelist_cost=10.0, detail="full") + assert [r["persona"] for r in data["results"]] == ["Alice"] + + @pytest.mark.asyncio + async def test_run_quick_poll_trip_envelope(self): + from synth_panel.mcp import server as _server + + captured: dict = {} + with ( + patch( + "synth_panel._runners.run_panel_parallel", + side_effect=_gate_aware_run_panel_parallel(10.0, captured), + ), + patch("synth_panel._runners.synthesize_panel", side_effect=_stub_synthesize_panel), + patch("synth_panel.mcp.server._shared_client", None), + patch("synth_panel.mcp.server.LLMClient", MagicMock()), + ): + raw = await _server.run_quick_poll( + question="What would you pay?", + personas=list(_PERSONAS), + model="haiku", + max_cost=5.0, + decision_being_informed=_VALID_DECISION, + ctx=_StubMcpContext(), + ) + data = json.loads(raw) + assert data["run_invalid"] is True + assert data["cost_exceeded"] is True + assert data["abort_reason"] == "cost_exceeded" + assert data["cost_gate"]["halted"] is True + assert data["resume"]["remaining_personas"] == ["Bob", "Cara"] + + +# --------------------------------------------------------------------------- +# extend_panel +# --------------------------------------------------------------------------- + + +def _extend_fake_parallel(per_panelist_cost: float, captured: dict): + from synth_panel.cost import TokenUsage + from synth_panel.orchestrator import PanelistResult + + def _fake(client=None, personas=None, questions=None, model=None, sessions=None, cost_gate=None, **kwargs): + captured["cost_gate"] = cost_gate + results = [] + for p in personas or []: + if cost_gate is not None and cost_gate.should_halt(): + break + results.append( + PanelistResult( + persona_name=p.get("name", "anon"), + responses=[{"question": "follow-up?", "response": "sure"}], + usage=TokenUsage(input_tokens=5, output_tokens=3), + model=model, + ) + ) + if cost_gate is not None: + cost_gate.record(per_panelist_cost) + return results, {}, dict(sessions or {}) + + return _fake + + +class TestExtendPanelMaxCost: + async def _call(self, max_cost, per_panelist_cost=10.0): + from synth_panel.mcp import server as _server + + captured: dict = {} + fake_existing = {"rounds": [], "path": [], "question_count": 0} + fake_sessions = {"Alice": object(), "Bob": object(), "Cara": object()} + synth_mock = MagicMock(side_effect=_stub_synthesize_panel) + with ( + patch("synth_panel.mcp.server._data_get_panel_result", return_value=fake_existing), + patch("synth_panel.mcp.server.load_panel_sessions", return_value=fake_sessions), + patch( + "synth_panel.mcp.server.run_panel_parallel", + side_effect=_extend_fake_parallel(per_panelist_cost, captured), + ), + patch("synth_panel.mcp.server.synthesize_panel", synth_mock), + patch("synth_panel.mcp.server.update_panel_result"), + patch("synth_panel.mcp.server._get_shared_client", return_value=object()), + ): + raw = await _server.extend_panel( + result_id="r-576", + questions=[{"text": "follow-up?"}], + model="haiku", + max_cost=max_cost, + decision_being_informed=_VALID_DECISION, + ctx=None, + ) + return json.loads(raw), captured, synth_mock + + @pytest.mark.asyncio + async def test_gate_plumbed_and_no_trip(self): + from synth_panel.cost import CostGate + + data, captured, _synth = await self._call(max_cost=100.0, per_panelist_cost=0.01) + gate = captured["cost_gate"] + assert isinstance(gate, CostGate) + assert gate.max_cost_usd == 100.0 + assert gate.total_panelists == 3 + assert "cost_exceeded" not in data + + @pytest.mark.asyncio + async def test_trip_envelope_and_synthesis_skip(self): + data, _captured, synth_mock = await self._call(max_cost=5.0, per_panelist_cost=10.0) + assert data["run_invalid"] is True + assert data["cost_exceeded"] is True + assert data["abort_reason"] == "cost_exceeded" + assert data["halted_at_panelist"] == 1 + assert data["cost_gate"]["halted"] is True + assert data["resume"]["partial_result_id"] == "r-576" + assert data["resume"]["remaining_personas"] == ["Bob", "Cara"] + synth_mock.assert_not_called() + assert data.get("synthesis") is None + # The partial round is still appended/returned. + assert [r["persona"] for r in data["results"]] == ["Alice"] + + @pytest.mark.asyncio + async def test_no_max_cost_passes_no_gate(self): + data, captured, _synth = await self._call(max_cost=None, per_panelist_cost=10.0) + assert captured["cost_gate"] is None + assert "cost_exceeded" not in data + + +# --------------------------------------------------------------------------- +# Boundary validation and unsupported combinations +# --------------------------------------------------------------------------- + + +class TestMaxCostValidation: + @pytest.mark.asyncio + @pytest.mark.parametrize("bad", [0, -1, -0.5]) + async def test_non_positive_rejected_run_panel(self, bad): + from synth_panel.mcp import server as _server + + data = json.loads( + await _server.run_panel( + personas=list(_PERSONAS), + questions=list(_QUESTIONS), + max_cost=bad, + decision_being_informed=_VALID_DECISION, + ctx=_StubMcpContext(), + ) + ) + assert data["error_code"] == "INVALID_TOOL_ARG" + assert data["field_path"] == "max_cost" + + @pytest.mark.asyncio + async def test_non_positive_rejected_run_quick_poll(self): + from synth_panel.mcp import server as _server + + data = json.loads( + await _server.run_quick_poll( + question="Q?", + max_cost=0, + decision_being_informed=_VALID_DECISION, + ctx=_StubMcpContext(), + ) + ) + assert data["error_code"] == "INVALID_TOOL_ARG" + assert data["field_path"] == "max_cost" + + @pytest.mark.asyncio + async def test_non_positive_rejected_extend_panel(self): + from synth_panel.mcp import server as _server + + data = json.loads( + await _server.extend_panel( + result_id="r-576", + questions=[{"text": "q?"}], + max_cost=-3, + decision_being_informed=_VALID_DECISION, + ctx=None, + ) + ) + assert data["error_code"] == "INVALID_TOOL_ARG" + assert data["field_path"] == "max_cost" + + @pytest.mark.asyncio + async def test_rejected_with_ensemble_models(self): + from synth_panel.mcp import server as _server + + data = json.loads( + await _server.run_panel( + personas=list(_PERSONAS), + questions=list(_QUESTIONS), + models=["haiku", "gpt-4o-mini"], + max_cost=5.0, + decision_being_informed=_VALID_DECISION, + ctx=_StubMcpContext(), + ) + ) + assert data["error_code"] == "INVALID_TOOL_ARG" + assert data["field_path"] == "max_cost" + assert "ensemble" in data["message"] + + @pytest.mark.asyncio + async def test_rejected_with_instrument(self): + from synth_panel.mcp import server as _server + + data = json.loads( + await _server.run_panel( + personas=list(_PERSONAS), + instrument={"questions": [{"text": "q?"}]}, + max_cost=5.0, + decision_being_informed=_VALID_DECISION, + ctx=_StubMcpContext(), + ) + ) + assert data["error_code"] == "INVALID_TOOL_ARG" + assert data["field_path"] == "max_cost" + assert "instrument" in data["message"] + + @pytest.mark.asyncio + async def test_rejected_with_variants(self): + from synth_panel.mcp import server as _server + + data = json.loads( + await _server.run_panel( + personas=list(_PERSONAS), + questions=list(_QUESTIONS), + variants=2, + max_cost=5.0, + decision_being_informed=_VALID_DECISION, + ctx=_StubMcpContext(), + ) + ) + assert data["error_code"] == "INVALID_TOOL_ARG" + assert data["field_path"] == "max_cost" + assert "variants" in data["message"] + + @pytest.mark.asyncio + async def test_rejected_in_sampling_mode_run_panel(self): + from synth_panel.mcp import server as _server + from synth_panel.mcp.sampling import SamplingDecision + + with patch( + "synth_panel.mcp.server._decide_sampling_mode", + return_value=SamplingDecision(mode="sampling"), + ): + data = json.loads( + await _server.run_panel( + personas=list(_PERSONAS), + questions=list(_QUESTIONS), + max_cost=5.0, + decision_being_informed=_VALID_DECISION, + ctx=_StubMcpContext(), + ) + ) + assert data["error_code"] == "INVALID_TOOL_ARG" + assert data["field_path"] == "max_cost" + assert "sampling" in data["message"] + + @pytest.mark.asyncio + async def test_rejected_in_sampling_mode_run_quick_poll(self): + from synth_panel.mcp import server as _server + from synth_panel.mcp.sampling import SamplingDecision + + with patch( + "synth_panel.mcp.server._decide_sampling_mode", + return_value=SamplingDecision(mode="sampling"), + ): + data = json.loads( + await _server.run_quick_poll( + question="Q?", + personas=list(_PERSONAS), + max_cost=5.0, + decision_being_informed=_VALID_DECISION, + ctx=_StubMcpContext(), + ) + ) + assert data["error_code"] == "INVALID_TOOL_ARG" + assert data["field_path"] == "max_cost" + assert "sampling" in data["message"]