Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
72 changes: 69 additions & 3 deletions docs/mcp.md
Original file line number Diff line number Diff line change
Expand Up @@ -227,9 +227,9 @@ and log/stream discipline — is documented in
| Tool | Description |
|------|-------------|
| `run_prompt` | Send a single prompt to an LLM. No personas required. The simplest tool — ask a quick research question. **Does not require `decision_being_informed`.** |
| `run_panel` | Run a full synthetic focus group panel. Each persona answers all questions independently in parallel, followed by synthesis. Accepts inline `questions`, an inline `instrument` dict (v1/v2/v3), or an `instrument_pack` name. Template placeholders in the instrument (e.g. the bundled packs' `{problem}` / `{candidates}`) are filled via the `vars` argument — the MCP equivalent of the CLI's `--var`. **Requires `decision_being_informed`.** |
| `run_quick_poll` | Quick single-question poll across personas. A simplified `run_panel` for one question with synthesis. Accepts inline `personas` and/or a saved `pack_id` (merged; falls back to a built-in diverse set when both are omitted). **Requires `decision_being_informed`.** |
| `extend_panel` | Append a single ad-hoc round to a saved panel result. Reuses each panelist's saved session for conversational context. **Not** a re-entry into the v3 DAG — use for human-in-the-loop follow-ups. **Requires `decision_being_informed`.** |
| `run_panel` | Run a full synthetic focus group panel. Each persona answers all questions independently in parallel, followed by synthesis. Accepts inline `questions`, an inline `instrument` dict (v1/v2/v3), or an `instrument_pack` name. Template placeholders in the instrument (e.g. the bundled packs' `{problem}` / `{candidates}`) are filled via the `vars` argument — the MCP equivalent of the CLI's `--var`. Optional `max_cost` (USD) arms the mid-run cost gate (see [`max_cost`](#max_cost-hard-spend-ceiling)). **Requires `decision_being_informed`.** |
| `run_quick_poll` | Quick single-question poll across personas. A simplified `run_panel` for one question with synthesis. Accepts inline `personas` and/or a saved `pack_id` (merged; falls back to a built-in diverse set when both are omitted). Optional `max_cost` (USD) spend ceiling. **Requires `decision_being_informed`.** |
| `extend_panel` | Append a single ad-hoc round to a saved panel result. Reuses each panelist's saved session for conversational context. **Not** a re-entry into the v3 DAG — use for human-in-the-loop follow-ups. Optional `max_cost` (USD) spend ceiling for the extension round. **Requires `decision_being_informed`.** |

### Tool-call examples

Expand Down Expand Up @@ -407,6 +407,72 @@ saved panel.
is not retrievable later — sampling always returns full transcripts regardless of
`detail`. `extend_panel` returns its single appended round in full.

### `max_cost`: hard spend ceiling

`run_panel`, `run_quick_poll`, and `extend_panel` accept an optional
**`max_cost`** argument — a hard ceiling on the run's total spend, in USD. It
is the MCP analog of the CLI's `--max-cost` and wires into the same `CostGate`
machinery (`src/synth_panel/cost.py`): after each panelist completes,
`running_cost / completed_n * total_n` is compared against the ceiling.

```jsonc
// run_panel with a $2 ceiling
{
"tool": "run_panel",
"arguments": {
"pack_id": "general-consumer",
"questions": [{ "text": "What would you pay for this?" }],
"max_cost": 2.0,
"decision_being_informed": "choosing launch tier price"
}
}
```

When the projection exceeds the ceiling the run **soft-halts**: the current
panelist(s) finish, no new panelists start, and synthesis is skipped (spending
more to summarize a deliberately-truncated panel would produce an
untrustworthy result). The response is still a **valid partial envelope** —
the completed prefix is persisted under `result_id` as usual — carrying:

```jsonc
// response (truncated) after a cost-gate trip
{
"run_invalid": true,
"cost_exceeded": true,
"abort_reason": "cost_exceeded",
"halted_at_panelist": 4,
"cost_gate": {
"max_cost_usd": 2.0,
"running_cost_usd": 1.62, // spend so far
"projected_total_usd": 4.05, // what tripped the gate
"completed": 4,
"total_panelists": 10,
"halted": true,
"halted_projection_usd": 4.05
},
"resume": {
"partial_result_id": "result-20260716-...",
"completed_panelists": ["..."],
"remaining_personas": ["..."],
"how_to_resume": "..." // fetch partial via get_panel_result;
// re-run remaining_personas with a raised cap
},
"synthesis": null // skipped on the partial
}
```

The partial envelope composes with `detail: "summary"` and still carries the
`panel_verdict` contract fields; the full partial transcript stays retrievable
via `get_panel_result(result_id)`.

Support matrix (mirrors the CLI's `--max-cost`, which the multi-round engine
refuses): `max_cost` applies to **BYOK** runs with inline `questions` (and
`extend_panel`'s single extension round). Combining it with sampling mode (the
host agent pays; the server sees no per-panelist costs), `models` ensembles,
`variants`, or `instrument` / `instrument_pack` inputs returns a typed
`INVALID_TOOL_ARG` on `max_cost` — a loud refusal before any spend, never a
silently unenforced ceiling. `max_cost` must be > 0.

### Typed error envelopes

Boundary errors return a typed `INVALID_TOOL_ARG` envelope
Expand Down
17 changes: 16 additions & 1 deletion docs/production-operations.md
Original file line number Diff line number Diff line change
Expand Up @@ -158,6 +158,18 @@ Two overflow safeguards on the synthesis step itself:
`halted_at_panelist`, and a `cost_gate` snapshot. Exit code `2`.
(`CostGate` in `src/synth_panel/cost.py`; wiring in
`src/synth_panel/cli/commands.py`.)
- **MCP `max_cost` (GH#576)** — the same gate on the MCP surface: the
`run_panel`, `run_quick_poll`, and `extend_panel` tools accept a
`max_cost` (USD) argument wired into the identical `CostGate` machinery
with the same soft-halt semantics. On a trip, the tool response is a
valid partial envelope with `run_invalid: true`, `cost_exceeded: true`,
`abort_reason: "cost_exceeded"`, `halted_at_panelist`, the `cost_gate`
snapshot, and an agent-legible `resume` block (persisted partial
`result_id`, completed panelists, remaining personas). Synthesis is
skipped on the partial. BYOK inline-`questions` runs only — sampling
mode, `models` ensembles, `variants`, and instrument inputs refuse
`max_cost` with a typed `INVALID_TOOL_ARG` (parity with the CLI's
multi-round refusal). See [docs/mcp.md](mcp.md#max_cost-hard-spend-ceiling).
- **Per-turn telemetry** — token usage is tracked per turn in four buckets
(input / output / cache-write / cache-read; `TokenUsage` and
`UsageTracker` in `src/synth_panel/cost.py`). Every panelist row and the
Expand Down Expand Up @@ -286,4 +298,7 @@ Kept here so this page stays trustworthy:
`--question-failure-budget` apply to single-round runs; multi-round
(branching) instruments refuse these flags loudly up front rather than
degrading silently (`_multi_round_flag_errors`,
`src/synth_panel/cli/commands.py`).
`src/synth_panel/cli/commands.py`). The MCP `max_cost` argument mirrors
the same matrix: instrument inputs (which always dispatch through the
multi-round engine on the MCP surface), ensembles, variants, and
sampling mode refuse it with a typed `INVALID_TOOL_ARG`.
33 changes: 23 additions & 10 deletions site/mcp/index.html
Original file line number Diff line number Diff line change
Expand Up @@ -1133,12 +1133,23 @@ <h3 class="mt-8 mb-3 text-base font-semibold text-white">
<code class="text-slate-300">cost</code>, so the loop is: pilot
with <code class="text-slate-300">run_quick_poll</code>, gate on
the returned cost, then scale to a full
<code class="text-slate-300">run_panel</code>. Note the hard
mid-run spend ceiling
(<code class="text-slate-300">--max-cost</code>) is a CLI flag —
there is no <code class="text-slate-300">max_cost</code> tool
argument on the MCP surface; unattended hard ceilings mean
driving the CLI.
<code class="text-slate-300">run_panel</code>. For unattended
hard ceilings, pass
<code class="text-slate-300">max_cost</code> (USD) to
<code class="text-slate-300">run_panel</code> /
<code class="text-slate-300">run_quick_poll</code> /
<code class="text-slate-300">extend_panel</code> — the MCP
analog of the CLI's
<code class="text-slate-300">--max-cost</code>, wired into the
same mid-run cost gate. On a trip the response is a valid
partial envelope with
<code class="text-slate-300">cost_exceeded: true</code>,
<code class="text-slate-300">abort_reason:
"cost_exceeded"</code>, a
<code class="text-slate-300">cost_gate</code> snapshot (spend
so far, cap, projection), and a
<code class="text-slate-300">resume</code> block naming the
persisted partial and the remaining personas.
</dd>
</div>
</dl>
Expand Down Expand Up @@ -1222,11 +1233,13 @@ <h3 class="mt-8 mb-3 text-base font-semibold text-white">
<tr>
<td>Cost ceiling hit</td>
<td>
<code>--max-cost</code> halts gracefully mid-run: pending
panelists cancelled, valid partial JSON with
<code>run_invalid: true</code>,
<code>--max-cost</code> (CLI) or the <code>max_cost</code>
tool argument (MCP) halts gracefully mid-run: pending
panelists cancelled, synthesis skipped, valid partial JSON
with <code>run_invalid: true</code>,
<code>cost_exceeded: true</code>,
<code>halted_at_panelist</code>. Exit <code>2</code>.
<code>halted_at_panelist</code>, and (MCP) a
<code>resume</code> block. CLI exit <code>2</code>.
</td>
</tr>
<tr>
Expand Down
4 changes: 2 additions & 2 deletions site/mcp/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -289,7 +289,7 @@ Summary first, transcripts on demand `run_panel` and `run_quick_poll` default to

Structured polling — no prose parsing Both run tools accept a `response_schema` argument (JSON Schema) that forces structured output at generation time — forced-choice, Likert, tagged themes, ranking — so downstream code branches on fields, not on regexes over prose. Pattern catalogue: [docs/structured-polling.md](https://github.com/DataViking-Tech/SynthPanel/blob/main/docs/structured-polling.md).

Budget-gated iteration MCP mode defaults to `haiku` and caps panels at 100 personas × 50 questions. Every response carries in-band `total_cost` and per-panelist `usage` / `cost`, so the loop is: pilot with `run_quick_poll`, gate on the returned cost, then scale to a full `run_panel`. Note the hard mid-run spend ceiling (`--max-cost`) is a CLI flag — there is no `max_cost` tool argument on the MCP surface; unattended hard ceilings mean driving the CLI.
Budget-gated iteration MCP mode defaults to `haiku` and caps panels at 100 personas × 50 questions. Every response carries in-band `total_cost` and per-panelist `usage` / `cost`, so the loop is: pilot with `run_quick_poll`, gate on the returned cost, then scale to a full `run_panel`. For unattended hard ceilings, pass `max_cost` (USD) to `run_panel` / `run_quick_poll` / `extend_panel` — the MCP analog of the CLI's `--max-cost`, wired into the same mid-run cost gate. On a trip the response is a valid partial envelope with `cost_exceeded: true`, `abort_reason: "cost_exceeded"`, a `cost_gate` snapshot (spend so far, cap, projection), and a `resume` block naming the persisted partial and the remaining personas.

Full tool semantics: [docs/mcp.md](https://github.com/DataViking-Tech/SynthPanel/blob/main/docs/mcp.md) · operational contract: [docs/production-operations.md](https://github.com/DataViking-Tech/SynthPanel/blob/main/docs/production-operations.md).

Expand All @@ -302,7 +302,7 @@ What actually happens when a panel run goes sideways. Failures return typed enve
| K of N panelists fail | Errors are recorded per persona and per question (`error` field per row), the panel continues, and `failure_stats` reports the rate. The run is invalid only past `--failure-threshold` (default 0.5) — then synthesis is auto-disabled and the CLI exits `2`. `--strict` = zero tolerance. |
| Synthesis fails | Never shaped like success: the envelope carries `synthesis_error` and the run is marked invalid — but panelist data is saved. Recover post-hoc without re-running the panel: `synthpanel panel synthesize <result-id> --synthesis-model sonnet`. A synthesis-model outage costs one cheap retry, not the panel spend. |
| Rate limits | `--max-concurrent` caps in-flight requests; `--rate-limit-rps` adds a token-bucket cap (fractional values accepted). Rate-limit exhaustion is classified distinctly in `abort_reason` — the signal is "raise the throttle," not "chase a model bug." |
| Cost ceiling hit | `--max-cost` halts gracefully mid-run: pending panelists cancelled, valid partial JSON with `run_invalid: true`, `cost_exceeded: true`, `halted_at_panelist`. Exit `2`. |
| Cost ceiling hit | `--max-cost` (CLI) or the `max_cost` tool argument (MCP) halts gracefully mid-run: pending panelists cancelled, synthesis skipped, valid partial JSON with `run_invalid: true`, `cost_exceeded: true`, `halted_at_panelist`, and (MCP) a `resume` block. CLI exit `2`. |
| SIGINT / SIGTERM | With checkpointing active, a final checkpoint is flushed and the run marked aborted; stdout still emits a complete, parseable envelope with `abort_reason: "sigint"`. Resume picks up where it stopped. |
| Every panelist fails | Loud, not shaped like success: a structured total-failure diagnostic names the upstream cause (bad model id, provider 400). MCP serializes `{"error", "run_invalid": true, "total_failure"}`; the CLI exits `2`. |

Expand Down
24 changes: 23 additions & 1 deletion src/synth_panel/_runners.py
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@

from pydantic import BaseModel

from synth_panel.cost import ZERO_USAGE, resolve_cost
from synth_panel.cost import ZERO_USAGE, CostGate, resolve_cost
from synth_panel.cost import TokenUsage as CostTokenUsage
from synth_panel.instrument import Instrument
from synth_panel.llm.client import LLMClient
Expand Down Expand Up @@ -691,6 +691,7 @@ def run_panel_sync(
attachment_bank: dict[str, dict[str, Any]] | None = None,
allow_empty_attachments: bool = False,
sessions_out: dict[str, Any] | None = None,
cost_gate: CostGate | None = None,
) -> tuple[
list[PanelistResult],
list[dict[str, Any]],
Expand All @@ -711,6 +712,16 @@ def run_panel_sync(
keyed by persona name. Kept as an out-param rather than a seventh
tuple element so existing unpack sites stay valid. The MCP server
uses this to persist AC-7 decision-stamped transcripts.

``cost_gate``: optional :class:`~synth_panel.cost.CostGate` (sp-utnk /
GH#576). Threaded to :func:`run_panel_parallel`, which records each
completing panelist's priced cost and soft-halts dispatch when the
projected run total exceeds the ceiling. When the gate has halted,
synthesis is skipped (same rationale as the CLI: synthesizing a
deliberately-truncated panel burns more budget without producing a
trustworthy result). The caller keeps the gate reference and is
expected to inspect ``cost_gate.should_halt()`` on return to shape
its partial-result envelope.
"""
all_personas = list(personas)
variant_names: set[str] = set()
Expand All @@ -728,6 +739,9 @@ def run_panel_sync(
variant_count += 1
logger.info("Running panel with %d base + %d variant personas", len(personas), variant_count)

# GH#576: pass the gate only when armed so existing monkeypatched
# test doubles / caller shims with explicit signatures keep working.
_gate_kwargs: dict[str, Any] = {"cost_gate": cost_gate} if cost_gate is not None else {}
panelist_results, _registry, _sessions = run_panel_parallel(
client=client,
personas=all_personas,
Expand All @@ -743,6 +757,7 @@ def run_panel_sync(
extract_schema=extract_schema,
attachment_bank=attachment_bank,
allow_empty_attachments=allow_empty_attachments,
**_gate_kwargs,
)
if sessions_out is not None:
sessions_out.update(_sessions)
Expand Down Expand Up @@ -773,6 +788,13 @@ def run_panel_sync(
# ``synthesis_dict``. Callers (MCP server / SDK) detect this and lift
# it to ``run_invalid=True`` + top-level ``synthesis_error`` on the
# result envelope.
# GH#576: mirror the CLI's cost-gate discipline — a halted run is a
# deliberately-truncated partial, so spending more budget synthesizing
# it would produce an untrustworthy result. Skip synthesis entirely.
cost_gate_halted = cost_gate is not None and cost_gate.should_halt()
if synthesis and cost_gate_halted:
logger.warning("cost gate halted the panel; skipping synthesis on the partial result")
synthesis = False
if synthesis:
Comment on lines +794 to 798

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Account for synthesis before spending beyond max_cost

When all panelist completions leave the projection at or below max_cost, this condition permits synthesis even though CostGate.record() is only called for panelist costs in orchestrator.py:1693-1698. For example, a panel costing $1 under a $2 cap can still issue a $5 synthesis request and return a $6 total without any cost_exceeded marker. Thus any capped call with synthesis enabled can exceed the documented hard total-spend ceiling; synthesis must be budgeted or refused when the remaining ceiling cannot cover it.

Useful? React with 👍 / 👎.

synth_model_for_check = synthesis_model or model
overflow = detect_synthesis_context_overflow(
Expand Down
Loading
Loading