Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
44 changes: 44 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,14 @@ For auto-generated release notes, see [GitHub Releases](https://github.com/DataV

### Added

- **`cost show <result-id>`.** Per-run cost breakdown for one saved
panel result — total/panelist cost, per-model token + USD rollup, and
synthesis cost — as a thin alias over the cost section of `panel
inspect` (same result resolution: ID or JSON path). Previously the
`cost` subcommand only had `summary` (cross-run aggregate) and per-run
cost required reading the full `panel inspect` output. The
"Result saved" follow-up hint now lists it.

- **Synthesis failure recovery ladder (sp-rcvr).** A thrice-reproduced
production failure — a 20-persona panel whose questions embed ~14k-char
page dumps passed the synthesis pre-flight yet deterministically failed
Expand Down Expand Up @@ -41,6 +49,42 @@ For auto-generated release notes, see [GitHub Releases](https://github.com/DataV

### Fixed

- **A one-entry `--models` spec no longer routes through the ensemble
path, silently skipping synthesis.** `panel run --models <one-model>`
(weight-free spec, single entry) was classified as an "ensemble" and
took the ensemble branch, which never runs synthesis, never engages the
standard-path cost summary, and exited 0 with a bare "Ensemble
complete" on stdout — a naive single-model run got no synthesis and no
signal that anything was skipped. Single-entry specs are now demoted to
the standard single-model path (identical to `--model X`, synthesis
included) with a stderr note; ensemble comparison requires two or more
models.
- **Runs with token usage no longer record $0.0000 when the provider
reports a zero cost.** OpenRouter returns `usage.cost: 0` for BYOK keys
(and some upstreams omit native cost); `resolve_cost` trusted the zero
verbatim, so a run with tens of thousands of tokens recorded
`total_cost $0.0000` / `cost_usd 0.0` — and, because the orchestrator's
cost gate accrues via the same path, `--max-cost` could never trip.
A provider-reported $0 for usage the local pricing table prices as
nonzero now falls back to the local estimate (with a stderr-visible
warning); genuinely free models still record $0. Dry-run estimates and
synthesis costs (which always used the local table) were already
correct — the mismatch between them and the $0 run cost was the tell.
- **Ensemble runs now state synthesis status and results location on
stdout.** Text-mode output for a completed (>=2 model) ensemble run was
the single line "Ensemble complete"; it now prints a terse summary —
models x personas x questions, incident count, recorded cost,
"Synthesis: skipped" with the exact `panel synthesize <result-id>`
follow-up (or the `--save` prerequisite), and where the results live.
JSON output carries `synthesis: null` + `synthesis_status: "skipped"`
explicitly. Single-model text runs likewise end with a RUN SUMMARY
block (personas, questions, errors, recorded cost, synthesis status,
results path or the `--save` hint).
- **`--personas` / `--instrument` help text documents bundled names.**
Both flags said "Path to a YAML file..." although bundled
pack/instrument names (e.g. `general-consumer`, `general-survey`) are
accepted and are the primary happy path; the help now points at
`pack list` / `instruments list` and keeps the YAML-path alternative.
- **Synthesis judge final-strike escalation now stays in the run's
provider family for every provider (GH#571, sy-549 class).** A native
`--model gemini` run with only `GEMINI_API_KEY` set could complete all
Expand Down
188 changes: 185 additions & 3 deletions src/synth_panel/cli/commands.py
Original file line number Diff line number Diff line change
Expand Up @@ -1690,6 +1690,29 @@ def system_prompt_fn(persona: dict) -> str:
except ValueError as exc:
print(f"Error parsing --models: {exc}", file=sys.stderr)
return 1
# gh-r4: a one-entry weight-free spec ("--models X") is a plain
# single-model run, not an ensemble. Routing it through the
# ensemble branch silently skipped synthesis, exited 0 with a bare
# "Ensemble complete", and bypassed the standard-path cost gate /
# summary. Demote it to the same path "--model X" takes; ensemble
# comparison needs >= 2 models. (--blend keeps its own validation.)
if ensemble_mode and not blend_mode and len(model_spec) == 1:
model = model_spec[0][0]
args.model = model
has_model = model
has_models = None
model_spec = None
ensemble_mode = False
print(
f"Note: --models with a single model ({model}) runs a standard "
"panel run (synthesis included). Ensemble comparison requires "
"two or more models.",
file=sys.stderr,
)
if has_models:
# The demotion above nulls has_models and model_spec together, so a
# truthy has_models implies a parsed spec; assert for mypy's sake.
assert model_spec is not None
# sp-zdul: warn if weighted sum is far from 1.0 — the user likely
# intended an exact-ratio split and a typo (e.g. "0.3,0.3,0.3")
# silently reweights their panel.
Expand Down Expand Up @@ -1968,6 +1991,12 @@ def system_prompt_fn(persona: dict) -> str:
from synth_panel._runners import format_panelist_result as _fmt_panelist

output = build_ensemble_output(ens_result, panelist_formatter=_fmt_panelist)
# gh-r4: the ensemble path never runs synthesis. Say so explicitly
# in the payload instead of leaving consumers to infer it from a
# missing key ("panel inspect" showed only 'Synthesis: not run'
# with no explanation, and text stdout said nothing at all).
output["synthesis"] = None
output["synthesis_status"] = "skipped"
# sp-atvc: attach a metadata bundle whose cost.per_model covers
# every ensemble model, not just the first in the spec.
ens_per_model_meta = {mr.model: (mr.usage, mr.cost) for mr in ens_result.model_results}
Expand Down Expand Up @@ -2021,7 +2050,41 @@ def system_prompt_fn(persona: dict) -> str:
output["result_id"] = ensemble_result_id
output["saved_path"] = str(_results_dir() / f"{ensemble_result_id}.json")

emit(fmt, message="Ensemble complete", extra=output)
# gh-r4: text-mode stdout used to be the single line "Ensemble
# complete" — no cost, no error count, no synthesis status, no
# pointer to the results. Print a terse run summary instead.
# JSON/NDJSON keep the same message and carry everything in extra.
if fmt is OutputFormat.TEXT:
n_models = len(ensemble_models)
incidents = output.get("ensemble_incidents") or []
print(
f"Ensemble complete: {n_models} model(s) x "
f"{ens_result.persona_count} persona(s) x "
f"{ens_result.question_count} question(s)"
)
print(f" Models: {', '.join(ensemble_models)}")
print(f" Errors: {len(incidents)} incident(s)")
print(f" Cost: {ens_result.total_cost.format_usd()} (recorded)")
if ensemble_result_id is not None:
print(
" Synthesis: skipped — ensemble runs never synthesize "
"automatically. Run: "
f"synthpanel panel synthesize {ensemble_result_id}"
)
print(f" Results: {output['saved_path']}")
else:
print(
" Synthesis: skipped — ensemble runs never synthesize "
"automatically. Re-run with --save, then run: "
"synthpanel panel synthesize <result-id>"
)
print(
" Results: not saved — re-run with --save to persist "
"them (full per-model payload available via "
"--output-format json)"
)
else:
emit(fmt, message="Ensemble complete", extra=output)

if ensemble_result_id is not None:
_print_saved_result_hint(ensemble_result_id)
Expand Down Expand Up @@ -3106,6 +3169,31 @@ def _on_complete(pr: PanelistResult) -> None:
hb_n = convergence_baseline_payload.get("converged_at")
if hb_n is not None:
print(f" human baseline: converged_at=n={hb_n}")

# gh-r4: terse end-of-run summary so a default (text) run states
# panel size, error count, recorded cost, synthesis status, and
# where the results live — previously synthesis status and result
# location were only discoverable via --save/--output-format json.
if synthesis_dict:
synthesis_status = f"completed (cost: {synthesis_dict['cost']})"
Comment on lines +3177 to +3178

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Report fallback synthesis as failed

When synthesis returns a fallback result, the earlier is_fallback branch explicitly marks the run invalid and populates synthesis_error_payload, but the fallback still produces a truthy synthesis_dict. This condition therefore prints Synthesis: completed in the new run summary immediately after stderr reported that synthesis was incomplete, even though the command exits 2. Check the synthesis error/fallback state before treating any dictionary as a successful completion.

Useful? React with 👍 / 👎.

elif getattr(args, "no_synthesis", False):
synthesis_status = "skipped (--no-synthesis)"
elif skip_synthesis:
synthesis_status = "skipped (run halted or invalid)"
else:
synthesis_status = "FAILED — see stderr above; recover with: synthpanel panel synthesize <result-id>"
print(f"\n{'=' * 60}")
print("RUN SUMMARY")
print(f"{'=' * 60}")
print(
f" Personas: {len(personas)} Questions: {len(questions)} Errors: {failure_stats['errored_pairs']}"
)
print(f" Cost: {total_cost_est.format_usd()} (recorded)")
print(f" Synthesis: {synthesis_status}")
if saved_result_path is not None:
print(f" Results: {saved_result_path}")
else:
print(" Results: not saved — re-run with --save to persist (then: synthpanel report <result-id>)")
else:
# sp-efip: mirror the TEXT-mode behaviour and print the fatal
# banner to stderr even when JSON/NDJSON is the primary output.
Expand Down Expand Up @@ -6110,6 +6198,101 @@ def handle_cost_summary(args: argparse.Namespace, fmt: OutputFormat) -> int:
return 0


def handle_cost_show(args: argparse.Namespace, fmt: OutputFormat) -> int:
"""Per-run cost breakdown for one saved result (gh-r4).

Thin alias over the cost section of ``panel inspect`` — same result
resolution (ID or JSON path), same :class:`InspectReport` data, but
only the cost/usage fields. ``cost summary`` aggregates across runs;
this answers "what did run X cost?" without wading through the full
inspect output.
"""
from synth_panel.analysis.inspect import build_inspect_report

result_ref = args.result
path = Path(result_ref)
if path.exists() and path.suffix == ".json":
try:
data = json.loads(path.read_text(encoding="utf-8"))
except json.JSONDecodeError as exc:
msg = f"Error: not valid JSON: {result_ref}: {exc}"
if fmt is OutputFormat.TEXT:
print(msg, file=sys.stderr)
else:
emit(fmt, message=msg, extra={"error": "invalid_json"})
return 1
data.setdefault("id", path.stem)
else:
from synth_panel.mcp.data import get_panel_result

try:
data = get_panel_result(result_ref)
except FileNotFoundError:
msg = f"Error: panel result not found: {result_ref}"
if fmt is OutputFormat.TEXT:
print(msg, file=sys.stderr)
else:
emit(fmt, message=msg, extra={"error": "not_found"})
return 1

if not isinstance(data, dict):
msg = "Error: panel result is not a JSON object"
if fmt is OutputFormat.TEXT:
print(msg, file=sys.stderr)
else:
emit(fmt, message=msg, extra={"error": "invalid_shape"})
return 1

report = build_inspect_report(data)
cost_payload: dict[str, Any] = {
"result_id": report.result_id,
"total_cost": report.total_cost,
"panelist_cost": report.panelist_cost,
"total_tokens": report.total_tokens,
Comment on lines +6249 to +6251

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Populate panelist cost for actual saved results

For a normal result created by save_panel_result, panelist_cost is not persisted at the top level, so build_inspect_report leaves this field None. Consequently the new cost show <result-id> command returns panelist_cost: null in JSON and silently omits the panelist-cost line in text for its primary input type; the test only passes because its hand-written fixture includes a field real saves lack. Derive the panelist amount from persisted metadata/synthesis cost or persist it when saving.

Useful? React with 👍 / 👎.

"per_model": [
{
"model": m.model,
"total_tokens": m.total_tokens,
"input_tokens": m.input_tokens,
"output_tokens": m.output_tokens,
"cost_usd": m.cost_usd,
}
for m in report.model_rollup
],
"synthesis": {
"ran": report.synthesis.ran,
"model": report.synthesis.model,
"cost": report.synthesis.cost,
},
}

if fmt is OutputFormat.TEXT:
print(f"Cost for {report.result_id or result_ref}")
print(f" Total cost: {report.total_cost or '(not recorded)'}")
if report.panelist_cost:
print(f" Panelist cost: {report.panelist_cost}")
if report.synthesis.ran:
synth_line = report.synthesis.cost or "(not recorded)"
if report.synthesis.model:
synth_line += f" ({report.synthesis.model})"
print(f" Synthesis: {synth_line}")
else:
print(" Synthesis: not run")
print(f" Total tokens: {report.total_tokens}")
if report.model_rollup:
print(" Per model:")
for m in report.model_rollup:
cost_str = f"${m.cost_usd:.4f}" if m.cost_usd is not None else "(unknown)"
print(
f" {m.model}: {cost_str} "
f"(tokens={m.total_tokens} input={m.input_tokens} output={m.output_tokens})"
)
else:
emit(fmt, message="Cost show", extra={"cost": cost_payload})

return 0


# ---------------------------------------------------------------------------
# Instruments subcommands (sp-xsu)
# ---------------------------------------------------------------------------
Expand Down Expand Up @@ -7444,8 +7627,7 @@ def _print_saved_result_hint(result_id: str) -> None:
print(f" synthpanel report {result_id} # full Markdown report (incl. cost rollup)", file=sys.stderr)
print(f" synthpanel results show {result_id} # raw saved result", file=sys.stderr)
print(" synthpanel results list # all saved results", file=sys.stderr)
# #538: there is no per-result `cost <id>` command — the per-run cost
# rollup lives in `report` above. `cost summary` aggregates across runs.
print(f" synthpanel cost show {result_id} # this run's cost breakdown", file=sys.stderr)
print(" synthpanel cost summary # cost rollup across saved runs", file=sys.stderr)


Expand Down
29 changes: 21 additions & 8 deletions src/synth_panel/cli/parser.py
Original file line number Diff line number Diff line change
Expand Up @@ -138,11 +138,12 @@ def build_parser() -> argparse.ArgumentParser:
panel_run_parser.add_argument(
"--personas",
default=None,
metavar="PATH",
metavar="NAME_OR_PATH",
help=(
"Path to a YAML file defining personas. Required unless "
"--resume <run-id> is given, in which case the original "
"personas path is recovered from the checkpoint."
"Name of a bundled/installed persona pack (see 'synthpanel pack "
"list', e.g. general-consumer) or a path to a YAML file defining "
"personas. Required unless --resume <run-id> is given, in which "
"case the original personas path is recovered from the checkpoint."
),
)
panel_run_parser.add_argument(
Expand Down Expand Up @@ -173,11 +174,13 @@ def build_parser() -> argparse.ArgumentParser:
panel_run_parser.add_argument(
"--instrument",
default=None,
metavar="PATH",
metavar="NAME_OR_PATH",
help=(
"Path to a YAML file defining the survey/instrument. "
"Required unless --resume <run-id> is given, in which case "
"the original instrument path is recovered from the checkpoint."
"Name of a bundled/installed instrument (see 'synthpanel "
"instruments list', e.g. general-survey) or a path to a YAML "
"file defining the survey/instrument. Required unless "
"--resume <run-id> is given, in which case the original "
"instrument path is recovered from the checkpoint."
),
)
panel_run_parser.add_argument(
Expand Down Expand Up @@ -1216,6 +1219,16 @@ def build_parser() -> argparse.ArgumentParser:
)
cost_subparsers = cost_parser.add_subparsers(dest="cost_command")

cost_show_parser = cost_subparsers.add_parser(
"show",
help="Per-run cost breakdown for one saved panel result (alias for the cost section of 'panel inspect').",
)
cost_show_parser.add_argument(
"result",
metavar="RESULT",
help="Panel result ID or path to a result JSON file.",
)

cost_summary_parser = cost_subparsers.add_parser(
"summary",
help="Summarize cost across saved panel runs (totals, by model, by month).",
Expand Down
19 changes: 19 additions & 0 deletions src/synth_panel/cost.py
Original file line number Diff line number Diff line change
Expand Up @@ -613,6 +613,25 @@ def resolve_cost(
provider_total = float(usage.provider_reported_cost)
local_total = local.total_cost

# gh-r4: a provider-reported cost of exactly $0 for a run that consumed
# tokens is almost never a real bill — OpenRouter returns ``usage.cost:
# 0`` for BYOK keys (the upstream provider bills directly) and some
# upstreams omit native cost. Trusting the zero verbatim recorded runs
# as $0.0000 and starved CostGate, so --max-cost could never trip.
# Fall back to the local pricing table whenever it prices the same
# usage as nonzero; genuinely free models (local table also prices $0)
# still record $0.
if provider_total == 0.0 and usage.total_tokens > 0 and local_total > 0:

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Preserve legitimate provider-reported zero costs

When a provider genuinely bills $0—such as an OpenRouter :free route or a promotion—this branch replaces the authoritative zero with paid local pricing. lookup_pricing has no special handling for :free, so such slugs either match the paid model family or fall back to nonzero DEFAULT_PRICING; the run is then misrecorded as paid and CostGate can halt a workload that has accrued no charge. A zero alone cannot distinguish BYOK from a genuinely free request, so the fallback needs an explicit BYOK/provider signal rather than overriding every zero.

Useful? React with 👍 / 👎.

logger.warning(
"Provider reported $0.00 for %d tokens on model=%s (BYOK or "
"missing native cost); recording the local pricing-table "
"estimate $%.6f instead.",
usage.total_tokens,
model,
local_total,
)
return local

if local_total > 0 and provider_total > 0:
ratio = abs(provider_total - local_total) / max(provider_total, local_total)
if ratio > _DIVERGENCE_WARN_RATIO:
Expand Down
4 changes: 4 additions & 0 deletions src/synth_panel/main.py
Original file line number Diff line number Diff line change
Expand Up @@ -215,6 +215,10 @@ def _main(argv: list[str] | None) -> int:
sub = getattr(args, "cost_command", None)
if sub == "summary":
return handle_cost_summary(args, output_format)
elif sub == "show":
from synth_panel.cli.commands import handle_cost_show

return handle_cost_show(args, output_format)
else:
parser.parse_args(["cost", "--help"])
return 1
Expand Down
Loading
Loading