diff --git a/.agents/skills/panel/SKILL.md b/.agents/skills/panel/SKILL.md new file mode 100644 index 0000000000..a1f0220596 --- /dev/null +++ b/.agents/skills/panel/SKILL.md @@ -0,0 +1,118 @@ +--- +name: panel +description: >- + Run a model panel: two analysts answer one question independently on different models without seeing each other's work, then a third model judges both and re-verifies their load-bearing claims itself. + Use when the captain invokes /panel, asks for a panel, a second opinion from another model, a cross-check, or an adversarial review of an answer, and when a contested or expensive-to-get-wrong question would otherwise be settled by one voice. + Owns the decision of when a panel is worth its cost, the operating sequence around bin/fm-model-panel.sh, and the honest handling of a home that cannot field two distinct models. +user-invocable: true +metadata: + internal: true +--- + +# Model panel + +Two analysts answer the same question independently on different models, neither able to see the other's work, and a third model judges them. +The value is not three opinions. +It is independence plus adversarial verification: the judge re-verifies the load-bearing claims itself instead of refereeing rhetoric, so it can catch the mistake both analysts made for the same reason, which neither could have caught alone. + +`bin/fm-model-panel.sh` owns the mechanics; its header and `--help` are authoritative for commands, flags, the configuration resolution order, and exit statuses. +This skill owns when to run one and how to carry it through. + +## Is this question worth a panel + +A panel costs two full investigations plus a judge that re-checks their evidence, and two rounds of supervision between dispatch and answer. +Spend that only where a second independent answer would change what happens next. + +Worth it: + +- The question is contested or expensive to get wrong, and would otherwise be settled by one confident voice. +- The answer depends on live state that written records describe unreliably, which is exactly where a single analyst inherits a stale claim and never notices. +- Several orderings or designs are defensible and the choice matters more than the analysis. +- An earlier answer read as confident but thin, and you want it attacked rather than repeated. + +Not worth it: + +- The question has one obvious answer, or one command settles it. +- The work is implementation against a clear specification; that is an ordinary ship task. +- The real bottleneck is a captain decision rather than analysis; register the decision through `decision-hold-lifecycle` instead. +- Speed matters more than confidence. +- You would act the same way whatever the judge concluded. + +Write the question down precisely before starting. +Both analysts receive byte-identical text and nothing else, so a vague question wastes the whole formation; put a long question in a file and pass `--question-file`. + +## Operating sequence + +1. **Start the panel.** + `bin/fm-model-panel.sh start [--project ] ""` resolves the lineup, writes the briefs, and dispatches the analysts concurrently. + Use `--dry-run` first when you want to show the captain which models will argue before spending anything. + `--project` defaults to the firstmate repo, which is what a question about the fleet needs; pass a project name for a question about that project's code. +2. **Supervise the analysts as ordinary scouts.** + They report through the normal status path and wake you the normal way. + Do not relay their individual reports to the captain and do not let one analyst see the other's work; the independence rule in their briefs is the whole point. +3. **Advance once every analyst has reported.** + `bin/fm-model-panel.sh advance ` refuses to create the judge until every analyst has both written a terminal status event and left a non-empty report, so run it after each analyst finishes and let it tell you whether it is still waiting. + A `waiting:` line means nothing to do yet, and it names which of the two facts is still missing for which analyst. +4. **Supervise the judge, then advance again.** + The second `advance` prints `complete: ` once the judge has both reported a terminal status event and left a non-empty report, which is the same gate its analysts passed. + A `wedged:`, `stood down:`, or `verdict lost:` block instead of a `waiting:` line means the panel needs a decision from you; read the next sections before doing anything about it. +5. **Read the judge's report and relay its findings.** + Relay the answer, the contested facts and what the evidence showed, any mistake shared by both analysts, and what remains unverified. + Say which models argued and which judged: that is substance the captain asked for, not internal machinery. + Never relay a raw report; follow the captain-facing translation contract in `AGENTS.md` section 9. +6. **Pass the shared completion gate before treating the panel as done.** + A panel reliably surfaces captain decisions, which is exactly the work that gets lost, so the judge's report ends with a decisions inventory. + Load `decision-hold-lifecycle` and register every unresolved decision from that inventory before calling the panel complete. +7. **Tear down the members normally.** + Every report survives teardown at `data//report.md`, and the panel record at `data//panel.meta` records which model filled which role. + +## When a member never signals that it finished + +Writing the report and appending the terminal `done:` or `failed:` line are two separate acts, and a member that is between them is doing something completely ordinary. +So a member that still has its runtime record gets the plain `waiting:` line, and there is nothing for you to do: it will append its own terminal line and the next `advance` proceeds. + +`advance` prints a `wedged:` block only when the member left a non-empty report and is GONE, with no runtime record left, because it was torn down or crashed after writing its report and before signalling. +That line will never arrive on its own, so the block names the member and prints the exact command: `bin/fm-model-panel.sh advance --accept-unfinished `. + +That override is per-member and deliberate. +It names one task id and waives nothing for any other member, it is recorded permanently in the panel record as `accepted_unfinished`, the judge's brief is told to treat that report as possibly truncated and to say so in its own report, and an accepted judge report puts the caveat on every `complete:` output. +It also stays available for a member that still looks present but you know is dead, such as one killed by a harness crash that left its runtime record behind; used that way it warns you that the member may still be writing, and you should be sure before overriding it. +There is no timeout anywhere in this formation: a panel waits forever rather than judging an unfinished report on its own, so the decision to accept one is always yours. +Say so when relaying the verdict, because a panel completed over an unfinished report is a weaker result than one that was not. + +## When an analyst finishes without a report + +`advance` prints a `stood down:` block and exits 1 when an ANALYST stops writing without leaving a report, either by ending with a terminal status event or by losing its runtime record. +That member has stopped writing, so no report can arrive, and the panel cannot produce a verdict. + +Stand the panel down and tell the captain it produced nothing. +The override deliberately refuses to waive a missing report: a verdict built on the one report that survived is not a panel, and presenting it as one would rebuild the echo-as-corroboration failure this formation exists to refuse. +If the captain still wants the surviving analysis, that is a NEW single-analyst review started deliberately with `--reduced`, which is labelled as such in the briefs, the record, and the judge's own report, and never this panel converted in place. + +## When the judge finishes without a verdict + +`advance` prints a `verdict lost:` block when the JUDGE stops writing without leaving a report, either by ending with a terminal status event or by losing its runtime record. +This is not a dead panel: both analyst reports are complete and untouched, and only the adjudication is missing, so the formation is intact. + +Run `bin/fm-model-panel.sh advance --rejudge` to dispatch one replacement judge over those same unchanged reports. +The superseded judge task is recorded in the panel record, no report is stamped incomplete by it, and nothing re-dispatches on its own. +A judge that still holds its runtime record and has written nothing is indistinguishable from one that is working, so `advance` prints the plain `waiting:` line and `--rejudge` refuses; if you know that judge is dead, such as one killed by a harness crash that left its runtime record behind, tear it down first and then `--rejudge`, which is permitted once the record is gone. +Mention when relaying that the verdict came from a replacement judge, because the captain is entitled to know a judge was lost. + +## When the panel cannot be a panel + +`start` refuses with exit 4 when both analysts would resolve to the same model. +That refusal is correct and must not be worked around: two identical analysts are not independent, and a panel that quietly is not one is worse than no panel. + +Tell the captain plainly that this home can currently reach only one model for the analyst seats, and offer the two real options: + +- Configure a second distinct model for the second analyst seat, then re-run the panel. +- Run the reduced form with `--reduced`: one analyst plus a judge that re-verifies its claims. + +The reduced form is a **single-analyst review**, not a panel. +Its briefs, its record, and its judge's own report all say so. +Never describe its output as a panel result, an independent cross-check, or a second opinion. + +When no third distinct model is available the judge shares a model with one analyst. +That prints a warning and proceeds, because the judge's independence comes from re-verifying against live state with every report in hand rather than from its runtime. +Mention it when relaying if the verdict is close. diff --git a/.gitignore b/.gitignore index 8aa8bde969..3ba90241a2 100644 --- a/.gitignore +++ b/.gitignore @@ -24,6 +24,7 @@ __pycache__/ .claude/settings.fm-task.json config/crew-harness config/crew-dispatch.json +config/model-panel.json config/secondmate-harness config/backlog-backend config/backend diff --git a/AGENTS.md b/AGENTS.md index 75013180e3..4c411577e9 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -78,6 +78,7 @@ bin/ helper scripts, committed; read each script's header before .env optional X-mode pairing token; LOCAL, gitignored; presence-gates section 14 config/crew-harness crewmate harness override; LOCAL, gitignored; absent or "default" = same as firstmate. Inherited as the literal file: a concrete primary adapter value also controls a secondmate home's own crewmates (section 4) config/crew-dispatch.json optional crewmate dispatch profiles; LOCAL, gitignored; firstmate-maintained but human-editable natural-language rules that choose a per-task harness/model/effort profile (section 4). Inherited by secondmate homes +config/model-panel.json optional model-panel role profiles; LOCAL, gitignored; maps the pinned analyst and judge roles to concrete models, falling back to config/crew-dispatch.json's default profile set. NOT inherited by secondmate homes, because it names the models one home can actually reach (docs/configuration.md "Model panel roles") config/secondmate-harness harness the PRIMARY uses to launch SECONDMATE agents, optionally followed by a model and effort token on the same line (" [] []"; section 4); LOCAL, gitignored; absent or "default" harness falls back to config/crew-harness then firstmate's own. The primary's own setting; NOT inherited into secondmate homes (secondmates do not spawn secondmates) config/firstmate-update-base comparison base for the instruction-surface currency check, naming the source this deployment updates from; LOCAL, gitignored; absent means the canonical upstream template; inherited by secondmate homes (docs/configuration.md "Upstream firstmate and curated-fork checks") config/fork-sync-upstream comparison base for the curated-fork check, naming the real upstream the fork tracks; LOCAL, gitignored; absent means the canonical upstream template; curator-only, NOT inherited by secondmate homes @@ -100,6 +101,8 @@ data/ personal fleet records; LOCAL, gitignored as a whole secondmates.md secondmate routing table; firstmate-private, maintained by fm-home-seed.sh (section 6) /brief.md per-task crewmate brief, or per-secondmate charter brief when kind=secondmate /report.md scout task deliverable, written by the crewmate; survives teardown + /panel.meta model-panel record written by bin/fm-model-panel.sh: form, project, question pointer, stage, member task ids, and each role's resolved profile + /question.md the one question text every member of that panel was given, written by bin/fm-model-panel.sh; survives teardown like the reports projects/ cloned repos; gitignored; READ-ONLY for you graphify-out/ local Graphify knowledge graph artifacts; gitignored, expected to drift after hook or incremental updates state/ volatile runtime signals; gitignored @@ -256,6 +259,9 @@ Classify the deliverable: - **Ship** is the default and produces a project change through the selected delivery mode. - **Scout** produces knowledge in `data//report.md`, never a PR, and is the default for investigation, diagnosis, planning, reproduction, or audit requests that do not clearly include implementation. +When the captain invokes `/panel`, asks for a panel, a second opinion from another model, or an adversarial cross-check of an answer, load the `panel` skill. +It owns that formation of independent scouts plus a judge, and when it is worth its cost. + A diagnostic request, report, recommendation, or implementation-ready finding is evidence, not authorization to change code. Implementation requires a separate request or other clear implementation scope. Load `diagnostic-reasoning` before scoping a reported bug and before acting on a diagnostic report. diff --git a/README.md b/README.md index e175519cc9..70a995d81f 100644 --- a/README.md +++ b/README.md @@ -174,6 +174,7 @@ Claude and grok use the slash form shown here; codex uses the same names with `$ | `/afk` | Enter away-mode supervision: the sub-supervisor self-handles routine notifications in bash, escalates captain-relevant events and bounded declared-external-wait rechecks as batched digests, and actively alerts if delivery gets stuck while you step away | | `/ahoy` | Recap only visible session events since the prior real captain message, falling back to Bearings when invoked as the session's first real captain message | | `/bearings` | Generate a standalone current-status report from bounded local fleet and registered-secondmate state, with live PR enrichment only when requested, written to a dated file in `data/` and surfaced concisely in chat; read-mostly, mutates no task state | +| `/panel` | Run a model panel on one question: two analysts investigate independently on different models without seeing each other's work, then a third model judges both and re-verifies their load-bearing claims; refuses rather than presenting two runs of one model as independent | | `/updatefirstmate` | Self-update the running firstmate and its secondmates to the latest from origin with fast-forward-only pulls, then re-read instructions and nudge secondmates | | `/stow` | Sweep the session for uncaptured durable knowledge, route each finding to its disk home per AGENTS.md, file undone next steps to the backlog, and report what is now safe to reset | diff --git a/bin/fm-classify-lib.sh b/bin/fm-classify-lib.sh index 5772658ebf..ab58d72aa6 100755 --- a/bin/fm-classify-lib.sh +++ b/bin/fm-classify-lib.sh @@ -93,6 +93,44 @@ status_is_terminal_verb() { esac } +# The verbs that declare a task STOPPED WRITING. Deliberately narrower than the +# terminal captain verbs above: needs-decision and blocked stop the agent's turn +# but expect it to resume and keep working (the scaffolds have it append +# `resolved:` and carry on), so they say nothing about the work being finished. +# Callers asking "must the captain see this" use status_is_terminal_verb; callers +# asking "will anything more be written" use the two helpers below. +# FM_CLASSIFY_FINISHED_VERBS overrides the set. +FM_CLASSIFY_FINISHED_VERBS_DEFAULT='done failed' + +# 0 if the given status line's leading verb declares the task finished. +status_is_finished_verb() { # + local line=$1 verb + [ -n "$line" ] || return 1 + verb=$(status_line_verb "$line") + [ -n "$verb" ] || return 1 + case " ${FM_CLASSIFY_FINISHED_VERBS:-$FM_CLASSIFY_FINISHED_VERBS_DEFAULT} " in + *" $verb "*) return 0 ;; + esac + return 1 +} + +# 0 if a status file EVER recorded a finished event. The status stream is an +# append-only EVENT log, so this deliberately scans every line instead of reading +# the last one: "this task finished at some point" is monotonic, and a later +# unrelated line never un-finishes it. Consumers that need the CURRENT state of a +# live crew use fm-crew-state.sh instead; this one answers a durable question that +# still has an answer after the pane is gone. +status_has_finished_event() { # + local f=$1 line + [ -e "$f" ] || return 1 + while IFS= read -r line || [ -n "$line" ]; do + if status_is_finished_verb "$line"; then + return 0 + fi + done < "$f" + return 1 +} + # 0 if the given (last) status line matches a captain-relevant verb. # Verb-aware by default: terminal verbs always match; nonterminal progress verbs # (working, resolved, captain-held) and paused never match from free-text prose; diff --git a/bin/fm-gate-refuse-lib.sh b/bin/fm-gate-refuse-lib.sh index 8e624408a5..7adaadfdf6 100644 --- a/bin/fm-gate-refuse-lib.sh +++ b/bin/fm-gate-refuse-lib.sh @@ -54,7 +54,7 @@ # tests/fm-gate-refuse.test.sh strips the bypass so it still verifies real refusal. # # Sourced by bin/fm-spawn.sh, bin/fm-send.sh, bin/fm-teardown.sh, -# bin/fm-sessionstart-nudge.sh, and the tests. +# bin/fm-model-panel.sh, bin/fm-sessionstart-nudge.sh, and the tests. # No side effects on source. set -u / set -e safe. The refusal is a hard exit, # not a return, because there is no safe way to continue a fleet mutation from a # gate context. diff --git a/bin/fm-model-panel.sh b/bin/fm-model-panel.sh new file mode 100755 index 0000000000..50a9a01724 --- /dev/null +++ b/bin/fm-model-panel.sh @@ -0,0 +1,1230 @@ +#!/usr/bin/env bash +# Run a model panel: two analysts answer one question independently on +# DIFFERENT models without seeing each other's work, then a third model judges +# both reports and re-verifies their load-bearing claims itself. +# Every panel member is an ordinary firstmate scout, so the panel adds a +# formation on top of the existing brief/spawn/report lifecycle and invents no +# second one. The `panel` skill owns when a panel is worth its cost. +# +# Usage: +# fm-model-panel.sh start [--id ] [--project ] +# [--reduced] [--dry-run] +# ( | --question-file ) +# fm-model-panel.sh advance [--accept-unfinished ] +# [--rejudge] +# fm-model-panel.sh status +# +# start resolve the three roles, write the question and both analyst +# briefs, and dispatch the analysts concurrently. The judge is NOT +# dispatched here: it is created only once every analyst has +# finished, which `advance` enforces. +# advance idempotent next step, gated on the SAME two conditions for every +# member, analysts and judge alike: the member wrote a terminal +# status event (a `done:` or `failed:` line in state/.status) +# AND its data//report.md exists and is non-empty. With both +# satisfied for every analyst it scaffolds and dispatches the judge; +# with both satisfied for the judge it prints +# `complete: ` and records the panel complete, which is +# the last thing it does. Until then it changes nothing and prints +# one of four outcomes: a `waiting:` line naming which of the two +# facts is missing for which member (exit 0); a `wedged:` block when a +# member left a report and is GONE without ever declaring itself +# finished, which is the one state that can never clear on its own +# (exit 0); a `stood down:` block when an ANALYST stopped writing +# with no report at all, which is the one state where the panel's +# premise has failed (recorded as `stage=stood-down`, exit 1); or a +# `verdict lost:` block when the JUDGE stopped writing with no report, +# where the evidence is intact and only the adjudication is missing, +# so `--rejudge` replaces it (exit 0). A member has stopped writing +# once it wrote a terminal event or lost its runtime record. +# status print the panel record without changing anything. +# +# --id explicit panel id (default: derived from the question +# plus a random suffix). Task ids are -a, +# -b, -judge, and -judge +# for a replacement judge, so the panel id is capped at 56 +# characters: the longest suffix is `-judge99` at 8, which +# keeps every task id within the 64-character task-id +# limit. +# --project +# the repo the scouts get a worktree of: an existing +# directory, or a name resolved under the home's +# projects/. Defaults to the firstmate repo itself, +# which is what a question about the fleet needs. +# --reduced run the NAMED REDUCED FORM: one analyst plus the +# judge, recorded and labelled everywhere as a +# single-analyst review rather than a panel. This is the +# honest degradation for a home with only one model, and +# it is never selected implicitly. +# --question-file +# read the question from a file instead of the +# positional argument. Exactly one of the two is +# required; a long question belongs in a file. +# --accept-unfinished +# with `advance`, accept ONE named member's existing +# report as final even though that member never wrote a +# terminal status event. It is explicit and per-member: it +# names one task id and waives nothing for any other +# member, it is never on by default, and it refuses both +# a member that has no report to accept and one that was +# ever observed to finish properly. Repeat the flag to +# accept more than one member. +# --rejudge with `advance` at the judge stage, dispatch ONE +# replacement judge over the existing, unchanged analyst +# reports after the current judge ended without writing a +# verdict. Explicit and recorded, never automatic: the +# superseded judge task is written to the panel record +# after the replacement is dispatched, and no report is +# stamped incomplete by it. It refuses when the judge did +# leave a verdict, and while the judge still LOOKS LIVE: +# a runtime record with no terminal status event yet. A +# judge with a terminal event, or one whose runtime record +# is gone, can be replaced. +# --dry-run with `start`, resolve and print the lineup without +# writing or dispatching anything. +# +# Roles are pinned here; models are not. `config/model-panel.json` maps each +# role to a dispatch profile, and docs/configuration.md "Model panel roles" owns +# that schema, the resolution order, and the degradation contract. Every role +# resolves through bin/fm-dispatch-select.sh, so panel profiles get the same +# validation and quota-aware array selection as crew dispatch profiles. +# +# Model identity for the distinctness test is the profile's model name with any +# provider prefix and any `:suffix` removed (the normalization bin/fm-dispatch- +# select.sh already uses), or `harness:` when the profile pins no model. +# Two profiles naming the same model through different harnesses are therefore +# correctly treated as ONE model, not two. +# +# Degradation is explicit, never silent: when the two analysts would resolve to +# the same model identity, `start` refuses with exit 4 and names both the +# configuration fix and the `--reduced` alternative. Two identical analysts are +# not a panel, and presenting one as a panel is worse than running none. When no +# third distinct model is available, the judge may share an analyst's model; that +# is a printed warning rather than a refusal, because the judge's independence +# comes from re-verifying against live state with both reports in hand. +# +# The two-condition judge gate is deliberate. A report file that EXISTS is not a +# report that is FINISHED, and dispatching the judge against a half-written +# analysis silently judges a truncated argument. The completion half is a status +# EVENT rather than a live crew-state read, because panel members are ordinary +# scouts that may be torn down before the judge is dispatched, and a gate with a +# fallback for that case would quietly become the fallback. `failed:` +# counts as terminal on purpose: the question is whether the analyst stopped +# writing, not whether it succeeded, and a failed analyst that still left a +# non-empty report is finished. The verbs are recognized through +# bin/fm-classify-lib.sh, this repo's owner of the status-event vocabulary, and +# the scan asks only whether such an event EVER appeared, which is monotonic - +# never what the last line currently says. The gate governs the judge's own +# report on exactly the same terms: the trap does not care which hop it is on, +# and handing the panel's final verdict out under a weaker rule than the one +# guarding its inputs would be absurd. +# +# ROOT CAUSE FOR ANY FUTURE CLASSIFICATION, stated once because it has already +# cost three separate repairs: bin/fm-teardown.sh removes a member's +# state/.status and state/.meta together, so the ABSENCE of either means +# TORN DOWN rather than NEVER EXISTED, and any new member classification must +# decide which of those two it is before it decides anything else. +# +# READINESS IS LATCHED, and the reason is a maintenance warning worth keeping. +# The gate was originally designed on the premise that the terminal status event +# is durable and survives cleanup. It is not: bin/fm-teardown.sh removes +# state/.status and state/.meta together, so a member that finished +# correctly and was then torn down became indistinguishable from one that +# vanished mid-write, and the only way past it stamped `accepted_unfinished` on a +# COMPLETE report and told the judge to distrust it. The premise was relayed and +# ruled on without anyone reading the code that deletes those files, which is +# exactly the verify-status-prose-against-live-state discipline the briefs below +# impose on every analyst; the capability's own design skipped it. So the FIRST +# time `advance` observes both conditions for a member, that readiness is written +# to `ready_members` in data//panel.meta, which does survive teardown, +# and every later invocation trusts the latch. Do not re-derive readiness from +# state/ alone. A narrow window remains: a member torn down before any `advance` +# observed it is never latched, so it can still be classified `unsignalled-gone` +# and stamped `accepted_unfinished` on a complete report, which is an accepted +# bounded tradeoff of this design. +# +# That gate can WEDGE, and the escape from it is deliberately manual and +# deliberately narrow. Writing the report and appending the terminal line are two +# separate acts, and the gap between them is the ORDINARY window: a member that +# still has its runtime record may simply be revising, so `advance` prints the +# plain `waiting:` line there and does not mention the override at all. Offering +# the escape in that window would invite the operator to recreate the exact race +# this gate closes, at the one moment it looks most reasonable to accept. +# +# The `wedged:` block is therefore gated on the member being GONE with no latched +# readiness: it left a report, nothing ever recorded it as finished, and its +# runtime record is no longer there, so only an operator can decide. That block +# names the member, gives the exact +# `--accept-unfinished ` command, and says plainly that accepting means +# judging a possibly INCOMPLETE report. The acceptance is recorded in the panel +# record as `accepted_unfinished`, so the provenance of the verdict carries the +# caveat forever, the judge's brief names the report it must treat as possibly +# truncated, and EVERY `complete:` output for an accepted judge repeats the +# caveat rather than only the invocation that recorded it. The override itself +# stays usable against a member that still looks present, because a crashed +# member keeps its runtime record and a hard refusal there would rebuild the +# no-escape dead end; used that way it warns loudly, on that invocation, that the +# member may still be writing. +# +# An ANALYST that stops writing with NO report is a different fact again, whether +# it declared itself finished or simply went away: no report can arrive and the +# panel's premise has failed. +# That prints a `stood down:` block naming the member, records `stage=stood-down` +# so the panel record says honestly what happened, and exits 1. The override +# never waives it, because a verdict built on the one report that survived is not +# a panel. If that surviving analysis is still wanted, it is a NEW single-analyst +# review started deliberately with `--reduced` and labelled as such everywhere, +# never this panel converted in place. +# +# A JUDGE that stops writing with no report costs something else entirely: the +# evidence is complete and untouched, and only the adjudication is missing, so +# standing the panel down would discard two finished investigations because the +# third member crashed. That prints a `verdict lost:` block instead, and +# `--rejudge` dispatches ONE replacement judge over the same unchanged reports, +# recording the superseded task in `superseded_judges`. It is explicit and +# recorded exactly like `--accept-unfinished`, never automatic, and it stamps no +# incompleteness on reports that were complete. +# +# There is deliberately NO time-based, retry-count-based, or attempt-count-based +# path that advances a panel on its own, and none may be added. An automatic +# timeout would reinstate exactly the half-written-report failure this gate +# closes, only on a delay and with nobody watching. Waiting waits forever until a +# human decides. +# +# Panel record: data//panel.meta (key=value), with the question at +# data//question.md. Both are durable and survive scout teardown, as do +# the reports at data//report.md. +# +# Retry safety: `start` refuses an existing panel id rather than clobbering a +# live panel. `advance` regenerates the judge brief only when the judge task was +# never dispatched (no state/.meta), so a failed dispatch can be retried +# without hand-editing, and never touches a brief whose agent is live. +# +# Exit status: 0 success (including `waiting:`), 2 usage error, 3 no-mistakes +# gate refusal (bin/fm-gate-refuse-lib.sh), 4 refused because the panel would not +# be a panel, 1 any other failure. +set -eu + +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" + +usage() { + awk ' + NR == 1 { next } + /^#/ { sub(/^# ?/, ""); print; next } + { exit } + ' "$0" +} + +case "${1:-}" in + -h|--help) usage; exit 0 ;; +esac + +# shellcheck source=bin/fm-gate-refuse-lib.sh +. "$SCRIPT_DIR/fm-gate-refuse-lib.sh" +fm_refuse_if_gate_agent + +# The owner of the status-event vocabulary the judge gate reads. +# shellcheck source=bin/fm-classify-lib.sh +. "$SCRIPT_DIR/fm-classify-lib.sh" + +FM_ROOT="${FM_ROOT_OVERRIDE:-$(cd "$SCRIPT_DIR/.." && pwd)}" +FM_HOME="${FM_HOME:-${FM_ROOT_OVERRIDE:-$FM_ROOT}}" +DATA="${FM_DATA_OVERRIDE:-$FM_HOME/data}" +STATE="${FM_STATE_OVERRIDE:-$FM_HOME/state}" +CONFIG="${FM_CONFIG_OVERRIDE:-$FM_HOME/config}" +PROJECTS="${FM_PROJECTS_OVERRIDE:-$FM_HOME/projects}" +PANEL_CONFIG="$CONFIG/model-panel.json" +CREW_DISPATCH="$CONFIG/crew-dispatch.json" + +die() { printf 'error: %s\n' "$1" >&2; exit "${2:-1}"; } +warn() { printf 'panel: %s\n' "$1" >&2; } + +# Every non-terminal outcome names the next command, so one entry point carries a +# panel from its question to its answer without the operator tracking stages. +panel_next_hint() { + # shellcheck disable=SC2016 # the backticks are markdown emphasis in the printed hint, not a command substitution + printf 'panel: next step is `%s advance %s` %s\n' "$0" "$1" "$2" +} + +command -v jq >/dev/null 2>&1 || die "jq is required to resolve panel roles" + +# --- role resolution -------------------------------------------------------- + +# Print the raw spec (profile object or profile array) for one role: the panel +# config's entry first, then config/crew-dispatch.json's default profile set. +# The crew-dispatch fallback is what lets a home that already declares which +# runtimes it dispatches on run a panel with no extra configuration. +role_spec() { + local role=$1 spec='' + if [ -f "$PANEL_CONFIG" ]; then + jq . "$PANEL_CONFIG" >/dev/null 2>&1 || die "$PANEL_CONFIG is not valid JSON" + spec=$(jq -c --arg role "$role" '.roles[$role] // empty' "$PANEL_CONFIG") + fi + if [ -z "$spec" ] && [ -f "$CREW_DISPATCH" ]; then + jq . "$CREW_DISPATCH" >/dev/null 2>&1 || die "$CREW_DISPATCH is not valid JSON" + spec=$(jq -c '.default // empty' "$CREW_DISPATCH") + fi + [ -n "$spec" ] || die "no profile for panel role '$role': add roles.$role to $PANEL_CONFIG (see docs/examples/model-panel.json) or a default profile set in $CREW_DISPATCH" + printf '%s\n' "$spec" +} + +# The ONE definition of a profile's model identity, interpolated by every jq +# program below so the exit-4 distinctness check and the exclusion filter cannot +# disagree about what counts as one model. Keep it in step with +# bin/fm-dispatch-select.sh's model_name normalization. +# shellcheck disable=SC2016 # $m is a jq binding, deliberately not shell-expanded +PANEL_IDENT_JQ='def ident: + (.model // "") as $m + | if ($m | length) > 0 then ($m | split("/") | last | split(":") | first | ascii_downcase) + else "harness:" + (.harness // "") + end;' + +profile_identity() { + printf '%s\n' "$1" | jq -r "$PANEL_IDENT_JQ"' ident' +} + +profile_field() { + printf '%s\n' "$1" | jq -r --arg field "$2" '.[$field] // ""' +} + +# Drop every candidate whose identity is in the newline-separated exclusion +# list, so a role backed by an array picks a model the panel is not already +# using instead of duplicating one. +filter_spec() { + local spec=$1 excluded=$2 + printf '%s\n' "$spec" | jq -c --arg excluded "$excluded" "$PANEL_IDENT_JQ"' + ($excluded | split("\n") | map(select(length > 0))) as $ex + | (if type == "array" then . else [.] end) + | map(select((ident) as $i | ($ex | index($i)) == null)) + ' +} + +# Resolve one role to a concrete profile through the shared dispatch selector, +# preferring a candidate whose model the panel is not already using. +resolve_role() { + local role=$1 excluded=${2:-} spec filtered count profile errors status=0 + spec=$(role_spec "$role") + if [ -n "$excluded" ]; then + filtered=$(filter_spec "$spec" "$excluded") + count=$(printf '%s\n' "$filtered" | jq 'length') + if [ "$count" -gt 0 ]; then + spec=$filtered + fi + fi + errors=$(mktemp "${TMPDIR:-/tmp}/fm-model-panel-select.XXXXXX") + profile=$("$FM_ROOT/bin/fm-dispatch-select.sh" "$spec" 2>"$errors") || status=$? + if [ "$status" -ne 0 ]; then + printf 'error: panel role %s could not be resolved:\n' "$role" >&2 + cat "$errors" >&2 + rm -f "$errors" + exit 1 + fi + rm -f "$errors" + printf '%s\n' "$profile" +} + +# --- panel record ----------------------------------------------------------- + +panel_dir() { printf '%s\n' "$DATA/$1"; } +panel_meta_path() { printf '%s\n' "$DATA/$1/panel.meta"; } + +# A `start` that fails before it attempts any dispatch must leave nothing +# behind, so a retry of the same panel id is not blocked by debris from a run +# that never reached a worker. Once a dispatch has been ATTEMPTED, nothing is +# rolled back: a failed spawn may still have left a live agent, and live work is +# never torn down as a side effect. +PANEL_TMP_FILES=() +PANEL_ROLLBACK_DIRS=() +PANEL_DISPATCH_STARTED=0 + +panel_rollback_register() { + case "$1" in + "$DATA"/?*) PANEL_ROLLBACK_DIRS+=("$1") ;; + *) die "refusing to register '$1' for rollback: it is not under $DATA" ;; + esac +} + +# shellcheck disable=SC2317,SC2329 # invoked through the EXIT trap +panel_cleanup() { + local status=$? path + for path in ${PANEL_TMP_FILES[@]+"${PANEL_TMP_FILES[@]}"}; do + rm -f "$path" + done + if [ "$status" -ne 0 ] && [ "$PANEL_DISPATCH_STARTED" -eq 0 ]; then + for path in ${PANEL_ROLLBACK_DIRS[@]+"${PANEL_ROLLBACK_DIRS[@]}"}; do + case "$path" in + "$DATA"/?*) rm -rf "$path" ;; + esac + done + fi + return "$status" +} + +panel_mktemp() { + local path + path=$(mktemp "${TMPDIR:-/tmp}/fm-model-panel.XXXXXX") + PANEL_TMP_FILES+=("$path") + printf '%s\n' "$path" +} + +# Values can contain '=' (a resolved profile is JSON), so split on the FIRST +# separator only and never re-join fields. +panel_meta_get() { + local file=$1 want=$2 line + while IFS= read -r line || [ -n "$line" ]; do + case "$line" in + "$want"=*) printf '%s\n' "${line#*=}"; return 0 ;; + esac + done < "$file" + printf '\n' +} + +panel_meta_set() { + local file=$1 want=$2 new=$3 tmp line found=0 + tmp="$file.tmp.$$" + : > "$tmp" + while IFS= read -r line || [ -n "$line" ]; do + case "$line" in + "$want"=*) printf '%s=%s\n' "$want" "$new" >> "$tmp"; found=1 ;; + *) printf '%s\n' "$line" >> "$tmp" ;; + esac + done < "$file" + [ "$found" -eq 1 ] || printf '%s=%s\n' "$want" "$new" >> "$tmp" + mv "$tmp" "$file" +} + +# --- brief composition ------------------------------------------------------ + +# The discipline that made the hand-run panels pay for themselves. Both runs +# were given a concrete example of a stale record misleading firstmate, and both +# found further stale records because of it, so this is a load-bearing section +# of the brief rather than decoration. +verify_prose_block() { + cat <<'EOF' +## Verify status prose, never trust it + +Written state in this fleet goes stale silently, and a stale record that still reads as current is the most reliable way to be confidently wrong here. +Backlog notes, task notes, status lines, earlier reports, code comments, and documentation are all claims about some past moment, not the current truth. +This is not a hypothetical: a hand-run of this panel was handed a backlog note describing work as still in flight long after it had landed, and only caught it by checking. +So for every load-bearing claim in your report, verify it yourself against the live source - the actual command output, the actual file at the actual path, the actual process or API state - and cite that evidence with the command you ran or the `file:line` you read. +Where prose and live state disagree, live state wins, and the disagreement is itself a finding worth reporting. +Where you could not verify something, say so explicitly instead of inheriting the claim. + +## Check your vantage point + +Confirm where you are actually running before concluding anything about the environment. +You are in a disposable worktree, which may sit on a different machine, container, network position, or account than the subject of the question. +A test run from the wrong vantage point returns a real result about the wrong system, and both analysts in one hand-run made exactly that mistake together. +EOF +} + +compose_analyst_task() { + local out=$1 label=$2 question_file=$3 report_path=$4 sibling_report=$5 form=$6 + { + if [ "$form" = panel ]; then + cat < "$out" +} + +compose_judge_task() { + local out=$1 question_file=$2 report_path=$3 form=$4 report_a=$5 report_b=$6 unfinished=${7:-} + { + if [ "$form" = panel ]; then + cat <<'EOF' +You are the judge on a firstmate model panel. +Two analysts answered the same question independently, on different models, with no access to each other's work. +Your job is not to referee their rhetoric, split the difference, or pick a winner wholesale. +It is to re-verify what they claim against live state and produce the answer the captain should act on. +EOF + else + cat <<'EOF' +You are the judge on a firstmate single-analyst review, the reduced form of the model panel. +One analyst answered the question, so there is no second independent report and no disagreement to adjudicate. +That makes your own re-verification more important, not less: you are the only independent check in this formation. +State in your report's first line that this was a single-analyst review rather than a panel, so its output is never presented as an independent panel result. +EOF + fi + cat <<'EOF' + +## The question every analyst was given + +EOF + cat "$question_file" + if [ "$form" = panel ]; then + cat < "$out" +} + +# Scaffold a scout brief through the shared scaffold, then replace its {TASK} +# placeholder with the composed panel task text. Failing loudly when the +# placeholder is absent keeps this honest if the scaffold ever changes shape. +scaffold_scout_brief() { + local id=$1 repo=$2 task_file=$3 brief + brief="$DATA/$id/brief.md" + "$FM_ROOT/bin/fm-brief.sh" "$id" "$repo" --scout >/dev/null \ + || die "could not scaffold the brief for $id" + awk -v taskfile="$task_file" ' + $0 == "{TASK}" && !filled { + while ((getline line < taskfile) > 0) print line + close(taskfile) + filled = 1 + next + } + { print } + ' "$brief" > "$brief.tmp" + mv "$brief.tmp" "$brief" + # Only a WHOLE line of "{TASK}" is the placeholder; the scaffold's Herdr + # declaration mentions the token inline and must not count as unfilled. + ! grep -Fxq '{TASK}' "$brief" \ + || die "the scout scaffold for $id still has an unfilled {TASK} line; the brief scaffold changed shape" +} + +dispatch_scout() { + local id=$1 project=$2 profile=$3 harness model effort + local -a args + harness=$(profile_field "$profile" harness) + model=$(profile_field "$profile" model) + effort=$(profile_field "$profile" effort) + args=("$id" "$project" --harness "$harness" --scout) + [ -z "$model" ] || args+=(--model "$model") + [ -z "$effort" ] || args+=(--effort "$effort") + "$FM_ROOT/bin/fm-spawn.sh" "${args[@]}" >/dev/null \ + || return 1 + printf 'dispatched: %s harness=%s model=%s effort=%s\n' \ + "$id" "$harness" "${model:-default}" "${effort:-default}" +} + +# --- start ------------------------------------------------------------------ + +derive_panel_id() { + local question_file=$1 slug suffix + slug=$(LC_ALL=C tr '[:upper:]' '[:lower:]' < "$question_file" \ + | LC_ALL=C tr -c 'a-z0-9' '-' \ + | LC_ALL=C tr -s '-' \ + | cut -c1-28) + slug=${slug#-} + slug=${slug%-} + [ -n "$slug" ] || slug=question + suffix=$(printf '%02x%02x' "$((RANDOM % 256))" "$((RANDOM % 256))") + printf 'panel-%s-%s\n' "$slug" "$suffix" +} + +resolve_project() { + local want=$1 + if [ -z "$want" ]; then + printf '%s\n' "$FM_ROOT" + return 0 + fi + if [ -d "$want" ]; then + (cd "$want" && pwd) + return 0 + fi + if [ -d "$PROJECTS/$want" ]; then + (cd "$PROJECTS/$want" && pwd) + return 0 + fi + die "no such project '$want': pass an existing directory or a name under $PROJECTS" +} + +cmd_start() { + local reduced=0 dry_run=0 + local panel_id='' project_arg='' question='' question_file='' + local arg + local want_value='' + for arg in "$@"; do + if [ -n "$want_value" ]; then + case "$want_value" in + id) panel_id=$arg ;; + project) project_arg=$arg ;; + question-file) question_file=$arg ;; + esac + want_value= + continue + fi + case "$arg" in + --id) want_value=id ;; + --id=*) panel_id=${arg#--id=} ;; + --project) want_value=project ;; + --project=*) project_arg=${arg#--project=} ;; + --question-file) want_value='question-file' ;; + --question-file=*) question_file=${arg#--question-file=} ;; + --reduced) reduced=1 ;; + --dry-run) dry_run=1 ;; + --*) die "unknown option $arg" 2 ;; + *) + [ -z "$question" ] || die "pass exactly one question (quote it, or use --question-file)" 2 + question=$arg + ;; + esac + done + [ -z "$want_value" ] || die "--$want_value requires a value" 2 + + if [ -n "$question_file" ]; then + [ -z "$question" ] || die "pass either a question argument or --question-file, not both" 2 + [ -r "$question_file" ] || die "cannot read question file: $question_file" 2 + else + [ -n "$question" ] || die "a question is required (as an argument or --question-file )" 2 + fi + + local project form + project=$(resolve_project "$project_arg") + form=panel + [ "$reduced" -eq 0 ] || form=single-analyst-review + + # Stage the question first so the panel id can be derived from it and both + # analysts provably receive byte-identical text. + local staged_question + trap panel_cleanup EXIT + staged_question=$(panel_mktemp) + if [ -n "$question_file" ]; then + cat "$question_file" > "$staged_question" + else + printf '%s\n' "$question" > "$staged_question" + fi + [ -s "$staged_question" ] || die "the question is empty" 2 + + [ -n "$panel_id" ] || panel_id=$(derive_panel_id "$staged_question") + case "$panel_id" in + ''|.*|*[!A-Za-z0-9._-]*) die "invalid panel id '$panel_id'" 2 ;; + esac + [ "${#panel_id}" -le 56 ] || die "panel id '$panel_id' is too long; task ids append up to -judge99 and must stay within 64 characters" 2 + + # Resolve the lineup. analyst-b excludes analyst-a's model so an array-backed + # role picks a second model rather than duplicating the first, and the judge + # excludes both. + local profile_a identity_a profile_judge identity_judge excluded + local profile_b='' identity_b='' + profile_a=$(resolve_role analyst_a) + identity_a=$(profile_identity "$profile_a") + excluded=$identity_a + if [ "$form" = panel ]; then + profile_b=$(resolve_role analyst_b "$identity_a") + identity_b=$(profile_identity "$profile_b") + if [ "$identity_b" = "$identity_a" ]; then + printf 'error: this would not be a panel: both analysts resolve to the model %s.\n' "$identity_a" >&2 + printf ' Two identical analysts are not independent, and presenting them as a panel is worse than running none.\n' >&2 + printf ' Configure a second distinct model as roles.analyst_b in %s (see docs/examples/model-panel.json),\n' "$PANEL_CONFIG" >&2 + printf ' or re-run with --reduced for the single-analyst review, which is labelled as such everywhere it appears.\n' >&2 + exit 4 + fi + excluded="$identity_a +$identity_b" + fi + profile_judge=$(resolve_role judge "$excluded") + identity_judge=$(profile_identity "$profile_judge") + case "$identity_judge" in + "$identity_a"|"${identity_b:-$identity_a}") + warn "no third distinct model is available, so the judge runs on $identity_judge, the same model as one analyst; it still re-verifies against live state with every report in hand" + ;; + esac + + local id_a="$panel_id-a" id_b="$panel_id-b" id_judge="$panel_id-judge" + local report_a="$DATA/$id_a/report.md" report_b="$DATA/$id_b/report.md" + local report_judge="$DATA/$id_judge/report.md" + + if [ "$dry_run" -eq 1 ]; then + printf 'panel: %s form=%s project=%s\n' "$panel_id" "$form" "$project" + printf 'panel: analyst-a %s\n' "$profile_a" + [ "$form" != panel ] || printf 'panel: analyst-b %s\n' "$profile_b" + printf 'panel: judge %s\n' "$profile_judge" + exit 0 + fi + + [ ! -e "$(panel_dir "$panel_id")" ] || die "panel '$panel_id' already exists at $(panel_dir "$panel_id")" + panel_rollback_register "$(panel_dir "$panel_id")" + # A task directory that already existed is NOT ours to roll back: reports + # survive teardown, so an earlier task's durable report can be sitting there. + # Only what this run creates is registered. + [ -e "$DATA/$id_a" ] || panel_rollback_register "$DATA/$id_a" + if [ "$form" = panel ] && [ ! -e "$DATA/$id_b" ]; then + panel_rollback_register "$DATA/$id_b" + fi + mkdir -p "$(panel_dir "$panel_id")" + cat "$staged_question" > "$DATA/$panel_id/question.md" + local question_path="$DATA/$panel_id/question.md" + local meta + meta=$(panel_meta_path "$panel_id") + { + printf 'panel=%s\n' "$panel_id" + printf 'form=%s\n' "$form" + printf 'project=%s\n' "$project" + printf 'question=%s\n' "$question_path" + printf 'created=%s\n' "$(date -u +%Y-%m-%dT%H:%M:%SZ)" + printf 'stage=analysts\n' + printf 'analyst_a_task=%s\n' "$id_a" + printf 'analyst_a_profile=%s\n' "$profile_a" + if [ "$form" = panel ]; then + printf 'analyst_b_task=%s\n' "$id_b" + printf 'analyst_b_profile=%s\n' "$profile_b" + fi + printf 'judge_task=%s\n' "$id_judge" + printf 'judge_profile=%s\n' "$profile_judge" + } > "$meta" + + local repo_label + repo_label=$(basename "$project") + local task_a task_b + task_a=$(panel_mktemp) + if [ "$form" = panel ]; then + task_b=$(panel_mktemp) + compose_analyst_task "$task_a" A "$question_path" "$report_a" "$report_b" "$form" + compose_analyst_task "$task_b" B "$question_path" "$report_b" "$report_a" "$form" + scaffold_scout_brief "$id_a" "$repo_label" "$task_a" + scaffold_scout_brief "$id_b" "$repo_label" "$task_b" + else + compose_analyst_task "$task_a" A "$question_path" "$report_a" "" "$form" + scaffold_scout_brief "$id_a" "$repo_label" "$task_a" + fi + + printf 'panel: %s form=%s project=%s\n' "$panel_id" "$form" "$project" + PANEL_DISPATCH_STARTED=1 + if ! dispatch_scout "$id_a" "$project" "$profile_a"; then + panel_meta_set "$meta" stage incomplete + die "could not dispatch analyst A ($id_a); no other panel member was dispatched" + fi + if [ "$form" = panel ]; then + if ! dispatch_scout "$id_b" "$project" "$profile_b"; then + panel_meta_set "$meta" stage incomplete + die "could not dispatch analyst B ($id_b) while analyst A ($id_a) is already running; decide whether to retry the dispatch or stand the panel down" + fi + fi + panel_next_hint "$panel_id" "once every analyst reports done" +} + +# --- advance / status ------------------------------------------------------- + +require_panel() { + local panel_id=$1 meta + meta=$(panel_meta_path "$panel_id") + [ -f "$meta" ] || die "no panel record at $meta" + printf '%s\n' "$meta" +} + +# 0 when this member's terminal status event was explicitly waived by an operator. +member_accepted_unfinished() { # + case " $2 " in + *" $1 "*) return 0 ;; + esac + return 1 +} + +# 0 when this member was ALREADY observed to satisfy both gate conditions and +# that observation was latched into the panel record. The latch is the durable +# half of the gate: state/.status does not survive teardown, so without it a +# member that finished correctly and was then cleaned up is indistinguishable +# from one that vanished mid-write. +member_ready_latched() { # + case " $2 " in + *" $1 "*) return 0 ;; + esac + return 1 +} + +# 0 when both gate conditions are observable RIGHT NOW from live signals. True +# only while state/.status still exists, which is why the result is latched +# the first time it is seen. +member_observed_ready() { # + [ -s "$DATA/$1/report.md" ] || return 1 + status_has_finished_event "$STATE/$1.status" +} + +# 0 while the fleet still tracks this task. Its absence does NOT prove the member +# died without declaring itself: teardown removes state/.meta and +# state/.status together, so this only separates a member that is still +# around from one that is not, and the latch above is what keeps a finished +# member from being misread as a vanished one. +member_still_present() { # + [ -f "$STATE/$1.meta" ] +} + +# One member's position against the two gate conditions. The same predicate +# serves every member, analysts and judge alike, because the judge is an ordinary +# scout that appends `done:` exactly as they do. status_has_finished_event comes +# from bin/fm-classify-lib.sh and asks only whether a terminal event EVER +# appeared. +# ready both conditions hold, now or at a latched earlier moment +# (or the terminal event was explicitly waived) +# finished-empty ended terminal with no report; nothing further can arrive +# gone-empty gone with no report and no terminal event; same fact +# unsignalled-gone a report is there, its author is gone and never declared +# unsignalled-live a report is there and its author is still present +# working still present and nothing written yet +member_state() { # + local id=$1 accepted=$2 latched=$3 finished=0 + if member_ready_latched "$id" "$latched"; then + printf 'ready\n' + return 0 + fi + if status_has_finished_event "$STATE/$id.status" \ + || member_accepted_unfinished "$id" "$accepted"; then + finished=1 + fi + if [ -s "$DATA/$id/report.md" ]; then + if [ "$finished" -eq 1 ]; then + printf 'ready\n' + elif member_still_present "$id"; then + printf 'unsignalled-live\n' + else + printf 'unsignalled-gone\n' + fi + else + if [ "$finished" -eq 1 ]; then + printf 'finished-empty\n' + elif member_still_present "$id"; then + printf 'working\n' + else + printf 'gone-empty\n' + fi + fi +} + +# The two conditions are different facts and the operator needs to know which one +# is missing, so each names itself. +member_wait_reason() { # + local id=$1 + case "$(member_state "$id" "$2" "$3")" in + finished-empty) + printf '%s has finished but its report %s is empty or absent\n' "$id" "$DATA/$id/report.md" ;; + gone-empty) + printf '%s is gone without ever writing a terminal status event or a report at %s\n' \ + "$id" "$DATA/$id/report.md" ;; + unsignalled-gone) + printf '%s left a report but is gone without ever writing a terminal status event\n' "$id" ;; + unsignalled-live) + printf '%s left a report but has not written its terminal status event yet\n' "$id" ;; + working) + printf '%s has written no terminal status event yet (no done: or failed: line in %s) and no report\n' \ + "$id" "$STATE/$id.status" ;; + esac +} + +# The one state that can never clear on its own: the member is gone, so the +# terminal line will never arrive and only an operator can decide. This advice is +# deliberately NOT printed for a member that is still present, because the gap +# between writing a report and declaring done is the ordinary write-then-declare +# window, and offering the override there would invite the operator to recreate +# the exact race this gate exists to close. +member_unsignalled_advice() { # + local panel_id=$1 id=$2 + printf 'panel: %s left a non-empty report at %s but never wrote a terminal status event, and it has no runtime record left, so it was torn down or died and that line will never arrive.\n' \ + "$id" "$DATA/$id/report.md" + printf 'panel: To accept its report as final, naming that one member explicitly:\n' + printf 'panel: %s advance %s --accept-unfinished %s\n' "$0" "$panel_id" "$id" + printf 'panel: That accepts a report which may be INCOMPLETE: it is recorded permanently in %s, and the judge is told to treat that report as possibly truncated.\n' \ + "$(panel_meta_path "$panel_id")" + printf 'panel: There is no timeout and nothing advances on its own; this panel waits until you decide.\n' +} + +# The panel's premise has failed: a member stopped writing without leaving the +# report the formation is built on, so no report can arrive and there is nothing +# to judge. +panel_stood_down_block() { # ... + local panel_id=$1 id + shift + printf 'stood down: panel %s cannot be completed.\n' "$panel_id" + for id in "$@"; do + printf 'panel: %s has stopped writing and left no report at %s, so no report can arrive.\n' \ + "$id" "$DATA/$id/report.md" + done + printf 'panel: --accept-unfinished deliberately refuses to waive a report that does not exist: a verdict built on what survives is not a panel.\n' + printf 'panel: Stand this panel down and tell the captain it produced no verdict.\n' + # shellcheck disable=SC2016 # the backticks are markdown emphasis in the printed block, not a command substitution + printf 'panel: If the surviving analysis is still wanted, that is a NEW single-analyst review started deliberately with `start --reduced`, which is labelled as such everywhere, never this panel converted in place.\n' +} + +# A dead JUDGE is a different failure from a dead analyst: the evidence is +# complete and intact, only the adjudication is missing, so re-judging the same +# unchanged reports preserves the formation's property exactly rather than +# salvaging around it. +panel_rejudge_block() { # + local panel_id=$1 id=$2 + printf 'verdict lost: panel %s has every analyst report but no verdict.\n' "$panel_id" + printf 'panel: %s has stopped writing and left no verdict at %s, so the adjudication is missing while the evidence is untouched.\n' \ + "$id" "$DATA/$id/report.md" + printf 'panel: Every analyst report is complete and unchanged, so a fresh judge can adjudicate exactly the same evidence:\n' + printf 'panel: %s advance %s --rejudge\n' "$0" "$panel_id" + printf 'panel: That dispatches ONE replacement judge over those same reports, records %s as superseded in %s, and stamps no incompleteness on any report.\n' \ + "$id" "$(panel_meta_path "$panel_id")" + printf 'panel: Nothing re-dispatches on its own; this panel waits until you decide.\n' +} + +# Print the gate outcome for these members and return: +# 0 the panel must not advance yet; a `waiting:` or `wedged:` outcome was printed +# 1 every member is ready +# 2 a member ended terminal with no report; the caller owns that outcome, +# because it costs a panel its premise but costs a judge only its verdict +# Every non-terminal outcome names the next command. Members observed ready for +# the first time are reported in PANEL_NEWLY_READY for the caller to latch. +PANEL_STOOD_DOWN='' +PANEL_NEWLY_READY='' +panel_gate_wait() { # ... + local panel_id=$1 sentence=$2 hint=$3 accepted=$4 latched=$5 + shift 5 + local id state reasons='' + local -a gone=() empty=() + PANEL_NEWLY_READY='' + for id in "$@"; do + if ! member_ready_latched "$id" "$latched" && member_observed_ready "$id"; then + PANEL_NEWLY_READY="${PANEL_NEWLY_READY:+$PANEL_NEWLY_READY }$id" + fi + state=$(member_state "$id" "$accepted" "$latched") + [ "$state" != ready ] || continue + [ "$state" != unsignalled-gone ] || gone+=("$id") + [ "$state" != finished-empty ] || empty+=("$id") + [ "$state" != gone-empty ] || empty+=("$id") + reasons="${reasons:+$reasons; }$(member_wait_reason "$id" "$accepted" "$latched")" + done + [ -n "$reasons" ] || return 1 + if [ "${#empty[@]}" -gt 0 ]; then + PANEL_STOOD_DOWN="${empty[*]}" + return 2 + fi + if [ "${#gone[@]}" -gt 0 ]; then + printf 'wedged: %s; %s\n' "$sentence" "$reasons" + for id in "${gone[@]}"; do + member_unsignalled_advice "$panel_id" "$id" + done + else + printf 'waiting: %s; %s\n' "$sentence" "$reasons" + fi + panel_next_hint "$panel_id" "$hint" + return 0 +} + +# The completion output, which must carry the accepted-unfinished caveat every +# time it is printed rather than only on the invocation that recorded it: a +# caveat the operator learns second-hand is not a caveat. +panel_complete_output() { # + printf 'complete: %s\n' "$1" + if member_accepted_unfinished "$2" "$3"; then + printf 'panel: CAVEAT: this verdict was accepted without %s ever writing a terminal status event, so the report may be INCOMPLETE; say so when you relay it.\n' "$2" + fi + printf 'panel: read that report, relay its findings, and pass the shared completion gate before standing the panel down\n' +} + +# Record the dead panel durably, so the record says honestly what happened +# instead of reading like a panel still waiting on a report. +panel_stand_down() { # + panel_meta_set "$1" stood_down "$PANEL_STOOD_DOWN" + panel_meta_set "$1" stage stood-down + exit 1 +} + +# Move the readiness just observed into the panel record, which lives under +# data/ and genuinely survives teardown, and echo the updated list. +panel_latch_ready() { # + local meta=$1 latched=$2 + if [ -n "$PANEL_NEWLY_READY" ]; then + latched="${latched:+$latched }$PANEL_NEWLY_READY" + panel_meta_set "$meta" ready_members "$latched" + fi + printf '%s\n' "$latched" +} + +# The task id for a replacement judge. The panel id is capped so that every id +# this can generate stays within the 64-character task-id limit. +panel_next_judge_id() { # + local panel_id=$1 n=2 id + for id in $2; do + n=$((n + 1)) + done + [ "$n" -le 99 ] || die "panel '$panel_id' has already replaced its judge $((n - 2)) times; stand it down rather than replacing it again" + printf '%s-judge%s\n' "$panel_id" "$n" +} + +cmd_status() { + local panel_id=${1:-} + [ -n "$panel_id" ] || die "status requires a panel id" 2 + local meta + meta=$(require_panel "$panel_id") + cat "$meta" +} + +cmd_advance() { + local panel_id='' want_value='' arg rejudge=0 + local -a accept=() + for arg in "$@"; do + if [ -n "$want_value" ]; then + accept+=("$arg") + want_value= + continue + fi + case "$arg" in + --accept-unfinished) want_value=accept ;; + --accept-unfinished=*) accept+=("${arg#--accept-unfinished=}") ;; + --rejudge) rejudge=1 ;; + --*) die "unknown option $arg" 2 ;; + *) + [ -z "$panel_id" ] || die "advance takes exactly one panel id" 2 + panel_id=$arg + ;; + esac + done + [ -z "$want_value" ] || die "--accept-unfinished requires a task id" 2 + [ -n "$panel_id" ] || die "advance requires a panel id" 2 + local meta form stage project id_a id_b id_judge profile_judge question_path + meta=$(require_panel "$panel_id") + form=$(panel_meta_get "$meta" form) + stage=$(panel_meta_get "$meta" stage) + project=$(panel_meta_get "$meta" project) + id_a=$(panel_meta_get "$meta" analyst_a_task) + id_b=$(panel_meta_get "$meta" analyst_b_task) + id_judge=$(panel_meta_get "$meta" judge_task) + profile_judge=$(panel_meta_get "$meta" judge_profile) + question_path=$(panel_meta_get "$meta" question) + local report_a="$DATA/$id_a/report.md" report_judge="$DATA/$id_judge/report.md" + local report_b='' + [ -z "$id_b" ] || report_b="$DATA/$id_b/report.md" + + local accepted latched + accepted=$(panel_meta_get "$meta" accepted_unfinished) + latched=$(panel_meta_get "$meta" ready_members) + + # The stage is settled before any acceptance is recorded, so the panel record + # never carries verdict provenance for a panel that cannot use it. + case "$stage" in + incomplete) + die "panel '$panel_id' never finished dispatching; decide whether to retry it or stand it down before advancing" + ;; + stood-down) + # shellcheck disable=SC2046 # the recorded member list is a space-separated field, deliberately split + panel_stood_down_block "$panel_id" $(panel_meta_get "$meta" stood_down) + exit 1 + ;; + complete) + panel_complete_output "$report_judge" "$id_judge" "$accepted" + return 0 + ;; + judge|analysts) ;; + *) die "panel '$panel_id' has an unknown stage '$stage'" ;; + esac + + if [ "${#accept[@]}" -gt 0 ]; then + local want + for want in "${accept[@]}"; do + case "$want" in + "$id_a"|"$id_judge") : ;; + "${id_b:-$id_a}") : ;; + *) die "'$want' is not a member of panel '$panel_id' (its members are $id_a${id_b:+, $id_b}, and $id_judge)" 2 ;; + esac + [ -s "$DATA/$want/report.md" ] \ + || die "$want has no report at $DATA/$want/report.md, so there is nothing to accept; a missing report is never waived" 2 + if member_ready_latched "$want" "$latched" || member_observed_ready "$want"; then + die "$want finished and left a report at $DATA/$want/report.md, so there is nothing to waive; the caveat is only for a report whose author never declared it finished" 2 + fi + if member_still_present "$want"; then + warn "$want still has a runtime record, so it may STILL BE WRITING $DATA/$want/report.md; accepting it now risks judging a report that is not finished" + fi + member_accepted_unfinished "$want" "$accepted" \ + || accepted="${accepted:+$accepted }$want" + done + panel_meta_set "$meta" accepted_unfinished "$accepted" + printf 'panel: recorded %s as accepted without a terminal status event; that report may be incomplete and the panel record now says so permanently\n' \ + "${accept[*]}" + fi + + local gate=0 superseded='' superseded_of='' + if [ "$rejudge" -eq 1 ]; then + [ "$stage" = judge ] \ + || die "--rejudge replaces a judge that was dispatched and left no verdict, but panel '$panel_id' is at stage '$stage'" 2 + [ ! -s "$report_judge" ] \ + || die "$id_judge left a verdict at $report_judge, so there is nothing to re-judge; discarding a written verdict is not this command's job" 2 + # Refused only while the judge still LOOKS live. Requiring a terminal event + # outright would make a torn-down judge unreplaceable forever, because + # teardown removes the status file that would prove it. + if member_still_present "$id_judge" && ! status_has_finished_event "$STATE/$id_judge.status"; then + printf 'error: %s still has a runtime record and has written no terminal status event, so it appears to be running and may still be writing its verdict.\n' "$id_judge" >&2 + printf ' Wait for it to append done: or failed:, and re-run --rejudge if it finishes without a verdict.\n' >&2 + printf ' If you know it is dead, tear it down first; a judge with no runtime record can be replaced immediately.\n' >&2 + exit 2 + fi + superseded=$(panel_meta_get "$meta" superseded_judges) + local replacement + replacement=$(panel_next_judge_id "$panel_id" "$superseded") + # A directory left by a failed replacement dispatch is ours to reuse, on the + # same terms as the judge path below: only a report or a runtime record means + # the id is genuinely taken. + [ ! -s "$DATA/$replacement/report.md" ] \ + || die "the replacement judge task $replacement already has a report at $DATA/$replacement/report.md; reconcile it before re-judging" + printf 'panel: replacing %s with %s over the same unchanged analyst reports\n' "$id_judge" "$replacement" + superseded_of=$id_judge + id_judge=$replacement + report_judge="$DATA/$id_judge/report.md" + elif [ "$stage" = judge ]; then + panel_gate_wait "$panel_id" \ + 'the panel is complete only once the judge has finished AND left a non-empty report' \ + "once the judge reports done" "$accepted" "$latched" "$id_judge" || gate=$? + latched=$(panel_latch_ready "$meta" "$latched") + case "$gate" in + 0) return 0 ;; + 2) + panel_rejudge_block "$panel_id" "$id_judge" + return 0 + ;; + esac + panel_meta_set "$meta" stage complete + panel_complete_output "$report_judge" "$id_judge" "$accepted" + return 0 + else + panel_gate_wait "$panel_id" \ + 'the judge is created only once every analyst has finished AND left a non-empty report' \ + "after the next analyst finishes" "$accepted" "$latched" "$id_a" ${id_b:+"$id_b"} || gate=$? + latched=$(panel_latch_ready "$meta" "$latched") + case "$gate" in + 0) return 0 ;; + 2) + # shellcheck disable=SC2086 # the member list is space-separated, deliberately split + panel_stood_down_block "$panel_id" $PANEL_STOOD_DOWN + panel_stand_down "$meta" + ;; + esac + fi + + # The judge task was never dispatched if it has no runtime record, so a brief + # left behind by a failed dispatch is ours to regenerate. + if [ -f "$STATE/$id_judge.meta" ]; then + die "the judge task $id_judge already has a runtime record but the panel stage is '$stage'; reconcile that task before advancing" + fi + rm -f "$DATA/$id_judge/brief.md" + + local unfinished='' label_a='Analyst A' + [ "$form" = panel ] || label_a='Analyst' + if member_accepted_unfinished "$id_a" "$accepted"; then + unfinished="- $label_a: \`$report_a\`" + fi + if [ -n "$id_b" ] && member_accepted_unfinished "$id_b" "$accepted"; then + unfinished="${unfinished:+$unfinished +}- Analyst B: \`$report_b\`" + fi + + local task_judge + trap panel_cleanup EXIT + task_judge=$(panel_mktemp) + compose_judge_task "$task_judge" "$question_path" "$report_judge" "$form" "$report_a" "$report_b" "$unfinished" + scaffold_scout_brief "$id_judge" "$(basename "$project")" "$task_judge" + dispatch_scout "$id_judge" "$project" "$profile_judge" \ + || die "could not dispatch the judge ($id_judge); every analyst report is still in place, so this is safe to retry" + if [ -n "$superseded_of" ]; then + panel_meta_set "$meta" superseded_judges "${superseded:+$superseded }$superseded_of" + panel_meta_set "$meta" judge_task "$id_judge" + fi + panel_meta_set "$meta" stage judge + panel_next_hint "$panel_id" "once the judge reports done" +} + +# --- dispatch --------------------------------------------------------------- + +COMMAND=${1:-} +[ "$#" -eq 0 ] || shift +case "$COMMAND" in + start) cmd_start "$@" ;; + advance) cmd_advance "$@" ;; + status) cmd_status "$@" ;; + '') usage >&2; exit 2 ;; + *) die "unknown command '$COMMAND' (start, advance, status)" 2 ;; +esac diff --git a/bin/fm-test-run.sh b/bin/fm-test-run.sh index df116be9b8..26db1b4ab3 100755 --- a/bin/fm-test-run.sh +++ b/bin/fm-test-run.sh @@ -123,7 +123,8 @@ family_for_basename() { fm-continuity-pretool-check.test.sh|fm-crew-state.test.sh|fm-decision-hold-lifecycle.test.sh|\ fm-dispatch-select.test.sh|fm-ensure-agents-md.test.sh|fm-grok-harness.test.sh|\ fm-herdr-lab.test.sh|fm-instruction-owners.test.sh|fm-lint.test.sh|\ - fm-install-herdr.test.sh|fm-nm-test-contract.test.sh|fm-no-mistakes-ownership.test.sh|\ + fm-install-herdr.test.sh|fm-model-panel.test.sh|fm-nm-test-contract.test.sh|\ + fm-no-mistakes-ownership.test.sh|\ fm-operational-input.test.sh|fm-pi-primary-types.test.sh|\ fm-send-popup-settle.test.sh|fm-send-settle.test.sh|fm-stow-contract.test.sh|\ fm-subagent-pretool-check.test.sh|\ @@ -637,8 +638,14 @@ families_for_changed_path() { printf '%s\n' backend-dispatch printf '%s\n' real-herdr-gated ;; + bin/fm-classify-lib.sh) + printf '%s\n' watcher-wake-lock + # The status-event vocabulary also backs bin/fm-model-panel.sh's judge + # gate, whose test lives in the pure-contract-unit family. + printf '%s\n' pure-contract-unit + ;; bin/fm-watch*|bin/fm-wake*|\ - bin/fm-classify-lib.sh|bin/fm-daemon*|bin/fm-turnend-guard*|bin/fm-guard.sh) + bin/fm-daemon*|bin/fm-turnend-guard*|bin/fm-guard.sh) printf '%s\n' watcher-wake-lock ;; bin/fm-afk*) @@ -679,7 +686,7 @@ families_for_changed_path() { printf '%s\n' real-herdr-gated ;; bin/fm-lint.sh|bin/fm-install-shellcheck.sh|\ - bin/fm-brief.sh|bin/fm-ensure-agents-md.sh|bin/fm-crew-state.sh|\ + bin/fm-brief.sh|bin/fm-ensure-agents-md.sh|bin/fm-crew-state.sh|bin/fm-model-panel.sh|\ bin/fm-decision-hold.sh|bin/fm-supervision*|bin/fm-transition-lib.sh|\ bin/fm-tmux-lib.sh|bin/fm-marker-lib.sh|bin/fm-operational-input.sh|bin/fm-tasks-axi-lib.sh|\ bin/fm-primary-scope-lib.sh|bin/fm-project-mode.sh|bin/fm-promote.sh|\ diff --git a/docs/architecture.md b/docs/architecture.md index 10c4df0c4c..9966fb2648 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -130,7 +130,7 @@ Ship briefs also tell the crewmate to verify `pwd -P` and `git rev-parse --show- Firstmate's own no-mistakes gate runs agents inside a checkout that also contains the fleet-captain identity in `AGENTS.md`, so gate execution needs an authority boundary separate from ordinary crewmate worktree isolation. The tracked `.no-mistakes.yaml` sets `disable_project_settings: true`; no-mistakes honors that setting only from the trusted default-branch copy, so a pushed branch cannot enable its own project instructions during validation. -Independently, `fm-spawn.sh`, `fm-send.sh`, and `fm-teardown.sh` source `bin/fm-gate-refuse-lib.sh` and exit with status 3 before fleet mutation when the gate environment marker is present or the current checkout matches the default no-mistakes gate-repository topology. +Independently, `fm-spawn.sh`, `fm-send.sh`, `fm-teardown.sh`, and `fm-model-panel.sh` source `bin/fm-gate-refuse-lib.sh` and exit with status 3 before fleet mutation when the gate environment marker is present or the current checkout matches the default no-mistakes gate-repository topology. A normal primary checkout or crewmate worktree has neither signal and remains unaffected. The helper's header owns the exact signal detection, relocated-home limitation, test-harness bypass, and relationship to no-mistakes' HEAD-continuity guard. diff --git a/docs/configuration.md b/docs/configuration.md index 553717e53b..fcc44b2ab9 100644 --- a/docs/configuration.md +++ b/docs/configuration.md @@ -338,6 +338,47 @@ Malformed JSON, an empty or malformed rule/default array, an unverified harness, Because the spawn backstop is gated by file presence, any fallback path after a missing match, validation error, or missing `jq` still passes a resolved harness explicitly until the file is fixed or removed. Secondmate homes inherit this file from the primary, so a secondmate's own crewmates apply the same dispatch profile behavior. +## Model panel roles (config/model-panel.json) + +A model panel has two analysts answering one question independently on different models and a third model judging both reports. +The roles are pinned in `bin/fm-model-panel.sh`; the models filling them are configuration, because a model name in tracked code rots silently and a home can only run a panel on the models it can actually reach. +This section is the single owner of the panel's configuration schema, resolution order, and degradation contract; the script's header owns its commands and flags, and the [`panel` skill](../.agents/skills/panel/SKILL.md) owns when a panel is worth its cost. + +`config/model-panel.json` is an optional local, gitignored file mapping each role to a dispatch profile. + +```json +{ + "roles": { + "analyst_a": { "harness": "", "model": "", "effort": "" }, + "analyst_b": { "harness": "", "model": "", "effort": "" }, + "judge": [ + { "harness": "", "model": "", "effort": "" } + ] + } +} +``` + +A role's value is one profile object or a non-empty array of them, exactly the shape "Crew dispatch profiles" above defines, and every role resolves through `bin/fm-dispatch-select.sh`. +Panel profiles therefore get the same validation and the same quota-aware array selection as crew dispatch profiles, with no second selector. +An omitted `model` or `effort` leaves that axis at the harness's own default; there is deliberately no injected effort default, because a profile carrying an effort its harness does not accept is rejected as an invalid pair. +Panels are ambiguous investigation work, so a high effort level is usually the right configured value - see [`docs/examples/model-panel.json`](examples/model-panel.json) for a starting point to copy. + +Each role resolves in this order: its entry in `config/model-panel.json`, then the top-level `default` profile set in `config/crew-dispatch.json`. +That fallback is the documented default, and it is why a home that already declares which runtimes it dispatches on can run a panel with no panel-specific configuration at all. +When neither file supplies a profile for a role, the panel refuses and names both files rather than guessing a model. +A role backed by an array prefers a candidate whose model the panel is not already using, so the second analyst picks a second model instead of duplicating the first. + +Model identity for that comparison is the profile's model name with any provider prefix and any `:suffix` removed - the normalization `bin/fm-dispatch-select.sh` already uses - or `harness:` when the profile pins no model. +Two profiles naming the same model through different harnesses are therefore correctly one model, not two. + +Degradation is explicit and never silent. +When both analysts would resolve to the same model identity, `start` refuses with exit 4 and names both the configuration fix and the reduced form; two identical analysts are not independent, and presenting them as a panel is worse than running none. +The reduced form is opt-in through `--reduced` and is recorded and labelled everywhere as a single-analyst review rather than a panel, in the briefs, in the panel record, and in the judge's own report. +When no third distinct model is available, the judge may share an analyst's model; that prints a warning and proceeds, because the judge's independence comes from re-verifying claims against live state with every report in hand. + +`config/model-panel.json` is deliberately NOT in the inheritable set that `bin/fm-config-inherit-lib.sh` declares, for the same reason as `config/backend`: it names the models a specific home can actually reach, and pushing the primary's list into every secondmate would overwrite exactly the local knowledge that lets each home field a real panel. +A secondmate home that needs a different lineup writes its own file, and a home that writes none still inherits the primary's `config/crew-dispatch.json` default profile set through the normal inheritance path. + ## Toolchain On session start the first mate detects what its required toolchain is missing or too old and lists each problem with either an exact install command or manual instructions. diff --git a/docs/examples/model-panel.json b/docs/examples/model-panel.json new file mode 100644 index 0000000000..0c959c87eb --- /dev/null +++ b/docs/examples/model-panel.json @@ -0,0 +1,10 @@ +{ + "roles": { + "analyst_a": { "harness": "claude", "model": "claude-opus-5", "effort": "xhigh" }, + "analyst_b": { "harness": "codex", "model": "gpt-5.6-sol", "effort": "xhigh" }, + "judge": [ + { "harness": "grok", "model": "grok-4-fast", "effort": "high" }, + { "harness": "claude", "model": "claude-opus-5", "effort": "xhigh" } + ] + } +} diff --git a/docs/scripts.md b/docs/scripts.md index de61bde060..a07d2ceff4 100644 --- a/docs/scripts.md +++ b/docs/scripts.md @@ -24,6 +24,7 @@ The shared no-mistakes gate refusal for fleet lifecycle entrypoints is summarize | `fm-backlog-handoff.sh` | Validate and delegate queued backlog-item moves into a secondmate home | | `fm-decision-hold.sh` | Create, verify, complete, and resolve durable captain-held decisions | | `fm-brief.sh` | Scaffold ship, scout, secondmate-charter, and Herdr-lab briefs | +| `fm-model-panel.sh` | Run a model panel: two independent analysts on different models, then a judge that re-verifies both | | `fm-herdr-lab.sh` | Provision and guardedly operate an isolated, never-default Herdr lab session | | `fm-install-herdr.sh` | Install CI's exact-version Herdr pin with official asset URL, SHA-256, and protocol checks | | `fm-install-treehouse.sh`| Install CI's exact-version Treehouse pin for real-Herdr E2E that needs spawn worktrees | diff --git a/tests/fm-gate-refuse.test.sh b/tests/fm-gate-refuse.test.sh index a61765d401..1ca4f20850 100755 --- a/tests/fm-gate-refuse.test.sh +++ b/tests/fm-gate-refuse.test.sh @@ -2,9 +2,10 @@ # Behavior tests for the no-mistakes GATE-agent fleet-lifecycle refusal. # # A confused no-mistakes gate agent runs inside a firstmate checkout, adopts the -# captain identity from AGENTS.md, and reaches for fm-spawn/fm-send/fm-teardown. +# captain identity from AGENTS.md, and reaches for a fleet-mutating entrypoint: +# fm-spawn, fm-send, fm-teardown, or fm-model-panel. # bin/fm-gate-refuse-lib.sh is the firstmate capability-removal half: sourced at -# the top of those three entrypoints and called before any fleet mutation, it +# the top of those entrypoints and called before any fleet mutation, it # fails closed on either of two independent signals: # 1. NO_MISTAKES_GATE set in the environment (the marker no-mistakes stamps); # 2. the current worktree's git-common-dir resolves under a no-mistakes gate @@ -34,6 +35,7 @@ GATE_LIB="$ROOT/bin/fm-gate-refuse-lib.sh" SPAWN="$ROOT/bin/fm-spawn.sh" SEND="$ROOT/bin/fm-send.sh" TEARDOWN="$ROOT/bin/fm-teardown.sh" +PANEL="$ROOT/bin/fm-model-panel.sh" fm_test_tmproot TMP fm-gate-refuse @@ -359,6 +361,83 @@ test_teardown_refuses_and_admits() { pass "fm-teardown: refuses on marker and gate-worktree backstop; a normal teardown is unaffected" } +# --- fm-model-panel --------------------------------------------------------- + +# make_panel_home -> echoes a home whose panel configuration resolves a +# full lineup, so a non-refused run gets all the way past role resolution. The +# quota reader is pointed at an absent command, so nothing reaches a live +# service; the refusal must fire before any of it anyway. +make_panel_home() { + local name=$1 home + home="$TMP/$name" + mkdir -p "$home/data" "$home/state" "$home/config" "$home/projects" "$home/subject" + cat > "$home/config/model-panel.json" <<'JSON' +{"roles":{ + "analyst_a":{"harness":"claude","model":"claude-opus-5"}, + "analyst_b":{"harness":"codex","model":"gpt-5.6-sol"}, + "judge":{"harness":"grok","model":"grok-4-fast"}}} +JSON + printf '%s\n' "$home" +} + +# The panel dispatches through the real bin/fm-spawn.sh, so it gets the SAME +# stubbed dispatch path the fm-spawn section builds: a regression that let the +# guard through could otherwise reach a real backend session before the +# assertions report it. +PANEL_FAKEBIN=$(make_spawn_fakebin "$TMP/panel-fake") + +# run_panel [ASSIGN...] -- -> combined output +run_panel() { + local cwd=$1 home=$2; shift 2 + local -a assigns=() + while [ "$#" -gt 0 ] && [ "$1" != "--" ]; do + assigns+=("$1"); shift + done + [ "$#" -eq 0 ] || shift + ( cd "$cwd" && env -u NO_MISTAKES_GATE -u FM_GATE_REFUSE_BYPASS \ + "FM_ROOT_OVERRIDE=$ROOT" "FM_HOME=$home" \ + "FM_DATA_OVERRIDE=$home/data" "FM_STATE_OVERRIDE=$home/state" \ + "FM_CONFIG_OVERRIDE=$home/config" "FM_PROJECTS_OVERRIDE=$home/projects" \ + "FM_DISPATCH_QUOTA_AXI=fm-test-absent-quota-axi" \ + "FM_SPAWN_NO_GUARD=1" "FM_FAKE_PANE_PATH=$home/subject" "TMUX=fake,1,0" \ + "PATH=$PANEL_FAKEBIN:$PATH" \ + ${assigns[@]+"${assigns[@]}"} \ + "$PANEL" "$@" ) 2>&1 +} + +test_panel_refuses_and_admits() { + local home out rc + + # env-marker refuse: neutral cwd, marker set. A real `start` is used, so the + # guard has to fire BEFORE role resolution and any panel record is written. + home=$(make_panel_home panel-envmark) + out=$(run_panel "$NORMAL_CWD" "$home" NO_MISTAKES_GATE=1 -- \ + start --id gp --project "$home/subject" "Anything?"); rc=$? + expect_code 3 "$rc" "panel: NO_MISTAKES_GATE must refuse" + assert_contains "$out" "$ENV_MSG" "panel: env-marker refusal message" + assert_absent "$home/data/gp" "panel: refused env-marker start must not write a panel record" + assert_absent "$home/data/gp-a" "panel: refused env-marker start must not scaffold an analyst" + + # path-backstop refuse: gate-worktree cwd, marker UNSET. + home=$(make_panel_home panel-backstop) + out=$(run_panel "$GATE_WT" "$home" -- start --id gp --project "$home/subject" "Anything?"); rc=$? + expect_code 3 "$rc" "panel: gate-worktree cwd must refuse with the marker unset" + assert_contains "$out" "$PATH_MSG" "panel: path-backstop refusal message" + assert_absent "$home/data/gp" "panel: refused backstop start must not write a panel record" + assert_absent "$home/data/gp-a" "panel: refused backstop start must not scaffold an analyst" + + # no-regression: neutral cwd, marker UNSET. --dry-run resolves the real lineup + # and prints it, which is everything up to dispatch without spending a model. + home=$(make_panel_home panel-ok) + out=$(run_panel "$NORMAL_CWD" "$home" -- \ + start --id gp --project "$home/subject" --dry-run "Anything?"); rc=$? + expect_code 0 "$rc" "panel: a normal session must still resolve a panel" + assert_contains "$out" "form=panel" "panel: normal dry run should print the resolved lineup" + assert_not_contains "$out" "$ENV_MSG" "panel: normal run must not print the gate refusal" + assert_not_contains "$out" "$PATH_MSG" "panel: normal run must not print the backstop refusal" + pass "fm-model-panel: refuses on marker and gate-worktree backstop; a normal panel is unaffected" +} + # --- tracked .no-mistakes.yaml ---------------------------------------------- test_no_mistakes_yaml_disables_project_settings() { @@ -396,4 +475,5 @@ test_helper_normal_is_noop test_spawn_refuses_and_admits test_send_refuses_and_admits test_teardown_refuses_and_admits +test_panel_refuses_and_admits test_no_mistakes_yaml_disables_project_settings diff --git a/tests/fm-model-panel.test.sh b/tests/fm-model-panel.test.sh new file mode 100755 index 0000000000..5895ac0392 --- /dev/null +++ b/tests/fm-model-panel.test.sh @@ -0,0 +1,792 @@ +#!/usr/bin/env bash +# Behavior tests for fm-model-panel.sh: role resolution, the honest +# single-model degradation, the different-models guarantee, the judge gate, and +# the brief contract both analysts and the judge receive. +# +# These drive the real script with the real fm-brief.sh scaffold and the real +# fm-dispatch-select.sh selector, against a copied bin/ whose fm-spawn.sh is +# replaced by a stub that records its argv. That keeps the assertions on what +# the panel ASKS the fleet to dispatch, without starting a backend, a worktree, +# or a harness. Quota data is deliberately unavailable, so an array-backed role +# falls back to the selector's uniform random choice and no test reaches a live +# quota service. +# shellcheck disable=SC2016 +set -u + +# shellcheck source=tests/lib.sh +. "$(dirname "${BASH_SOURCE[0]}")/lib.sh" + +fm_test_tmproot TMP_ROOT fm-model-panel + +FAKE_ROOT="$TMP_ROOT/root" +mkdir -p "$FAKE_ROOT/.agents/skills/decision-hold-lifecycle" +cp -a "$ROOT/bin" "$FAKE_ROOT/bin" +printf 'decision-hold stub\n' > "$FAKE_ROOT/.agents/skills/decision-hold-lifecycle/SKILL.md" +cat > "$FAKE_ROOT/bin/fm-spawn.sh" <<'SH' +#!/usr/bin/env bash +# Records the spawn request and writes the runtime record fm-model-panel.sh +# checks, so the panel's dispatch contract is observable without a backend. +set -eu +printf '%s\n' "$*" >> "$FM_FAKE_SPAWN_LOG" +if [ -n "${FM_FAKE_SPAWN_FAIL_ID:-}" ] && [ "$1" = "$FM_FAKE_SPAWN_FAIL_ID" ]; then + echo "fake spawn refused $1" >&2 + exit 1 +fi +printf 'window=fake:%s\nkind=scout\n' "$1" > "$FM_STATE_OVERRIDE/$1.meta" +echo "spawned $1" +SH +chmod +x "$FAKE_ROOT/bin/fm-spawn.sh" + +PANEL="$FAKE_ROOT/bin/fm-model-panel.sh" + +# new_home [panel-config-json]: build an isolated home and echo its path. +new_home() { + local config_json=${2:-} home="$TMP_ROOT/$1" + mkdir -p "$home/data" "$home/state" "$home/config" "$home/projects" "$home/subject" + [ -z "$config_json" ] || printf '%s\n' "$config_json" > "$home/config/model-panel.json" + printf '%s\n' "$home" +} + +run_panel() { + local home=$1 + shift + FM_ROOT_OVERRIDE="$FAKE_ROOT" FM_HOME="$home" \ + FM_DATA_OVERRIDE="$home/data" FM_STATE_OVERRIDE="$home/state" \ + FM_CONFIG_OVERRIDE="$home/config" FM_PROJECTS_OVERRIDE="$home/projects" \ + FM_FAKE_SPAWN_LOG="$home/spawn.log" \ + FM_DISPATCH_QUOTA_AXI=fm-test-absent-quota-axi \ + "$PANEL" "$@" 2>&1 +} + +spawn_log() { + cat "$1/spawn.log" 2>/dev/null || true +} + +# Append one status EVENT for a panel member, exactly as its scout scaffold tells +# it to. The judge gate reads these, so tests must produce them the same way. +say_status() { # + printf '%s\n' "$3" >> "$1/state/$2.status" +} + +TWO_MODELS='{"roles":{ + "analyst_a":{"harness":"claude","model":"claude-opus-5","effort":"xhigh"}, + "analyst_b":{"harness":"codex","model":"gpt-5.6-sol","effort":"xhigh"}, + "judge":{"harness":"grok","model":"grok-4-fast","effort":"high"}}}' + +ONE_MODEL='{"roles":{ + "analyst_a":{"harness":"claude","model":"claude-opus-5"}, + "analyst_b":{"harness":"claude","model":"claude-opus-5"}, + "judge":{"harness":"claude","model":"claude-opus-5"}}}' + +# The same model reached through two different harnesses is still one model. +ONE_MODEL_TWO_HARNESSES='{"roles":{ + "analyst_a":{"harness":"claude","model":"claude-opus-5"}, + "analyst_b":{"harness":"pi","model":"anthropic/claude-opus-5:1m"}, + "judge":{"harness":"claude","model":"claude-opus-5"}}}' + +SKILL="$ROOT/.agents/skills/panel/SKILL.md" +CONFIG_DOC="$ROOT/docs/configuration.md" +AGENTS="$ROOT/AGENTS.md" + +test_skill_owns_the_cost_decision_and_one_trigger() { + local count + assert_present "$SKILL" "the panel skill is missing" + assert_grep 'name: panel' "$SKILL" "the panel skill metadata has the wrong name" + assert_grep 'user-invocable: true' "$SKILL" "the panel skill must be captain-invocable" + assert_grep ' internal: true' "$SKILL" "the panel skill must be internal" + assert_grep 'bin/fm-model-panel.sh' "$SKILL" "the panel skill does not point at its entry point" + assert_grep 'Is this question worth a panel' "$SKILL" \ + "the panel skill does not own the cost decision" + assert_grep 'Worth it:' "$SKILL" "the panel skill does not say when a panel earns its cost" + assert_grep 'Not worth it:' "$SKILL" "the panel skill does not say when a panel wastes its cost" + assert_grep 'decision-hold-lifecycle' "$SKILL" "the panel skill does not fold in the completion gate" + assert_grep 'single-analyst review' "$SKILL" "the panel skill does not name the reduced form" + count=$(grep -Fc 'load the `panel` skill' "$AGENTS") + [ "$count" -eq 1 ] || fail "the panel skill needs exactly one AGENTS.md trigger, found $count" + pass "the panel skill owns the cost decision and has one AGENTS.md trigger" +} + +test_configuration_doc_owns_the_schema_and_degradation() { + assert_grep '## Model panel roles (config/model-panel.json)' "$CONFIG_DOC" \ + "docs/configuration.md does not own the panel configuration schema" + assert_grep 'analyst_a' "$CONFIG_DOC" "the documented schema does not name the analyst roles" + assert_grep 'bin/fm-dispatch-select.sh' "$CONFIG_DOC" \ + "the documented schema does not point at the shared selector" + assert_grep 'refuses with exit 4' "$CONFIG_DOC" \ + "the documented degradation contract does not state the refusal" + assert_grep 'config/model-panel.json` is deliberately NOT in the inheritable set' "$CONFIG_DOC" \ + "the documented contract does not explain why the panel config is not inherited" + assert_grep 'config/model-panel.json' "$ROOT/.gitignore" \ + "the local panel configuration file is not gitignored" + assert_present "$ROOT/docs/examples/model-panel.json" "the copyable example config is missing" + pass "docs/configuration.md owns the panel schema, default, and degradation contract" +} + +test_analysts_dispatch_on_different_models() { + local home out log + home=$(new_home different-models "$TWO_MODELS") + out=$(run_panel "$home" start --id trio --project "$home/subject" "Which queued work is blocked?") \ + || fail "start failed: $out" + log=$(spawn_log "$home") + assert_contains "$log" "trio-a $home/subject --harness claude --scout --model claude-opus-5 --effort xhigh" \ + "analyst A was not dispatched on its configured profile" + assert_contains "$log" "trio-b $home/subject --harness codex --scout --model gpt-5.6-sol --effort xhigh" \ + "analyst B was not dispatched on its configured profile" + assert_not_contains "$log" "trio-judge" "the judge must not be dispatched before the reports exist" + assert_grep 'form=panel' "$home/data/trio/panel.meta" "the panel record does not say it is a panel" + pass "analysts dispatch concurrently on the two configured models" +} + +test_identical_analyst_models_refuse() { + local home out status=0 + home=$(new_home identical-models "$ONE_MODEL") + out=$(run_panel "$home" start --id solo --project "$home/subject" "Anything?") || status=$? + expect_code 4 "$status" "an identical-model panel must refuse with the degraded-refusal status" + assert_contains "$out" "this would not be a panel" "the refusal does not say why" + assert_contains "$out" "--reduced" "the refusal does not offer the named reduced form" + assert_contains "$out" "claude-opus-5" "the refusal does not name the duplicated model" + [ -z "$(spawn_log "$home")" ] || fail "a refused panel must dispatch nothing" + assert_absent "$home/data/solo" "a refused panel must leave no panel record behind" + pass "a single-model home refuses rather than faking independence" +} + +test_same_model_through_two_harnesses_refuses() { + local home status=0 out + home=$(new_home cross-harness "$ONE_MODEL_TWO_HARNESSES") + out=$(run_panel "$home" start --id xh --project "$home/subject" "Anything?") || status=$? + expect_code 4 "$status" "one model reached through two harnesses must still refuse" + assert_contains "$out" "claude-opus-5" "the refusal does not name the shared model" + pass "two harnesses onto one model are not treated as two models" +} + +test_reduced_form_is_named_not_a_panel() { + local home out log brief + home=$(new_home reduced "$ONE_MODEL") + out=$(run_panel "$home" start --id rd --project "$home/subject" --reduced "Anything?") \ + || fail "reduced start failed: $out" + log=$(spawn_log "$home") + assert_contains "$log" "rd-a " "the single analyst was not dispatched" + assert_not_contains "$log" "rd-b " "the reduced form must not dispatch a second analyst" + assert_grep 'form=single-analyst-review' "$home/data/rd/panel.meta" \ + "the reduced run must be recorded as a single-analyst review" + brief="$home/data/rd-a/brief.md" + assert_grep 'single-analyst review' "$brief" "the reduced analyst brief does not name the reduced form" + assert_no_grep 'analyst A on a firstmate model panel' "$brief" \ + "the reduced analyst brief must not claim to be a panel" + pass "the reduced form runs one analyst and is labelled as such" +} + +test_analyst_briefs_share_the_question_and_forbid_peeking() { + local home out brief_a brief_b question + home=$(new_home independence "$TWO_MODELS") + question="$TMP_ROOT/question.md" + printf 'Line one of the question.\nLine two with detail.\n' > "$question" + out=$(run_panel "$home" start --id ind --project "$home/subject" --question-file "$question") \ + || fail "start failed: $out" + brief_a="$home/data/ind-a/brief.md" + brief_b="$home/data/ind-b/brief.md" + for brief in "$brief_a" "$brief_b"; do + assert_grep 'Line one of the question.' "$brief" "an analyst brief lost the question" + assert_grep 'Line two with detail.' "$brief" "an analyst brief truncated the question" + assert_grep 'Verify status prose, never trust it' "$brief" \ + "an analyst brief lost the verify-status-prose discipline" + assert_grep 'Check your vantage point' "$brief" "an analyst brief lost the vantage-point check" + assert_grep 'Highest-conviction calls' "$brief" "an analyst brief lost the scored-calls contract" + assert_grep 'Unresolved captain decisions' "$brief" "an analyst brief lost the decisions inventory" + assert_grep 'decision-hold-lifecycle' "$brief" "an analyst brief lost the completion gate" + done + assert_grep "$home/data/ind-b/report.md" "$brief_a" "analyst A was not told which report to avoid" + assert_grep "$home/data/ind-a/report.md" "$brief_b" "analyst B was not told which report to avoid" + assert_grep 'Do not read their report at' "$brief_a" "analyst A was not forbidden to read the other report" + assert_grep 'Do not read their report at' "$brief_b" "analyst B was not forbidden to read the other report" + pass "both analysts get the identical question and an explicit independence rule" +} + +test_judge_waits_for_every_report_then_gets_both() { + local home out log brief + home=$(new_home judge-gate "$TWO_MODELS") + out=$(run_panel "$home" start --id jg --project "$home/subject" "Which claim holds up?") \ + || fail "start failed: $out" + + out=$(run_panel "$home" advance jg) || fail "advance failed: $out" + assert_contains "$out" "waiting:" "advance must wait while a report is missing" + assert_not_contains "$(spawn_log "$home")" "jg-judge" "the judge must not be dispatched early" + + printf 'analyst a findings\n' > "$home/data/jg-a/report.md" + out=$(run_panel "$home" advance jg) || fail "advance failed: $out" + assert_contains "$out" "waiting:" "advance must still wait while the second report is missing" + assert_contains "$out" "jg-b" "the waiting line must name the analyst with no report" + assert_not_contains "$(spawn_log "$home")" "jg-judge" "the judge must wait for every analyst report" + + # Both reports EXIST now, but neither analyst has said it is finished. A file + # that exists is not a file that is finished, so this must still wait, and a + # nonterminal progress line must not be mistaken for a terminal event. Both + # analysts are still present, so this is the ordinary write-then-declare window + # and the override must not be advertised in it. + printf 'analyst b findings\n' > "$home/data/jg-b/report.md" + say_status "$home" jg-a 'working: still drafting' + out=$(run_panel "$home" advance jg) || fail "advance failed: $out" + assert_contains "$out" "waiting:" "the ordinary write-then-declare window must print the plain waiting line" + assert_contains "$out" "terminal status event" "the outcome must name the missing terminal event" + assert_contains "$out" "jg-a" "the outcome must name the analyst that has not finished" + assert_not_contains "$out" "--accept-unfinished" \ + "the override must not be advertised while the analyst is still present and may be writing" + assert_not_contains "$(spawn_log "$home")" "jg-judge" \ + "the judge must not be dispatched against a report whose analyst is still writing" + + say_status "$home" jg-a 'done: analyst a finished' + say_status "$home" jg-b 'done: analyst b finished' + out=$(run_panel "$home" advance jg) || fail "advance failed: $out" + log=$(spawn_log "$home") + assert_contains "$log" "jg-judge $home/subject --harness grok --scout --model grok-4-fast --effort high" \ + "the judge was not dispatched on its configured profile" + assert_grep 'stage=judge' "$home/data/jg/panel.meta" "the panel record did not advance to the judge stage" + + brief="$home/data/jg-judge/brief.md" + assert_grep "$home/data/jg-a/report.md" "$brief" "the judge was not given analyst A's report" + assert_grep "$home/data/jg-b/report.md" "$brief" "the judge was not given analyst B's report" + assert_grep 'Re-verify, do not referee' "$brief" "the judge brief lost its re-verification instruction" + assert_grep 'Verify status prose, never trust it' "$brief" \ + "the judge brief lost the verify-status-prose discipline" + assert_grep 'Scored highest-conviction calls' "$brief" "the judge brief lost the scoring contract" + assert_grep 'Shared mistakes' "$brief" "the judge brief lost the shared-mistake hunt" + assert_grep 'Unresolved captain decisions' "$brief" "the judge brief lost the decisions inventory" + assert_grep 'decision-hold-lifecycle' "$brief" "the judge brief lost the completion gate" + assert_no_grep 'claude-opus-5' "$brief" "the judge must not be told which model wrote which report" + assert_no_grep 'gpt-5.6-sol' "$brief" "the judge must not be told which model wrote which report" + + # The judge's own report passes the SAME two-condition gate: a report that + # exists is not a report that is finished, on the final hop either. + printf 'partial verdict\n' > "$home/data/jg-judge/report.md" + out=$(run_panel "$home" advance jg) || fail "advance failed: $out" + assert_contains "$out" "waiting:" "a half-written judge report must not complete the panel" + assert_not_contains "$out" "complete:" "a half-written judge report must not complete the panel" + assert_contains "$out" "jg-judge" "the outcome must name the judge that has not finished" + assert_no_grep 'stage=complete' "$home/data/jg/panel.meta" \ + "the panel must not record itself complete over an unfinished judge" + + say_status "$home" jg-judge 'done: verdict written' + out=$(run_panel "$home" advance jg) || fail "advance failed: $out" + assert_contains "$out" "complete: $home/data/jg-judge/report.md" \ + "a finished panel does not report its judge report path" + assert_not_contains "$out" "CAVEAT" "a panel nobody overrode must not carry an accepted-unfinished caveat" + assert_grep 'stage=complete' "$home/data/jg/panel.meta" "the panel record did not reach the complete stage" + pass "the judge is created only once every analyst report exists, and sees both blind" +} + +test_terminal_member_without_a_report_stands_the_panel_down() { + local home out status=0 + home=$(new_home stand-down "$TWO_MODELS") + out=$(run_panel "$home" start --id sd --project "$home/subject" "Anything?") \ + || fail "start failed: $out" + + # A member that ended terminal with no report has stopped writing, so no report + # can arrive: the panel's premise has failed and it cannot be waited out. + printf 'analyst a findings\n' > "$home/data/sd-a/report.md" + say_status "$home" sd-a 'done: analyst a finished' + say_status "$home" sd-b 'failed: ran out of budget before writing anything' + out=$(run_panel "$home" advance sd) || status=$? + [ "$status" -ne 0 ] || fail "a panel that can never complete must not report success" + assert_contains "$out" "stood down:" "a member that finished with no report must not print the bland waiting line" + assert_contains "$out" "sd-b" "the stand-down block must name the member that left no report" + assert_contains "$out" "Stand this panel down" "the stand-down block must name the operator's action" + assert_contains "$out" "--reduced" "the stand-down block must name the deliberate salvage path" + assert_not_contains "$out" "next step is" "a dead panel must not hint at a next advance" + assert_grep 'stage=stood-down' "$home/data/sd/panel.meta" "the dead panel was not recorded durably" + assert_grep 'stood_down=sd-b' "$home/data/sd/panel.meta" "the record does not name the member that killed the panel" + assert_not_contains "$(spawn_log "$home")" "sd-judge" "a stood-down panel must never dispatch a judge" + + # Idempotent: the recorded stand-down keeps saying the same thing. + status=0 + out=$(run_panel "$home" advance sd) || status=$? + [ "$status" -ne 0 ] || fail "a recorded stand-down must keep reporting failure" + assert_contains "$out" "stood down:" "the recorded stand-down must re-print its block" + assert_contains "$out" "sd-b" "the recorded stand-down must keep naming the member" + + # The override never manufactures a panel out of the one report that survived. + status=0 + out=$(run_panel "$home" advance sd --accept-unfinished sd-b) || status=$? + [ "$status" -ne 0 ] || fail "a missing report must never be waivable" + assert_no_grep 'accepted_unfinished' "$home/data/sd/panel.meta" \ + "a stood-down panel must not record an acceptance it can never use" + pass "a member that finishes with no report stands the panel down, durably and unwaivably" +} + +test_gone_analyst_with_nothing_stands_the_panel_down() { + local home out status=0 + home=$(new_home gone-empty "$TWO_MODELS") + out=$(run_panel "$home" start --id ge --project "$home/subject" "Anything?") \ + || fail "start failed: $out" + + # Both analysts were just dispatched and still hold their runtime records, so + # this is the ordinary early window and must stay a plain wait. + out=$(run_panel "$home" advance ge) || fail "advance failed: $out" + assert_contains "$out" "waiting:" "a member that has only just started must still be a plain wait" + assert_not_contains "$out" "stood down:" "the early window must not be mistaken for a member that is gone" + assert_not_contains "$out" "--accept-unfinished" "the early window must not advertise the override" + + # Analyst B is torn down before writing anything, and teardown removes its + # status file and runtime record together, so nothing further can ever arrive. + printf 'analyst a findings\n' > "$home/data/ge-a/report.md" + say_status "$home" ge-a 'done: a finished' + rm -f "$home/state/ge-b.status" "$home/state/ge-b.meta" + out=$(run_panel "$home" advance ge) || status=$? + [ "$status" -ne 0 ] || fail "a panel whose analyst is gone with nothing must not report success" + assert_contains "$out" "stood down:" "a gone analyst that produced nothing must not wait forever" + assert_contains "$out" "ge-b" "the stand-down must name the analyst that produced nothing" + assert_not_contains "$out" "next step is" "a dead panel must not hint at a next advance" + assert_grep 'stage=stood-down' "$home/data/ge/panel.meta" "the dead panel was not recorded durably" + assert_grep 'stood_down=ge-b' "$home/data/ge/panel.meta" "the record does not name the member that killed the panel" + assert_not_contains "$(spawn_log "$home")" "ge-judge" "a stood-down panel must never dispatch a judge" + pass "an analyst that is gone having produced nothing stands the panel down instead of waiting forever" +} + +test_gone_judge_with_nothing_offers_the_rejudge() { + local home out + home=$(new_home gone-judge "$TWO_MODELS") + out=$(run_panel "$home" start --id gj --project "$home/subject" "Anything?") \ + || fail "start failed: $out" + printf 'analyst a findings\n' > "$home/data/gj-a/report.md" + printf 'analyst b findings\n' > "$home/data/gj-b/report.md" + say_status "$home" gj-a 'done: a finished' + say_status "$home" gj-b 'done: b finished' + out=$(run_panel "$home" advance gj) || fail "advance failed: $out" + + # The judge is torn down before writing anything. The evidence is intact, so + # this is the same lost verdict as a judge that declared itself finished. + rm -f "$home/state/gj-judge.status" "$home/state/gj-judge.meta" + out=$(run_panel "$home" advance gj) \ + || fail "a panel that only lost its verdict must not report failure: $out" + assert_contains "$out" "verdict lost:" "a gone judge that produced nothing must be offered the re-judge" + assert_not_contains "$out" "stood down:" "a healthy panel must not be stood down over its judge" + assert_contains "$out" "advance gj --rejudge" "the block must print the exact re-judge command" + assert_no_grep 'stage=stood-down' "$home/data/gj/panel.meta" "a healthy panel must not be recorded dead" + + out=$(run_panel "$home" advance gj --rejudge) || fail "the re-judge must be permitted for a gone judge: $out" + assert_contains "$(spawn_log "$home")" "gj-judge2 " "the offered re-judge must actually dispatch" + assert_grep 'superseded_judges=gj-judge' "$home/data/gj/panel.meta" "the superseded judge was not recorded" + assert_no_grep 'accepted_unfinished' "$home/data/gj/panel.meta" \ + "replacing a gone judge must not stamp the analysts' complete reports" + pass "a judge that is gone having produced nothing is offered the re-judge the state actually permits" +} + +test_readiness_latch_survives_teardown() { + local home out status=0 + home=$(new_home latch "$TWO_MODELS") + out=$(run_panel "$home" start --id lt --project "$home/subject" "Anything?") \ + || fail "start failed: $out" + printf 'analyst a findings\n' > "$home/data/lt-a/report.md" + say_status "$home" lt-a 'done: a finished' + + # The first observation latches lt-a even while the panel waits for lt-b. + out=$(run_panel "$home" advance lt) || fail "advance failed: $out" + assert_contains "$out" "waiting:" "the panel must still wait for the second analyst" + assert_grep 'ready_members=lt-a' "$home/data/lt/panel.meta" \ + "readiness was not latched on first observation" + + # Real teardown removes the status file and the runtime record together, so a + # gate that re-derived readiness from state/ would now call lt-a wedged. + rm -f "$home/state/lt-a.status" "$home/state/lt-a.meta" + out=$(run_panel "$home" advance lt) || fail "advance failed: $out" + assert_contains "$out" "waiting:" "a torn-down member that was observed ready must not change the outcome" + assert_not_contains "$out" "wedged:" "a torn-down member that was observed ready must not become wedged" + assert_not_contains "$out" "lt-a" "a latched member must not be reported as missing anything" + assert_not_contains "$out" "--accept-unfinished" "a latched member must never be offered the override" + + out=$(run_panel "$home" advance lt --accept-unfinished lt-a) || status=$? + expect_code 2 "$status" "a member already observed ready must not be waivable" + assert_contains "$out" "nothing to waive" "the refusal must say the report was already finished" + assert_no_grep 'accepted_unfinished' "$home/data/lt/panel.meta" \ + "a refused override must never stamp a complete report" + + printf 'analyst b findings\n' > "$home/data/lt-b/report.md" + say_status "$home" lt-b 'done: b finished' + out=$(run_panel "$home" advance lt) || fail "advance failed: $out" + assert_contains "$(spawn_log "$home")" "lt-judge " "the judge must be dispatched on the latched readiness" + assert_no_grep 'accepted_unfinished' "$home/data/lt/panel.meta" \ + "no acceptance may be recorded for members that finished properly" + assert_no_grep 'Reports accepted without their author finishing' "$home/data/lt-judge/brief.md" \ + "the judge brief must never call a complete report possibly incomplete" + pass "readiness is latched on first observation and survives teardown of the status file" +} + +test_judge_without_a_verdict_offers_a_recorded_rejudge() { + local home out log status=0 + home=$(new_home rejudge "$TWO_MODELS") + out=$(run_panel "$home" start --id rj --project "$home/subject" "Anything?") \ + || fail "start failed: $out" + printf 'analyst a findings\n' > "$home/data/rj-a/report.md" + printf 'analyst b findings\n' > "$home/data/rj-b/report.md" + say_status "$home" rj-a 'done: a finished' + say_status "$home" rj-b 'done: b finished' + out=$(run_panel "$home" advance rj) || fail "advance failed: $out" + + # A judge that still looks live must not be superseded out from under itself. + out=$(run_panel "$home" advance rj --rejudge) || status=$? + expect_code 2 "$status" "re-judging a judge that still looks live must be refused" + assert_contains "$out" "rj-judge" "the refusal must name the judge it will not replace" + assert_contains "$out" "appears to be running" "the refusal must say why it refuses" + assert_contains "$out" "tear it down" "the refusal must name what the operator can do instead" + assert_not_contains "$(spawn_log "$home")" "rj-judge2" "a refused re-judge must dispatch nothing" + assert_no_grep 'superseded_judges' "$home/data/rj/panel.meta" "a refused re-judge must record nothing" + assert_grep 'judge_task=rj-judge' "$home/data/rj/panel.meta" "a refused re-judge must leave the judge in place" + + # The judge dies without a verdict. The evidence is complete and untouched, so + # this must not discard two finished investigations. + say_status "$home" rj-judge 'failed: ran out of budget' + out=$(run_panel "$home" advance rj) \ + || fail "a panel that only lost its verdict must not report failure: $out" + assert_contains "$out" "verdict lost:" "a reportless judge must not print the analyst stand-down" + assert_not_contains "$out" "stood down:" "a healthy panel must not be stood down over its judge" + assert_contains "$out" "rj-judge" "the block must name the judge that left no verdict" + assert_contains "$out" "advance rj --rejudge" "the block must print the exact re-judge command" + assert_no_grep 'stage=stood-down' "$home/data/rj/panel.meta" "a healthy panel must not be recorded dead" + + out=$(run_panel "$home" advance rj --rejudge) || fail "rejudge failed: $out" + log=$(spawn_log "$home") + assert_contains "$log" "rj-judge2 $home/subject --harness grok --scout --model grok-4-fast --effort high" \ + "the replacement judge was not dispatched on the judge profile" + assert_grep 'judge_task=rj-judge2' "$home/data/rj/panel.meta" "the record does not point at the replacement judge" + assert_grep 'superseded_judges=rj-judge' "$home/data/rj/panel.meta" "the superseded judge was not recorded" + assert_no_grep 'accepted_unfinished' "$home/data/rj/panel.meta" \ + "replacing a judge must not stamp the analysts' complete reports" + assert_grep "$home/data/rj-a/report.md" "$home/data/rj-judge2/brief.md" \ + "the replacement judge was not given the unchanged analyst reports" + assert_no_grep 'Reports accepted without their author finishing' "$home/data/rj-judge2/brief.md" \ + "the replacement judge must not be told a complete report is truncated" + assert_grep 'analyst a findings' "$home/data/rj-a/report.md" "the analyst reports must be left untouched" + + printf 'verdict\n' > "$home/data/rj-judge2/report.md" + say_status "$home" rj-judge2 'done: verdict written' + out=$(run_panel "$home" advance rj --rejudge) || status=$? + expect_code 2 "$status" "re-judging a judge that left a verdict must be refused" + assert_contains "$out" "nothing to re-judge" "the refusal must say a verdict already exists" + + out=$(run_panel "$home" advance rj) || fail "advance failed: $out" + assert_contains "$out" "complete: $home/data/rj-judge2/report.md" \ + "the replacement judge must be able to complete the panel" + assert_not_contains "$out" "CAVEAT" "a replaced judge is not an accepted-unfinished one" + pass "a judge that leaves no verdict is replaced by an explicit recorded re-judge" +} + +test_rejudge_replaces_a_torn_down_judge_and_records_only_after_dispatch() { + local home out status=0 + home=$(new_home rejudge-torndown "$TWO_MODELS") + out=$(run_panel "$home" start --id rf --project "$home/subject" "Anything?") \ + || fail "start failed: $out" + printf 'analyst a findings\n' > "$home/data/rf-a/report.md" + printf 'analyst b findings\n' > "$home/data/rf-b/report.md" + say_status "$home" rf-a 'done: a finished' + say_status "$home" rf-b 'done: b finished' + out=$(run_panel "$home" advance rf) || fail "advance failed: $out" + + # Real teardown removes the status file and the runtime record together, so a + # torn-down judge has neither. It must still be replaceable: a precondition + # that required the status file would make it unreplaceable forever. + rm -f "$home/state/rf-judge.status" "$home/state/rf-judge.meta" + + # A replacement whose dispatch fails must leave the record exactly as it was. + out=$(FM_FAKE_SPAWN_FAIL_ID=rf-judge2 run_panel "$home" advance rf --rejudge) || status=$? + [ "$status" -ne 0 ] || fail "a failed replacement dispatch must not report success" + assert_no_grep 'superseded_judges' "$home/data/rf/panel.meta" \ + "a failed replacement dispatch must not record a supersede that did not happen" + assert_grep 'judge_task=rf-judge' "$home/data/rf/panel.meta" \ + "a failed replacement dispatch must leave judge_task naming the original judge" + + out=$(run_panel "$home" advance rf --rejudge) || fail "rejudge of a torn-down judge failed: $out" + assert_contains "$(spawn_log "$home")" "rf-judge2 " "a torn-down judge must be replaceable" + assert_grep 'judge_task=rf-judge2' "$home/data/rf/panel.meta" "the record does not point at the replacement judge" + assert_grep 'superseded_judges=rf-judge' "$home/data/rf/panel.meta" "the superseded judge was not recorded" + assert_no_grep 'accepted_unfinished' "$home/data/rf/panel.meta" \ + "replacing a torn-down judge must not stamp any report" + pass "a torn-down judge is replaceable, and the record is written only after the dispatch succeeds" +} + +test_generated_task_ids_stay_within_the_task_id_limit() { + local home out id longest=panel-id-length-check status=0 + home=$(new_home id-length "$TWO_MODELS") + while [ "${#longest}" -lt 56 ]; do longest="${longest}x"; done + out=$(run_panel "$home" start --id "$longest" --project "$home/subject" "Anything?") \ + || fail "the documented maximum panel id must be accepted: $out" + for id in "$longest-a" "$longest-b" "$longest-judge" "$longest-judge99"; do + [ "${#id}" -le 64 ] || fail "generated task id $id exceeds the 64-character limit (${#id})" + done + + out=$(run_panel "$home" start --id "${longest}x" --project "$home/subject" "Anything?") || status=$? + expect_code 2 "$status" "a panel id one character over the cap must be refused" + assert_contains "$out" "64 characters" "the refusal must name the task-id limit" + pass "every generated task id stays within the 64-character task-id limit" +} + +test_accepted_judge_caveat_repeats_on_every_complete() { + local home out + home=$(new_home judge-caveat "$TWO_MODELS") + out=$(run_panel "$home" start --id jc --project "$home/subject" "Anything?") \ + || fail "start failed: $out" + printf 'analyst a findings\n' > "$home/data/jc-a/report.md" + printf 'analyst b findings\n' > "$home/data/jc-b/report.md" + say_status "$home" jc-a 'done: a finished' + say_status "$home" jc-b 'done: b finished' + out=$(run_panel "$home" advance jc) || fail "advance failed: $out" + + # The judge wrote a verdict and was then torn down without declaring itself + # finished, so the operator accepts it explicitly. + printf 'partial verdict\n' > "$home/data/jc-judge/report.md" + rm -f "$home/state/jc-judge.meta" + out=$(run_panel "$home" advance jc --accept-unfinished jc-judge) || fail "advance failed: $out" + assert_contains "$out" "complete: $home/data/jc-judge/report.md" "the accepted judge must complete the panel" + assert_contains "$out" "CAVEAT" "the completing invocation must carry the caveat" + + # A later idempotent advance reads on a fresh context and must say the same. + out=$(run_panel "$home" advance jc) || fail "advance failed: $out" + assert_contains "$out" "complete: $home/data/jc-judge/report.md" "the repeat advance must still report completion" + assert_contains "$out" "CAVEAT" "every complete: output must carry the caveat, not only the one that recorded it" + assert_contains "$out" "jc-judge" "the caveat must name the member whose report may be incomplete" + pass "an accepted judge report carries its caveat on every complete: output" +} + +test_acceptance_is_not_recorded_when_the_stage_cannot_use_it() { + local home out status=0 + home=$(new_home accept-stage "$TWO_MODELS") + out=$(FM_FAKE_SPAWN_FAIL_ID=as-b run_panel "$home" start --id as --project "$home/subject" "Anything?") \ + || status=$? + [ "$status" -ne 0 ] || fail "a failed analyst dispatch must not report success" + printf 'analyst a findings\n' > "$home/data/as-a/report.md" + + status=0 + out=$(run_panel "$home" advance as --accept-unfinished as-a) || status=$? + [ "$status" -ne 0 ] || fail "advance must still refuse a panel that never finished dispatching" + assert_contains "$out" "never finished dispatching" "the stage refusal must still be the reported failure" + assert_no_grep 'accepted_unfinished' "$home/data/as/panel.meta" \ + "an acceptance the stage can never use must not be recorded as verdict provenance" + pass "an acceptance is recorded only on a stage that can actually use it" +} + +test_wedged_member_names_its_override_and_the_override_is_per_member() { + local home out brief + home=$(new_home wedged "$TWO_MODELS") + out=$(run_panel "$home" start --id wd --project "$home/subject" "Which claim holds up?") \ + || fail "start failed: $out" + + # Both analysts left a report and were then torn down without ever signalling + # that they finished: the one state that can never clear on its own. + printf 'analyst a findings\n' > "$home/data/wd-a/report.md" + printf 'analyst b findings\n' > "$home/data/wd-b/report.md" + out=$(run_panel "$home" advance wd) || fail "advance failed: $out" + assert_contains "$out" "waiting:" "a member that is still present is in the ordinary window, not wedged" + assert_not_contains "$out" "--accept-unfinished" \ + "the override must not be advertised while both analysts are still present" + + rm -f "$home/state/wd-a.meta" "$home/state/wd-b.meta" + out=$(run_panel "$home" advance wd) || fail "advance failed: $out" + assert_contains "$out" "wedged:" "a member with a report and no runtime record left must not print the bland waiting line" + assert_contains "$out" "torn down or died" "the wedged block must say why the terminal line will never arrive" + assert_contains "$out" "wd-a" "the wedged block must name the wedged member" + assert_contains "$out" "advance wd --accept-unfinished wd-a" \ + "the wedged block must print the exact override command" + assert_contains "$out" "INCOMPLETE" "the wedged block must say the accepted report may be incomplete" + assert_contains "$out" "no timeout" "the wedged block must say nothing advances on its own" + + # Per-member: accepting one analyst waives nothing for the other. + out=$(run_panel "$home" advance wd --accept-unfinished wd-a) || fail "advance failed: $out" + assert_contains "$out" "wedged:" "overriding one member must not waive the gate for another" + assert_contains "$out" "wd-b" "the still-wedged member must still be named" + assert_not_contains "$out" "advance wd --accept-unfinished wd-a" \ + "an already-accepted member must not still be offered the override" + assert_not_contains "$(spawn_log "$home")" "wd-judge" \ + "one accepted member must not be enough to dispatch the judge" + assert_grep 'accepted_unfinished=wd-a' "$home/data/wd/panel.meta" \ + "the override was not recorded durably in the panel record" + + out=$(run_panel "$home" advance wd --accept-unfinished wd-b) || fail "advance failed: $out" + assert_contains "$(spawn_log "$home")" "wd-judge " "accepting every member must let the judge be dispatched" + assert_grep 'accepted_unfinished=wd-a wd-b' "$home/data/wd/panel.meta" \ + "the panel record must carry every accepted member" + + brief="$home/data/wd-judge/brief.md" + assert_grep 'Reports accepted without their author finishing' "$brief" \ + "the judge brief does not disclose that a report was accepted unfinished" + assert_grep "- Analyst A: \`$home/data/wd-a/report.md\`" "$brief" \ + "the judge brief does not name the possibly incomplete report" + assert_grep 'truncated or incomplete' "$brief" \ + "the judge brief does not tell the judge to treat that report as possibly incomplete" + assert_no_grep 'claude-opus-5' "$brief" "disclosing completeness must not disclose the model" + assert_no_grep 'gpt-5.6-sol' "$brief" "disclosing completeness must not disclose the model" + pass "the wedge names its per-member override, which is recorded and disclosed to the judge" +} + +test_accept_unfinished_refuses_a_nonmember_and_a_missing_report() { + local home out status=0 + home=$(new_home accept-guard "$TWO_MODELS") + out=$(run_panel "$home" start --id ag --project "$home/subject" "Anything?") \ + || fail "start failed: $out" + + out=$(run_panel "$home" advance ag --accept-unfinished not-a-member) || status=$? + expect_code 2 "$status" "accepting a task that is not a panel member must be a usage error" + assert_contains "$out" "not a member" "the refusal does not say the task is not a panel member" + assert_no_grep 'accepted_unfinished' "$home/data/ag/panel.meta" \ + "a refused override must not be recorded" + + status=0 + out=$(run_panel "$home" advance ag --accept-unfinished ag-a) || status=$? + expect_code 2 "$status" "accepting a member with no report at all must be a usage error" + assert_contains "$out" "nothing to accept" "the refusal does not say there is no report to accept" + assert_no_grep 'accepted_unfinished' "$home/data/ag/panel.meta" \ + "a refused override must not be recorded" + pass "the override refuses a non-member and refuses to waive a report that does not exist" +} + +test_failed_analyst_with_a_report_still_reaches_the_judge() { + local home out + home=$(new_home failed-analyst "$ONE_MODEL") + out=$(run_panel "$home" start --id fa --project "$home/subject" --reduced "Anything?") \ + || fail "reduced start failed: $out" + + # The reduced form's single analyst is gated exactly like a panel's two. + printf 'partial findings\n' > "$home/data/fa-a/report.md" + out=$(run_panel "$home" advance fa) || fail "advance failed: $out" + assert_contains "$out" "waiting:" "the single analyst of the reduced form must be gated too" + assert_contains "$out" "fa-a" "the waiting line must name the single analyst" + assert_not_contains "$(spawn_log "$home")" "fa-judge" \ + "the reduced form must not dispatch the judge before its analyst has finished" + + # The gate asks whether the analyst stopped writing, not whether it succeeded. + say_status "$home" fa-a 'failed: ran out of budget partway' + out=$(run_panel "$home" advance fa) || fail "advance failed: $out" + assert_contains "$(spawn_log "$home")" "fa-judge " \ + "a failed analyst that still left a report must reach the judge" + assert_grep 'stage=judge' "$home/data/fa/panel.meta" \ + "the panel record did not advance to the judge stage after a terminal failure" + pass "failed: is terminal too, and the reduced form's single analyst is gated the same way" +} + +test_crew_dispatch_default_is_the_documented_fallback() { + local home out log + home=$(new_home crew-fallback) + printf '%s\n' '{"default":[{"harness":"claude","model":"claude-opus-5"},{"harness":"codex","model":"gpt-5.6-sol"}]}' \ + > "$home/config/crew-dispatch.json" + out=$(run_panel "$home" start --id cd --project "$home/subject" "Fallback?") \ + || fail "start failed: $out" + log=$(spawn_log "$home") + assert_contains "$log" "cd-a " "analyst A was not dispatched from the crew-dispatch default" + assert_contains "$log" "cd-b " "analyst B was not dispatched from the crew-dispatch default" + case "$log" in + *"claude-opus-5"*) : ;; + *) fail "the crew-dispatch fallback did not use a configured model" ;; + esac + case "$log" in + *"gpt-5.6-sol"*) : ;; + *) fail "the crew-dispatch fallback did not pick a second distinct model for analyst B" ;; + esac + pass "an unconfigured home falls back to the crew-dispatch default profile set" +} + +test_no_configuration_refuses_with_both_paths_named() { + local home status=0 out + home=$(new_home unconfigured) + out=$(run_panel "$home" start --id nc --project "$home/subject" "Anything?") || status=$? + [ "$status" -ne 0 ] || fail "an unconfigured home must not silently invent a lineup" + assert_contains "$out" "model-panel.json" "the refusal does not name the panel configuration file" + assert_contains "$out" "crew-dispatch.json" "the refusal does not name the fallback configuration file" + pass "a home with no configuration refuses and names both configuration files" +} + +test_dry_run_shows_the_lineup_without_spending_anything() { + local home out + home=$(new_home dry-run "$TWO_MODELS") + out=$(run_panel "$home" start --id dr --project "$home/subject" --dry-run "Anything?") \ + || fail "dry run failed: $out" + assert_contains "$out" "claude-opus-5" "the dry run does not show the analyst A model" + assert_contains "$out" "gpt-5.6-sol" "the dry run does not show the analyst B model" + assert_contains "$out" "grok-4-fast" "the dry run does not show the judge model" + [ -z "$(spawn_log "$home")" ] || fail "a dry run must dispatch nothing" + assert_absent "$home/data/dr" "a dry run must write no panel record" + pass "the dry run shows the resolved lineup and changes nothing" +} + +test_failed_second_dispatch_is_reported_and_blocks_advance() { + local home status=0 out + home=$(new_home partial-dispatch "$TWO_MODELS") + out=$(FM_FAKE_SPAWN_FAIL_ID=pd-b run_panel "$home" start --id pd --project "$home/subject" "Anything?") \ + || status=$? + [ "$status" -ne 0 ] || fail "a failed analyst dispatch must not report success" + assert_contains "$out" "pd-a" "the failure does not name the analyst that is already running" + assert_grep 'stage=incomplete' "$home/data/pd/panel.meta" \ + "a half-dispatched panel must record that it never finished dispatching" + status=0 + out=$(run_panel "$home" advance pd) || status=$? + [ "$status" -ne 0 ] || fail "advance must refuse a panel that never finished dispatching" + pass "a half-dispatched panel reports the failure and refuses to advance" +} + +test_failed_first_dispatch_records_incomplete() { + local home status=0 out + home=$(new_home first-dispatch "$TWO_MODELS") + out=$(FM_FAKE_SPAWN_FAIL_ID=fd-a run_panel "$home" start --id fd --project "$home/subject" "Anything?") \ + || status=$? + [ "$status" -ne 0 ] || fail "a failed first dispatch must not report success" + assert_contains "$out" "fd-a" "the failure does not name the analyst that never dispatched" + assert_grep 'stage=incomplete' "$home/data/fd/panel.meta" \ + "a panel whose first analyst never dispatched must record that it never finished dispatching" + status=0 + out=$(run_panel "$home" advance fd) || status=$? + [ "$status" -ne 0 ] || fail "advance must refuse a panel whose first analyst never dispatched" + pass "a failed analyst-A dispatch is recorded, so advance refuses instead of waiting on a report nobody will write" +} + +test_rollback_never_deletes_a_directory_it_did_not_create() { + local home status=0 out + home=$(new_home rollback "$TWO_MODELS") + # An earlier scout's durable report survived teardown under the task id this + # panel is about to use, and so did its brief, so the shared scaffold refuses + # and start dies before it attempts any dispatch. + mkdir -p "$home/data/rb-a" + printf 'an earlier scout report\n' > "$home/data/rb-a/report.md" + printf 'an earlier brief\n' > "$home/data/rb-a/brief.md" + out=$(run_panel "$home" start --id rb --project "$home/subject" "Anything?") || status=$? + [ "$status" -ne 0 ] || fail "start must fail when a task directory already holds a brief: $out" + assert_present "$home/data/rb-a/report.md" "the rollback deleted a report this run did not create" + assert_grep 'an earlier scout report' "$home/data/rb-a/report.md" \ + "the pre-existing report was clobbered by the failed start" + assert_absent "$home/data/rb" "a start that failed before dispatch must roll back the record it did create" + pass "a failed start rolls back only what it created and leaves an earlier task's report intact" +} + +test_start_refuses_to_clobber_an_existing_panel() { + local home status=0 out + home=$(new_home existing "$TWO_MODELS") + out=$(run_panel "$home" start --id dup --project "$home/subject" "First") || fail "start failed: $out" + out=$(run_panel "$home" start --id dup --project "$home/subject" "Second") || status=$? + [ "$status" -ne 0 ] || fail "a second start on a live panel id must refuse" + assert_contains "$out" "already exists" "the refusal does not say the panel already exists" + assert_grep 'First' "$home/data/dup/question.md" "the live panel's question was overwritten" + pass "start refuses an existing panel id instead of clobbering it" +} + +test_skill_owns_the_cost_decision_and_one_trigger +test_configuration_doc_owns_the_schema_and_degradation +test_analysts_dispatch_on_different_models +test_identical_analyst_models_refuse +test_same_model_through_two_harnesses_refuses +test_reduced_form_is_named_not_a_panel +test_analyst_briefs_share_the_question_and_forbid_peeking +test_judge_waits_for_every_report_then_gets_both +test_terminal_member_without_a_report_stands_the_panel_down +test_gone_analyst_with_nothing_stands_the_panel_down +test_gone_judge_with_nothing_offers_the_rejudge +test_readiness_latch_survives_teardown +test_judge_without_a_verdict_offers_a_recorded_rejudge +test_rejudge_replaces_a_torn_down_judge_and_records_only_after_dispatch +test_generated_task_ids_stay_within_the_task_id_limit +test_accepted_judge_caveat_repeats_on_every_complete +test_acceptance_is_not_recorded_when_the_stage_cannot_use_it +test_wedged_member_names_its_override_and_the_override_is_per_member +test_accept_unfinished_refuses_a_nonmember_and_a_missing_report +test_failed_analyst_with_a_report_still_reaches_the_judge +test_crew_dispatch_default_is_the_documented_fallback +test_no_configuration_refuses_with_both_paths_named +test_dry_run_shows_the_lineup_without_spending_anything +test_failed_second_dispatch_is_reported_and_blocks_advance +test_failed_first_dispatch_records_incomplete +test_rollback_never_deletes_a_directory_it_did_not_create +test_start_refuses_to_clobber_an_existing_panel