Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
118 changes: 118 additions & 0 deletions .agents/skills/panel/SKILL.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,118 @@
---
name: panel
description: >-
Run a model panel: two analysts answer one question independently on different models without seeing each other's work, then a third model judges both and re-verifies their load-bearing claims itself.
Use when the captain invokes /panel, asks for a panel, a second opinion from another model, a cross-check, or an adversarial review of an answer, and when a contested or expensive-to-get-wrong question would otherwise be settled by one voice.
Owns the decision of when a panel is worth its cost, the operating sequence around bin/fm-model-panel.sh, and the honest handling of a home that cannot field two distinct models.
user-invocable: true
metadata:
internal: true
---

# Model panel

Two analysts answer the same question independently on different models, neither able to see the other's work, and a third model judges them.
The value is not three opinions.
It is independence plus adversarial verification: the judge re-verifies the load-bearing claims itself instead of refereeing rhetoric, so it can catch the mistake both analysts made for the same reason, which neither could have caught alone.

`bin/fm-model-panel.sh` owns the mechanics; its header and `--help` are authoritative for commands, flags, the configuration resolution order, and exit statuses.
This skill owns when to run one and how to carry it through.

## Is this question worth a panel

A panel costs two full investigations plus a judge that re-checks their evidence, and two rounds of supervision between dispatch and answer.
Spend that only where a second independent answer would change what happens next.

Worth it:

- The question is contested or expensive to get wrong, and would otherwise be settled by one confident voice.
- The answer depends on live state that written records describe unreliably, which is exactly where a single analyst inherits a stale claim and never notices.
- Several orderings or designs are defensible and the choice matters more than the analysis.
- An earlier answer read as confident but thin, and you want it attacked rather than repeated.

Not worth it:

- The question has one obvious answer, or one command settles it.
- The work is implementation against a clear specification; that is an ordinary ship task.
- The real bottleneck is a captain decision rather than analysis; register the decision through `decision-hold-lifecycle` instead.
- Speed matters more than confidence.
- You would act the same way whatever the judge concluded.

Write the question down precisely before starting.
Both analysts receive byte-identical text and nothing else, so a vague question wastes the whole formation; put a long question in a file and pass `--question-file`.

## Operating sequence

1. **Start the panel.**
`bin/fm-model-panel.sh start [--project <name-or-path>] "<question>"` resolves the lineup, writes the briefs, and dispatches the analysts concurrently.
Use `--dry-run` first when you want to show the captain which models will argue before spending anything.
`--project` defaults to the firstmate repo, which is what a question about the fleet needs; pass a project name for a question about that project's code.
2. **Supervise the analysts as ordinary scouts.**
They report through the normal status path and wake you the normal way.
Do not relay their individual reports to the captain and do not let one analyst see the other's work; the independence rule in their briefs is the whole point.
3. **Advance once every analyst has reported.**
`bin/fm-model-panel.sh advance <panel-id>` refuses to create the judge until every analyst has both written a terminal status event and left a non-empty report, so run it after each analyst finishes and let it tell you whether it is still waiting.
A `waiting:` line means nothing to do yet, and it names which of the two facts is still missing for which analyst.
4. **Supervise the judge, then advance again.**
The second `advance` prints `complete: <report path>` once the judge has both reported a terminal status event and left a non-empty report, which is the same gate its analysts passed.
A `wedged:`, `stood down:`, or `verdict lost:` block instead of a `waiting:` line means the panel needs a decision from you; read the next sections before doing anything about it.
5. **Read the judge's report and relay its findings.**
Relay the answer, the contested facts and what the evidence showed, any mistake shared by both analysts, and what remains unverified.
Say which models argued and which judged: that is substance the captain asked for, not internal machinery.
Never relay a raw report; follow the captain-facing translation contract in `AGENTS.md` section 9.
6. **Pass the shared completion gate before treating the panel as done.**
A panel reliably surfaces captain decisions, which is exactly the work that gets lost, so the judge's report ends with a decisions inventory.
Load `decision-hold-lifecycle` and register every unresolved decision from that inventory before calling the panel complete.
7. **Tear down the members normally.**
Every report survives teardown at `data/<task-id>/report.md`, and the panel record at `data/<panel-id>/panel.meta` records which model filled which role.

## When a member never signals that it finished

Writing the report and appending the terminal `done:` or `failed:` line are two separate acts, and a member that is between them is doing something completely ordinary.
So a member that still has its runtime record gets the plain `waiting:` line, and there is nothing for you to do: it will append its own terminal line and the next `advance` proceeds.

`advance` prints a `wedged:` block only when the member left a non-empty report and is GONE, with no runtime record left, because it was torn down or crashed after writing its report and before signalling.
That line will never arrive on its own, so the block names the member and prints the exact command: `bin/fm-model-panel.sh advance <panel-id> --accept-unfinished <task-id>`.

That override is per-member and deliberate.
It names one task id and waives nothing for any other member, it is recorded permanently in the panel record as `accepted_unfinished`, the judge's brief is told to treat that report as possibly truncated and to say so in its own report, and an accepted judge report puts the caveat on every `complete:` output.
It also stays available for a member that still looks present but you know is dead, such as one killed by a harness crash that left its runtime record behind; used that way it warns you that the member may still be writing, and you should be sure before overriding it.
There is no timeout anywhere in this formation: a panel waits forever rather than judging an unfinished report on its own, so the decision to accept one is always yours.
Say so when relaying the verdict, because a panel completed over an unfinished report is a weaker result than one that was not.

## When an analyst finishes without a report

`advance` prints a `stood down:` block and exits 1 when an ANALYST stops writing without leaving a report, either by ending with a terminal status event or by losing its runtime record.
That member has stopped writing, so no report can arrive, and the panel cannot produce a verdict.

Stand the panel down and tell the captain it produced nothing.
The override deliberately refuses to waive a missing report: a verdict built on the one report that survived is not a panel, and presenting it as one would rebuild the echo-as-corroboration failure this formation exists to refuse.
If the captain still wants the surviving analysis, that is a NEW single-analyst review started deliberately with `--reduced`, which is labelled as such in the briefs, the record, and the judge's own report, and never this panel converted in place.

## When the judge finishes without a verdict

`advance` prints a `verdict lost:` block when the JUDGE stops writing without leaving a report, either by ending with a terminal status event or by losing its runtime record.
This is not a dead panel: both analyst reports are complete and untouched, and only the adjudication is missing, so the formation is intact.

Run `bin/fm-model-panel.sh advance <panel-id> --rejudge` to dispatch one replacement judge over those same unchanged reports.
The superseded judge task is recorded in the panel record, no report is stamped incomplete by it, and nothing re-dispatches on its own.
A judge that still holds its runtime record and has written nothing is indistinguishable from one that is working, so `advance` prints the plain `waiting:` line and `--rejudge` refuses; if you know that judge is dead, such as one killed by a harness crash that left its runtime record behind, tear it down first and then `--rejudge`, which is permitted once the record is gone.
Mention when relaying that the verdict came from a replacement judge, because the captain is entitled to know a judge was lost.

## When the panel cannot be a panel

`start` refuses with exit 4 when both analysts would resolve to the same model.
That refusal is correct and must not be worked around: two identical analysts are not independent, and a panel that quietly is not one is worse than no panel.

Tell the captain plainly that this home can currently reach only one model for the analyst seats, and offer the two real options:

- Configure a second distinct model for the second analyst seat, then re-run the panel.
- Run the reduced form with `--reduced`: one analyst plus a judge that re-verifies its claims.

The reduced form is a **single-analyst review**, not a panel.
Its briefs, its record, and its judge's own report all say so.
Never describe its output as a panel result, an independent cross-check, or a second opinion.

When no third distinct model is available the judge shares a model with one analyst.
That prints a warning and proceeds, because the judge's independence comes from re-verifying against live state with every report in hand rather than from its runtime.
Mention it when relaying if the verdict is close.
1 change: 1 addition & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -24,6 +24,7 @@ __pycache__/
.claude/settings.fm-task.json
config/crew-harness
config/crew-dispatch.json
config/model-panel.json
config/secondmate-harness
config/backlog-backend
config/backend
Expand Down
6 changes: 6 additions & 0 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -78,6 +78,7 @@ bin/ helper scripts, committed; read each script's header before
.env optional X-mode pairing token; LOCAL, gitignored; presence-gates section 14
config/crew-harness crewmate harness override; LOCAL, gitignored; absent or "default" = same as firstmate. Inherited as the literal file: a concrete primary adapter value also controls a secondmate home's own crewmates (section 4)
config/crew-dispatch.json optional crewmate dispatch profiles; LOCAL, gitignored; firstmate-maintained but human-editable natural-language rules that choose a per-task harness/model/effort profile (section 4). Inherited by secondmate homes
config/model-panel.json optional model-panel role profiles; LOCAL, gitignored; maps the pinned analyst and judge roles to concrete models, falling back to config/crew-dispatch.json's default profile set. NOT inherited by secondmate homes, because it names the models one home can actually reach (docs/configuration.md "Model panel roles")
config/secondmate-harness harness the PRIMARY uses to launch SECONDMATE agents, optionally followed by a model and effort token on the same line ("<harness> [<model>] [<effort>]"; section 4); LOCAL, gitignored; absent or "default" harness falls back to config/crew-harness then firstmate's own. The primary's own setting; NOT inherited into secondmate homes (secondmates do not spawn secondmates)
config/firstmate-update-base comparison base for the instruction-surface currency check, naming the source this deployment updates from; LOCAL, gitignored; absent means the canonical upstream template; inherited by secondmate homes (docs/configuration.md "Upstream firstmate and curated-fork checks")
config/fork-sync-upstream comparison base for the curated-fork check, naming the real upstream the fork tracks; LOCAL, gitignored; absent means the canonical upstream template; curator-only, NOT inherited by secondmate homes
Expand All @@ -100,6 +101,8 @@ data/ personal fleet records; LOCAL, gitignored as a whole
secondmates.md secondmate routing table; firstmate-private, maintained by fm-home-seed.sh (section 6)
<id>/brief.md per-task crewmate brief, or per-secondmate charter brief when kind=secondmate
<id>/report.md scout task deliverable, written by the crewmate; survives teardown
<panel-id>/panel.meta model-panel record written by bin/fm-model-panel.sh: form, project, question pointer, stage, member task ids, and each role's resolved profile
<panel-id>/question.md the one question text every member of that panel was given, written by bin/fm-model-panel.sh; survives teardown like the reports
projects/ cloned repos; gitignored; READ-ONLY for you
graphify-out/ local Graphify knowledge graph artifacts; gitignored, expected to drift after hook or incremental updates
state/ volatile runtime signals; gitignored
Expand Down Expand Up @@ -256,6 +259,9 @@ Classify the deliverable:
- **Ship** is the default and produces a project change through the selected delivery mode.
- **Scout** produces knowledge in `data/<id>/report.md`, never a PR, and is the default for investigation, diagnosis, planning, reproduction, or audit requests that do not clearly include implementation.

When the captain invokes `/panel`, asks for a panel, a second opinion from another model, or an adversarial cross-check of an answer, load the `panel` skill.
It owns that formation of independent scouts plus a judge, and when it is worth its cost.

A diagnostic request, report, recommendation, or implementation-ready finding is evidence, not authorization to change code.
Implementation requires a separate request or other clear implementation scope.
Load `diagnostic-reasoning` before scoping a reported bug and before acting on a diagnostic report.
Expand Down
1 change: 1 addition & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -174,6 +174,7 @@ Claude and grok use the slash form shown here; codex uses the same names with `$
| `/afk` | Enter away-mode supervision: the sub-supervisor self-handles routine notifications in bash, escalates captain-relevant events and bounded declared-external-wait rechecks as batched digests, and actively alerts if delivery gets stuck while you step away |
| `/ahoy` | Recap only visible session events since the prior real captain message, falling back to Bearings when invoked as the session's first real captain message |
| `/bearings` | Generate a standalone current-status report from bounded local fleet and registered-secondmate state, with live PR enrichment only when requested, written to a dated file in `data/` and surfaced concisely in chat; read-mostly, mutates no task state |
| `/panel` | Run a model panel on one question: two analysts investigate independently on different models without seeing each other's work, then a third model judges both and re-verifies their load-bearing claims; refuses rather than presenting two runs of one model as independent |
| `/updatefirstmate` | Self-update the running firstmate and its secondmates to the latest from origin with fast-forward-only pulls, then re-read instructions and nudge secondmates |
| `/stow` | Sweep the session for uncaptured durable knowledge, route each finding to its disk home per AGENTS.md, file undone next steps to the backlog, and report what is now safe to reset |

Expand Down
38 changes: 38 additions & 0 deletions bin/fm-classify-lib.sh
Original file line number Diff line number Diff line change
Expand Up @@ -93,6 +93,44 @@ status_is_terminal_verb() {
esac
}

# The verbs that declare a task STOPPED WRITING. Deliberately narrower than the
# terminal captain verbs above: needs-decision and blocked stop the agent's turn
# but expect it to resume and keep working (the scaffolds have it append
# `resolved:` and carry on), so they say nothing about the work being finished.
# Callers asking "must the captain see this" use status_is_terminal_verb; callers
# asking "will anything more be written" use the two helpers below.
# FM_CLASSIFY_FINISHED_VERBS overrides the set.
FM_CLASSIFY_FINISHED_VERBS_DEFAULT='done failed'

# 0 if the given status line's leading verb declares the task finished.
status_is_finished_verb() { # <status-line>
local line=$1 verb
[ -n "$line" ] || return 1
verb=$(status_line_verb "$line")
[ -n "$verb" ] || return 1
case " ${FM_CLASSIFY_FINISHED_VERBS:-$FM_CLASSIFY_FINISHED_VERBS_DEFAULT} " in
*" $verb "*) return 0 ;;
esac
return 1
}

# 0 if a status file EVER recorded a finished event. The status stream is an
# append-only EVENT log, so this deliberately scans every line instead of reading
# the last one: "this task finished at some point" is monotonic, and a later
# unrelated line never un-finishes it. Consumers that need the CURRENT state of a
# live crew use fm-crew-state.sh instead; this one answers a durable question that
# still has an answer after the pane is gone.
status_has_finished_event() { # <status-file>
local f=$1 line
[ -e "$f" ] || return 1
while IFS= read -r line || [ -n "$line" ]; do
if status_is_finished_verb "$line"; then
return 0
fi
done < "$f"
return 1
}

# 0 if the given (last) status line matches a captain-relevant verb.
# Verb-aware by default: terminal verbs always match; nonterminal progress verbs
# (working, resolved, captain-held) and paused never match from free-text prose;
Expand Down
2 changes: 1 addition & 1 deletion bin/fm-gate-refuse-lib.sh
Original file line number Diff line number Diff line change
Expand Up @@ -54,7 +54,7 @@
# tests/fm-gate-refuse.test.sh strips the bypass so it still verifies real refusal.
#
# Sourced by bin/fm-spawn.sh, bin/fm-send.sh, bin/fm-teardown.sh,
# bin/fm-sessionstart-nudge.sh, and the tests.
# bin/fm-model-panel.sh, bin/fm-sessionstart-nudge.sh, and the tests.
# No side effects on source. set -u / set -e safe. The refusal is a hard exit,
# not a return, because there is no safe way to continue a fleet mutation from a
# gate context.
Expand Down
Loading
Loading