Skip to content

fix(bin): enforce clone identity at spawn and verify recorded worktree on recovery resume - #878

Open
sparkus wants to merge 3 commits into
kunchenguid:mainfrom
sparkus:fm/spawn-clone-identity
Open

fix(bin): enforce clone identity at spawn and verify recorded worktree on recovery resume#878
sparkus wants to merge 3 commits into
kunchenguid:mainfrom
sparkus:fm/spawn-clone-identity

Conversation

@sparkus

@sparkus sparkus commented Jul 22, 2026

Copy link
Copy Markdown
Contributor

Intent

Fix the confirmed cross-home worktree-pool isolation defect in Firstmate's shared tooling. A ship/scout spawn must canonicalize Git common-directory identity for the selected project clone and proposed worktree, reject a foreign home's pooled worktree before hooks, agent launch, or task metadata publication, and record the selected clone identity for recovery. Recovery or harness resume paths that can restore historical cwd, especially Grok resume, must verify the live agent cwd, Git top-level, and repository identity against the recorded worktree before any project mutation, failing closed and requiring a fresh launch when they differ. Add regressions using two independent clones of the same remote under one shared pool namespace and a simulated historical-cwd restoration. Keep the fix Firstmate-side without changing Treehouse, preserve supported backend behavior, keep shell scripts shellcheck-clean, use one sentence per line in Markdown, avoid incident restatement beyond the operational contract, base and target main, and never add an agent co-author.

What Changed

  • bin/fm-spawn.sh now canonicalizes the Git common-directory identity of the selected project clone and the proposed worktree (via new bin/fm-worktree-identity-lib.sh), rejects a foreign home's pooled worktree before any hooks, agent launch, or task metadata publication, and records the selected clone identity as project_git_common_dir= in task metadata for recovery.
  • New bin/fm-recovery-verify.sh checks a resumed task's live agent cwd, Git top-level, and repository identity against the recorded worktree before any project mutation; it resolves the recovery tmux target to a unique live window id (new helper in bin/backends/tmux.sh) and fails closed — requiring a fresh launch — on missing fields, lost/ambiguous windows, unreadable cwd, or identity mismatch. The stuck-crewmate-recovery skill now requires running this verification after any harness resume.
  • Added tests/fm-spawn-clone-identity.test.sh (6 regressions covering foreign-pool rejection, clone-identity metadata, historical-cwd resume rejection, lost/ambiguous window fail-closed, and owned-worktree acceptance) and documented the spawn clone-identity boundary in AGENTS.md, docs/architecture.md, and docs/scripts.md.

Risk Assessment

✅ Low: The round-2 commit cleanly resolves the only actionable prior finding by resolving the recorded tmux target to exactly one live stable window id with fail-closed zero/ambiguous handling, adds focused regressions for lost and ambiguous window names, and introduces no new material risk across the re-inspected branch.

Testing

Ran the new clone-identity regression suite plus the surrounding spawn/backend/orca/test-runner suites (all green), then demonstrated the fix end-to-end with a real two-clone shared-pool fixture: fm-spawn.sh refuses the foreign clone's pooled worktree before hooks, launch, or metadata, records the selected clone identity for the owned worktree, and fm-recovery-verify.sh fails closed on a simulated historical-cwd restoration and on lost/ambiguous tmux windows while accepting the recorded worktree; transcript captured as evidence and demo fixtures cleaned up. Shellcheck was not run per testing-stage rules.

Evidence: End-to-end CLI transcript: two-clone shared-pool spawn refusal, clone-identity metadata, and recovery fail-closed/verify scenarios
=== Fixture: two independent clones of ONE remote sharing one worktree-pool namespace ===
home-a and home-b origin:  file:///tmp/fm-clone-identity-demo.GIweT5/remote.git  (identical remote URL)
home-a clone common dir:   /private/tmp/fm-clone-identity-demo.GIweT5/home-a/projects/shared/.git
home-b clone common dir:   /private/tmp/fm-clone-identity-demo.GIweT5/home-b/projects/shared/.git
pool slots: shared-pool/owned-slot (home-a's worktree), shared-pool/foreign-slot (home-b's worktree)

=== 1. Spawn resolving to the FOREIGN home's pooled worktree: refused before hooks/launch/metadata ===
$ fm-spawn.sh demo-foreign home-a/projects/shared   # pool hands back home-b's foreign-slot
error: treehouse get yielded worktree '/private/tmp/fm-clone-identity-demo.GIweT5/shared-pool/foreign-slot' owned by clone '/private/tmp/fm-clone-identity-demo.GIweT5/home-b/projects/shared/.git', but task demo-foreign selected clone '/private/tmp/fm-clone-identity-demo.GIweT5/home-a/projects/shared/.git' from '/tmp/fm-clone-identity-demo.GIweT5/home-a/projects/shared'; refusing before hooks, agent launch, or metadata publication. Inspect the shared worktree pool binding for target firstmate:fm-demo-foreign
exit code: 1
agent launched?          no
task metadata published? no

=== 2. Spawn into the OWNED pooled worktree: succeeds and records selected clone identity ===
$ fm-spawn.sh demo-owned home-a/projects/shared   # pool hands back home-a's owned-slot
warn: no registry at /tmp/fm-clone-identity-demo.GIweT5/home-a/data/projects.md; defaulting shared to no-mistakes off
spawned demo-owned harness=claude kind=ship mode=no-mistakes yolo=off window=firstmate:fm-demo-owned worktree=/tmp/fm-clone-identity-demo.GIweT5/shared-pool/owned-slot
exit code: 0
agent launched?          YES
--- state/demo-owned.meta (selected clone identity recorded for recovery) ---
window=firstmate:fm-demo-owned
worktree=/tmp/fm-clone-identity-demo.GIweT5/shared-pool/owned-slot
project=/tmp/fm-clone-identity-demo.GIweT5/home-a/projects/shared
project_git_common_dir=/private/tmp/fm-clone-identity-demo.GIweT5/home-a/projects/shared/.git
harness=claude
kind=ship

=== 3. Recovery resume (Grok-style historical-cwd restoration): live cwd is the FOREIGN worktree -> fail closed ===
$ fm-recovery-verify.sh demo-owned   # backend pane cwd restored to home-b's foreign-slot
error: resumed task demo-owned foreground cwd '/tmp/fm-clone-identity-demo.GIweT5/shared-pool/foreign-slot' resolves to worktree '/private/tmp/fm-clone-identity-demo.GIweT5/shared-pool/foreign-slot' and clone '/private/tmp/fm-clone-identity-demo.GIweT5/home-b/projects/shared/.git', expected worktree '/private/tmp/fm-clone-identity-demo.GIweT5/shared-pool/owned-slot' and selected clone '/private/tmp/fm-clone-identity-demo.GIweT5/home-a/projects/shared/.git'; exit this resumed agent and relaunch fresh in '/private/tmp/fm-clone-identity-demo.GIweT5/shared-pool/owned-slot'
exit code: 1

=== 4. Recovery resume: recorded tmux window name lost/renamed -> fail closed (no active-client fallback) ===
$ fm-recovery-verify.sh demo-owned   # window renamed away
error: task demo-owned tmux target 'firstmate:fm-demo-owned' does not resolve to exactly one live window; refusing recovery resume
exit code: 1

=== 5. Recovery resume: live cwd matches recorded worktree AND selected clone -> verified ===
$ fm-recovery-verify.sh demo-owned
verified task demo-owned worktree=/private/tmp/fm-clone-identity-demo.GIweT5/shared-pool/owned-slot clone=/private/tmp/fm-clone-identity-demo.GIweT5/home-a/projects/shared/.git
exit code: 0

Pipeline

Updates from git push no-mistakes

✅ **intent** - passed

✅ No issues found.

✅ **Rebase** - passed

✅ No issues found.

🔧 **Review** - 3 issues found → auto-fixed ✅
  • ⚠️ bin/fm-recovery-verify.sh:78 - fm-recovery-verify.sh reads the resumed task's live cwd via tmux display-message -p -t against the name-based window= target from meta (fm_backend_target_of_meta returns "$SES:$W", bin/fm-spawn.sh:889). fm-spawn.sh:1085's own comment documents that display-message with a lost/renamed window name falls back to the active client's window — the reason the spawn settle-poll targets the stable window id ($WID) instead. In the verifier the common failure direction is a spurious refusal (fail closed), but a false accept exists: if the task window's name is lost and the active client's pane happens to be inside the recorded worktree (plausible while an operator inspects that worktree during recovery), verification passes while the actual resumed agent runs elsewhere. Harden by resolving the window name to exactly one live window id (failing closed on zero or ambiguous matches) before reading pane_current_path.
  • ℹ️ .agents/skills/stuck-crewmate-recovery/SKILL.md:34 - The resume-side verification is enforced procedurally: SKILL.md:34 instructs the recovering agent to run bin/fm-recovery-verify.sh after any harness resume, but nothing mechanically gates fm-send instruction delivery on a passed verification. Since resumes are agent-operated, the skill is the available Firstmate-side seam; noting that the fail-closed guarantee for Grok resume still relies on the operating agent following the skill.
  • ℹ️ bin/fm-spawn.sh:301 - ORCA_ABORT_METADATA_ALLOWED=0 spans the entire validate_spawn_worktree call, so when an unvalidated Orca worktree cannot be removed during abort, fallback metadata is now skipped for both the new clone-identity failure and the pre-existing isolation failure. The leaked Orca worktree is reported only on stderr and is no longer discoverable from state by later teardown/fleet-sync. This is a deliberate consequence of the no-metadata-before-validation boundary, but operators lose the previous state breadcrumb for the isolation-failure mode.

🔧 Fix: resolve recovery tmux target to unique window id, fail closed
✅ Re-checked - no issues remain.

✅ **Test** - passed

✅ No issues found.

  • bash tests/fm-spawn-clone-identity.test.sh — all 6 regression tests pass (foreign-pool rejection, clone-identity metadata, historical-cwd resume rejection, lost/ambiguous window fail-closed, owned-worktree acceptance)
  • bin/fm-test-run.sh tests/fm-spawn-batch.test.sh tests/fm-spawn-dispatch-profile.test.sh tests/fm-spawn-worktree-settle.test.sh tests/fm-backend.test.sh tests/fm-backend-tmux-smoke.test.sh tests/fm-spawn-clone-identity.test.sh — backend-dispatch family, failed=0
  • bin/fm-test-run.sh tests/fm-backend-orca.test.sh tests/fm-test-run.test.sh — orca abort-metadata paths and runner family mapping, failed=0
  • Manual end-to-end CLI demo: built two independent clones of one remote under a shared worktree-pool namespace, ran real bin/fm-spawn.sh (foreign slot refused with no agent launch/metadata/hooks; owned slot succeeds and records project_git_common_dir=) and real bin/fm-recovery-verify.sh (foreign historical cwd refused, lost window name fails closed, recorded worktree verifies)
  • Checked git log 593e3a2..b190820 commit messages for agent co-author trailers (none) and confirmed Markdown doc changes use one sentence per line with no Treehouse-side edits
✅ **Document** - passed

✅ No issues found.

⚠️ **Lint** - 1 warning
  • ⚠️ linter found issues (exit code 127)
✅ **Push** - passed

✅ No issues found.

@sparkus

sparkus commented Jul 22, 2026

Copy link
Copy Markdown
Contributor Author

Additional field evidence from the originating incident (post-submission): after a foreign-slot rejection, a stale TREEHOUSE_DIR environment variable in the recovered session still pointed at the rejected foreign slot; only a marker file was touched before detection, but it shows worktree rejection/reallocation must also REBIND the session environment (TREEHOUSE_DIR and any worktree-derived env), not just the cwd. Worth covering in the post-resume verification this PR adds.

@kunchenguid

kunchenguid commented Jul 24, 2026

Copy link
Copy Markdown
Owner

Automated reminder: thanks for the PR! This branch currently has a merge conflict with the base branch.

When you get a chance, please rebase onto (or merge) the latest base branch, resolve the conflict, and push. After that, checks will re-run and the PR will get looked at again.

Noted for firstmate#878 at a63f0df8.

@sparkus
sparkus force-pushed the fm/spawn-clone-identity branch from a63f0df to 2d26711 Compare July 25, 2026 01:14
@kunchenguid kunchenguid removed the wheelhouse:pending-contributor-action Managed by Wheelhouse label Jul 25, 2026
@kunchenguid

kunchenguid commented Jul 25, 2026

Copy link
Copy Markdown
Owner

Automated reminder: thanks for the PR! This branch currently has a merge conflict with the base branch.

When you get a chance, please rebase onto (or merge) the latest base branch, resolve the conflict, and push. After that, checks will re-run and the PR will get looked at again.

Noted for firstmate#878 at 2d267117.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants