Skip to content

fix(bin): stop evicting live watchers on ps truncation and path spelling - #851

Open
m0d7 wants to merge 8 commits into
kunchenguid:mainfrom
m0d7:fm/fix-watcher-lockmatch
Open

fix(bin): stop evicting live watchers on ps truncation and path spelling#851
m0d7 wants to merge 8 commits into
kunchenguid:mainfrom
m0d7:fm/fix-watcher-lockmatch

Conversation

@m0d7

@m0d7 m0d7 commented Jul 22, 2026

Copy link
Copy Markdown

Intent

The developer wanted to fix a watcher-continuity false-deny bug in the firstmate repo's bin/fm-wake-lib.sh, where a healthy watcher was wrongly judged dead and every fleet command was denied, causing a supervision deadlock diagnosed live on macOS. The fix covers two stacked defects: adding -ww to the ps call in fm_pid_identity so narrow-COLUMNS hook contexts no longer truncate the command identity (closing upstream issue #799, with the sibling ps call in bin/fm-afk-start.sh also patched), and adding a same-file -ef fallback to the lock's path string comparisons so case-unstable getcwd spellings on case-insensitive filesystems don't evict a live watcher, while preserving fail-closed semantics. Requirements included regression tests in tests/ verified to fail on old code and pass on new, a clean bin/fm-lint.sh run, a passing full test suite, and driving the no-mistakes validation pipeline (review, tests, docs, lint) to completion rather than hand-fixing findings. The PR body had to document both defects with the captured live evidence, use "Fixes #799" wording, and note the case-stability defect was previously unreported; explicit constraints included no em dashes, no agent co-author on commits, working only from a clean isolated worktree, and never pushing to the default branch. When the final push failed with a 403 because the authenticated account lacked write access to kunchenguid/firstmate, the developer approved a fork-based PR, pushing the branch to a fork and opening a cross-repo PR against the upstream repo.

What Changed

  • Added -ww to the liveness ps calls in fm_pid_identity (bin/fm-wake-lib.sh) and daemon_pid_matches (bin/fm-afk-start.sh) so a narrow-COLUMNS caller (e.g. a PreToolUse hook under the UI terminal width) can no longer truncate command=, recompute a non-matching identity, and evict a live watcher/daemon as stale.
  • Introduced an fm_same_path helper that compares by string first and falls back to same-file identity (-ef), and routed the lock's fm-home/watcher-path checks in bin/fm-wake-lib.sh and bin/fm-watch-arm.sh through it, so case-unstable getcwd() spellings on a case-insensitive filesystem stop evicting a live watcher while a genuinely different or missing path still fails closed.
  • Added regression coverage in tests/fm-watcher-lock.test.sh for both defects, documented the invariants in docs/watcher-continuity.md, and hardened the flaky unit_signal_exits_with_lock_cleanup test in tests/fm-afk-launch.test.sh to wait for the launcher's TERM-trap installation before signaling.

Risk Assessment

✅ Low: A narrow, fail-closed-preserving fix applied symmetrically at both lock-comparison sites, with deterministic regression tests for both stacked defects and no correctness, portability, or security issues found in the changed code.

Testing

Exercised the fix at the end-user surface by driving the watcher-continuity PreToolUse gate against a live healthy watcher fixture: base-commit code reproduces both live false-denies (case-respelled home spelling and narrow-width ps identity truncation) while the fixed code allows the fleet command and still refuses genuinely different or missing recorded homes; the two new regression tests were verified to fail on old code and pass on new, the round-1 flaky-test fix in tests/fm-afk-launch.test.sh held stable across repeated runs, and the repo's full changed-set suite of 21 test files passed with zero failures at the final HEAD.

Evidence: Gate E2E transcript, OLD code (base 673b6ad): both live-incident shapes DENIED (supervision deadlock reproduced)
### watcher-continuity gate, OLD code (base 673b6ad) ###

--- Demo 1: same home directory, case-respelled by the hook context ---
lock records fm-home: /var/folders/4l/t9dqj4zj4gzcqxnthzv1yt2r0000gp/T//fm-gate-demo.eoP8ve/sc/firstmate
hook context resolves: /var/folders/4l/t9dqj4zj4gzcqxnthzv1yt2r0000gp/T//fm-gate-demo.eoP8ve/SC/firstmate (same directory on this case-insensitive APFS volume)
gate exit for 'bin/fm-crew-state.sh task': 2
gate deny object (stderr):
{"hookSpecificOutput":{"hookEventName":"PreToolUse","permissionDecision":"deny"},"systemMessage":"[watcher-continuity] tasks are in flight and no live watcher holds this home lock; drain wakes with bin/fm-wake-drain.sh, use fail-closed bin/fm-teardown.sh for completed tasks when needed, then re-arm with bin/fm-watch-arm.sh as a tracked Claude background task before running other fleet commands (blocked: fm-crew-state.sh)"}
RESULT: DENIED - the healthy watcher was falsely evicted; every fleet command is refused (supervision deadlock)

--- Demo 2: width-honoring ps under a narrow (77-column) hook context ---
recorded pid-identity (full width): Wed Jul 22 13:35:35 2026     bash -c sleep 300 # long-watcher-invocation-trailer-that-pushes-the-recorded-command-line-well-past-seventy-seven-columns\012true
gate exit for 'bin/fm-crew-state.sh task': 2
gate deny object (stderr):
{"hookSpecificOutput":{"hookEventName":"PreToolUse","permissionDecision":"deny"},"systemMessage":"[watcher-continuity] tasks are in flight and no live watcher holds this home lock; drain wakes with bin/fm-wake-drain.sh, use fail-closed bin/fm-teardown.sh for completed tasks when needed, then re-arm with bin/fm-watch-arm.sh as a tracked Claude background task before running other fleet commands (blocked: fm-crew-state.sh)"}
RESULT: DENIED - the healthy watcher was falsely evicted; every fleet command is refused (supervision deadlock)

### end OLD code (base 673b6ad) ###
Evidence: Gate E2E transcript, NEW code at final HEAD 739309d: both shapes ALLOWED

--- Demo 1: same home directory, case-respelled by the hook context --- gate exit for 'bin/fm-crew-state.sh task': 0 RESULT: ALLOWED - the live watcher is recognized and the fleet command proceeds --- Demo 2: width-honoring ps under a narrow (77-column) hook context --- gate exit for 'bin/fm-crew-state.sh task': 0 RESULT: ALLOWED - the live watcher is recognized and the fleet command proceeds

### watcher-continuity gate, new-code-round2-head-739309d ###

--- Demo 1: same home directory, case-respelled by the hook context ---
lock records fm-home: /var/folders/4l/t9dqj4zj4gzcqxnthzv1yt2r0000gp/T//fm-gate-demo.t9yHt9/sc/firstmate
hook context resolves: /var/folders/4l/t9dqj4zj4gzcqxnthzv1yt2r0000gp/T//fm-gate-demo.t9yHt9/SC/firstmate (same directory on this case-insensitive APFS volume)
gate exit for 'bin/fm-crew-state.sh task': 0
RESULT: ALLOWED - the live watcher is recognized and the fleet command proceeds

--- Demo 2: width-honoring ps under a narrow (77-column) hook context ---
recorded pid-identity (full width): Wed Jul 22 14:30:37 2026     bash -c sleep 300 # long-watcher-invocation-trailer-that-pushes-the-recorded-command-line-well-past-seventy-seven-columns\012true
gate exit for 'bin/fm-crew-state.sh task': 0
RESULT: ALLOWED - the live watcher is recognized and the fleet command proceeds

### end new-code-round2-head-739309d ###
Evidence: Gate fail-closed transcript, NEW code: genuinely different / missing recorded home still deny; exact home allows
### watcher-continuity gate fail-closed check, new-code-head-739309d ###

--- sanity: lock records the exact home -> healthy watcher allows ---
lock records fm-home: /var/folders/4l/t9dqj4zj4gzcqxnthzv1yt2r0000gp/T//fm-gate-failclosed.tKzCh9/sc/firstmate
hook context home:    /var/folders/4l/t9dqj4zj4gzcqxnthzv1yt2r0000gp/T//fm-gate-failclosed.tKzCh9/sc/firstmate
gate result: allow (exit 0)   expected: allow   -> OK, matches expectation

--- lock records a genuinely different existing home -> still refuses ---
lock records fm-home: /var/folders/4l/t9dqj4zj4gzcqxnthzv1yt2r0000gp/T//fm-gate-failclosed.tKzCh9/genuinely-other-home
hook context home:    /var/folders/4l/t9dqj4zj4gzcqxnthzv1yt2r0000gp/T//fm-gate-failclosed.tKzCh9/sc/firstmate
gate result: deny (exit 2)   expected: deny   -> OK, matches expectation
gate deny object (stderr):
{"hookSpecificOutput":{"hookEventName":"PreToolUse","permissionDecision":"deny"},"systemMessage":"[watcher-continuity] tasks are in flight and no live watcher holds this home lock; drain wakes with bin/fm-wake-drain.sh, use fail-closed bin/fm-teardown.sh for completed tasks when needed, then re-arm with bin/fm-watch-arm.sh as a tracked Claude background task before running other fleet commands (blocked: fm-crew-state.sh)"}

--- lock records a home that no longer exists -> still refuses ---
lock records fm-home: /var/folders/4l/t9dqj4zj4gzcqxnthzv1yt2r0000gp/T//fm-gate-failclosed.tKzCh9/recorded-home-that-no-longer-exists
hook context home:    /var/folders/4l/t9dqj4zj4gzcqxnthzv1yt2r0000gp/T//fm-gate-failclosed.tKzCh9/sc/firstmate
gate result: deny (exit 2)   expected: deny   -> OK, matches expectation
gate deny object (stderr):
{"hookSpecificOutput":{"hookEventName":"PreToolUse","permissionDecision":"deny"},"systemMessage":"[watcher-continuity] tasks are in flight and no live watcher holds this home lock; drain wakes with bin/fm-wake-drain.sh, use fail-closed bin/fm-teardown.sh for completed tasks when needed, then re-arm with bin/fm-watch-arm.sh as a tracked Claude background task before running other fleet commands (blocked: fm-crew-state.sh)"}

RESULT: fail-closed semantics preserved (mismatch and missing-path locks still deny; only the same-directory respelling is accepted)
### end new-code-head-739309d ###
Evidence: New regression tests run against OLD code: both fail with the expected eviction messages
### two new regression tests, run individually against OLD code (base 673b6ad bin/, new tests) ###

$ bash tests/.tmp-single-test.sh test_pid_identity_unlimited_width_under_narrow_terminal
not ok - fm_pid_identity truncated the command at a narrow width (got 'Wed Jul 22 13:35:50 2026     bash -c sle')
exit: 1

$ bash tests/.tmp-single-test.sh test_lock_match_accepts_same_dir_different_spelling
not ok - a different spelling of the same home and watcher path evicted a live watcher
exit: 1
Evidence: New regression tests against NEW code: both pass
### same two regression tests against NEW code (target 709ab62) ###

$ bash tests/.tmp-single-test.sh test_pid_identity_unlimited_width_under_narrow_terminal
ok - fm_pid_identity forces unlimited ps width in a narrow-terminal context
exit: 0

$ bash tests/.tmp-single-test.sh test_lock_match_accepts_same_dir_different_spelling
ok - lock match tolerates same-directory spelling differences and still refuses real mismatches
exit: 0
Evidence: Real macOS ps truncation in a 77-column pty (defect class confirmed with the system ps; -ww returns the full command)
### real macOS ps truncation in a narrow pty (Darwin 25.4.0, APFS host that diagnosed the live bug class) ###
# A long-command process is inspected by ps inside a real 77-column pty.
# Plain ps truncates command= at the terminal width; ps -ww returns it in full.

$ script -q /dev/null sh -c "stty cols 77; ps -p 88412 -o lstart= -o command="
��Wed Jul 22 13:36:43 2026     bash -c sleep 30 # extremely-long-trailer-annot

$ script -q /dev/null sh -c "stty cols 77; ps -ww -p 88412 -o lstart= -o command="
��Wed Jul 22 13:36:43 2026     bash -c sleep 30 # extremely-long-trailer-annotation-that-should-push-the-command-line-well-past-narrow-width-abcdefghijklmnopqrstuvwxyz\012true

# Note: on this host ps only honors width when stdout is a tty; fm_pid_identity
# reads ps through a pipe, so the deterministic regression test and the gate demo
# use a width-sensitive ps stand-in to reproduce the live width-honoring ps.
Evidence: Flaky afk-launch signal test: root cause, base-commit reproduction, fix, 10/10 clean under 4x load (round 1)
### pre-existing flake: tests/fm-afk-launch.test.sh unit_signal_exits_with_lock_cleanup ###

# Failure as seen in the changed-set runner (new code) and in 1 of 5 standalone reruns:
not ok - launcher signal: interrupted lifecycle resumed or retained its lock

# Same failure on UNCHANGED base code 673b6ad (bin/ reverted, 8 standalone runs):
# base run 3 exit: 1, base run 6 exit: 1, all others 0
not ok - launcher signal: interrupted lifecycle resumed or retained its lock
not ok - launcher signal: interrupted lifecycle resumed or retained its lock

# Root cause: fm_afk_launch_main acquires the lifecycle lock (bin/fm-afk-launch.sh:606)
# before installing its TERM trap (line 609). The test waited for the lock dir and
# TERMed immediately, so under load the signal could land in that window and kill the
# child by default action with the lock retained.

# Fix applied (test-only): the stand-in start hook, which runs strictly after the traps,
# writes a ready marker the test now waits for; the sleeper is backgrounded so the trapped
# TERM interrupts the wait builtin, and its pid is recorded and reaped.

# Verification: 10 consecutive standalone runs under 4x CPU load, 0 failures:
ok - launcher signal: TERM exits and releases the lifecycle lock
failures: 0 / 10
Evidence: Round-2 stability re-check of the fixed signal test at final HEAD: 0/5 failures
### round 2: stability re-check of the previously flaky signal test at final HEAD 739309d ###

# tests/fm-afk-launch.test.sh, 5 consecutive standalone runs (after the round-1
# ready-marker hardening landed as commit 739309d):

run 1: exit=0 | ok - launcher signal: TERM exits and releases the lifecycle lock
run 2: exit=0 | ok - launcher signal: TERM exits and releases the lifecycle lock
run 3: exit=0 | ok - launcher signal: TERM exits and releases the lifecycle lock
run 4: exit=0 | ok - launcher signal: TERM exits and releases the lifecycle lock
run 5: exit=0 | ok - launcher signal: TERM exits and releases the lifecycle lock
failures: 0 / 5

# Round 1 had already verified 10 consecutive clean runs under 4x CPU load; this
# re-check confirms the fix as committed at the branch head.
Evidence: Changed-set suite summary at HEAD 739309d (21 scripts, 0 failed)
FM_TEST_SUMMARY total=21 failed=0 skipped_gate=1 duration_ms=789261
FM_TEST_SUMMARY_FAMILY family=afk count=2 duration_ms=7549 failed=0
FM_TEST_SUMMARY_FAMILY family=real-herdr-gated count=9 duration_ms=399028 failed=0
FM_TEST_SUMMARY_FAMILY family=watcher-wake-lock count=10 duration_ms=381124 failed=0
Evidence: Reproducible gate E2E demo script (old/new code, both defects)
#!/usr/bin/env bash
# E2E demo of the watcher-continuity deadlock (upstream issue #799 plus the
# unreported case-stability defect) at the real end-user surface: the Claude
# PreToolUse gate bin/fm-continuity-pretool-check.sh.
#
# Both demos build a genuine primary home with tasks in flight and a LIVE,
# HEALTHY watcher-lock holder, then invoke the gate exactly as the hook does
# for an ordinary fleet command. A DENY here is the live-observed supervision
# deadlock: the healthy watcher is falsely judged dead and every fleet command
# is refused.
#
# Demo 1 - case-unstable getcwd spelling: the lock records .../sc/firstmate,
#   the hook context resolves the SAME directory as .../SC/firstmate
#   (case-insensitive APFS, as observed live 2026-07-21).
# Demo 2 - width-honoring ps in a narrow (77-column) hook context, as observed
#   live (COLUMNS=77) with a watcher command line longer than 77 columns, so
#   the recomputed identity is truncated and mismatches the recorded one.
#   This host's ps only truncates when its stdout is a tty, and fm_pid_identity
#   reads ps through a command-substitution pipe, so the live width-honoring
#   behavior is reproduced with the same deterministic ps stand-in the
#   regression test uses (it truncates to FM_TEST_PS_WIDTH unless -ww asks for
#   unlimited width). The real macOS ps truncating at 77 columns in a real pty
#   is captured separately in ps-truncation-real-pty.txt.
#
# Usage: ROOT=<repo> bash gate-e2e-demo.sh <label>
set -u
ROOT=${ROOT:?set ROOT to the firstmate checkout}
LABEL=${1:?pass a label, e.g. old-code or new-code}
GATE="$ROOT/bin/fm-continuity-pretool-check.sh"
LIB="$ROOT/bin/fm-wake-lib.sh"
T=$(mktemp -d "${TMPDIR:-/tmp}/fm-gate-demo.XXXXXX")
HOLDER=
HOLDER2=
cleanup() {
  [ -n "$HOLDER" ] && kill "$HOLDER" 2>/dev/null
  [ -n "$HOLDER2" ] && kill "$HOLDER2" 2>/dev/null
  wait 2>/dev/null
  rm -rf "$T"
}
trap cleanup EXIT

echo "### watcher-continuity gate, $LABEL ###"

setup_primary() {
  local primary=$1
  mkdir -p "$primary/bin" "$primary/state"
  printf '# fixture\n' > "$primary/AGENTS.md"
  git -C "$primary" init -q
  printf 'project=fixture\n' > "$primary/state/task.meta"
}

record_lock() {
  local state=$1 pid=$2 home=$3 identity
  identity=$(FM_STATE_OVERRIDE="$state" bash -c '. "$1"; fm_pid_identity "$2"' _ "$LIB" "$pid") || return 1
  mkdir -p "$state/.watch.lock"
  printf '%s\n' "$pid" > "$state/.watch.lock/pid"
  printf '%s\n' "$home" > "$state/.watch.lock/fm-home"
  printf '%s\n' "$ROOT/bin/fm-watch.sh" > "$state/.watch.lock/watcher-path"
  printf '%s\n' "$identity" > "$state/.watch.lock/pid-identity"
}

verdict() {
  local rc=$1 err=$2
  echo "gate exit for 'bin/fm-crew-state.sh task': $rc"
  if [ "$rc" = 0 ]; then
    echo "RESULT: ALLOWED - the live watcher is recognized and the fleet command proceeds"
  else
    echo "gate deny object (stderr):"
    cat "$err"
    echo "RESULT: DENIED - the healthy watcher was falsely evicted; every fleet command is refused (supervision deadlock)"
  fi
}

echo
echo "--- Demo 1: same home directory, case-respelled by the hook context ---"
PRIMARY="$T/sc/firstmate"
setup_primary "$PRIMARY"
sleep 300 &
HOLDER=$!
record_lock "$PRIMARY/state" "$HOLDER" "$PRIMARY" || { echo "FATAL: could not record holder identity"; exit 1; }
RESPELLED="$T/SC/firstmate"
if [ "$PRIMARY" -ef "$RESPELLED" ]; then
  echo "lock records fm-home: $PRIMARY"
  echo "hook context resolves: $RESPELLED (same directory on this case-insensitive APFS volume)"
else
  echo "FATAL: filesystem is case-sensitive here; demo 1 not applicable"
  exit 1
fi
rc=0
FM_ROOT_OVERRIDE="$RESPELLED" FM_HOME="$RESPELLED" FM_STATE_OVERRIDE="$PRIMARY/state" \
  "$GATE" --command 'bin/fm-crew-state.sh task' 2> "$T/err1" || rc=$?
verdict "$rc" "$T/err1"
kill "$HOLDER" 2>/dev/null
wait "$HOLDER" 2>/dev/null
HOLDER=

echo
echo "--- Demo 2: width-honoring ps under a narrow (77-column) hook context ---"
PRIMARY2="$T/home2/firstmate"
setup_primary "$PRIMARY2"
bash -c 'sleep 300 # long-watcher-invocation-trailer-that-pushes-the-recorded-command-line-well-past-seventy-seven-columns
true' >/dev/null 2>&1 &
HOLDER2=$!
sleep 0.3
record_lock "$PRIMARY2/state" "$HOLDER2" "$PRIMARY2" || { echo "FATAL: could not record holder identity"; exit 1; }
echo "recorded pid-identity (full width): $(cat "$PRIMARY2/state/.watch.lock/pid-identity")"
FAKEBIN="$T/fakebin"
mkdir -p "$FAKEBIN"
REAL_PS=$(command -v ps)
cat > "$FAKEBIN/ps" <<'SH'
#!/usr/bin/env bash
# Width-sensitive ps stand-in (same shape as the regression test): truncate to
# FM_TEST_PS_WIDTH columns unless -ww requests unlimited width.
ww=false
for arg in "$@"; do [ "$arg" = -ww ] && ww=true; done
out=$("$FM_TEST_REAL_PS" "$@") || exit $?
if $ww || [ -z "${FM_TEST_PS_WIDTH:-}" ]; then
  printf '%s\n' "$out"
else
  printf '%s\n' "$out" | cut -c1-"$FM_TEST_PS_WIDTH"
fi
SH
chmod +x "$FAKEBIN/ps"
rc2=0
PATH="$FAKEBIN:$PATH" FM_TEST_REAL_PS="$REAL_PS" FM_TEST_PS_WIDTH=77 \
  FM_ROOT_OVERRIDE="$PRIMARY2" FM_HOME="$PRIMARY2" FM_STATE_OVERRIDE="$PRIMARY2/state" \
  "$GATE" --command 'bin/fm-crew-state.sh task' 2> "$T/err2" || rc2=$?
verdict "$rc2" "$T/err2"
kill "$HOLDER2" 2>/dev/null
wait "$HOLDER2" 2>/dev/null
HOLDER2=
echo
echo "### end $LABEL ###"
Evidence: Reproducible gate fail-closed demo script
#!/usr/bin/env bash
# Fail-closed companion to gate-e2e-demo.sh: proves the fm_same_path (-ef)
# fallback did NOT weaken the watcher-continuity gate. With tasks in flight and
# a live lock holder, a lock that records a GENUINELY different home, or a home
# that no longer exists, must still deny fleet commands under the fixed code.
#
# Usage: ROOT=<repo> bash gate-e2e-fail-closed-demo.sh <label>
set -u
ROOT=${ROOT:?set ROOT to the firstmate checkout}
LABEL=${1:?pass a label, e.g. new-code}
GATE="$ROOT/bin/fm-continuity-pretool-check.sh"
LIB="$ROOT/bin/fm-wake-lib.sh"
T=$(mktemp -d "${TMPDIR:-/tmp}/fm-gate-failclosed.XXXXXX")
HOLDER=
FAILED=0
cleanup() {
  [ -n "$HOLDER" ] && kill "$HOLDER" 2>/dev/null
  wait 2>/dev/null
  rm -rf "$T"
}
trap cleanup EXIT

echo "### watcher-continuity gate fail-closed check, $LABEL ###"

PRIMARY="$T/sc/firstmate"
mkdir -p "$PRIMARY/bin" "$PRIMARY/state"
printf '# fixture\n' > "$PRIMARY/AGENTS.md"
git -C "$PRIMARY" init -q
printf 'project=fixture\n' > "$PRIMARY/state/task.meta"
sleep 300 &
HOLDER=$!

record_lock_home() { # <recorded-home>
  local home=$1 identity
  identity=$(FM_STATE_OVERRIDE="$PRIMARY/state" bash -c '. "$1"; fm_pid_identity "$2"' _ "$LIB" "$HOLDER") || return 1
  mkdir -p "$PRIMARY/state/.watch.lock"
  printf '%s\n' "$HOLDER" > "$PRIMARY/state/.watch.lock/pid"
  printf '%s\n' "$home" > "$PRIMARY/state/.watch.lock/fm-home"
  printf '%s\n' "$ROOT/bin/fm-watch.sh" > "$PRIMARY/state/.watch.lock/watcher-path"
  printf '%s\n' "$identity" > "$PRIMARY/state/.watch.lock/pid-identity"
}

run_case() { # <label> <recorded-home> <expected: allow|deny>
  local label=$1 recorded=$2 expect=$3 rc=0 outcome verdict
  record_lock_home "$recorded" || { echo "FATAL: could not record holder identity"; exit 1; }
  FM_ROOT_OVERRIDE="$PRIMARY" FM_HOME="$PRIMARY" FM_STATE_OVERRIDE="$PRIMARY/state" \
    "$GATE" --command 'bin/fm-crew-state.sh task' 2> "$T/err" || rc=$?
  case "$rc" in
    0) outcome=allow ;;
    2) outcome=deny ;;
    *) outcome="exit-$rc" ;;
  esac
  if [ "$outcome" = "$expect" ]; then verdict="OK, matches expectation"; else verdict="UNEXPECTED"; FAILED=1; fi
  echo
  echo "--- $label ---"
  echo "lock records fm-home: $recorded"
  echo "hook context home:    $PRIMARY"
  echo "gate result: $outcome (exit $rc)   expected: $expect   -> $verdict"
  if [ -s "$T/err" ]; then
    echo "gate deny object (stderr):"
    cat "$T/err"
  fi
}

run_case "sanity: lock records the exact home -> healthy watcher allows" "$PRIMARY" allow
OTHER="$T/genuinely-other-home"
mkdir -p "$OTHER"
run_case "lock records a genuinely different existing home -> still refuses" "$OTHER" deny
run_case "lock records a home that no longer exists -> still refuses" "$T/recorded-home-that-no-longer-exists" deny

echo
if [ "$FAILED" -eq 0 ]; then
  echo "RESULT: fail-closed semantics preserved (mismatch and missing-path locks still deny; only the same-directory respelling is accepted)"
else
  echo "RESULT: ONE OR MORE CASES DID NOT MATCH EXPECTATIONS"
fi
echo "### end $LABEL ###"
exit "$FAILED"
- Outcome: 🔧 1 issue found → auto-fixed ✅ across 2 runs (1h20m2s)

Pipeline

Updates from git push no-mistakes

✅ **intent** - passed

✅ No issues found.

✅ **Rebase** - passed

✅ No issues found.

⚠️ **Review** - 1 info
  • ℹ️ bin/fm-lock.sh:24 - The width-truncation defect class fixed here survives in pre-existing siblings not touched by this branch: bin/fm-lock.sh:24 and :42 regex-match ps -o args= output for harness liveness without -ww (a narrow-COLUMNS caller could misjudge a live interpreter-based lock holder as not-a-harness, i.e. a false-stale session lock), the live-e2e helpers (tests/fm-opencode-primary-live-e2e.test.sh:77, tests/fm-pi-primary-live-e2e.test.sh:62, tests/fm-grok-continuity-live-e2e.test.sh:46) read ps -o command= without -ww, and bin/fm-pending-reply-lib.sh:700 mitigates the same class with COLUMNS=10000 instead of -ww (works, but inconsistent with the new convention). All pre-existing; reasonable follow-up candidates.
🔧 **Test** - 1 issue found → auto-fixed ✅
  • ⚠️ tests/fm-afk-launch.test.sh:240 - Pre-existing flaky test unit_signal_exits_with_lock_cleanup (not caused by this diff: reproduced 2/8 runs on unchanged base commit 673b6ad with identical failure). The test TERMed the launcher child as soon as the lock dir appeared, but fm_afk_launch_main installs its TERM trap only after lock acquisition (bin/fm-afk-launch.sh:606-609), so under load the signal could land in that window and kill the child with the lock retained. Fixed in commit 739309d using the same ready-marker hardening this branch applied in 814a2f7; verified 10 consecutive clean runs under 4x CPU load.
  • bash tests/fm-watcher-lock.test.sh on target code (all 29 tests pass, including the 2 new regression tests)
  • test_pid_identity_unlimited_width_under_narrow_terminal and test_lock_match_accepts_same_dir_different_spelling run individually against base-commit bin/ (both fail with the defect messages) and against target bin/ (both pass)
  • E2E gate demo gate-e2e-demo.sh: live healthy watcher + bin/fm-continuity-pretool-check.sh --command &#39;bin/fm-crew-state.sh task&#39; with a case-respelled home (real case-insensitive APFS) and with a width-honoring ps at 77 columns; base code denies both (deny JSON captured), target code allows both
  • Real-platform probe: macOS ps truncates command= at 77 columns inside a real 77-column pty while ps -ww returns the full command (script -q /dev/null sh -c &#34;stty cols 77; ps ...&#34;)
  • bash tests/fm-continuity-pretool-check.test.sh (all 4 gate tests pass)
  • bin/fm-test-run.sh --changed --base 673b6ad (21 scripts: 20 pass, 1 expected herdr gate skip, 1 flaky failure investigated below)
  • Flake attribution for tests/fm-afk-launch.test.sh unit_signal_exits_with_lock_cleanup: 1/5 failures on target code, 2/8 failures on unchanged base code (pre-existing), then 10/10 clean runs under 4x CPU load after the test hardening committed as 739309d

🔧 Fix: verify changed-set suite green after afk-launch flake fix
✅ Re-checked - no issues remain.

  • bin/fm-test-run.sh --changed --base 673b6ad (repo's conservative changed-set: 21 test files, 0 failed, 1 expected gate-skip)
  • bin/fm-test-run.sh tests/fm-watcher-lock.test.sh (includes new regression tests test_pid_identity_unlimited_width_under_narrow_terminal and test_lock_match_accepts_same_dir_different_spelling)
  • bin/fm-test-run.sh tests/fm-continuity-pretool-check.test.sh (gate allow/deny contract unchanged)
  • bin/fm-test-run.sh tests/fm-afk-launch.test.sh plus 5 consecutive standalone reruns of the previously flaky unit_signal_exits_with_lock_cleanup (0/5 failures at final HEAD; round 1 also verified 10/10 under 4x CPU load)
  • Gate E2E demo at HEAD (gate-e2e-demo.sh): healthy watcher + case-respelled home and + 77-column width-honoring ps both ALLOW under new code; same scenarios DENY under base-commit bin/ (round 1 transcripts)
  • New fail-closed gate demo at HEAD (gate-e2e-fail-closed-demo.sh): exact-home sanity allows; genuinely different existing home and missing recorded home still deny with the standard deny object
  • Real macOS ps probe in a narrow pty: plain ps truncates command= at terminal width, ps -ww returns it in full (defect class confirmed on the diagnosing host)
  • Round 1: both new regression tests run individually against base 673b6ad bin/ fail with the expected messages and pass against the fixed code
✅ **Document** - passed

✅ No issues found.

✅ **Lint** - passed

✅ No issues found.

✅ **Push** - passed

✅ No issues found.

@kunchenguid

kunchenguid commented Jul 24, 2026

Copy link
Copy Markdown
Owner

Automated reminder: thanks for the PR! This branch currently has a merge conflict with the base branch.

When you get a chance, please rebase onto (or merge) the latest base branch, resolve the conflict, and push. After that, checks will re-run and the PR will get looked at again.

Noted for firstmate#851 at 739309df.

m0d7 added 7 commits July 25, 2026 08:29
fm_pid_identity recomputed the identity with ps without -ww, so a
narrow-COLUMNS context (a Claude Code PreToolUse hook inheriting the UI's
terminal width, observed live at COLUMNS=77) truncated command= and judged
a live watcher dead. fm_watcher_lock_matches_pid compared fm-home and
watcher-path as strings, so on a case-insensitive filesystem two processes
resolving the same directory with different case (observed live:
Sc/firstmate vs sc/firstmate) evicted a live watcher the same way.
Together these made fm-continuity-pretool-check deny every fleet command
with a remedy (re-arm) that cannot help.

Add -ww for unlimited ps width (also in fm-afk-start.sh's
daemon_pid_matches fallback), and fall back to a same-file -ef test when
the path string comparison fails; a missing path keeps -ef false, so a
genuine mismatch still refuses.

Fixes kunchenguid#799
test_watch_restart_attaches_to_healthy_peer's TERM-resistant node peer
could die if --restart's TERM arrived before node installed its SIGTERM
handler, making the arm start a fresh watcher instead of attaching
(observed flaking under parallel test load). The peer now writes a
readiness marker after installing the handler and the test waits for it
before launching the arm.
… afk-launch signal test

unit_signal_exits_with_lock_cleanup TERMed the launcher child as soon as the
lifecycle lock dir appeared, but fm_afk_launch_main installs its TERM trap
only after fm_afk_launch_lock_acquire returns, so on a loaded host the signal
could land in that window and kill the child by default action with the lock
retained (observed flaking on unchanged base code too, 2 of 8 runs). The
stand-in start hook now writes a ready marker the test waits for, which runs
strictly after the traps are installed, and the sleeper is backgrounded so
the trapped TERM interrupts the wait builtin immediately and its recorded
pid is reaped. Verified 10 consecutive runs under 4x CPU load with 0
failures. Same defect class as the peer-attach hardening in 814a2f7.
@m0d7
m0d7 force-pushed the fm/fix-watcher-lockmatch branch from 739309d to fa8f594 Compare July 25, 2026 05:37
@kunchenguid kunchenguid removed the wheelhouse:pending-contributor-action Managed by Wheelhouse label Jul 25, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Watcher-continuity deadlock: fm_pid_identity ps command truncated in narrow-COLUMNS hook context

2 participants