fix(bin): stop evicting live watchers on ps truncation and path spelling - #851
Open
m0d7 wants to merge 8 commits into
Open
fix(bin): stop evicting live watchers on ps truncation and path spelling#851m0d7 wants to merge 8 commits into
m0d7 wants to merge 8 commits into
Conversation
5 tasks
Owner
|
Automated reminder: thanks for the PR! This branch currently has a merge conflict with the base branch. When you get a chance, please rebase onto (or merge) the latest base branch, resolve the conflict, and push. After that, checks will re-run and the PR will get looked at again. Noted for firstmate#851 at |
fm_pid_identity recomputed the identity with ps without -ww, so a narrow-COLUMNS context (a Claude Code PreToolUse hook inheriting the UI's terminal width, observed live at COLUMNS=77) truncated command= and judged a live watcher dead. fm_watcher_lock_matches_pid compared fm-home and watcher-path as strings, so on a case-insensitive filesystem two processes resolving the same directory with different case (observed live: Sc/firstmate vs sc/firstmate) evicted a live watcher the same way. Together these made fm-continuity-pretool-check deny every fleet command with a remedy (re-arm) that cannot help. Add -ww for unlimited ps width (also in fm-afk-start.sh's daemon_pid_matches fallback), and fall back to a same-file -ef test when the path string comparison fails; a missing path keeps -ef false, so a genuine mismatch still refuses. Fixes kunchenguid#799
test_watch_restart_attaches_to_healthy_peer's TERM-resistant node peer could die if --restart's TERM arrived before node installed its SIGTERM handler, making the arm start a fresh watcher instead of attaching (observed flaking under parallel test load). The peer now writes a readiness marker after installing the handler and the test waits for it before launching the arm.
…s in continuity coverage
… afk-launch signal test unit_signal_exits_with_lock_cleanup TERMed the launcher child as soon as the lifecycle lock dir appeared, but fm_afk_launch_main installs its TERM trap only after fm_afk_launch_lock_acquire returns, so on a loaded host the signal could land in that window and kill the child by default action with the lock retained (observed flaking on unchanged base code too, 2 of 8 runs). The stand-in start hook now writes a ready marker the test waits for, which runs strictly after the traps are installed, and the sleeper is backgrounded so the trapped TERM interrupts the wait builtin immediately and its recorded pid is reaped. Verified 10 consecutive runs under 4x CPU load with 0 failures. Same defect class as the peer-attach hardening in 814a2f7.
m0d7
force-pushed
the
fm/fix-watcher-lockmatch
branch
from
July 25, 2026 05:37
739309d to
fa8f594
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Intent
The developer wanted to fix a watcher-continuity false-deny bug in the firstmate repo's bin/fm-wake-lib.sh, where a healthy watcher was wrongly judged dead and every fleet command was denied, causing a supervision deadlock diagnosed live on macOS. The fix covers two stacked defects: adding -ww to the ps call in fm_pid_identity so narrow-COLUMNS hook contexts no longer truncate the command identity (closing upstream issue #799, with the sibling ps call in bin/fm-afk-start.sh also patched), and adding a same-file -ef fallback to the lock's path string comparisons so case-unstable getcwd spellings on case-insensitive filesystems don't evict a live watcher, while preserving fail-closed semantics. Requirements included regression tests in tests/ verified to fail on old code and pass on new, a clean bin/fm-lint.sh run, a passing full test suite, and driving the no-mistakes validation pipeline (review, tests, docs, lint) to completion rather than hand-fixing findings. The PR body had to document both defects with the captured live evidence, use "Fixes #799" wording, and note the case-stability defect was previously unreported; explicit constraints included no em dashes, no agent co-author on commits, working only from a clean isolated worktree, and never pushing to the default branch. When the final push failed with a 403 because the authenticated account lacked write access to kunchenguid/firstmate, the developer approved a fork-based PR, pushing the branch to a fork and opening a cross-repo PR against the upstream repo.
What Changed
-wwto the livenesspscalls infm_pid_identity(bin/fm-wake-lib.sh) anddaemon_pid_matches(bin/fm-afk-start.sh) so a narrow-COLUMNScaller (e.g. a PreToolUse hook under the UI terminal width) can no longer truncatecommand=, recompute a non-matching identity, and evict a live watcher/daemon as stale.fm_same_pathhelper that compares by string first and falls back to same-file identity (-ef), and routed the lock'sfm-home/watcher-pathchecks inbin/fm-wake-lib.shandbin/fm-watch-arm.shthrough it, so case-unstablegetcwd()spellings on a case-insensitive filesystem stop evicting a live watcher while a genuinely different or missing path still fails closed.tests/fm-watcher-lock.test.shfor both defects, documented the invariants indocs/watcher-continuity.md, and hardened the flakyunit_signal_exits_with_lock_cleanuptest intests/fm-afk-launch.test.shto wait for the launcher's TERM-trap installation before signaling.Risk Assessment
✅ Low: A narrow, fail-closed-preserving fix applied symmetrically at both lock-comparison sites, with deterministic regression tests for both stacked defects and no correctness, portability, or security issues found in the changed code.
Testing
Exercised the fix at the end-user surface by driving the watcher-continuity PreToolUse gate against a live healthy watcher fixture: base-commit code reproduces both live false-denies (case-respelled home spelling and narrow-width ps identity truncation) while the fixed code allows the fleet command and still refuses genuinely different or missing recorded homes; the two new regression tests were verified to fail on old code and pass on new, the round-1 flaky-test fix in tests/fm-afk-launch.test.sh held stable across repeated runs, and the repo's full changed-set suite of 21 test files passed with zero failures at the final HEAD.
Evidence: Gate E2E transcript, OLD code (base 673b6ad): both live-incident shapes DENIED (supervision deadlock reproduced)
Evidence: Gate E2E transcript, NEW code at final HEAD 739309d: both shapes ALLOWED
--- Demo 1: same home directory, case-respelled by the hook context --- gate exit for 'bin/fm-crew-state.sh task': 0 RESULT: ALLOWED - the live watcher is recognized and the fleet command proceeds --- Demo 2: width-honoring ps under a narrow (77-column) hook context --- gate exit for 'bin/fm-crew-state.sh task': 0 RESULT: ALLOWED - the live watcher is recognized and the fleet command proceedsEvidence: Gate fail-closed transcript, NEW code: genuinely different / missing recorded home still deny; exact home allows
Evidence: New regression tests run against OLD code: both fail with the expected eviction messages
Evidence: New regression tests against NEW code: both pass
Evidence: Real macOS ps truncation in a 77-column pty (defect class confirmed with the system ps; -ww returns the full command)
Evidence: Flaky afk-launch signal test: root cause, base-commit reproduction, fix, 10/10 clean under 4x load (round 1)
Evidence: Round-2 stability re-check of the fixed signal test at final HEAD: 0/5 failures
Evidence: Changed-set suite summary at HEAD 739309d (21 scripts, 0 failed)
Evidence: Reproducible gate E2E demo script (old/new code, both defects)
Evidence: Reproducible gate fail-closed demo script
Pipeline
Updates from git push no-mistakes
✅ **intent** - passed
✅ No issues found.
✅ **Rebase** - passed
✅ No issues found.
bin/fm-lock.sh:24- The width-truncation defect class fixed here survives in pre-existing siblings not touched by this branch: bin/fm-lock.sh:24 and :42 regex-matchps -o args=output for harness liveness without -ww (a narrow-COLUMNS caller could misjudge a live interpreter-based lock holder as not-a-harness, i.e. a false-stale session lock), the live-e2e helpers (tests/fm-opencode-primary-live-e2e.test.sh:77, tests/fm-pi-primary-live-e2e.test.sh:62, tests/fm-grok-continuity-live-e2e.test.sh:46) readps -o command=without -ww, and bin/fm-pending-reply-lib.sh:700 mitigates the same class with COLUMNS=10000 instead of -ww (works, but inconsistent with the new convention). All pre-existing; reasonable follow-up candidates.🔧 **Test** - 1 issue found → auto-fixed ✅
tests/fm-afk-launch.test.sh:240- Pre-existing flaky test unit_signal_exits_with_lock_cleanup (not caused by this diff: reproduced 2/8 runs on unchanged base commit 673b6ad with identical failure). The test TERMed the launcher child as soon as the lock dir appeared, but fm_afk_launch_main installs its TERM trap only after lock acquisition (bin/fm-afk-launch.sh:606-609), so under load the signal could land in that window and kill the child with the lock retained. Fixed in commit 739309d using the same ready-marker hardening this branch applied in 814a2f7; verified 10 consecutive clean runs under 4x CPU load.bash tests/fm-watcher-lock.test.shon target code (all 29 tests pass, including the 2 new regression tests)test_pid_identity_unlimited_width_under_narrow_terminalandtest_lock_match_accepts_same_dir_different_spellingrun individually against base-commit bin/ (both fail with the defect messages) and against target bin/ (both pass)E2E gate demogate-e2e-demo.sh: live healthy watcher +bin/fm-continuity-pretool-check.sh --command 'bin/fm-crew-state.sh task'with a case-respelled home (real case-insensitive APFS) and with a width-honoring ps at 77 columns; base code denies both (deny JSON captured), target code allows bothReal-platform probe: macOSpstruncatescommand=at 77 columns inside a real 77-column pty whileps -wwreturns the full command (script -q /dev/null sh -c "stty cols 77; ps ...")bash tests/fm-continuity-pretool-check.test.sh(all 4 gate tests pass)bin/fm-test-run.sh --changed --base 673b6ad(21 scripts: 20 pass, 1 expected herdr gate skip, 1 flaky failure investigated below)Flake attribution fortests/fm-afk-launch.test.shunit_signal_exits_with_lock_cleanup: 1/5 failures on target code, 2/8 failures on unchanged base code (pre-existing), then 10/10 clean runs under 4x CPU load after the test hardening committed as 739309d🔧 Fix: verify changed-set suite green after afk-launch flake fix
✅ Re-checked - no issues remain.
bin/fm-test-run.sh --changed --base 673b6ad (repo's conservative changed-set: 21 test files, 0 failed, 1 expected gate-skip)bin/fm-test-run.sh tests/fm-watcher-lock.test.sh (includes new regression tests test_pid_identity_unlimited_width_under_narrow_terminal and test_lock_match_accepts_same_dir_different_spelling)bin/fm-test-run.sh tests/fm-continuity-pretool-check.test.sh (gate allow/deny contract unchanged)bin/fm-test-run.sh tests/fm-afk-launch.test.sh plus 5 consecutive standalone reruns of the previously flaky unit_signal_exits_with_lock_cleanup (0/5 failures at final HEAD; round 1 also verified 10/10 under 4x CPU load)Gate E2E demo at HEAD (gate-e2e-demo.sh): healthy watcher + case-respelled home and + 77-column width-honoring ps both ALLOW under new code; same scenarios DENY under base-commit bin/ (round 1 transcripts)New fail-closed gate demo at HEAD (gate-e2e-fail-closed-demo.sh): exact-home sanity allows; genuinely different existing home and missing recorded home still deny with the standard deny objectReal macOS ps probe in a narrow pty: plain ps truncates command= at terminal width, ps -ww returns it in full (defect class confirmed on the diagnosing host)Round 1: both new regression tests run individually against base 673b6ad bin/ fail with the expected messages and pass against the fixed code✅ **Document** - passed
✅ No issues found.
✅ **Lint** - passed
✅ No issues found.
✅ **Push** - passed
✅ No issues found.