Skip to content

fix(afk): refuse bare-shell supervisor targets and bound watcher shutdown#862

Open
ManveerBhullar wants to merge 1 commit into
kunchenguid:mainfrom
ManveerBhullar:fm/afk-daemon-reliability-scout
Open

fix(afk): refuse bare-shell supervisor targets and bound watcher shutdown#862
ManveerBhullar wants to merge 1 commit into
kunchenguid:mainfrom
ManveerBhullar:fm/afk-daemon-reliability-scout

Conversation

@ManveerBhullar

Copy link
Copy Markdown

What

Two away-mode supervision reliability repairs, both reproduced and root-caused in scout afk-daemon-reliability-scout (report in that task), implemented on the daemon only — no broad runtime-backend redesign, no manual target guessing, no broad tmux sweeps.

1. Fail fast at startup for a confidently dead/bare-shell supervisor target

fm_backend_target_exists only proved the pane was present, so an explicit FM_SUPERVISOR_TARGET pointing at a bare shell (e.g. the wrong tmux window such as firstmate:0.0, or an agent that had exited to its login shell) passed startup validation. The composer guard then read that shell as pending/unknown and deferred every escalation until the max-defer wedge (~300 s).

The daemon's startup now also probes fm_backend_agent_alive (bin/fm-supervise-daemon.sh):

  • dead → refuse fast (log startup failed: ... bare shell, exit 1, release lock + pidfile), mirroring the secondmate-liveness policy that gates on dead only;
  • unknown (pi's generic node, or an unreadable pane) → warn and continue, so a real primary is never falsely refused.

The composer guard is unchanged — it is what correctly prevented typing into a shell; this just surfaces the problem immediately instead of at the max-defer wedge.

Test seam: FM_SUPERVISOR_ALLOW_DEAD_TARGET=1 downgrades the refusal to a warn. No external signal can distinguish an isolated E2E test's simulated agent pane (a shell-named process drawing a composer) from a real bare shell, so the opt-out is explicit. It is never set in production, and the composer guard still prevents any injection into a shell regardless.

2. Bound watcher shutdown so AFK stop completes inside its budget

The daemon's cleanup wait()ed its watcher child without a bound. fm-watch.sh blocks each cycle in an external sleep $FM_POLL (default 15) and bash defers a trapped SIGTERM until that sleep finishes, so on fm-afk-launch.sh stop the daemon could block ~FM_POLL and exceed stop's 10 s SIGTERM budget — surfacing as away-mode daemon did not exit after SIGTERM; preserving lifecycle state (the daemon did exit shortly after; a later stop found it absent).

Cleanup now SIGKILLs the watcher after a short grace (FM_WATCHER_SHUTDOWN_GRACE_SECS, default 5), well inside the 10 s budget. The watcher's durable state is the on-disk wake queue and suppression markers, so a forced kill never loses work.

Why these are the right scope

  • Defect 1 root cause is caller target selection (wrong explicit target) + weak startup validation (presence-only). The proven path is $TMUX_PANE auto-discovery; the fix makes a wrong-but-existing target fail fast.
  • Defect 2 root cause is daemon shutdown signaling (the watcher's deferred SIGTERM during external sleep + the daemon's unbounded wait). The "terminal-close warning" on the later stop is a benign consequence (the detached session auto-collapses on pane-process exit), not a separate defect.

Verification

  • Reproduced both defects in isolated homes pre-fix (defect 1: a bash pane passes target_exists, reads composer pending; defect 2: stop = 10.4 s with the timeout message; minimal trap … TERM; sleep 15 fires its trap only once the external sleep completes).
  • Post-fix:
    • defect 1: stop completes in ~6.3–6.5 s with the normal "away mode stopped" line;
    • defect 2: a bare-shell target is refused at startup (exit 1, no pidfile).
  • bin/fm-lint.sh (pinned shellcheck 0.11.0, CI's gate): clean.
  • Regression tests added:
    • tests/fm-daemon.test.sh: test_daemon_refuses_bare_shell_supervisor_target, test_daemon_does_not_refuse_unknown_supervisor_target, test_daemon_allow_dead_opt_out_permits_simulated_pane (real tmux, real daemon executed directly);
    • tests/fm-afk-launch.test.sh: e2e_tmux_daemon_stop_bounded_inside_budget (real daemon, not the sleeper placeholder, so a watcher child actually exists).
  • Existing daemon/inject E2E suites pass; the simulated-agent-pane suites set the new opt-out (tests/fm-afk-inject-e2e.test.sh, tests/fm-afk-inject-herdr-e2e.test.sh, tests/fm-afk-pi-herdr-return-e2e.test.sh).
  • Evidence recorded in docs/tmux-backend.md ("Agent liveness probe") and docs/herdr-backend.md ("Daemon shutdown bounds the watcher wait").

Notes

…down

Two away-mode reliability defects (scout afk-daemon-reliability-scout):

1. Startup validation checked only that the supervisor pane existed
   (fm_backend_target_exists), so an explicit FM_SUPERVISOR_TARGET pointing
   at a bare shell (a wrong tmux window, or an agent that exited to its
   login shell) started cleanly, then deferred every escalation until the
   max-defer wedge. Now also probe fm_backend_agent_alive and fail fast on a
   confident `dead` (mirroring the secondmate-liveness policy that gates on
   `dead` only); `unknown` (pi's node, unreadable panes) warns and continues
   so a real primary is never refused. A test-seam opt-out
   FM_SUPERVISOR_ALLOW_DEAD_TARGET covers isolated E2E tests that simulate
   an agent pane with a shell-named process; the composer guard still
   prevents any injection into a shell regardless.

2. The daemon's cleanup wait()ed its watcher child without a bound. The
   watcher blocks in an external `sleep $FM_POLL` and bash defers a trapped
   SIGTERM until that sleep finishes, so on fm-afk-launch stop the daemon
   could block ~FM_POLL (15s) and exceed stop's 10s budget, surfacing as
   "away-mode daemon did not exit after SIGTERM". Cleanup now SIGKILLs the
   watcher after a short grace (FM_WATCHER_SHUTDOWN_GRACE_SECS, default 5),
   well inside the budget; the watcher's durable state is the on-disk wake
   queue and suppression markers, so a forced kill never loses work.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant