fix(watcher): stop false stale and possible-wedge wakes for healthy crew#897
Open
mattwwatson wants to merge 3 commits into
Open
fix(watcher): stop false stale and possible-wedge wakes for healthy crew#897mattwwatson wants to merge 3 commits into
mattwwatson wants to merge 3 commits into
Conversation
4 tasks
The watcher could wake the supervisor for crew that were healthy: a declared pause was re-sighted by pane hash, so any pane repaint reset or re-escalated it, and repeat possible-wedge escalations fired on a timer without any new evidence. Anchor pause sightings on the declared pause marker instead of the pane hash, require fresh evidence (a probe re-read) before re-escalating a wedge, cap the doubling escalation window with FM_WEDGE_ESCALATE_MAX_SECS, and teach crew_absorb_class to distinguish an unreadable pane from a stopped one so each is classified and absorbed correctly.
…fication call sites
mattwwatson
force-pushed
the
up/watcher-false-wedge
branch
from
July 23, 2026 21:11
315bff3 to
4175bf8
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Intent
Upstream contribution from the mattwwatson/firstmate fork: stop the watcher waking the supervisor for healthy crew. Anchor declared-pause sightings on the pause marker rather than the pane hash so repaints do not reset or re-escalate a pause, require fresh probe evidence before repeating a possible-wedge escalation (doubling window capped by FM_WEDGE_ESCALATE_MAX_SECS), and make crew_absorb_class distinguish unreadable panes from stopped ones. This addresses the same watcher-flooding pain reported in upstream issue #882. New tests/fm-watch-false-wedge.test.sh passes isolated on the upstream base (9/9 locally, plus fm-watch-triage 34/34); it runs in the portable serial lane unless maintainers prefer shard placement.
What Changed
FM_WEDGE_ESCALATE_MAX_SECSknob (validated as numeric, documented indocs/configuration.mdanddocs/architecture.md).crew_absorb_classinbin/fm-classify-lib.shnow distinguishes unreadable panes from stopped ones, and a new regression suitetests/fm-watch-false-wedge.test.sh(9 cases, registered in thewatcher-wake-lockchanged-file family and portable serial lane) plus expandedfm-watch-triagetests cover the changed classification paths.Risk Assessment
✅ Low: All substantiated round-1 findings were fixed exactly as recommended (test-family registration, backoff-cap validation, key-derivation dedup) with no new issues introduced, and the core watcher rework remains intent-conformant with comprehensive regression tests (9 new false-wedge tests plus the updated triage matrix) covering every fail-safe direction.
Testing
Ran the new fm-watch-false-wedge suite (9/9) and fm-watch-triage (34/34) plus the four adjacent watcher/daemon/wake suites (all green), confirmed both suites sit in the portable-serial lane as stated, and captured product-level before/after transcripts by running the base-commit watcher next to the fixed one: the fix collapses six bare stale wakes for a healthy paused crew into one labeled sighting, stops repeat wedge escalations without fresh probe evidence (doubling window shown in the triage log), and distinguishes unreadable crew-state reads from stopped crews — no failures or regressions found. No visual artifact applies: this is a headless CLI watcher, so its supervisor-facing wake queue and triage-log transcripts are the end-user surface.
Evidence: Symptom A before/after: 6 bare stale wakes on base vs 1 labeled declared-pause wake after the fix
Evidence: Symptom B before/after: base re-escalates on elapsed time alone; fix absorbs the unchanged repeat with a doubled escalation window
Evidence: crew_absorb_class before/after: failed read now classified 'unreadable', not folded into 'none'
reader exits non-zero (worktree unreadable): BEFORE (base 82a7943): none AFTER (this change) : unreadable reader returns 'state: parked' (crew really stopped): BEFORE (base 82a7943): none AFTER (this change) : nonePipeline
Updates from git push no-mistakes
✅ **intent** - passed
✅ No issues found.
✅ **Rebase** - passed
✅ No issues found.
🔧 **Review** - 4 issues found → auto-fixed ✅
bin/fm-test-run.sh:136- The new regression test tests/fm-watch-false-wedge.test.sh is not registered in family_for_basename() in bin/fm-test-run.sh, so it falls into the 'unclassified' family. The changed-file map routes bin/fm-watch* and bin/fm-classify-lib.sh to the 'watcher-wake-lock' family only, so futurefm-test-run.sh --changedruns after edits to the exact files this test guards will silently skip it (it only runs via --all or the portable-serial lane). Add fm-watch-false-wedge.test.sh to the watcher-wake-lock family list (bin/fm-test-run.sh:136 area) so watcher changes re-run their regression cover.bin/fm-watch.sh:158- WEDGE_ESCALATE_MAX_SECS is not validated as numeric, while WEDGE_UNREADABLE_SURFACE_COUNT one line below gets a case guard. If FM_WEDGE_ESCALATE_MAX_SECS is set empty or non-numeric,[ "$secs" -gt "$WEDGE_ESCALATE_MAX_SECS" ]in wedge_escalate_interval errors and evaluates false, silently removing the cap: the repeat-escalation backoff then doubles unchecked up to STALE_ESCALATE_SECS*2^12 (~11 days with defaults), effectively silencing repeat wedge escalations — the missed-alarm mode this change elsewhere guards against. Apply the same numeric-fallback case guard used for WEDGE_UNREADABLE_SURFACE_COUNT.bin/fm-watch.sh:1092- The main loop still derives the per-window marker key with the inline triple substitution (key=${w//:/}; key=${key////}; key=${key//./_}) at lines 1092-1094 even though this commit introduces state_key() as the documented single owner of that derivation and converts every other site (including line 1104 a few lines below, which recomputes the same key via state_key). Folding this last site into state_key() would complete the dedup and remove the double computation.bin/fm-watch.sh:557- Upgrade transition note: pre-existing .paused-resurfaced-* markers written by the old code contain an epoch timestamp, while pause_declared_class now compares the marker content against the current pause line. On the first watcher run after upgrade, every already-surfaced live-agent pause will mismatch and be surfaced once more before re-anchoring on the pause instance. One extra labeled wake per paused window, self-healing; acceptable, but worth knowing when validating the rollout.🔧 Fix: register wedge test in changed map, guard cap, dedup key
✅ Re-checked - no issues remain.
✅ **Test** - passed
✅ No issues found.
bash tests/fm-watch-false-wedge.test.sh(9/9 ok, exit 0)bash tests/fm-watch-triage.test.sh(34/34 ok, exit 0)bin/fm-test-run.sh --list --lane portable-serialconfirms both suites are registered in the portable serial lanebash tests/fm-watch-checkpoint.test.sh,tests/fm-watcher-lock.test.sh,tests/fm-daemon.test.sh,tests/fm-wake-queue.test.sh(4/27/97/12 ok, all exit 0) covering the other changed watcher/daemon pathsManual e2e before/after: ran base-commit (82a7943) and fixed fm-watch.sh against a simulated healthy paused crew with a repainting pane for 6 rounds, capturing wake queues and triage logs (symptom A evidence)Manual e2e before/after: primed a quiet provably-working stale pane past FM_STALE_ESCALATE_SECS and re-ran with unchanged probe evidence, showing base re-escalates while the fix absorbs with a doubled next-escalation window (symptom B evidence)Manual before/after ofcrew_absorb_classwith a failing crew-state reader vs a parked crew, base lib vs fixed libVerifiedFM_WEDGE_ESCALATE_MAX_SECSis documented in docs/configuration.md and wired in bin/fm-watch.sh✅ **Document** - passed
✅ No issues found.
✅ **Lint** - passed
✅ No issues found.
✅ **Push** - passed
✅ No issues found.