Skip to content

fix(watch): keep a declared pause on its bounded cadence - #1153

Open
august-agent wants to merge 1 commit into
kunchenguid:mainfrom
august-agent:fm/watcher-paused-churn-w4
Open

fix(watch): keep a declared pause on its bounded cadence#1153
august-agent wants to merge 1 commit into
kunchenguid:mainfrom
august-agent:fm/watcher-paused-churn-w4

Conversation

@august-agent

Copy link
Copy Markdown

The problem

A crew that declares paused: and then sits on long external monitors made the watcher churn. Every arm cycle started a watcher that immediately closed on stale: <window> for that same declared wait, so no watcher stayed up to hold the home lock, and the turn-end path then reported watcher: FAILED - cycle ended without an actionable reason and demanded a supervision repair. This repeated every few seconds and never settled. Declaring the pause helped the first surface only; the repetition continued.

Root cause

Two independent defects, both reproduced before changing any code.

Watcher, bin/fm-watch.sh. pause_state_class demoted a still-declared pause to none whenever the crew's agent was still live, so a live declaration was designed to surface exactly once per stale hash. That holds only for a byte-static pane. A live crew's pane text keeps drifting, and on every drift the changed-hash branch reclassified the declaration as none and called clear_pause_tracking, which deletes .paused-<key> and the .paused-resurfaced-<key> cadence anchor. Two polls later the pane was stably stale again with no suppressor, so the same declared wait surfaced as a bare stale: wake and the cycle closed on it. Nothing bounded the repetition: the FM_PAUSE_RESURFACE_SECS cadence never applied to a live paused crew at all. A briefly busy pane wiped the same markers through the second clear in the busy branch.

Reproduction, five consecutive watcher cycles over a live paused crew whose pane gained one line between cycles:

round 1: EXITED -> stale: test:fm-op
round 2: EXITED -> stale: test:fm-op
round 3: EXITED -> stale: test:fm-op
round 4: EXITED -> stale: test:fm-op
round 5: EXITED -> stale: test:fm-op
stale wakes queued: 5

The same fixture with a byte-static pane surfaced once and absorbed the remaining four cycles, which isolates pane drift as the amplifier.

Arm layer, bin/fm-watch-arm.sh. Several arms can follow one watcher cycle. Only the arm that owns the watcher prints the wake reason; a follower attaches, then sees the cycle close with no successor and reported the typed failure for a close supervision had handled correctly:

round 1: armA rc=1 [watcher: already running pid N|watcher: attached pid=N (beacon 0s)|watcher: FAILED - cycle ended without an actionable reason|]
round 1: armB rc=0 [watcher: started pid=N (beacon fresh)|signal: .../op.status|]

The fix

A declaration stays on its bounded FM_PAUSE_RESURFACE_SECS cadence once it has been surfaced inside the current window, whether or not its agent is still live, and its cadence markers survive both pane drift and a briefly busy pane. The withdrawal check at the top of the stale loop remains the one owner of clearing them.

Preserved: an active run still outranks the declaration and takes the ordinary wedge timer; a crew with no declaration still trips stale immediately; a forgotten pause still re-surfaces once per window as an awaiting-external recheck; a live declaration firstmate has not seen yet still surfaces once on first sight.

A follower arm now consults the durable wake queue, the shared record of what a cycle surfaced. A record appended at or after the cycle began, together with a fresh liveness beacon, explains the close, so the follower prints those queued reason lines and exits zero. A stale or absent beacon never qualifies, so genuinely down supervision still fails loudly and the guard keeps its real protection.

After the fix, the same five-cycle reproduction surfaces once and absorbs the rest, and both concurrent arms return zero with the queued reason.

Tests

  • tests/fm-watch-triage.test.sh - a live declared pause surfaces once, absorbs across pane drift with its cadence markers and wedge-timer exemption intact, and returns to ordinary stale surfacing once withdrawn. Fails on the pre-fix watcher at drift cycle 2.
  • tests/fm-watcher-lock.test.sh - a follower arm whose cycle queued a wake reports that wake instead of a typed failure, and the same setup with an ancient beacon still fails loudly. The first fails on the pre-fix arm with exit 1.
  • tests/fm-turnend-guard.test.sh - the guard stays silent for a live beating watcher with wakes queued for the next turn, and still blocks when no beacon exists.

Also fixes a pre-existing race in the watcher-lock suite: the TERM-resistant peer fixture is now waited on until it has installed its signal handler, since the restart's signal could otherwise land during interpreter startup, kill the peer, and fail the attach assertion for reasons unrelated to the code under test. That failure reproduces on main.

Validation

bin/fm-lint.sh clean, bin/fm-doc-audience-check.sh clean, and bin/fm-test-run.sh --family watcher-wake-lock green (10 suites, 0 failed), plus the auto-arm, instruction-owner, and documentation-audience suites.

A crew that declares `paused:` and then sits on long external monitors kept
re-surfacing the same stale wake on every watcher cycle.

`pause_state_class` demoted a still-declared pause to `none` whenever the
crew's agent was still live, so only the first sighting of a stale hash was
ever meant to surface. A live crew's pane text keeps drifting, though, and
every drift reclassified the declaration, cleared the pause cadence markers,
and let the very next stable hash surface the same wait again. Nothing bounded
the repetition, so the watcher closed its cycle on that wake over and over and
no watcher stayed up to hold the home lock.

A declaration now stays on its bounded `FM_PAUSE_RESURFACE_SECS` cadence once
it has been surfaced inside the current window, whether or not its agent is
still live, and the cadence markers survive pane drift and a briefly busy
pane. An active run still outranks the declaration, a crew with no declaration
still trips stale normally, and a forgotten pause still re-surfaces once per
window for a recheck.

The same fleet also made `bin/fm-watch-arm.sh` report a false failure. Several
arms can follow one watcher cycle, and only the arm that owns the watcher
prints its wake reason; every follower saw the same close with no successor and
reported `watcher: FAILED - cycle ended without an actionable reason`, which
each turn then had to treat as a supervision repair. A follower now consults
the durable wake queue, the shared record of what a cycle surfaced: a record
appended at or after the cycle began, together with a fresh liveness beacon,
explains the close, so the follower reports those queued reason lines and exits
zero. A stale or absent beacon never qualifies, so genuinely down supervision
still fails loudly.

Also waits for the TERM-resistant peer fixture in the watcher-lock suite to
install its signal handler before arming. Without that wait the restart's
signal could land during interpreter startup and kill the peer, failing the
attach assertion for reasons unrelated to the code under test.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant