fix(watch): keep a declared pause on its bounded cadence - #1153
Open
august-agent wants to merge 1 commit into
Open
fix(watch): keep a declared pause on its bounded cadence#1153august-agent wants to merge 1 commit into
august-agent wants to merge 1 commit into
Conversation
A crew that declares `paused:` and then sits on long external monitors kept re-surfacing the same stale wake on every watcher cycle. `pause_state_class` demoted a still-declared pause to `none` whenever the crew's agent was still live, so only the first sighting of a stale hash was ever meant to surface. A live crew's pane text keeps drifting, though, and every drift reclassified the declaration, cleared the pause cadence markers, and let the very next stable hash surface the same wait again. Nothing bounded the repetition, so the watcher closed its cycle on that wake over and over and no watcher stayed up to hold the home lock. A declaration now stays on its bounded `FM_PAUSE_RESURFACE_SECS` cadence once it has been surfaced inside the current window, whether or not its agent is still live, and the cadence markers survive pane drift and a briefly busy pane. An active run still outranks the declaration, a crew with no declaration still trips stale normally, and a forgotten pause still re-surfaces once per window for a recheck. The same fleet also made `bin/fm-watch-arm.sh` report a false failure. Several arms can follow one watcher cycle, and only the arm that owns the watcher prints its wake reason; every follower saw the same close with no successor and reported `watcher: FAILED - cycle ended without an actionable reason`, which each turn then had to treat as a supervision repair. A follower now consults the durable wake queue, the shared record of what a cycle surfaced: a record appended at or after the cycle began, together with a fresh liveness beacon, explains the close, so the follower reports those queued reason lines and exits zero. A stale or absent beacon never qualifies, so genuinely down supervision still fails loudly. Also waits for the TERM-resistant peer fixture in the watcher-lock suite to install its signal handler before arming. Without that wait the restart's signal could land during interpreter startup and kill the peer, failing the attach assertion for reasons unrelated to the code under test.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The problem
A crew that declares
paused:and then sits on long external monitors made the watcher churn. Every arm cycle started a watcher that immediately closed onstale: <window>for that same declared wait, so no watcher stayed up to hold the home lock, and the turn-end path then reportedwatcher: FAILED - cycle ended without an actionable reasonand demanded a supervision repair. This repeated every few seconds and never settled. Declaring the pause helped the first surface only; the repetition continued.Root cause
Two independent defects, both reproduced before changing any code.
Watcher,
bin/fm-watch.sh.pause_state_classdemoted a still-declared pause tononewhenever the crew's agent was still live, so a live declaration was designed to surface exactly once per stale hash. That holds only for a byte-static pane. A live crew's pane text keeps drifting, and on every drift the changed-hash branch reclassified the declaration asnoneand calledclear_pause_tracking, which deletes.paused-<key>and the.paused-resurfaced-<key>cadence anchor. Two polls later the pane was stably stale again with no suppressor, so the same declared wait surfaced as a barestale:wake and the cycle closed on it. Nothing bounded the repetition: theFM_PAUSE_RESURFACE_SECScadence never applied to a live paused crew at all. A briefly busy pane wiped the same markers through the second clear in the busy branch.Reproduction, five consecutive watcher cycles over a live paused crew whose pane gained one line between cycles:
The same fixture with a byte-static pane surfaced once and absorbed the remaining four cycles, which isolates pane drift as the amplifier.
Arm layer,
bin/fm-watch-arm.sh. Several arms can follow one watcher cycle. Only the arm that owns the watcher prints the wake reason; a follower attaches, then sees the cycle close with no successor and reported the typed failure for a close supervision had handled correctly:The fix
A declaration stays on its bounded
FM_PAUSE_RESURFACE_SECScadence once it has been surfaced inside the current window, whether or not its agent is still live, and its cadence markers survive both pane drift and a briefly busy pane. The withdrawal check at the top of the stale loop remains the one owner of clearing them.Preserved: an active run still outranks the declaration and takes the ordinary wedge timer; a crew with no declaration still trips stale immediately; a forgotten pause still re-surfaces once per window as an awaiting-external recheck; a live declaration firstmate has not seen yet still surfaces once on first sight.
A follower arm now consults the durable wake queue, the shared record of what a cycle surfaced. A record appended at or after the cycle began, together with a fresh liveness beacon, explains the close, so the follower prints those queued reason lines and exits zero. A stale or absent beacon never qualifies, so genuinely down supervision still fails loudly and the guard keeps its real protection.
After the fix, the same five-cycle reproduction surfaces once and absorbs the rest, and both concurrent arms return zero with the queued reason.
Tests
tests/fm-watch-triage.test.sh- a live declared pause surfaces once, absorbs across pane drift with its cadence markers and wedge-timer exemption intact, and returns to ordinary stale surfacing once withdrawn. Fails on the pre-fix watcher at drift cycle 2.tests/fm-watcher-lock.test.sh- a follower arm whose cycle queued a wake reports that wake instead of a typed failure, and the same setup with an ancient beacon still fails loudly. The first fails on the pre-fix arm with exit 1.tests/fm-turnend-guard.test.sh- the guard stays silent for a live beating watcher with wakes queued for the next turn, and still blocks when no beacon exists.Also fixes a pre-existing race in the watcher-lock suite: the TERM-resistant peer fixture is now waited on until it has installed its signal handler, since the restart's signal could otherwise land during interpreter startup, kill the peer, and fail the attach assertion for reasons unrelated to the code under test. That failure reproduces on
main.Validation
bin/fm-lint.shclean,bin/fm-doc-audience-check.shclean, andbin/fm-test-run.sh --family watcher-wake-lockgreen (10 suites, 0 failed), plus the auto-arm, instruction-owner, and documentation-audience suites.