Skip to content

Fix flaky RemoteForegroundResolver tests that failed on loaded CI runners - #181

Merged
thdxg merged 1 commit into
mainfrom
claude/main-branch-test-failure-01fe8d
Jul 21, 2026
Merged

Fix flaky RemoteForegroundResolver tests that failed on loaded CI runners#181
thdxg merged 1 commit into
mainfrom
claude/main-branch-test-failure-01fe8d

Conversation

@thdxg

@thdxg thdxg commented Jul 21, 2026

Copy link
Copy Markdown
Owner

What

Fix RemoteForegroundResolverTests flaking on loaded CI runners. The async tests now wait for the resolver's fire-and-forget probe Task by polling until the expected state holds, instead of a fixed Task.yield() count.

Why

The Checks run on main (commit e2b9a0a, PR #180) failed intermittently:

✘ RemoteForegroundResolverTests / applies_probe_names_to_matching_panes()
  Expectation failed: (pane.foregroundProcessName → nil) == "btop"

refresh() spawns a fire-and-forget Task { let map = await probe(...); finish(...) }. The test's flush() waited for it with for _ in 0..<4 { await Task.yield() }. Four yields is enough on an idle machine but not on a loaded CI runner, where the child Task's await probe(...) may not have resumed yet — so the probed name wasn't applied and the assertion saw nil. It passes locally every time, which is exactly why it slipped through.

Confirmed the cause deterministically before fixing: dropping the yield count to 0 reproduces the exact CI assertion locally (three of the four async tests fail identically).

How

Replaced the fixed-count flush() with waitUntil { condition } that polls until the assertion's own condition holds, bounded by a ~2s ceiling so a genuine regression still fails fast. The same fragile pattern lived in all four async tests in the file, so this hardens every one, not just the one that tripped.

Two non-obvious points, both worked through against a deterministic repro (injecting a 50ms Task.sleep into the probe to simulate CI scheduling latency):

  • The poll sleeps, it doesn't yield. A tight Task.yield() loop on @MainActor never lets the clock advance, so it starves a timer-based continuation — a yield-based waitUntil gave up after ~17ms without the 50ms sleep ever firing. Sleeping 1ms between checks hands the actor back long enough for pending work to run; the same test then correctly waits it out (passes at 0.054s).
  • The negative-assertion test (failed_probe_keeps_last_known_names — name must not change) has no state edge to poll. Added var isIdle: Bool { inflight.isEmpty } to RemoteForegroundResolver — an honest read of existing state, visible via @testable import — and poll on that. It works because refresh() inserts into inflight synchronously before spawning the Task, so isIdle is false right after refresh() and flips true only after finish() runs.

No production behavior changes: the only non-test change is the read-only isIdle accessor.

Verified

  • mise run format, mise run lint, and mise run test all pass
  • Added or updated tests for new model / persistence / palette / hotkey logic — reworked the affected tests; validated the fix against a deterministic reproduction of the race (0-yield repro fails as the CI did; 50ms-slow-probe repro passes only with the sleep-based poll)

Notes for reviewers

Pre-existing warnings in the same CI run are not touched here (redundant #require in RemoteSpawnTests, a nonisolated(unsafe) on AppState.observerTokens) — they don't fail the build. Happy to fold in a cleanup if you'd prefer.

Summary by CodeRabbit

  • Tests
    • Improved asynchronous test synchronization by waiting for specific state changes instead of relying on scheduling assumptions.
    • Added coverage to confirm resolver idle state after probe failures.
    • Strengthened validation for probe completion, distinct destinations, and foreground process updates.

…ners

The tests waited for the resolver's fire-and-forget probe Task with a
fixed `for _ in 0..<4 { await Task.yield() }`. Four yields is enough on
an idle machine but not on a loaded CI runner, where the child Task's
`await probe(...)` may not have resumed yet — so the probed name wasn't
applied and the assertion saw nil (the #180 flake on main).

Replace the fixed-count flush with a `waitUntil { condition }` poll that
loops until the assertion's own condition holds, bounded by a ~2s ceiling
so a genuine regression still fails fast. The poll *sleeps* between
checks rather than yielding: a tight Task.yield() loop on @mainactor
never lets the clock advance, so it would starve a timer-based
continuation. The one negative-assertion test (name must not change) has
no state edge to poll, so it waits on a new `isIdle` accessor that
reflects whether a probe Task is still outstanding.
@github-actions github-actions Bot added the area:tests Test changes label Jul 21, 2026
@coderabbitai

coderabbitai Bot commented Jul 21, 2026

Copy link
Copy Markdown

Review Change Stack

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: c8772990-2c2a-4da2-aaee-9acaf285b5b2

📥 Commits

Reviewing files that changed from the base of the PR and between e2b9a0a and 9d3d736.

📒 Files selected for processing (2)
  • Macterm/System/RemoteForegroundResolver.swift
  • MactermTests/System/RemoteForegroundResolverTests.swift

📝 Walkthrough

Walkthrough

RemoteForegroundResolver now exposes probe idleness, and its async tests use bounded polling to await probe calls, process-name updates, and failed-probe completion instead of fixed task yields.

Changes

Remote foreground synchronization

Layer / File(s) Summary
Expose resolver idle state
Macterm/System/RemoteForegroundResolver.swift
Adds read-only isIdle, derived from whether the resolver’s in-flight probe set is empty.
Replace fixed async flushing
MactermTests/System/RemoteForegroundResolverTests.swift
Adds waitUntil polling and updates probe, host, foreground-name, and failure-path tests to await specific completion conditions.

Estimated code review effort: 2 (Simple) | ~10 minutes

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Title check ✅ Passed The title clearly summarizes the main change: fixing flaky RemoteForegroundResolver tests on loaded CI runners.
Description check ✅ Passed The description matches the template with What, Why, How, Verified, and Notes sections filled in.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
✨ Finishing Touches
📝 Generate docstrings
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch claude/main-branch-test-failure-01fe8d

Comment @coderabbitai help to get the list of available commands.

@github-actions github-actions Bot added the benchmark:regression CI benchmark: significant resource regression vs main label Jul 21, 2026
@github-actions

Copy link
Copy Markdown
Contributor

Window-state benchmark

State Metric main@e2b9a0afc this branch Δ
focused CPU % 1.00 1.20 +20%
Memory (RSS MB) 108.8 109.4 +1%
CPU ms/s (powermetrics) 9.1 11.7 +29%
Wakeups/s (powermetrics) 151.7 114.5 -25%
unfocused CPU % 1.00 1.20 +20%
Memory (RSS MB) 112.9 113.9 +1%
CPU ms/s (powermetrics) 10.1 11.8 +17%
Wakeups/s (powermetrics) 140.8 173.6 +23%
minimized CPU % 0.20 0.20 +0%
Memory (RSS MB) 113.1 114.1 +1%
CPU ms/s (powermetrics) 1.3 1.9 +41%
Wakeups/s (powermetrics) 66.9 69.5 +4%
workload-focused CPU % 2.00 2.70 +35% 🔺
Memory (RSS MB) 153.1 159.8 +4%
CPU ms/s (powermetrics) 18.5 26.2 +42% 🔺
Wakeups/s (powermetrics) 192.1 263.4 +37% 🔺
workload-unfocused CPU % 1.70 2.70 +59% 🔺
Memory (RSS MB) 127.6 160.0 +25% 🔺
CPU ms/s (powermetrics) 16.8 27.3 +63% 🔺
Wakeups/s (powermetrics) 185.0 267.1 +44% 🔺
workload-minimized CPU % 0.40 0.40 +0%
Memory (RSS MB) 127.7 160.1 +25% 🔺
CPU ms/s (powermetrics) 3.2 3.4 +6%
Wakeups/s (powermetrics) 112.8 153.2 +36%

⚠️ Labeled benchmark:regression

This PR is labeled benchmark:regression because these metrics regressed by ≥25% vs main@e2b9a0afc (beyond each metric's absolute noise floor):

  • workload-focused — CPU %: 2.00 → 2.70 (+35%)
  • workload-focused — CPU ms/s (powermetrics): 18.5 → 26.2 (+42%)
  • workload-focused — Wakeups/s (powermetrics): 192.1 → 263.4 (+37%)
  • workload-unfocused — CPU %: 1.70 → 2.70 (+59%)
  • workload-unfocused — Memory (RSS MB): 127.6 → 160.0 (+25%)
  • workload-unfocused — CPU ms/s (powermetrics): 16.8 → 27.3 (+63%)
  • workload-unfocused — Wakeups/s (powermetrics): 185.0 → 267.1 (+44%)
  • workload-minimized — Memory (RSS MB): 127.7 → 160.1 (+25%)

Reported value is the median of 3×10s windows per state (splitting the window and taking the median keeps one co-scheduled spike from skewing a state); CPU % is the process CPU-time delta over a window. Runs land on different shared runners, so treat small deltas as noise — 🔺/🔻 marks changes ≥25% that also clear the metric's absolute noise floor (CPU % ≥0.5, Memory (RSS MB) ≥25, CPU ms/s ≥5, Wakeups/s ≥50); CPU deltas off a noise-dominated baseline aren't flagged (CPU % baseline ≥1.5, CPU ms/s baseline ≥15). Flagged changes add the benchmark:regression / benchmark:improvement label.

@thdxg
thdxg merged commit 36930ec into main Jul 21, 2026
9 checks passed
@thdxg
thdxg deleted the claude/main-branch-test-failure-01fe8d branch July 21, 2026 11:16
onliner10 added a commit to onliner10/macterm that referenced this pull request Jul 27, 2026
The scheduled wake fired at 3s and the settle it triggers requires
now - lastActivityAt >= 3s, so it only worked while timer jitter ran
positive. An early wake would no-op the settle, and with every window
occluded there is no timer left to retry — the run would sit at .running
forever, which is exactly what this machinery exists to prevent. Derive both
from one constant with an explicit margin so they can't drift apart or meet.

preserveProgrammaticCommandInput carried a payload's content evidence for
2s, and the evidence only ORs in — so a `pane run "cmd"` followed by a
genuinely blank Return inside that window reported content it didn't have.
The carry is only needed where a bracketed paste can swallow the newline,
which is the same agent-TUI foreground the in-place heuristic requires, so
gate it on that and keep the ambiguous window out of the shell case.

Also: reschedule the wake with a work item rather than cancelling and
respawning a Task at ~2 Hz per live pane, matching the idiom the view
already uses; state the interaction-before-submission ordering contract at
both ends, since recordUserInteraction clears the arming that
recordCommandSubmission sets; and wait for the wake by polling instead of a
fixed 100ms sleep against a 20ms timer (the shape that flaked in thdxg#181).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:tests Test changes benchmark:regression CI benchmark: significant resource regression vs main

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant