test: mark PTY-dependent hooks tests as xfail on macOS (#821)

[mangelajo]

Problem

The alleged macOS PTY kernel buffer timing race condition continues to cause random test failures in CI despite multiple fix attempts:

• test_exec_bash hook test fails on macOS due to PTY output not captured by mocked logger #560 — Added synchronous drain after async reader cancellation
• Flaky test_exec_bash in hooks_test.py on macOS due to PTY output race condition #733 — Added select()-based drain with timeout
• fix: retry PTY drain on empty select() to prevent data loss on macOS #826 — Increased DRAIN_MAX_EMPTY_POLLS from 1 to 10

Latest failing CI run: https://github.com/jumpstarter-dev/jumpstarter/actions/runs/28084374100/job/83146665529

Each fix reduced the frequency but did not eliminate the race. The CI runs parallel test suites with concurrent package installations, creating enough I/O contention for the macOS PTY buffer to miss the drain window. We haven't managed to reproduce it locally, but it happens on CI.

Workaround

Mark all 19 tests that spawn real subprocesses via PTY and assert on captured logger output as xfail(strict=False) on macOS. This unblocks CI while the root cause is investigated.

Using strict=False means:

• If the test passes on macOS → reported as xpass (no failure)
• If the test fails on macOS → reported as xfail (no failure)
• On Linux → marker has no effect, tests run normally

Tests marked with @macos_pty_xfail

Output capture tests:

• test_hook_environment_variables
• test_real_time_output_logging
• test_post_lease_hook_execution_on_completion
• test_exec_bash
• test_exec_python3
• test_script_file_sh
• test_script_file_py_autodetects_python
• test_script_file_py_exec_override
• test_noninteractive_environment
• test_drain_captures_output_without_trailing_newline

Drain behavior tests (patched PTY):

• test_drain_reads_data_remaining_in_pty_buffer
• test_drain_select_oserror_exits_gracefully
• test_drain_select_valueerror_exits_gracefully
• test_drain_exits_when_deadline_exceeded_before_select
• test_drain_exception_is_suppressed
• test_drain_retries_empty_select_then_captures_data
• test_drain_terminates_after_max_empty_polls
• test_drain_empty_counter_resets_on_data

Regression tests:

• test_infrastructure_messages_at_debug_not_info

Related

• Re-opened Flaky test_exec_bash in hooks_test.py on macOS still reproducing after previous fixes #821 with updated description and moved to 0.10.0 milestone
• Added comments to Flaky test_exec_bash in hooks_test.py on macOS due to PTY output race condition #733 and test_exec_bash hook test fails on macOS due to PTY output not captured by mocked logger #560 noting the issue persists

Handles #821 (targeted for proper fix in 0.10.0)

[coderabbitai]

📝 Walkthrough

Walkthrough

Imports sys and defines a new macos_pty_xfail pytest marker (strict=False, conditioned on sys.platform == "darwin") in hooks_test.py. Applies this marker to all PTY/subprocess-dependent HookExecutor test methods and the TestHookExecutorPRRegressions class to mark them as expected failures on macOS due to a PTY output timing race condition.

Changes

macOS PTY xfail marker for HookExecutor tests

Layer / File(s)	Summary
sys import and macos_pty_xfail marker definition python/packages/jumpstarter/jumpstarter/exporter/hooks_test.py	Adds import sys and defines the macos_pty_xfail module-level variable as a pytest.mark.xfail conditioned on sys.platform == "darwin" with strict=False and a PTY timing-race reason string.
Apply @macos_pty_xfail to PTY-dependent tests python/packages/jumpstarter/jumpstarter/exporter/hooks_test.py	Decorates HookExecutor test methods covering environment variable propagation, real-time logging, after-lease hook logging, timeout/warn behavior, exec/bin detection, script/file interpreter autodetection, PTY drain/selection edge cases, and the TestHookExecutorPRRegressions class with @macos_pty_xfail.

Estimated code review effort

🎯 1 (Trivial) | ⏱️ ~3 minutes

Possibly related PRs

• jumpstarter-dev/jumpstarter#561: Modifies PTY output draining and line flushing in hooks.py, directly related to the same PTY timing race these xfail markers address.
• jumpstarter-dev/jumpstarter#826: Increased DRAIN_MAX_EMPTY_POLLS in hooks.py and added drain regression coverage in hooks_test.py to address the same PTY drain timing race on macOS.

Suggested reviewers

• kirkbrauer

Poem

🐇 On macOS the PTY plays tricks so mean,
A race condition haunts the buffer unseen.
So I stamp each test with an xfail mark,
No more red CI runs in the dark!
Strict=False means hope still flickers bright,
One day we'll fix the drain, and all will be right. 🍎

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title accurately states the main change: marking PTY-dependent hooks tests as xfail on macOS.
Linked Issues check	✅ Passed	The PR marks the affected hooks tests as xfail on macOS, matching the issue's request to stop flaky CI failures.
Out of Scope Changes check	✅ Passed	The only change is the macOS xfail marker for the affected tests, with no unrelated additions.
Docstring Coverage	✅ Passed	Docstring coverage is 90.91% which is sufficient. The required threshold is 80.00%.
Description check	✅ Passed	The description matches the change: it explains marking macOS PTY-dependent hooks tests as xfail to work around CI flakiness.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch fix/xfail-macos-hooks-tests-821

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[mangelajo]

Let's see if #837 will fix it before merging this one

[mangelajo]

Let's see if #837 will fix it before merging this one

#837 made MacOS better, but it's creating some issues in E2E/linux which we don't want. As we talked about on the matrix channel xfail is acceptable for the hooks on macos until we investigate during 0.10.0