fix(server): preserve PTY exit status from the zombie reaper#77
Merged
Conversation
The periodic reap_zombies() backstop called waitpid(-1) and discarded the status. When it reaped a PTY's child before that PTY's collect_exit_status() ran, collect_exit_status() found nothing to wait on and returned EXIT_STATUS_UNKNOWN, which the client maps to exit code 1 — a bogus failure even for commands that succeeded (e.g. a completed git clone reported as 'clone failed'). Record each reaped child's status in a small shared table that collect_exit_status() drains, and hold the table lock across its own waitpid so the two reapers can't race. Adds a fork-based repro test.
Coverage
|
pcarrier
approved these changes
Jul 13, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The bug
reap_zombies()runs every 5s (crates/server/src/lib.rs:2464) and calledwaitpid(-1, …, WNOHANG), discarding each child's status. When it reaped a PTY's child before that PTY'scollect_exit_status()ran,collect_exit_status()had nothing left to wait on and returnedEXIT_STATUS_UNKNOWN. The client mapsEXIT_STATUS_UNKNOWN→ exit code 1 — a fabricated failure even for commands that succeeded.This surfaced as
use_integration"git clone failed but produced no output": in the read replica, of the exit-1 clones, 24 have complete, successful git output (Resolving deltas: 100% … done., a fast-forwardinggit pull, evenAlready up to date.) yet reported exit 1. git returns0on those — so the1was never git's. It's regular, not flaky: it's a race against a 5s timer, and it widens when the box is busy (cleanup is delayed under load).The fix
reap_zombies()now records each reaped child's status in a small shared table (reaped_statuses), andcollect_exit_status()reads that table first.collect_exit_status()holds the table lock across its ownwaitpidso the two reapers can't interleave — whichever reaps the child, the real status survives. The table is drained bycollect_exit_status(), so in the normal EOF path (where a PTY reaps its own child) it stays near-empty.Unix-only; Windows
reap_zombiesis already a no-op.Repro test
collect_exit_status_survives_backstop_reapforks a child that exits42, letsreap_zombies()reap it first (the production race), then assertscollect_exit_statusreturns42.left: -2147483648(EXIT_STATUS_UNKNOWN) vs42.blit-serversuite green (156 tests), fmt + clippy clean.Tag
@indentto continue the conversation here.