Fix loadscope-family schedulers hanging the run after a worker crash

[davidheff]

Fixes #784.

The hang

When a worker crashes under --dist=loadscope, loadfile or loadgroup, the run can hang indefinitely with every worker idle, the controller waiting on its queue, and a handful of tests never run. We hit this repeatedly in a ~14,000-test suite under --dist=loadgroup: the run stopped at 99% with all sixteen workers parked in torun.get(), the controller parked in its own queue.get(), and nineteen collected tests stranded.

Two faults in LoadScopeScheduling combine to cause it:

1. remove_node re-queues completed work units. A node's workload keeps every work unit it has ever been given — mark_test_complete only flips a flag, and nothing prunes finished units. When a worker crashes, remove_node puts the entire workload back with self.workqueue.update(workload), so the queue gains one already-finished work unit for every scope the dead worker got through. Assigning one of those to a node sends it an empty runtests command (_assign_work_unit filters completed tests out of the indices), so the node reports nothing and waits forever. Since a test report is the only thing that drives scheduling onwards, the run wedges — at the end of a run, because re-queued units join the back of the queue.

2. A node holding a single test cannot make progress. A worker holds its final pending test in reserve, not starting it until further work or a shutdown notice arrives (the same behaviour that Ensure loadscope and loadfile work when the first scope has one work unit #277 addressed for initial distribution). _reschedule assigns exactly one work unit, so a previously idle node (e.g. the replacement worker) topped up with a single-test unit never starts it, never reports, and is never scheduled again.

The fix

• remove_node returns only work units that still hold pending tests to the queue.
• _reschedule keeps assigning until the node holds at least two pending tests, and shuts the node down when the queue runs dry — which is what releases its reserved test.
• The crashed test itself is no longer re-queued: it is already reported as failed by handle_crashitem, and re-running it just crashed the replacement worker in turn. Marking it complete matches LoadScheduling, which pops the crashed item.

All three changes live in LoadScopeScheduling, so loadscope, loadfile and loadgroup are all covered.

Tests

• Two scheduler-level unit tests in test_dsession.py (the first such tests for LoadScopeScheduling): one pins the re-queue behaviour after a crash (completed units dropped, crashed test marked complete, pending work preserved), the other pins the top-up guarantee for a node that cannot otherwise report.
• An acceptance test in TestNodeFailure crashes a real worker after it has completed a whole scope, under -n1 --dist=loadfile. Without the fix it hangs indefinitely (guarded by a 120 s subprocess timeout); with it, the replacement worker runs the remaining tests and the run completes with 1 failed, 3 passed.

All three new tests were verified to fail against the unfixed scheduler, and the full testing/ suite passes with the fix (one pre-existing non-strict xfail, TestNodeFailure::test_each_multiple "#20: xdist race condition on node restart", xpasses on my machine).

We have been running the _reschedule logic here as a monkeypatch in production for a while: a crash injected into the real 14,000-test suite now recovers cleanly where it previously wedged the run.

🤖 Generated with Claude Code