Fix macOS fork-safety crash: default to spawn on darwin#443
Conversation
On macOS, fork() in a multi-threaded process is unsafe — forked children
inherit the parent's memory but not its threads, leaving shared resources
(locks, CoreFoundation state, ObjC runtime) in an inconsistent state.
macOS Tahoe (26.0) enforces this strictly, causing SIGSEGV crashes in
Celery workers.
CPython's multiprocessing already changed the default start method on
macOS from fork to spawn in Python 3.8 (bpo-33725). This commit brings
billiard in line with that upstream fix by:
- Defaulting to 'spawn' on macOS (sys.platform == 'darwin')
- Updating get_all_start_methods() to list 'spawn' first on macOS
- Adding unit tests for default context selection, forking_is_enabled(),
set_start_method override, and get_all_start_methods() ordering
Non-macOS platforms remain unchanged (default is still 'fork').
Users can still explicitly set_start_method('fork') if needed.
The importlib.reload approach corrupted module state because reloading billiard.context creates new class objects while other modules retain references to old ones. This caused super() to fail in DefaultContext with 'obj is not an instance or subtype of type'. Using subprocess to test import-time conditionals is fully isolated: each test spawns a fresh Python process that patches sys.platform before importing billiard.context for the first time.
There was a problem hiding this comment.
Pull request overview
This PR fixes a macOS fork-safety crash by changing the default multiprocessing start method from 'fork' to 'spawn' on darwin platforms. This aligns billiard with CPython's behavior change introduced in Python 3.8 (bpo-33725), which addressed reliability issues with fork() on macOS 10.14 (Mojave) and later versions where arbitrary code execution after fork() is no longer reliable.
Changes:
- Modified default context initialization to use 'spawn' on darwin instead of 'fork'
- Added comprehensive test coverage for platform-specific default start methods
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated no comments.
| File | Description |
|---|---|
| billiard/context.py | Added conditional logic to set default context to spawn on darwin, fork on other Unix systems |
| t/unit/test_context.py | New test file with comprehensive coverage for darwin spawn defaults, linux fork defaults, override capability, and forking_is_enabled checks |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
|
@auvipy I’m thinking about releasing a beta or RC with this fix so ppl can easily test it, WDYT? |
|
I am using the version before tahoe for stability reason... you can go with an rc for this |
|
Potential fix for celery/celery#9894