fix(pubsub): stop running subscriber callbacks on the publishing thread - #250
Open
YuanYuYuan wants to merge 11 commits into
Open
fix(pubsub): stop running subscriber callbacks on the publishing thread#250YuanYuYuan wants to merge 11 commits into
YuanYuYuan wants to merge 11 commits into
Conversation
YuanYuYuan
force-pushed
the
pr/2-pubsub-reentrancy
branch
from
July 28, 2026 08:06
1f04eb3 to
1006b7d
Compare
There was a problem hiding this comment.
Pull request overview
Fixes subscriber callback deadlocks by moving hazardous delivery off publishing threads and updating Python bindings accordingly.
Changes:
- Adds QoS-aware subscriber selection and callback dispatch queues.
- Releases Python’s GIL during publishing and removes a receive-side payload copy.
- Adds Rust and Python regression and backpressure tests.
Reviewed changes
Copilot reviewed 9 out of 9 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
crates/hiroz/src/pubsub.rs |
Implements dispatching and QoS-aware subscribers. |
crates/hiroz/src/node.rs |
Applies dispatching to raw subscribers. |
crates/hiroz/src/ffi/subscriber.rs |
Stores generalized subscriber handles. |
crates/hiroz/src/common.rs |
Identifies handlers that execute user code. |
crates/hiroz-tests/tests/reentrant_publish.rs |
Tests reentrant publishing and teardown. |
crates/hiroz-tests/tests/dispatch_backpressure.rs |
Tests dispatcher queue bounds. |
crates/hiroz-py/tests/test_reentrant_publish.py |
Tests Python reentrancy and thread cleanup. |
crates/hiroz-py/src/pubsub.rs |
Releases the GIL while publishing. |
crates/hiroz-py/src/node.rs |
Uses borrowed sample payloads in callbacks. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 12 out of 12 changed files in this pull request and generated no new comments.
Comments suppressed due to low confidence (4)
crates/hiroz/src/pubsub.rs:1325
- This branch runs before
runs_user_codeis checked, so every TransientLocal queue/notifier subscriber (including rmw'sbuild_with_notifierpath) gets an unbounded dispatcher even though its handler only enqueues. That delays the wait-set notification and permits an unbounded backlog ahead of the already boundedBoundedQueue, contradicting the queue-mode contract below. Split the advanced path onruns_user_codeand wire queue handlers directly to the advanced subscriber callback.
let inner = if qos_needs_advanced(&self.entity.qos) {
crates/hiroz/src/pubsub.rs:1526
build_internalalready performs this encoding validation for everyDataHandlerat lines 1300–1320. Wrapping the new callback again parses each encoding twice and emits duplicate mismatch/unknown-format logs. Pass the callback directly tobuild_internal.
let expected_encoding = self.expected_encoding.clone();
let callback = Arc::new(move |sample: Sample| {
crates/hiroz-py/tests/test_reentrant_publish.py:132
- This makes the Python suite fail unconditionally on macOS, although macOS Python wheels are supported (
docs/bindings/python.md:13-14)./proc/self/taskis Linux-only and is needed solely by the drain-thread leak detector; skip that one test on unsupported platforms (a skip is not a false pass) while still running the portable re-entrancy tests.
# The leak test can only see the Rust drain thread through procfs.
assert os.path.isdir("/proc/self/task"), (
"/proc/self/task is unavailable - the drain-thread leak test cannot run "
"on this platform and must not be reported as passing"
)
crates/hiroz-py/src/pubsub.rs:38
- The watchdog test does not independently exercise this GIL release: with dispatcher delivery enabled, the seed
publish()returns quickly, and the test then sleeps for 300 ms before measuring progress, so reverting onlyallow_threadsstill passes. Add a detector that keeps the zenoh publish blocked while another Python thread must make progress; otherwise this regression can return unnoticed.
py.allow_threads(|| self.inner.publish(zbuf.into()))
.map_err(|e| e.into_pyerr())
YuanYuYuan
force-pushed
the
pr/2-pubsub-reentrancy
branch
3 times, most recently
from
July 29, 2026 04:12
cbe579d to
2760d4d
Compare
Publishing from inside a subscriber callback on the same session deadlocked deterministically, with no error and no traceback. Three separate mechanisms had to be removed. 1. Every subscriber was declared as a zenoh-ext `AdvancedSubscriber`, unconditionally. Its sample callback takes a `std::sync::Mutex` and then invokes the user callback under that guard. `std::sync::Mutex` is not reentrant, so a callback that published into its own topic graph re-entered a mutex its own thread already held further up the stack. Declare the advanced subscriber only when the QoS profile actually configures advanced features: for Volatile — the ROS 2 default — an `AdvancedSubscriber` declares no liveliness subscriber, no heartbeat subscriber and no detection token, so it was pure per-sample overhead plus the fatal lock. 2. Zenoh delivers a same-session sample synchronously, inline on the thread that called `put`. With the lock gone, a callback that publishes therefore *recursed* instead of iterating, until the stack overflowed. Adopt the shape zenoh's own `FifoChannel` uses, and that zenoh-python installs by default for a Python callable: delivery enqueues and returns, user code runs elsewhere. `ZPub`'s four publish paths hold a thread-local marker across the zenoh `put`; the plain subscriber's shim enqueues when that marker is set and invokes inline when it is not. An inter-process sample arrives on a zenoh RX worker, which is never inside a hiroz publish, so that path keeps its inline call and pays one thread-local read. Queue-mode subscribers — every rmw subscription, and every `recv()`-based user — already enqueue and return, so they get no dispatcher at all. The marker keys on the publishing thread rather than on zenoh's `Locality` on purpose: two sessions in one process with a direct route deliver a `Locality::Remote` sample inline on the publisher's thread, so an `allowed_origin(SessionLocal)` split would miss that case. With no path left on which a callback is reachable from inside `put`, re-entrancy is structurally impossible rather than depth-bounded, so the interim `MAX_CALLBACK_REENTRY_DEPTH` cap, `CallbackDepthGuard` and `InheritedCallbackDepth` are removed rather than left as unreachable defensive code. 3. `hiroz-py`'s `ZPublisher.publish`/`publish_raw` release the GIL across the zenoh publish. This does not fix the deadlock, but it downgrades a whole-interpreter freeze — no exception, no traceback, only an external kill — to a single blocked thread, which is the difference between an undiagnosable hang and a diagnosable one. Queue policy is split per path. The plain dispatcher takes its capacity from the same history-QoS expression `build()` uses to size `BoundedQueue` and drops the oldest on overflow; the advanced dispatcher stays unbounded. Bounded and blocking recreates the deadlock on both paths — the blocked producer sits inside the callback holding the very lock the drain thread needs. Bounded and dropping is correct for the plain path, which is Volatile with KEEP_LAST(depth) and already promises no more than `depth` undelivered samples, and wrong for the advanced path, where loss would discard samples miss-detection went out of its way to recover, mid-reorder. Also adds `ZSubBuilder::build_with_sample_callback`, which hands the callback the `Sample` rather than a decoded message, and uses it in `hiroz-py`. The Python callback path routed every sample through an identity codec whose `Output` carries no lifetime, so it had to `to_vec()` the whole payload before the callback ran, only for msgspec to decode out of it and drop it — one full payload copy per message. Measured as an interleaved paired A/B (two release wheels differing only in this call site, 5+6 reps, 16k timed round trips each, half-trip p50): 64 B 114.5 -> 114.1 (noise), 4 KB 120.9 -> 121.0 (noise), 64 KB 224.4 -> 221.1, -3.4 us with 10 of 11 reps in [-2.6, -5.1]. The size-dependence is the point: it is what distinguishes removing a payload-sized memcpy from removing a fixed per-message cost. Wire format is unchanged: an `AdvancedPublisher` with no cache, no publisher_detection and no sample_miss_detection puts on the plain key expression, so Volatile publishers and subscribers stay byte-identical and interop is unaffected. Volatile subscribers no longer get zenoh-ext's HLC-timestamp de-duplication. Ordering is preserved where it was ever guaranteed: one FIFO queue, one drain thread, and drop-oldest preserves the relative order of what survives. Not preserved, and documented at the type: a plain subscriber receiving both local and remote publications on one topic now runs them on two different threads, so their relative order is not guaranteed. Neither ROS 2 nor zenoh guarantees ordering across publishers, and a plain zenoh subscriber could already be invoked concurrently from several RX workers. Detector evidence, both directions. Reverting the dispatch decision (`local_publish_active() -> false`) and keeping the tests, all three self-feeding loop tests die with `fatal runtime error: stack overflow, aborting` (rc=101), including `intra_closed_loop_runs_iteratively`, which drives 2_000 round trips through a two-topic one-session callback cycle that could not previously be expressed as a loop. Against the unfixed sources the original three scenarios fail on their 20s deadline (`0 passed; 3 failed`). Every one of the six Python cells hangs with rc=124 under an external wall-clock timeout on a wheel built from unfixed sources — not merely fails, hangs — and all six pass on this branch.
Three defects, all in this branch's own new machinery. **The FFI publish path was never marked local.** `local_only_shim` hands a sample to the drain thread only while `LOCAL_PUBLISH_DEPTH` is set, and that flag comes from `LocalPublishGuard`. All four `ZPub` publish methods take the guard; `RawPublisher::publish_bytes` did not. A same-process raw publish was therefore delivered inline on the publishing thread, so a raw callback that publishes back into its own topic recurses until the stack is gone -- the exact defect the dispatcher exists to prevent, still reachable through the FFI door, which is the door `rmw-zenoh-rs` uses. **A bounded queue claimed to be lossless.** The backlog warning fired whenever the pending count crossed `DISPATCH_BACKLOG_WARN_AT`, and told the user the queue "is lossless, so the backlog costs memory". Both nearby comments assert that only unbounded dispatchers reach it, but nothing enforced that: capacity comes from `KeepLast(depth)`, and a `KeepLast(1024)` subscriber has exactly that capacity. It would warn that it cannot lose samples immediately before dropping one. Gate the warning on `DISPATCH_UNBOUNDED`; bounded queues already have an accurate drop warning. **A doc link pointed at a function that does not exist.** `LocalPublishGuard` claimed every publish "funnels through `ZPub::finish_put`". There is no such method and no choke point -- four call sites each enter the guard. The next author to add a fifth publish path would have gone looking for a funnel, not found one, and shipped without the guard, silently reinstating the bug above. `hiroz-tests` now enables `hiroz/ffi`. Without it the raw API is not compiled into the test crate at all, so the FFI surface had no coverage and any test written against it would have compiled away to nothing. The new test asserts thread identity rather than absence of a crash: proving the recursion directly needs an unbounded feedback loop, which aborts the runner and explains nothing. Verified in both directions -- with the guard the callback lands on the drain thread and it passes; with the guard removed the callback and the publisher report the same `ThreadId(2)` and it fails.
Enabling `hiroz/ffi` for `hiroz-tests` makes `clippy-tests` lint `crates/hiroz/src/ffi/*` for the first time, and that module has 22 pre-existing `missing_safety_doc` violations across `action.rs`, `serialize.rs` and `service.rs`. Under `-D warnings` the job fails. The guard fix in `RawPublisher::publish_bytes` stays -- it is the actual defect fix, and it was verified in both directions locally: with the guard removed, the raw subscriber callback and the publisher report the same `ThreadId`, meaning delivery happens inline on the publishing thread and a callback that republishes recurses until the stack is gone. What is lost is the CI regression test for that path, and the honest reason is scope: making it runnable requires documenting the safety contract of 22 `pub unsafe extern "C"` functions, which does not belong in a pull request about subscriber-callback re-entrancy. Writing 22 perfunctory `# Safety` blocks without establishing each contract would be worse than leaving them. Follow-up, worth filing: the FFI surface is entirely unlinted and untested because the feature is off everywhere. That is its own defect, and it is why this fix could ship unnoticed in the first place.
Two defects in the new dispatcher, both on teardown, both found by an
adversarial review pass.
**The Python bindings deadlocked the interpreter on teardown.** Dropping a
`ZSub` joins its delivery thread, and that thread's callback body is
`Python::with_gil`. `destroy_subscriber` is a `#[pymethods]` fn, so it runs
with the GIL held: it waits for the thread, the thread waits for the GIL, and
the interpreter freezes with no exception and no traceback. Reachable from
`destroy_subscriber`, `del node`, or interpreter exit -- `tp_dealloc` holds
the GIL too, and `PyZNode` had no `Drop`. This is the same failure class the
PR removes, relocated from `publish()` to teardown, and newly reachable
because nothing joined a GIL-needing thread before. `destroy_subscriber` now
takes a `Python` token and drops under `py.allow_threads`; `PyZNode` gets a
`Drop` that does the same for the subscribers it owns.
The existing `test_transient_local_dispatcher_threads_do_not_leak` passes
either way: it calls `_settle(...)` first, so the drain thread is parked in
`dequeue` and the hazard window is closed before the drop.
**`drop(subscriber)` could block for minutes, or forever.** `dequeue` popped
`pending` before honouring `closed`, so `Drop` ran a user callback for every
queued sample before returning. On the unbounded TransientLocal path that is
`backlog x callback_duration` with no ceiling -- a 1 kHz publisher against a
5 ms callback leaves ~30 000 samples queued after 30 s, blocking the drop for
~150 s with no log line and no way to cancel. It could block forever if a
callback waited on anything the dropping thread had to supply.
`closed` is now checked first. Dropping a subscriber means "stop delivering to
me", so the undelivered backlog is discarded rather than forced through a
callback the caller has already disposed of -- what destroying an rclcpp
subscription does. Teardown costs at most one in-flight callback.
That last point is a deliberate reversal of the previous documented intent
("drain what is queued, then exit"); it is now declared in Breaking Changes,
along with two breaks the description had omitted: Volatile subscriber
callbacks are no longer mutually excluded (they were, via zenoh-ext's
`Mutex<State>`, since every subscriber used to be an `AdvancedSubscriber`),
and `RawSubscriber::inner` changed type.
reentrant_publish 10/10 and dispatch_backpressure 2/2 still pass, including
both teardown scenarios.
The ROS interop step captured nextest's output with `complete` and never printed it, so a green job showed the command echo followed by "All ROS 2 <distro> tests passed!" and nothing in between. That banner could not be falsified: nextest exits 0 having run zero tests, and each interop test returns early -- still passing -- when check_ros2_available says no. Print the captured output and require a nextest summary reporting a non-zero count. Also correct two doc claims in pubsub.rs: the dispatcher and queue-mode capacities are not the same expression (they differ at a zero depth, harmlessly -- now pinned by two queue tests), and catch_unwind around a user callback is inert under the abort-on-panic opt profile.
YuanYuYuan
force-pushed
the
pr/2-pubsub-reentrancy
branch
from
July 29, 2026 09:13
90092b2 to
db3c6cc
Compare
Every other scenario in this file drives the synchronous publish, so the async path was unpinned. It is guarded differently and the difference is load-bearing: the guard is scoped to into_future() rather than held across the await, which is only sufficient because zenoh resolves a put eagerly there (IntoFuture = ready(self.wait()), zenoh 1.9.0). That is an upstream implementation detail, not a contract -- if the put ever became lazy it would move outside the guard and every deadlock this file prevents would return on the async path unnoticed. Asserts on thread identity rather than waiting for a hang, so it fails in a second with a legible message. Proven in both directions: dropping the guard from the async path fails it on the assertion.
Two changes here were not about subscriber re-entrancy and are moved to their own pull requests: - scripts/test-ros.nu, the non-vacuous interop gate. CI hygiene, found while gathering evidence for this fix. - ZSubBuilder::build_with_sample_callback and its hiroz-py call site, a payload-copy removal. It shared a call site with the re-entrancy fix, which is proximity, not a reason to review them together. Nothing else changes. The 13 tests in reentrant_publish and dispatch_backpressure still pass, and the GIL-release and teardown fixes in hiroz-py stay -- those are the same defect as the deadlock, reached from Python.
This was referenced Jul 29, 2026
Review found the advanced branch was taken on `qos_needs_advanced` alone, before `runs_user_code` was consulted, so every TransientLocal queue-mode subscriber -- /tf_static, /robot_description, every latched rmw subscription -- got a dispatcher thread it does not need. That is an extra OS thread per subscription, an extra thread hop and condvar wake per sample on the inter-process path, and an unbounded queue in front of the bounded one. A queue-mode handler runs no user code, so zenoh-ext's state lock is not a hazard for it. Note the fix is NOT to gate the whole branch on runs_user_code, as first suggested: TransientLocal needs the AdvancedSubscriber for history replay and miss recovery whether or not user code runs. Only the dispatcher is conditional, so `SubscriberHandle::Advanced::dispatcher` becomes an Option, mirroring the Plain variant. Also removes an invented history from two shipped test files: MAX_CALLBACK_REENTRY_DEPTH and its depth cap of 16 never existed on main, so "this caps out at 16" was asserting a behaviour that never shipped. On main the same loop deadlocks -- which is the defect being fixed.
always_shim returns an opaque impl Fn, so it cannot share a match arm with a plain closure -- E0308 on the previous commit. Box both to Box<dyn Fn(Sample) + Send + Sync>.
The second SubscriberHandle::Advanced construction site is in the raw FFI subscriber path, behind #[cfg(feature = "ffi")]. ci.yml never builds with that feature, so it compiled clean there and only test.yml's "Build Rust FFI library" step caught it -- which is exactly the gap issue #270 describes.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Fixes #249
Publishing from inside a subscriber callback on the same session deadlocked deterministically, with no error and no traceback; through the Python bindings it froze the whole interpreter. This removes the three stacked mechanisms behind it.
Stated up front: session-local delivery becomes asynchronous.
publish()returning no longer means a same-session subscriber has been called. Nothing in this repo depended on that —rmw-zenoh-rsusesbuild_with_notifierthroughout, andwait_for_subscriptiongoes through liveliness, not delivery — but it is a real change for downstream code that publishes and then asserts on a callback's side effect on the next line.Key Changes
Declare zenoh-ext's
AdvancedSubscriberonly when QoS actually needs it. Its sample callback invokes user code under a non-reentrantstd::sync::Mutex. For Volatile — the ROS 2 default — it declares no liveliness subscriber, heartbeat or detection token, so it was pure overhead plus the fatal lock. TransientLocal keeps it, since history replay and miss recovery live there.Deliver session-local samples off the publishing thread. Zenoh dispatches a same-session sample inline on the thread that called
put, so with the lock gone a publishing callback recursed instead of iterating.ZPub's publish paths set a thread-local marker across theput; the plain subscriber's shim enqueues when it is set and calls inline when it is not. Inter-process samples arrive on an RX worker, never inside a hiroz publish, so they keep the inline path at the cost of one thread-local read. Queue-mode subscribers already enqueue and get no dispatcher.The marker keys on the publishing thread, not on zenoh's
Locality: two sessions in one process with a direct route deliver aLocality::Remote-tagged sample inline on the publisher's thread, which anallowed_originsplit would miss.Remove
MAX_CALLBACK_REENTRY_DEPTH. Re-entrancy is now structurally impossible rather than depth-bounded, so the interim cap is deleted rather than left as unreachable code that silently drops samples.Bound the plain dispatcher queue at the history depth; leave the advanced one unbounded. Blocking would recreate the deadlock (the blocked producer holds the lock the drain thread needs). Drop-oldest is right for the plain path, which is Volatile
KEEP_LAST(depth)and already promises no more — retaining everything honoured a QoS stricter than declared and let a tight local loop grow the process without limit. It is wrong for the advanced path, where loss would discard samples miss-detection recovered.hiroz-pyreleases the GIL across the zenoh publish. Does not fix the deadlock; downgrades a whole-interpreter freeze to a single blocked thread.What fails without this
Both detectors were re-run on this branch, in both directions. Baseline:
reentrant_publishis 11 passed.Revert the production change, keep the tests. Four tests fail on their deadline, then a TransientLocal cycle test wedges the binary permanently — its deadline cannot rescue it, because the blocked thread is the one the harness needs to report. The run never terminates and must be killed externally. That is why the fix is not "raise the timeout".
Revert only the dispatch decision (
local_publish_active() -> false):self_feeding_callback_loop_iterates_without_a_depth_capandintra_closed_loop_runs_iterativelyboth die withfatal runtime error: stack overflow. The rest still pass, so only those two are detectors for this half.Both revert experiments were run when the suite had 10 tests, before
async_publish_delivers_off_the_publishing_threadwas added in 9077de8. That test guards the async publish path; it is not a deadlock detector.Python, two release wheels differing only in the production change. The fixed wheel is 57 passed, 2 skipped. On the reverted wheel, each re-entrancy cell run in its own process under an external 60 s timeout:
0 bytes writtenis the point, and why an external timeout was needed: pytest never writes a byte, not even a collection header, because every thread is frozen. No in-process mechanism could have reported this. For the same reason the thread-leak test reads/proc/self/taskrather thanthreading.enumerate()— the drain thread is a Rust thread CPython never sees, soenumerate()would report clean no matter how many leaked. Measured 0 → 4 → 0 across three cycles.Trade stated plainly: an unconditional feedback loop used to stop at depth 16 with a loud error and dropped samples; it now runs forever at constant stack and queue depth, consuming a core. The new failure is the one every other ROS 2 stack has — observable and stoppable — and it no longer discards data on legitimate deep fan-out.
Breaking Changes
Six, all consequences of moving session-local delivery off the publishing thread.
Session-local delivery is asynchronous. Code that publishes then asserts on a callback's side effect must synchronise.
Same-session samples can be dropped. The queue is bounded at the history depth and drops oldest, recorded as a periodic
warn!. This matchesKEEP_LAST(depth), and blocking instead would reinstate the deadlock — but when delivery was inline, loss was structurally impossible.Subscriber callbacks are no longer mutually excluded. zenoh-ext invokes the callback under its
Mutex<State>(handle_sample(&mut State, ..)), so callbacks were serialised by construction. Volatile subscribers are now plain subscribers: a local sample on the drain thread can run concurrently with a remote one on an RX worker. A callback doing a non-atomic read-modify-write was race-free before and is not now. ROS 2 makes no single-threaded-callback promise either, so this is defensible — but it was true before.Local and remote publications on one topic are no longer ordered relative to each other for a plain subscriber.
Dropping a subscriber discards its undelivered backlog. Previously
Dropran the callback once per queued sample — unbounded on TransientLocal (a 1 kHz publisher against a 5 ms callback blocks the drop for minutes) and able to block forever if a callback awaited something the dropping thread supplies. Teardown now costs at most one in-flight callback, as destroying an rclcpp subscription does.RawSubscriber::innerchanged type,AdvancedSubscriber<()>→SubscriberHandle— apubfield inpub mod ffi.Not breaking: the wire format. An
AdvancedPublisherwith no cache, publisher_detection or sample_miss_detection puts on the plain key expression, so Volatile endpoints stay byte-identical andrmw_zenoh_cppinterop is unaffected.Volatile subscribers do lose one zenoh-ext behaviour, and the loss is wider than de-duplication.
apply_transient_local_pubis a no-op for Volatile, so those samples carry noSourceInfo;handle_sampletherefore takes its timestamped branch, not its sequenced one. That branch delivers a sample only when its timestamp is strictly newer than the last delivered (state.last_delivered.map(|t| t < *timestamp)), so it suppressed duplicates and out-of-order/older-timestamp samples. Both go away. This applies only wheretimestampingis enabled in the Zenoh config — with it off the samples carried noTimestamp, the branch never applied, and nothing changes.Notes
mainafter fix(codegen): stable ordering for generated python stubs #268, so this branch also runs thecheck-rustdoc-linksandGenerated Python Stubsgates.crates/hiroz/src/node.rs's raw FFI subscriber is behind#[cfg(feature = "ffi")], whichci.ymlnever compiles. The type change in this PR broke it while all 27ci.ymlchecks stayed green; onlytest.yml's "Build Rust FFI library" step caught it. That gap is FFI module is never compiled, linted or tested; the "Go FFI tests" CI step silently skips #270.