fix(lifecycle): run transition callbacks outside the state-machine lock - #253
fix(lifecycle): run transition callbacks outside the state-machine lock#253YuanYuYuan wants to merge 3 commits into
Conversation
ed77e5b to
de0039e
Compare
There was a problem hiding this comment.
Pull request overview
Fixes lifecycle callback deadlocks by releasing the state-machine lock during callbacks and exposing accurate intermediate states.
Changes:
- Splits transitions into locked
begin/finishphases. - Corrects lifecycle client transition-state mappings.
- Adds re-entrancy regression tests and lock-free managed-entity notifications.
Reviewed changes
Copilot reviewed 4 out of 4 changed files in this pull request and generated 2 comments.
| File | Description |
|---|---|
state_machine.rs |
Adds split transition APIs. |
node.rs |
Runs callbacks outside mutex guards. |
client.rs |
Maps intermediate state IDs correctly. |
reentrant_lifecycle.rs |
Tests callback state queries. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 4 out of 4 changed files in this pull request and generated no new comments.
Comments suppressed due to low confidence (3)
crates/hiroz-tests/tests/reentrant_lifecycle.rs:16
- This says observers retain the old behavior, but previously the mutex prevented observers from reading any state until the callback completed, and the client also mapped busy-state IDs to
Unconfigured. The PR intentionally changes observers to see the intermediate state, so describe that new behavior rather than claiming it was already observable.
//! the guard while the callback runs. Observers keep seeing exactly what they
//! saw before: the intermediate state (`configuring`, `activating`, …).
crates/hiroz/src/lifecycle/node.rs:212
- If
on_errorpanics after an error from an Active-state transition, this branch moves the machine toFinalizedand immediately unwinds past the managed-entity synchronization below. The node then reportsFinalizedwhile its lifecycle publishers remain activated and can still publish. Deactivate the managed entities before resuming the panic (ideally through the same helper used by normal completion) so the externally visible state and entity gating remain consistent.
self.state_machine
.lock()
.unwrap()
.finish_error_processing(CallbackReturn::Error);
resume_unwind(payload);
crates/hiroz/src/lifecycle/state_machine.rs:216
finishis a new public safe API that can bypass every transition invariant: for example, callingStateMachine::new().finish(Activate, Inactive, Success)moves directly fromUnconfiguredtoActive, because neither the matchingbeginnor the expected intermediate state is validated. The documented caller obligation does not prevent accidental state corruption. Keep the split methods crate-private if they are only forZLifecycleNode, or makebeginreturn an opaque token thatfinishconsumes and validates; apply the same restriction tofinish_error_processing.
/// Second half of [`Self::trigger`]: apply the user callback's verdict.
///
/// `start_state` must be the value returned by the matching [`Self::begin`].
pub fn finish(
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 4 out of 4 changed files in this pull request and generated no new comments.
Comments suppressed due to low confidence (1)
crates/hiroz/src/lifecycle/node.rs:205
- The panic-recovery branch for
on_erroris not covered. If thiscatch_unwindor its settling call regresses, the node remains silently stuck inErrorProcessing; the new test only checks anon_errorcallback that returns normally, while the normal transition callback has a dedicated panic test. Please add the equivalent panicking-on_errortest and assert both panic propagation and the documented settled state (Finalized).
let ret = match catch_unwind(cb) {
`ZLifecycleNode::trigger_transition` invoked the user's transition callback
from inside `state_machine.lock().unwrap().trigger(..)`. That mutex is shared
with the node's own `~/get_state`, `~/change_state` and
`~/get_available_transitions` handlers and with `get_current_state()`, so a
node could not answer any question about its own state while a transition
callback ran — and a callback that asks waits forever.
Split `StateMachine::trigger` into `begin` (validate, enter the intermediate
"busy" state) and `finish` (apply the callback's verdict), with `trigger` kept
as a wrapper for callers that own the machine outright. The error path gets the
same `finish_error_processing` split. The callback now runs between the two
locked steps, with no guard held, and observers see the genuine intermediate
state while it runs.
The `begin` call is bound to its own `let` on purpose: writing it as a `match`
scrutinee keeps the temporary guard alive for the whole `match` body and
silently reinstates the deadlock. That mistake was made and caught by these
tests during development.
Also fixes `state_from_lc`, which mapped every transition state onto
`Unconfigured`. A node genuinely reports `configuring` / `activating` while a
callback runs; telling a lifecycle manager it had reset itself instead is wrong
and was indistinguishable from the real thing. Without this the intermediate
state is unobservable over the wire, so the deadlock test could not check it.
Managed entities get the same collect-then-invoke treatment:
`trigger_transition` called `e.on_activate()` / `e.on_deactivate()` while
iterating under `managed_entities.lock()`, which `create_publisher` also takes.
This is not reachable today — the only way to register an entity is
`create_publisher`, so every element is a `ZLifecyclePublisher` whose
activate/deactivate only flip an atomic — and it ships without a test on
purpose, because a test for it would have to fabricate a registration path that
does not exist, and a detector that can only fail against invented code proves
nothing. The comment says exactly that so the next reader does not re-derive it.
Detector evidence, both directions. Two deadline-guarded tests in
`crates/hiroz-tests/tests/reentrant_lifecycle.rs`. With the fix reverted and
the tests kept, both fail:
transition_callback_querying_own_state_does_not_deadlock
~/get_state did not answer while the transition callback was
running - the state-machine mutex was held across the callback: Timeout(8s)
failing_transition_callback_still_observes_intermediate_state (same)
With the fix, both pass, and `lifecycle` stays at 30/30.
Moving the transition callback out of the state-machine lock left a worse failure mode than the deadlock it removed. `begin` enters the intermediate state and drops its guard; if the callback then panics, the unwind escapes `trigger_transition` before `finish` runs, so `current` stays at `Configuring`/`Activating`/... forever. Because the guard was already gone, the mutex is not poisoned either, so nothing reports it: every later `begin` returns `None` and `~/get_state` answers `configuring` for the life of the process. While the callback ran under the guard, the same panic unwound through a live `MutexGuard` and poisoned the mutex, so the next access failed loudly. Fail-fast had become a silent permanent wedge. Catch the unwind, settle on `Failure` — revert to the start state, the same "nothing changed" outcome as the invalid-transition arm — and re-raise the payload so the panic is still as loud as before. `on_error` is deliberately not run on that path: we are already unwinding, and running more user code on the way out risks a second panic. The `on_error` invocation gets the same guard, since a panic there would otherwise strand the node in `ErrorProcessing`.
…mmary The error path splits the same way the normal path does -- `finish` records `CallbackReturn::Error` and drops the guard, `on_error` runs unlocked, `finish_error_processing` applies its verdict -- but nothing exercised it. Both existing scenarios return `Success` or `Failure`, so a regression that ran `on_error` back under the mutex would have passed the entire suite. The new scenario uses the same detector as the working one: `on_error` asks its own node, through a real client in another context, what state it is in. Verified in both directions -- reintroducing the guard across `on_error` fails it on the client timeout rather than hanging the suite. It also pins the state that should be observable there, `ErrorProcessing`. The module summary claimed observers "keep seeing exactly what they saw before". They do not, and that is the point of the change: previously a `~/get_state` during a transition blocked for the callback's whole duration, and `ZLifecycleClient` mapped every transition-state id onto `Unconfigured`. The intermediate state is newly observable, which is what rclcpp reports.
3fdbc4b to
67d4ca7
Compare
Description
Fixes #252
ZLifecycleNode::trigger_transitioninvoked the user's transition callback from insidestate_machine.lock().unwrap().trigger(..). That mutex is shared with the node's own~/get_state,~/change_stateand~/get_available_transitionshandlers and withget_current_state(), so a node could not answer any question about its own state while a transition callback ran — and a callback that asks waits forever.Branches off
mainand stands alone.Key Changes
crates/hiroz/src/lifecycle/state_machine.rs— splitStateMachine::triggerintobegin(validate, enter the intermediate "busy" state) andfinish(apply the callback's verdict), withtriggerkept as a wrapper for callers that own the machine outright. The error path gets the samefinish_error_processingsplit. The callback now runs between the two locked steps with no guard held.The
begincall is bound to its ownleton purpose. Writing it as amatchscrutinee keeps the temporary guard alive for the wholematchbody and silently reinstates the deadlock — that mistake was made and caught by these tests during development, and the code says so.crates/hiroz/src/lifecycle/client.rs— fixstate_from_lc, the client-side parser, which mapped every transition-state id ontoUnconfiguredvia its catch-all arm. Ids 10–15 (Configuring,CleaningUp,ShuttingDown,Activating,Deactivating,ErrorProcessing) now map to themselves. This is also what makes the intermediate state observable to the new test.crates/hiroz/src/lifecycle/node.rs— managed entities get collect-then-invoke as well:trigger_transitioncallede.on_activate()/e.on_deactivate()while iterating undermanaged_entities.lock(), whichcreate_publisheralso takes. This one is not reachable today and is fixed preventively: the only way to register an entity iscreate_publisher, so every element is aZLifecyclePublisherwhose activate/deactivate only flip an atomic. It ships without a test on purpose — a test would have to fabricate a registration path that does not exist, and a detector that can only fail against invented code proves nothing. The comment says exactly that so the next reader does not have to re-derive it.Splitting
triggermeans a panicking callback could strand the machine in its intermediate state, with no lock held and no verdict ever applied —Configuringforever, and every subsequent transition rejected. Both callback sites are therefore wrapped incatch_unwind: the machine is finished withFailure(orfinish_error_processing(Error)on the error path) and the payload is re-raised withresume_unwind, so a panicking callback still fails exactly as loudly as before. See "Breaking Changes" for the one profile where this guard does not apply.What fails without this
Detector proven in both directions. With the fix reverted and the two deadline-guarded tests in
crates/hiroz-tests/tests/reentrant_lifecycle.rskept, both fail:With the fix, both pass.
No regression:
lifecycle30/30,hiroz --lib282/282.Breaking Changes
One observable change, and it is the intended fix — but it is narrower than "the node's reporting changed". The node always reported its genuine intermediate state over the wire. The
idit publishes was correct before this PR; what was wrong was hiroz's ownZLifecycleClient, which discarded ids 10–15 in a catch-all and surfacedUnconfiguredto its caller.So:
ZLifecycleClientnow seesConfiguring/Activating/ … where it previously sawUnconfigured. Anything readingUnconfiguredas "transition in progress" was reading a bug in this client.ros2 lifecycle get) are unaffected — they parse the wire value themselves and always saw the correct state.Panic semantics, stated because they are new: a panic in a transition or
on_errorcallback is now caught, the state machine is advanced to a terminal verdict, and the panic is re-raised. The net effect for a caller is unchanged (the panic still propagates), but the machine is no longer left mid-transition. This guard relies on unwinding, so it does not apply under the[profile.opt]profile, which setspanic = "abort"— there a panicking callback aborts the process, as it did before.dev,testandreleaseall unwind and are covered.No API is removed.
StateMachine::triggerandtrigger_error_processingare retained as wrappers.Notes
cargo docis red onmaintoday for unrelated reasons (three unresolved intra-doc links inerror.rs);pr/0-rustdoc-linksclears it. This branch adds no doc links.Checklist
./scripts/check-local.shsuccessfully