Ensure gateway cleanup completes after controller shutdown failures

[dmulcahey]

Problem

Gateway.shutdown() currently treats controller shutdown and ZHA-owned cleanup as one fallible sequence. It marks the gateway as shutting down, tears down device and group wrappers, and then awaits application_controller.shutdown(). Only after that await succeeds does it cancel the gateway's remaining tasks and timers and clear its device and group maps.

If controller shutdown raises or is cancelled, execution leaves the method before that local finalization runs. The gateway is then stuck in a half-shut state:

• gateway-owned background work can remain active;
• device and group maps can remain populated;
• the controller remains attached to the gateway and can still deliver callbacks;
• a second shutdown call returns immediately because the single shutting_down flag cannot distinguish “in progress,” “failed,” and “complete.”

Cancellation during the short post-controller settling delay has the same cleanup-skipping risk.

Root cause

The shutdown path has two independently fallible domains—controller teardown and local ZHA teardown—but represents them with one boolean and one linear control flow. There is no unconditional finalization boundary, no serialized retry state, and no protection against initialization replacing a controller whose shutdown is still incomplete.

What changes

This change gives shutdown explicit, independently retryable phases:

• serialize initialization and shutdown transitions with one lock;
• stop local producers and detach both controller listener registrations before the fallible controller await;
• retain interrupted device and group teardown work until preparation can finish, even though the public maps are cleared unconditionally;
• use the retained controller reference itself to represent incomplete controller shutdown, clearing it only after that exact controller shuts down successfully;
• record controller completion before the post-controller delay so cancellation during the delay cannot shut the controller down twice;
• always attempt gateway-owned task and timer cleanup and always clear the device and group maps;
• preserve the first controller or cancellation error when local cleanup also fails, while logging the secondary cleanup failure;
• retry only phases that remain incomplete;
• refuse reinitialization while incomplete shutdown work still owns the current controller or wrapper state.

A completed duplicate shutdown remains harmless, while a concurrent caller waits for the active attempt instead of returning while cleanup is still running.

Resulting behavior

After shutdown begins, a controller exception or cancellation can no longer bypass ZHA-owned finalization. Successfully completed phases are not repeated. If a local phase itself fails or is interrupted, its state remains explicitly incomplete so a later call has a deterministic recovery path rather than falsely reporting completion.

Real-world examples

USB coordinator disappears during unload

A USB adapter is unplugged or its serial transport fails while the radio library is disconnecting. Controller shutdown raises, but ZHA still cancels its polling and background work, clears local device and group ownership, and detaches controller callbacks. The exact failed controller remains available for a later controller-only retry instead of leaving the gateway permanently half shut.

Unload is cancelled during the radio settling delay

The controller has already shut down, but the caller is cancelled during the short delay that lets a radio thread callback settle. The controller remains marked complete, local finalization still runs, and the cancellation is propagated afterward. A later call does not invoke controller shutdown a second time.

Controller teardown and local cleanup both fail

A radio backend raises while a separate local cleanup operation also encounters an error. The radio failure remains the primary exception reported to the caller, maps are still cleared, and the incomplete local phase can be retried. The secondary failure is logged instead of masking the more useful original cause.

Reload follows a failed shutdown

A reload is requested immediately after controller shutdown failed. Initialization cannot replace the retained controller with a new instance while the old one still needs teardown. The lifecycle transition is serialized, preserving the exact controller and the deterministic retry path.

Wrapper teardown is interrupted

Cancellation arrives while a device wrapper is removing entity subscriptions and owned work. The wrapper reference is retained outside the cleared public maps, so a later shutdown call can resume that incomplete teardown rather than silently classifying the local phase as complete.

Scope

This is limited to gateway shutdown finalization, retry state, and the initialization boundary required to preserve that state. It does not add device lifecycle generations, change startup task ownership, or alter entity behavior.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 97.28%. Comparing base (82a14cd) to head (9c57820).

Additional details and impacted files

@@            Coverage Diff             @@
##              dev     #842      +/-   ##
==========================================
+ Coverage   97.27%   97.28%   +0.01%     
==========================================
  Files          55       55              
  Lines       10941    10994      +53     
==========================================
+ Hits        10643    10696      +53     
  Misses        298      298              

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[dmulcahey]

Not sure if this is something we want to fix / support in practice.