feat: detect scanners stuck reporting zero free connection slots

[bluetoothbot]

What

Detect when a Bluetooth scanner (typically an ESPHome proxy) is stuck reporting zero free connection slots, and surface it as a warning + diagnostic instead of letting it silently break BLE connections.

Why

Closes #340. Proxies can enter a state where they report all slots as occupied even though no devices are actually connected. Recovery requires a proxy reboot, but the symptom is hidden — the user has to enable debug logging on habluetooth.wrappers and grep INFO output for slots=0/N free to even know which proxy to power-cycle.

How

• BluetoothManager now tracks _allocations_zero_since per source: the timestamp when a source first reported free == 0 (cleared on any non-zero update or scanner unregister).
• The existing periodic _async_check_unavailable calls a new _async_check_stuck_allocations which logs a one-shot WARNING once a source has stayed at zero free slots for STUCK_ALLOCATION_THRESHOLD_SECONDS (30 min) — but only if habluetooth has no in-flight connections through that scanner, to avoid false positives for legitimately busy proxies.
• New async_stuck_allocations() public query and a stuck_allocations field in async_diagnostics() for HA's diagnostics download.
• STUCK_ALLOCATION_THRESHOLD_SECONDS is exposed in __init__ for tuning/tests.
• .pxd updated to keep the cython build in sync.

Testing

• 5 new tests covering: detection at threshold, warning fires once per stuck cycle, recovery clears state, in-flight connections suppress the warning, unregister cleans tracking, diagnostics surface stuck sources.
• Full suite: 230 passed, 1 skipped.

🤖 Generated with Claude Code

[codecov]

Codecov Report

❌ Patch coverage is 81.57895% with 7 lines in your changes missing coverage. Please review.
✅ Project coverage is 94.25%. Comparing base (931296d) to head (b99ca8d).

Files with missing lines	Patch %	Lines
src/habluetooth/manager.py	81.08%	3 Missing and 4 partials ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #407      +/-   ##
==========================================
- Coverage   94.50%   94.25%   -0.26%     
==========================================
  Files          14       14              
  Lines        1911     1949      +38     
  Branches      230      240      +10     
==========================================
+ Hits         1806     1837      +31     
- Misses         62       65       +3     
- Partials       43       47       +4     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[codspeed-hq]

Merging this PR will degrade performance by 7.73%

⚠️ Different runtime environments detected

Some benchmarks with significant performance changes were compared across different runtime environments,
which may affect the accuracy of the results.

Open the report in CodSpeed to investigate

❌ 1 regressed benchmark
✅ 10 untouched benchmarks

Warning

Please fix the performance issues or acknowledge them on CodSpeed.

Performance Changes

	Benchmark	BASE	HEAD	Efficiency
❌	test_inject_100_simple_advertisements	188.8 µs	204.6 µs	-7.73%

Tip

Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.

---

_{Comparing koan/detect-stuck-allocations (b99ca8d) with main (931296d)}

[bdraco]

@bluetoothbot plan This proposed solution is the wrong fix. We should figure out why they are getting stuck in the first place

[bluetoothbot]

I have what I need. Writing the updated plan now.

Pivot: investigate root cause of stuck slot allocations, not detect-and-warn

Changes in this iteration

• Dropped the symptom-detection approach per @bdraco's "wrong fix" comment. No _allocations_zero_since tracking, no 30-minute warning, no async_stuck_allocations() public API. The original PR (feat: detect scanners stuck reporting zero free connection slots #407, branch koan/detect-stuck-allocations) should be closed without merge.
• Reframed as a multi-repo root-cause investigation. The slot accounting lives on the ESPHome proxy itself (firmware), surfaces via aioesphomeapi, lands in bleak-esphome (ESPHomeBluetoothDevice.async_update_ble_connection_limits at bleak-esphome/src/bleak_esphome/backend/device.py:43), and is reported to habluetooth via BluetoothManager.async_on_allocation_changed at manager.py:929. habluetooth treats the count as opaque truth, so a habluetooth-only fix can't address the cause.
• Added a narrow, diagnostics-only first phase so the next user who hits this can produce evidence (transition log + scanner-level counters in async_diagnostics()). No user-facing warnings. The expensive Cython-touching .pxd changes from the previous plan are not needed.
• Added an audit phase for habluetooth's connection lifecycle to rule out a habluetooth-side leak (e.g. connect() paths that never reach disconnect() on the proxy side, scanner unregister mid-connection) before pointing at upstream.
• Codspeed regression noted in PR comments (-7.73% on test_inject_100_simple_advertisements) confirms that adding per-source bookkeeping in a hot path is the wrong layer to do this work in — another reason to back out.

Summary

Issue #340 reports that ESPHome proxies sometimes report slots=0/N free indefinitely with zero free connections, even after HA restarts, until the proxy is power-cycled. The previously proposed PR detects the symptom and warns the user. @bdraco's review says that's the wrong fix and asks us to find the root cause. This plan replaces the symptom-detection work with an investigation: add minimum-cost diagnostics so we can capture the divergence in the wild, audit habluetooth's contribution to the connection lifecycle, then push the actual fix to whichever layer is responsible (most likely bleak-esphome or ESPHome firmware, not habluetooth).

Alternatives Considered

• Approach A (chosen) — Root-cause investigation across the stack. Add diagnostics-only instrumentation in habluetooth, audit the connection lifecycle for leaks habluetooth could be causing, then move the actual fix to the correct layer (bleak-esphome, aioesphomeapi, or ESPHome firmware) once we know where the divergence originates. Trade-off: takes longer and is multi-repo; produces a real fix instead of a workaround.
• Approach B — Symptom detection in habluetooth (original PR). Track _allocations_zero_since and warn after 30 min. Rejected by @bdraco; also added a 7.73% benchmark regression and missed coverage on the new branches. Doesn't fix anything — users still have to reboot the proxy.
• Approach C — Auto-recovery: send bluetooth_device_disconnect for phantom-allocated addresses. Tempting, but (a) needs to live in bleak-esphome not habluetooth, (b) without root-cause understanding, "phantom" is just a guess and we might disconnect real connections, (c) belongs after Phase 1 produces evidence.

Implementation Phases

Phase 1: Diagnostics-only instrumentation in habluetooth (small, additive)

• What:
  • In BluetoothManager.async_on_allocation_changed (src/habluetooth/manager.py:929), at DEBUG only, log allocation transitions with deltas: previous free/slots/allocated → new, plus the per-scanner _connect_in_progress and _connect_failures snapshot from the affected BaseHaScanner. This is the bare minimum the next Bluetooth proxy connection slots can become permanently stuck, blocking all BLE connections until proxy reboot #340 reporter would need to attach to a bug report.
  • Extend BluetoothManager.async_diagnostics (manager.py:240) so the existing "allocations" entry per source also carries a small history record: timestamp of last free→0 transition, last 0→non-zero transition, count of zero→zero updates received (i.e. allocation updates that arrived while we already had free == 0). Bounded to per-source scalars; no per-event list.
  • No new public methods, no warnings, no .pxd changes, no new constants in __init__.
• Why: Lets us collect concrete evidence from a stuck system without adding load-bearing behavior. The previous PR's 7.73% regression came from threading new bookkeeping through the manager-class slots and the periodic unavailable-check — Phase 1 deliberately avoids both: the bookkeeping piggybacks on existing allocation callbacks and is read-only from the diagnostics path.
• Gotchas:
  • async_on_allocation_changed is called on every BlueZ slot change and every ESPHome BluetoothDeviceConnectionResponse, so anything beyond scalar updates and a debug log will show up in benchmarks. Keep it allocation-free in the steady state.
  • Don't re-introduce __slots__ churn: add at most two new attributes (_allocations_last_zero_monotonic, _allocations_zero_repeat_count) per source, stored inside the existing _allocations dict via a wider dataclass or a sibling dict — pick whichever is less invasive to manager.pxd. If it would force a .pxd change, prefer a sibling dict[str, tuple[float, int]] declared as object.
• Done when:
  • DEBUG-level logs include allocation deltas with scanner-state context.
  • async_diagnostics() output for an affected source shows the last zero/non-zero transition times and the repeat count.
  • No regression > 1% on the test_inject_100_simple_advertisements and test_inject_* benchmarks (verify locally with pytest tests/test_benchmark_* before pushing).

Phase 2: Audit habluetooth's contribution to the lifecycle

• What:
  • Walk every code path that calls a remote-scanner backend's connect() and disconnect() in src/habluetooth/wrappers.py (HaBleakClientWrapper.connect at wrappers.py:410, disconnect at wrappers.py:651, _async_get_backend_for_ble_device at wrappers.py:542).
  • Specifically verify the cases:
    1. Connect raises mid-flight (e.g. _get_services fails). Does the proxy see a clean bluetooth_device_disconnect? (bleak-esphome _disconnect at client.py:406 does this — confirm habluetooth never bypasses the bleak disconnect.)
    2. HaBleakClientWrapper is GC'd while self._backend.is_connected is True. Does the destructor chain reach the proxy?
    3. Scanner unregister fires while a connection through that scanner is active (_async_unregister_scanner_internal at manager.py:835). What happens to in-flight BleakClient instances pointed at that scanner — does the slot release ever propagate?
    4. manager.async_release_connection_slot is only called for local adapters when connect fails (wrappers.py:495). Confirm this is correct and that we don't need an analogous release-hint for the remote path (the remote scanner manages its own counts, but a sanity check is cheap).
  • Capture findings in the PR description for the eventual fix PR — no code yet unless an actual leak is found.
• Why: Before blaming ESPHome firmware, prove habluetooth isn't leaking a "phantom in-progress" or losing a disconnect call. If a leak is found, the fix lands in habluetooth.
• Gotchas:
  • _finished_connecting in base_scanner.py:174 decrements _connect_in_progress. If connect() raises after _add_connecting but the exception path skips _finished_connecting, the in-progress count will leak — verify the try/finally at wrappers.py:482-495 covers every path.
  • The "10 proxies enabled but only 4 ever listed as a valid connection path" note from Bluetooth proxy connection slots can become permanently stuck, blocking all BLE connections until proxy reboot #340 suggests scoring may be excluding stuck scanners by design (_score_connection_paths returns NO_RSSI_VALUE when free == 0 at base_scanner.py:222-225) — that's a symptom of the upstream stuck-state, not a habluetooth bug, but document it so we don't chase it.
• Done when: An audit note (PR description, not a doc file) lists each lifecycle path and either "verified clean" or "fix needed: ".

Phase 3: Upstream coordination (bleak-esphome / aioesphomeapi / ESPHome firmware)

• What:
  • File a follow-up issue in bleak-esphome referencing Bluetooth proxy connection slots can become permanently stuck, blocking all BLE connections until proxy reboot #340 + the Phase 1 diagnostics, asking maintainers (likely also @bdraco) to confirm whether the proxy ever re-sends BluetoothConnectionsFreeResponse after a stuck state, and whether the API client emits a re-subscribe on reconnect that would force a state refresh.
  • The most plausible mechanisms to investigate, in priority order:
    1. ESPHome firmware never releases slot when the peer disappears mid-connection (no BLE_GAP_EVENT_DISCONNECT). Fix: firmware-side timeout / GATT keepalive. Tracked in esphome/esphome repo.
    2. aioesphomeapi loses the unsubscribe when HA<->ESP TCP reconnects, leaving the proxy with stale "subscriber" state that pins slots. Fix: explicit re-subscribe with reset semantics.
    3. bleak-esphome ESPHomeBluetoothDevice.async_update_ble_connection_limits treats every push as authoritative but never reconciles against actual _disconnect_callbacks membership. Fix: cross-check on update.
  • From habluetooth's side, ship Phase 1 diagnostics and ask a Bluetooth proxy connection slots can become permanently stuck, blocking all BLE connections until proxy reboot #340 reporter (or a synthetic repro by killing/restarting the ESP proxy mid-connection) to capture the diagnostic dump.
• Why: The slot count is owned by the proxy and reported through two intermediate libraries before habluetooth ever sees it. Without upstream cooperation, habluetooth can at best paper over the bug.
• Gotchas:
  • Reproduction is hard — Bluetooth proxy connection slots can become permanently stuck, blocking all BLE connections until proxy reboot #340 reporter mentions it persists across HA restarts. Easiest synthetic repro: open a connection through the proxy, kill the HA process before disconnect, restart HA, observe slot state on the proxy.
  • Don't merge Phase 1 + upstream issues without a clear "we'll revisit habluetooth once root cause is known" note, so we don't ship diagnostics-only code and then leave it stranded.
• Done when: Upstream tracking issue exists with a Phase 1 diagnostic dump attached, and a hypothesis is narrowed to one of firmware / aioesphomeapi / bleak-esphome.

Phase 4: Apply the real fix in the responsible layer (conditional)

• What: Once Phase 3 identifies the source, implement the fix there. If — and only if — the audit in Phase 2 turns up a habluetooth-side contributor, do that work in habluetooth.
• Why: A fix in the wrong layer is just a different workaround.
• Gotchas: Resist the urge to add habluetooth-side recovery (e.g. "force-disconnect phantom slots") until the cause is known; that's Approach C and has real-disconnect risk.
• Done when: Stuck-slot state no longer occurs without a proxy reboot, validated by a Bluetooth proxy connection slots can become permanently stuck, blocking all BLE connections until proxy reboot #340 reporter or synthetic repro.

Corner Cases

• Sources that legitimately stay at free == 0 for long periods because they're saturated with real connections — must not get false "evidence of stuck" markers. Phase 1 records transitions, not duration; saturation looks identical to stuck in transition data, which is fine because no automated action is taken.
• Allocation updates may arrive before a scanner reports any advertisements (proxies announce slot state at connect time). Make sure the transition record handles "first ever update is free == 0" without crashing.
• Scanner unregistered while a transition record exists: clean up the sibling dict alongside self._allocations.pop(scanner.source, None) at manager.py:848.
• The original PR's async_stuck_allocations() public method is intentionally not part of this plan — no public API is added in Phase 1.

Testing Strategy

• Phase 1:
  • Unit test in tests/test_manager.py that drives async_on_allocation_changed with a 5→0 transition followed by repeats and confirms the diagnostics dump reflects the timestamps and repeat count.
  • Confirm DEBUG log line format under caplog.
  • Run pytest tests/test_benchmark_base_scanner.py and ensure test_inject_100_simple_advertisements is back within 1% of main.
• Phase 2: Add no new tests yet; lifecycle audit produces findings, not code. If a leak is found, add a regression test for that specific path.
• Phase 3: Out of repo.

Risks & Alternatives

• Risk: Phase 1 ships and then Phase 3 stalls because no one with a stuck proxy attaches a diagnostic dump. Mitigation: link Phase 1 to Bluetooth proxy connection slots can become permanently stuck, blocking all BLE connections until proxy reboot #340 explicitly and ask the original reporter to capture.
• Risk: We discover the fix really does belong in habluetooth (e.g. an in-progress count leak). Mitigation: Phase 2 audit; the fix is small and targeted, not the 220-line PR proposed originally.
• Alternative: Approach C (auto-recovery via forced disconnect) remains an option if Phase 3 finds it's the proxy and ESPHome upstream can't / won't fix promptly. Would live in bleak-esphome, not habluetooth, and would require Phase 1's diagnostic data to identify safe-to-disconnect addresses.

Open Questions

• Should the original PR (feat: detect scanners stuck reporting zero free connection slots #407) be closed before Phase 1 lands, or converted in place? Recommendation: close with a comment linking this plan; open a fresh, smaller PR for Phase 1 so the codspeed and coverage history are clean.
• Are there existing repros in the test suite (e.g. mocked ESPHomeScanner) we can extend to simulate a stuck proxy without needing real hardware?

---

Generated by Kōan /plan — iteration on existing issue