E2E flake: asyncio.InvalidStateError logged during gRPC teardown after lease timeout

[mangelajo]

Summary

The E2E test G2: lease timeout during slow beforeLease hook exits cleanly fails intermittently because an asyncio.exceptions.InvalidStateError is logged to stderr during gRPC channel teardown after a lease expires. The test assertion Expect(out).NotTo(ContainSubstring("Error:")) catches the substring InvalidStateError:.

Failed run: https://github.com/jumpstarter-dev/jumpstarter/actions/runs/28078498190/job/83127941530#step:10:1008

Log snippet

[06/24/26 06:25:57] INFO     [jumpstarter.client.lease] Acquiring lease
                             019ef84e-8cde-789f-8310-3f9579a30929 for selector
                             example.com/board=hooks for duration 0:00:05
[06/24/26 06:25:58] INFO     Lease 019ef84e-8cde-789f-8310-3f9579a30929 acquired
                             successfully! (0:00:00)
                    INFO     Waiting for ready connection at
                             /run/user/1001/jumpstarter-u2c12h_m/socket
[06/24/26 06:26:02] INFO     Lease 019ef84e-8cde-789f-8310-3f9579a30929 ended at
                             2026-06-24 06:26:02+00:00
                    INFO     Lease 019ef84e-8cde-789f-8310-3f9579a30929 ended at
                             2026-06-24 06:26:02.014601+00:00
[06/24/26 06:26:18] ERROR    [asyncio] Exception in callback
                             Future.set_result(None)
                             handle: <Handle Future.set_result(None)>
                             Traceback (most recent call last):
                               File
                             ".../lib/python3.12/asyncio/events.py", line 88, in _run
                                 self._context.run(self._callback, *self._args)
                             asyncio.exceptions.InvalidStateError: invalid state

Investigation

Scenario

1. Exporter config exporter-hooks-slow-before.yaml has a beforeLease hook with sleep 8
2. Client requests a lease with 5s duration
3. Lease expires after 5s while the beforeLease hook is still running (~second 4 of 8)
4. Client detects the lease ended and begins graceful teardown
5. ~16 seconds after the lease ends, a gRPC internal callback fires Future.set_result(None) on a Future that's already been cancelled/completed
6. Python's asyncio default exception handler logs ERROR [asyncio] ... asyncio.exceptions.InvalidStateError: invalid state

Root cause

This is a race condition in gRPC Python's async channel teardown. When the lease expires and the client exits, the gRPC aio.secure_channel is being closed. During teardown, an internal gRPC callback tries to resolve a Future that's already in a terminal state (cancelled or already resolved). This races with anyio's task group cancellation.

The application code already handles InvalidStateError in several places:

• streams/router.py:60,66 — contextlib.suppress(asyncio.exceptions.InvalidStateError) in send_eof and aclose
• streams/common.py:29 — catches asyncio.InvalidStateError in copy_stream

However, the error in this failure is not raised in any application code path. It fires as an asyncio event loop callback (_run in asyncio/events.py:88), which bypasses all try/except blocks and is logged directly by asyncio's default exception handler.

Fix

Install a scoped custom asyncio exception handler that suppresses InvalidStateError during gRPC teardown. The handler should be installed around the shell session lifecycle in _shell_with_signal_handling and should only suppress InvalidStateError, forwarding all other exceptions to the default handler.

Test file

e2e/test/hooks_test.go:371-380

It("G2: lease timeout during slow beforeLease hook exits cleanly", func() {
    startHooksExporter("exporter-hooks-slow-before.yaml")
    out, err := RunCmd("timeout", "60", "jmp", "shell",
        "--client", "test-client-hooks",
        "--selector", "example.com/board=hooks",
        "--duration", "5s", "--", "sleep", "30")
    _ = err
    Expect(out).NotTo(ContainSubstring("Error:"))
})