[Bug] vsock: 100% CPU main-loop livelock and hung connections after Resume with in-flight traffic (v1.16.x, EVENT_IDX)

[sergii-tereshchenko]

Describe the bug

After a PATCH /vm {"state":"Paused"} → PATCH /vm {"state":"Resumed"} cycle, if there was in-flight bidirectional vsock traffic at pause time, the vsock device wedges on resume:

1. The Firecracker main/event-loop thread spins at 100% CPU in a tight epoll_pwait loop, and
2. All vsock connections stop making progress — existing connections hang and new host-initiated connections are never accepted by the guest.

The vCPUs run (guest is not paused) but block in KVM_RUN waiting for device interrupts that the wedged event loop never delivers, so the guest goes effectively dark (no serial output, no network).

This looks like the Pause/Resume-path analogue of #5969 / #5958, which fixed the equivalent hang only for the snapshot LoadSnapshot path. The Resume API path has no equivalent re-arm and additionally exhibits a busy-poll livelock. Both appear rooted in the vsock VIRTIO_RING_F_EVENT_IDX support added in #5872 (first shipped in v1.16.0).

To Reproduce

Minimal shape (does not require snapshots):

1. Boot a microVM with a vsock device and a guest agent listening on a vsock port.
2. Establish steady bidirectional vsock traffic — e.g. a guest process writing to stdout streamed host→guest over one connection (host→guest RX active), while the host periodically sends requests (guest→host TX active). The key is that both an RX descriptor and a TX descriptor are in-flight/unconsumed at the instant of pause.
3. PATCH /vm {"state":"Paused"}.
4. PATCH /vm {"state":"Resumed"}.
5. From the host, open a new vsock connection to the guest agent (e.g. an exec-style control connection).

Observed: step 5's CONNECT is never accepted; the host side blocks indefinitely. Firecracker's main thread is pinned at 100% CPU. Existing connections also stop advancing.

It is intermittent across cycles — it triggers on whichever pause happens to catch a descriptor in-flight, so a workload with continuous traffic (a 1 Hz stdout writer plus periodic control RPCs) reproduces it reliably within one or two pause/resume cycles.

Diagnostics captured on a wedged instance

Firecracker v1.16.1, x86_64, MMIO transport, two vCPUs. Captured live while the VM was hung.

Main thread — 100% CPU, tight epoll loop. strace -c -p <fc-main-tid> over 3 s:

% time     seconds  usecs/call     calls    errors syscall
------ ----------- ----------- --------- --------- ----------------
 99.86    0.491726           1    330917           epoll_pwait
  0.14    0.000703         117         6           write
  0.00    0.000008           2         3           read

~110k epoll_pwait/s, each returning immediately. Per-event trace shows a nested-epoll cycle that never clears:

epoll_pwait(30, [{events=EPOLLIN, data={u32=13, u64=13}}], 32, 0, ...) = 1   # inner eventpoll
epoll_pwait(9,  [{events=EPOLLIN, data={u32=30, ...}}],     256, -1, ...) = 1  # outer -> inner
# repeats forever; fd 13 keeps reporting EPOLLIN and is never drained/deregistered

fd 13 is an accepted connection on the vsock backend Unix socket (the UDS the muxer accepts host-side connections on) — i.e. pending host→guest data on a muxed vsock connection that the device muxer never consumes and never removes from the epoll set → level-triggered busy-poll.

vCPU threads — idle, blocked in KVM_RUN.

fc_vcpu 0: State S (sleeping), blocked in ioctl(KVM_RUN), ~0 CPU growth over 10s
fc_vcpu 1: State S (sleeping), blocked in ioctl(KVM_RUN), ~0 CPU growth over 10s

Guest is waiting for device interrupts that the livelocked event loop never injects (ping to guest dead, serial silent after resume).

Firecracker warnings at every resume (guest console / firecracker log):

[vm:fc_vcpu 0:WARN] Received a VcpuEvent::Resume message with immediate_exit enabled. immediate_exit was disabled before proceeding
[vm:fc_vcpu 1:WARN] Received a VcpuEvent::Resume message with immediate_exit enabled. immediate_exit was disabled before proceeding
[vm:main:WARN] Got a spurious notification from api thread

Expected behaviour

After Resume, the vsock device should re-process any in-flight RX/TX descriptors and re-arm notifications (as the LoadSnapshot kick() path was fixed to do in #5958), so existing connections resume and new connections are accepted. The event loop must not busy-poll a level-triggered backend fd whose data cannot currently be delivered.

Hypothesis (for maintainers)

Two stacked effects, both downstream of vsock EVENT_IDX (#5872):

1. Notification state not re-armed across Resume. With VIRTIO_RING_F_EVENT_IDX negotiated, a descriptor in-flight at pause leaves the used/avail-event indices in a state where, after resume, the guest's TX kick and/or the host's RX interrupt is suppressed and never retried — the same class of bug fix(vsock): kick tx queue on restore #5958 fixed for restore, but the Resume API path has no equivalent replay/re-arm hook.
2. Muxer busy-poll under backpressure. When host→guest RX cannot be delivered (guest not consuming because of (1)), the muxer leaves the connection's backend UDS fd registered with level-triggered EPOLLIN and never drains or deregisters it, pinning the main loop at 100% CPU and starving all other device servicing — which is why the whole VM (net, serial) goes dark, not just vsock.

The restore fix (#5958) replays the TX queue kick in the device's restore kick(). Resume appears to need the analogous re-arm, plus (or including) correct edge/level handling of the muxer connection fd so a non-drainable connection doesn't livelock the loop.

Note on scope: effect (2), the busy-poll, may be at least partly independent of EVENT_IDX — a level-triggered backend fd that is never drained or deregistered is a muxer event-handling issue that EVENT_IDX exposes (by leaving RX undeliverable) rather than causes. If so, disabling vsock EVENT_IDX would hide the livelock but not fix the underlying fd-handling. This is a reason to fix the re-arm + muxer handling rather than treat it purely as an EVENT_IDX regression.

Environment

• Firecracker: v1.16.1 (release binary). Also expected on v1.16.0 (EVENT_IDX for vsock landed in feat(vsock): Add VIRTIO_RING_F_EVENT_IDX support #5872 → v1.16.0).
• Host arch: x86_64. vsock transport: MMIO. 2 vCPUs.
• Setup: Firecracker running inside a virtualized host (nested virtualization); the vsock device backs a containerd-shim (nerdbox) control channel to an in-guest agent. The bug is in Firecracker's vsock device event loop and is not specific to nesting.
• Guest kernel / rootfs: custom guest kernel + minimal initrd (details available on request; not believed relevant — the guest driver is stock virtio-vsock).
• Not using snapshots for this repro — plain PauseVM/ResumeVM API.

Additional context

• Related: feat(vsock): Add VIRTIO_RING_F_EVENT_IDX support #5872 (adds VIRTIO_RING_F_EVENT_IDX to vsock, v1.16.0), fix(vsock): kick tx queue on restore #5958 / [Bug] vsock guest-to-host connections hang after snapshot restore #5969 (restore-path vsock hang, fixed for LoadSnapshot only in v1.16.1).
• The 100% CPU main-loop symptom (distinct from the plain connection hang in [Bug] vsock guest-to-host connections hang after snapshot restore #5969) may be the more actionable signal, since it's a deterministic busy-poll on a specific fd rather than a race — happy to capture additional traces (full per-event epoll_pwait dump, muxer state, queue indices) or provide a self-contained repro harness on request.

[Manciukic]

Hey @sergii-tereshchenko, thanks for the report!
Are you able to reproduce the issue on v1.15, before the EVENT_IDX addition?
I didn't take a deep look yet, so I'm not sure what's happening, but could you try the current main? I recently fixed a case where the epoll could spin indefinitely on EPOLLIN. My hunch is that it won't fix the issue but at least remove the busy spin. Running it with debug logging might show why it's hanging.

[Manciukic]

Alright, I was able to give this a spin in the background and confirm my hunch:

• v1.15.x works fine
• the current main fixes the spin but not the hang

Furthermore, it seems the bug is deterministic and doesn't depend on activity on the device.

The bug was introduced in b27cdd025 (#5882, first shipped in v1.16.0), which fixed a real MSI-X reset race on the restore path. The PR introduced a pending_event_ack gate to avoid sending RX to a guest processing the transport_reset, but that was added in a kick() shared by both restore and plain resume, where only the restore caller should be arming it.

The fix seems to be to move arming the pending_event_ack in a different place (maybe restore()?), but we need to be careful we don't introduce other issues (what happens if we pause/snapshot/resume? etc).

I've also verified there's no workaround for this issue on v1.16. I'll make sure this gets picked up by the next release.

cc: @JackThomson2 which is more familiar with the pending_event_ack mechanism

[sergii-tereshchenko]

Yeah. We are on 1.15.1 right now and it is working well. We have tried to update to 1.16.1 and our integration tests start to hang. I had similar conclusion on root cause and potential fix. We are planning to stay on 1.15.1 until it is fixed. Thank you for picking it up so quickly.

[maggie-lou]

This is also blocking our upgrade from 1.15.1 -> 1.16. Thanks for identifying the root cause and fix 🙏

[Manciukic]

The fix was merged in #6100, it will land in the next release in a few weeks. We'll consider a patch release for the v1.16 branch as well. I'll keep this issue open until a version with the fix is officially released.