Skip to content

fix: recover zombie connections on KeepAliveTimeout - #126

Open
FlavioPulli wants to merge 1 commit into
evolution-foundation:mainfrom
FlavioPulli:fix/keepalive-timeout-handling
Open

fix: recover zombie connections on KeepAliveTimeout#126
FlavioPulli wants to merge 1 commit into
evolution-foundation:mainfrom
FlavioPulli:fix/keepalive-timeout-handling

Conversation

@FlavioPulli

@FlavioPulli FlavioPulli commented Jul 24, 2026

Copy link
Copy Markdown

With EnableAutoReconnect disabled (whatsmeow.go:464), a TCP connection that dies silently (no FIN/RST from the server — NAT timeout, network flap) never emits events.Disconnected: keepalive pings just keep timing out and the instance stays flagged as connected while being unable to send or receive. Only a manual restart fixes it. whatsmeow's own docs on KeepAliveTimeout suggest exactly this: "Clients may use this event to decide to force a disconnect+reconnect faster."

This handles KeepAliveTimeout and KeepAliveRestored:

  • both are forwarded to the CONNECTION webhook subscription so operators can observe socket health;
  • after 3 consecutive timeouts the instance is restarted through the same non-blocking ReconnectClient path the Disconnected handler already uses (exact-match on the count so an ongoing restart isn't duplicated).

We run multiple production instances behind this and it eliminated our "zombie instance" incidents (instance reported connected, no traffic flowing, external watchdog required).

Summary by Sourcery

Handle keepalive timeout and restoration events by exposing them as CONNECTION events and triggering controlled instance restarts after repeated timeouts to avoid zombie connections when auto-reconnect is disabled.

New Features:

  • Forward KeepAliveTimeout and KeepAliveRestored events to CONNECTION subscriptions and global queues for visibility into socket health.

Bug Fixes:

  • Automatically restart instances after three consecutive keepalive timeouts when auto-reconnect is disabled to recover silently dead TCP connections.

Enhancements:

  • Add structured logging around keepalive timeouts and restorations to aid monitoring and troubleshooting.

With EnableAutoReconnect disabled, a TCP connection that dies silently
(no FIN/RST from the server) never emits events.Disconnected: keepalive
pings just keep timing out and the instance stays flagged as connected
while being unable to send or receive — a zombie that only a manual
restart fixes. whatsmeow's docs explicitly suggest using KeepAliveTimeout
to force a faster disconnect+reconnect.

Handle KeepAliveTimeout and KeepAliveRestored: both are forwarded to the
CONNECTION webhook subscription so operators can observe socket health,
and after 3 consecutive timeouts the instance is restarted through the
same non-blocking ReconnectClient path the Disconnected handler already
uses (exact-match on the count so ongoing restarts aren't duplicated).
@sourcery-ai

sourcery-ai Bot commented Jul 24, 2026

Copy link
Copy Markdown

Reviewer's Guide

Handle KeepAlive timeout/restoration events to surface socket health via CONNECTION webhooks and automatically restart zombie instances after repeated keepalive failures.

Sequence diagram for KeepAliveTimeout handling and automatic reconnect

sequenceDiagram
    participant WhatsAppServer
    participant MyClient
    participant WhatsmeowService
    participant WebhookSubscriber

    WhatsAppServer->>MyClient: events.KeepAliveTimeout
    MyClient->>MyClient: myEventHandler(KeepAliveTimeout)
    MyClient->>WhatsmeowService: CallWebhook(eventType=KeepAliveTimeout)
    WhatsmeowService->>WebhookSubscriber: sendToQueueOrWebhook(globalEventType=CONNECTION)

    alt [ErrorCount == 3]
        MyClient->>MyClient: LogWarn("3 consecutive keepalive timeouts")
        MyClient->>WhatsmeowService: ReconnectClient(instanceID)
    end

    WhatsAppServer->>MyClient: events.KeepAliveRestored
    MyClient->>MyClient: myEventHandler(KeepAliveRestored)
    MyClient->>WhatsmeowService: CallWebhook(eventType=KeepAliveRestored)
    WhatsmeowService->>WebhookSubscriber: sendToQueueOrWebhook(globalEventType=CONNECTION)
Loading

File-Level Changes

Change Details Files
Handle KeepAliveTimeout events, including webhook emission, logging, and conditional instance restart after repeated failures.
  • Add KeepAliveTimeout case to myEventHandler that marks the event for webhook delivery and populates event payload with error count and last success timestamp.
  • Log a warning on each KeepAliveTimeout with the consecutive error count and last successful keepalive time in RFC3339 format.
  • Trigger a non-blocking instance restart via ReconnectClient when the keepalive error count reaches exactly 3, reusing the Disconnected handler restart path and guarding against duplicate restarts.
pkg/whatsmeow/service/whatsmeow.go
Handle KeepAliveRestored events and classify both KeepAliveTimeout and KeepAliveRestored as CONNECTION events for routing to the appropriate webhooks and global queues.
  • Add KeepAliveRestored case to myEventHandler that marks the event for webhook delivery and logs an informational message when keepalive pings resume.
  • Extend CallWebhook event-type routing so KeepAliveTimeout and KeepAliveRestored are sent to CONNECTION subscriptions.
  • Extend SendToGlobalQueues routing (for both variants) to classify KeepAliveTimeout and KeepAliveRestored under the CONNECTION global event type.
pkg/whatsmeow/service/whatsmeow.go

Tips and commands

Interacting with Sourcery

  • Trigger a new review: Comment @sourcery-ai review on the pull request.
  • Continue discussions: Reply directly to Sourcery's review comments.
  • Generate a GitHub issue from a review comment: Ask Sourcery to create an
    issue from a review comment by replying to it. You can also reply to a
    review comment with @sourcery-ai issue to create an issue from it.
  • Generate a pull request title: Write @sourcery-ai anywhere in the pull
    request title to generate a title at any time. You can also comment
    @sourcery-ai title on the pull request to (re-)generate the title at any time.
  • Generate a pull request summary: Write @sourcery-ai summary anywhere in
    the pull request body to generate a PR summary at any time exactly where you
    want it. You can also comment @sourcery-ai summary on the pull request to
    (re-)generate the summary at any time.
  • Generate reviewer's guide: Comment @sourcery-ai guide on the pull
    request to (re-)generate the reviewer's guide at any time.
  • Resolve all Sourcery comments: Comment @sourcery-ai resolve on the
    pull request to resolve all Sourcery comments. Useful if you've already
    addressed all the comments and don't want to see them anymore.
  • Dismiss all Sourcery reviews: Comment @sourcery-ai dismiss on the pull
    request to dismiss all existing Sourcery reviews. Especially useful if you
    want to start fresh with a new review - don't forget to comment
    @sourcery-ai review to trigger a new review!

Customizing Your Experience

Access your dashboard to:

  • Enable or disable review features such as the Sourcery-generated pull request
    summary, the reviewer's guide, and others.
  • Change the review language.
  • Add, remove or edit custom review instructions.
  • Adjust other review settings.

Getting Help

@sourcery-ai sourcery-ai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hey - I've left some high level feedback:

  • The restart-on-timeout logic currently triggers purely on ErrorCount == 3; if this is intended only when EnableAutoReconnect is disabled, consider explicitly checking that flag to avoid overlapping with the built-in reconnect behavior.
  • The threshold of 3 consecutive keepalive timeouts is hard-coded; consider making this configurable so operators can tune sensitivity based on their deployment characteristics.
  • For KeepAliveTimeout/KeepAliveRestored, you might want to include consistent, richer data in the webhook payload (e.g., last success timestamp and error count for restored events as well) to make monitoring and alerting easier.
Prompt for AI Agents
Please address the comments from this code review:

## Overall Comments
- The restart-on-timeout logic currently triggers purely on ErrorCount == 3; if this is intended only when EnableAutoReconnect is disabled, consider explicitly checking that flag to avoid overlapping with the built-in reconnect behavior.
- The threshold of 3 consecutive keepalive timeouts is hard-coded; consider making this configurable so operators can tune sensitivity based on their deployment characteristics.
- For KeepAliveTimeout/KeepAliveRestored, you might want to include consistent, richer data in the webhook payload (e.g., last success timestamp and error count for restored events as well) to make monitoring and alerting easier.

Sourcery is free for open source - if you like our reviews please consider sharing them ✨
Help me be more useful! Please click 👍 or 👎 on each comment and I'll use the feedback to improve your reviews.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant