UDP socket continues to use potentially broken socket after error

[jcantrill]

A note for the community

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Problem

With v0.47, sending over a socket using UDP would handle errors (i.e. os error 111) from receiver restarts, by recreating the socket. The meant that vector would fail over to the new receiver when it was available and continue to emit messages. Currently, since UDP is a protocol that by nature returns no indicators of successful delivery, it is possible for vector to continue to write to a socket that is in a bad state.

Vector logs a message like:

2026-07-10T15:33:43.319328Z ERROR sink{component_kind="sink" component_id=output_rsyslog component_type=socket}: vector::internal_events::socket: Error sending data. error=Connection refused (os error 111) error_code="socket_send" error_type="writer_failed" stage="sending" mode=udp

• It generates discarded and error metrics at this point in time
• Afterwards it continues to generate send metrics with no discard or error (which is expected given its UDP)

Configuration

[sinks.example]
type = "socket"
inputs = ["example_parse_encoding"]
address = "logserver:514"
mode = "udp"

Version

v0.54

Debug Output

Example Data

Steps to Reproduce:

1. Deploy a syslog server pod in a kubernetes cluster with a NodePort Service
2. Configure vector to forward logs to the service using a UDP socket sink
3. Verify the receiver is receiving logs
4. Delete the syslog receiver pod
5. Verify the receiver is receiving logs

Note: The referenced issue stated it was necessary to debug the node and 'we had to clear conntrack manually to resume syslog forwarding.'

Additional Context

Regression introduced by: #23728

Claude assesssment:

  Before commit a98693c625 (the GELF chunking commit):
  - When a UDP send error occurred:
    - The error was logged via emit!(SocketSendError)
    - The event was marked as EventStatus::Errored
    - The function immediately returned (return;), exiting the loop
    - This caused the socket to be recreated by the outer loop in udp.rs::run()
    - Subsequent events would use the new socket

  At commit a98693c625 (current HEAD - GELF chunking):
  - When a UDP send error occurs:
    - The error is logged via emit!(SocketSendError)
    - The event is marked as EventStatus::Errored
    - The function continues processing (continue is implicit, no return) with the same socket
    - The socket is NOT recreated on error
    - All subsequent events continue using the same (potentially broken) socket
    - Socket recreation only happens when the entire input stream ends naturally

  Key Behavioral Change:

  The introduction of GELF chunking support in a98693c625 removed the socket recreation behavior on send errors.

  - Before a98693c625: Send error → return → socket recreated → subsequent sends succeed if the issue was transient
  - At a98693c625: Send error → continue with same socket → all subsequent sends likely fail if socket is in a bad state (e.g., ICMP Port Unreachable cached in kernel)

  This is actually a regression - the automatic recovery mechanism was lost when chunking support was added. The code now just logs errors and keeps trying with the same broken socket,
  which won't recover from kernel-cached ICMP errors until the entire stream ends.

References

• Originally Reported as part of https://redhat.atlassian.net/browse/LOG-9560
• Regression was introduced as part of feat(codecs): Support GELF chunking for encoding #23728

[jcantrill]

cc @vparfonov