feat(connlib): request larger buffers for UDP sockets (#8731)

Sufficiently large receive buffers are important to sustain
high-throughput as latency increases. If the receive buffer in the
kernel is too small, packets need to be dropped on arrival.

Firefox uses 1MB in its QUIC stack [0]. `quic-go` recommends to set send
and receive buffers to 7.5 MB [1]. Power users of Firezone are likely
receiving a lot more traffic than the average Firefox user (especially
with Internet Resource activated) so setting it to 10 MB seems
reasonable. Sending packets is likely not as critical because we have
back-pressure through our system such that we will stop reading IP
packets when we cannot write to our UDP socket. The UDP socket is
sitting in a separate thread and those threads are connected with
dedicated queues which act as another buffer. However, as the data below
shows, some systems have really small send buffers which are currently
likely a speed bottleneck because we need to suspend writing so
frequently.

Assuming a 50ms latency, the bandwidth-delay product tells us that we
can (in theory) saturate a 1.6 Gbps link with a 10MB receive buffer
(assuming the OS also has large enough buffer sizes in its TCP or QUIC
stack):

```
80 Mb / 0.05s = 1600Mbps
```

Experiments and research [2] show the following:

|OS|Receive buffer (default)|Receive buffer (this PR)|Send buffer
(default)|Send buffer (this PR)|
|---|---|---|---|---|
|Windows|65KB|10MB|65KB|1MB|
|MacOS|786KB|8MB|9KB|1MB|
|Linux|212KB|212KB|212KB|212KB|

With the exception of Linux, the OSes appear to be quite generous with
how big they allow receive buffers to be. On Linux, these limit can be
changed by setting the `core.net.rmem_max` and `core.net.wmem_max`
parameters using `sysctl`.

Most of our users are on Windows and MacOS, meaning they immediately
benefit from this without having to change any system settings. Larger
client-side UDP receive buffers are critical for any "download" scenario
which is likely the majority of usecases that Firezone is used for.

On Windows, increasing this receive buffer almost doubles the throughput
in an iperf3 download test.

[0]: mozilla/neqo#2470
[1]: https://github.com/quic-go/quic-go/wiki/UDP-Buffer-Sizes
[2]: https://unix.stackexchange.com/a/424381

---------

Signed-off-by: Thomas Eizinger <thomas@eizinger.io>
Co-authored-by: Jamil <jamilbk@users.noreply.github.com>

Author: thomaseizinger

rust/connlib/tunnel/src/sockets.rs

@@ -173,7 +173,7 @@ impl ThreadedUdpSocket {
                     .build()
                     .expect("Failed to spawn tokio runtime on UDP thread")
                     .block_on(async move {
-                        let socket = match sf(&addr) {
+                        let mut socket = match sf(&addr) {
                             Ok(s) => {
                                 let _ = error_tx.send(Ok(()));
 
@@ -185,6 +185,13 @@ impl ThreadedUdpSocket {
                             }
                         };
 
+                        match socket.set_buffer_sizes(socket_factory::SEND_BUFFER_SIZE, socket_factory::RECV_BUFFER_SIZE) {
+                            Ok(()) => {},
+                            Err(e) => {
+                                let _ = error_tx.send(Err(e));
+                                return;
+                            },
+                        }
 
                         let send = pin!(async {
                             while let Ok(datagram) = outbound_rx.recv_async().await {

rust/socket-factory/src/lib.rs

@@ -23,6 +23,10 @@ use tokio::io::Interest;
 
 pub trait SocketFactory<S>: Fn(&SocketAddr) -> io::Result<S> + Send + Sync + 'static {}
 
+pub const SEND_BUFFER_SIZE: usize = ONE_MB;
+pub const RECV_BUFFER_SIZE: usize = 10 * ONE_MB;
+const ONE_MB: usize = 1024 * 1024;
+
 impl<F, S> SocketFactory<S> for F where F: Fn(&SocketAddr) -> io::Result<S> + Send + Sync + 'static {}
 
 pub fn tcp(addr: &SocketAddr) -> io::Result<TcpSocket> {
@@ -183,6 +187,28 @@ impl UdpSocket {
         })
     }
 
+    pub fn set_buffer_sizes(
+        &mut self,
+        requested_send_buffer_size: usize,
+        requested_recv_buffer_size: usize,
+    ) -> io::Result<()> {
+        let socket = socket2::SockRef::from(&self.inner);
+
+        socket.set_send_buffer_size(requested_send_buffer_size)?;
+        socket.set_recv_buffer_size(requested_recv_buffer_size)?;
+
+        let send_buffer_size = socket.send_buffer_size()?;
+        let recv_buffer_size = socket.recv_buffer_size()?;
+
+        tracing::info!(%requested_send_buffer_size, %send_buffer_size, %requested_recv_buffer_size, %recv_buffer_size, port = %self.port, "Set UDP socket buffer sizes");
+
+        Ok(())
+    }
+
+    pub fn port(&self) -> u16 {
+        self.port
+    }
+
     /// Configures a new source IP resolver for this UDP socket.
     ///
     /// In case [`DatagramOut::src`] is [`None`], this function will be used to set a source IP given the destination IP of the datagram.

website/src/app/kb/deploy/gateways/readme.mdx

@@ -125,6 +125,17 @@ Firezone's
 [automatic load balancing](/kb/architecture/critical-sequences#high-availability)
 to distribute Client connections across them.
 
+### Performance tuning
+
+The default receive buffer size on Linux is quite small which can limit the
+maximum throughput that users perceive in "upload scenarios" (i.e. where the
+Gateway needs to receive large volumes of traffic).
+
+On startup, the Gateway attempts to increase the size of the UDP receive buffers
+to 10 MB. However, the actual size of the receive buffer is limited by the
+`net.core.rmem_max` kernel parameter. For the increased buffer size to take
+effect, you may need to increase the `net.core.rmem_max` parameter on the
+Gateway's host system.
 ## Deploy a single Gateway
 
 Deploying a single Gateway can be accomplished in the admin portal.

website/src/components/Changelog/Apple.tsx

@@ -23,7 +23,14 @@ export default function Apple() {
   return (
     <Entries downloadLinks={downloadLinks} title="macOS / iOS">
       {/* When you cut a release, remove any solved issues from the "known issues" lists over in `client-apps`. This must not be done when the issue's PR merges. */}
-      <Unreleased></Unreleased>
+      <Unreleased>
+        <ChangeItem pull="8731">
+          Improves throughput performance by requesting socket receive buffers
+          of 10MB. The actual size of the buffers is capped by the operating
+          system. You may need to adjust <code>kern.ipc.maxsockbuf</code> for this to take
+          full effect.
+        </ChangeItem>
+      </Unreleased>
       <Entry version="1.4.12" date={new Date("2025-04-21")}>
         <ChangeItem pull="8798">
           Improves performance of relayed connections on IPv4-only systems.

website/src/components/Changelog/GUI.tsx

@@ -8,7 +8,22 @@ export default function GUI({ os }: { os: OS }) {
   return (
     <Entries downloadLinks={downloadLinks(os)} title={title(os)}>
       {/* When you cut a release, remove any solved issues from the "known issues" lists over in `client-apps`. This must not be done when the issue's PR merges. */}
-      <Unreleased></Unreleased>
+      <Unreleased>
+        {os === OS.Linux && (
+          <ChangeItem pull="8731">
+            Improves throughput performance by requesting socket receive buffers
+            of 10MB. The actual size of the buffers is capped by the operating
+            system. You may need to adjust <code>net.core.rmem_max</code> for this to take
+            full effect.
+          </ChangeItem>
+          )}
+        {os === OS.Windows && (
+          <ChangeItem pull="8731">
+            Improves throughput performance by requesting socket receive buffers
+            of 10MB.
+          </ChangeItem>
+          )}
+      </Unreleased>
       <Entry version="1.4.11" date={new Date("2025-04-21")}>
         <ChangeItem pull="8798">
           Improves performance of relayed connections on IPv4-only systems.

website/src/components/Changelog/Gateway.tsx

@@ -26,6 +26,12 @@ export default function Gateway() {
         <ChangeItem pull="8798">
           Improves performance of relayed connections on IPv4-only systems.
         </ChangeItem>
+        <ChangeItem pull="8731">
+          Improves throughput performance by requesting socket receive buffers
+          of 10MB. The actual size of the buffers is capped by the operating
+          system. You may need to adjust <code>net.core.rmem_max</code> for this to take
+          full effect.
+        </ChangeItem>
       </Unreleased>
       <Entry version="1.4.6" date={new Date("2025-04-15")}>
         <ChangeItem pull="8383">

website/src/components/Changelog/Headless.tsx

@@ -13,6 +13,20 @@ export default function Headless({ os }: { os: OS }) {
         <ChangeItem pull="8798">
           Improves performance of relayed connections on IPv4-only systems.
         </ChangeItem>
+        {os === OS.Linux && (
+          <ChangeItem pull="8731">
+            Improves throughput performance by requesting socket receive buffers
+            of 10MB. The actual size of the buffers is capped by the operating
+            system. You may need to adjust <code>net.core.rmem_max</code> for this to take
+            full effect.
+          </ChangeItem>
+          )}
+        {os === OS.Windows && (
+          <ChangeItem pull="8731">
+            Improves throughput performance by requesting socket receive buffers
+            of 10MB.
+          </ChangeItem>
+          )}
       </Unreleased>
       <Entry version="1.4.6" date={new Date("2025-04-15")}>
         {os == OS.Linux && (