Skip to content

feat: Client auto-sense fails if a DTLS 1.3 ServerHello is fragmented (e.g., PQ keys + low MTU) #86

@xnorpx

Description

@xnorpx

Description:
The client-side DTLS version auto-sense relies on a lightweight, stateless packet sniffer (server_hello_version_inner in src/detect.rs) to inspect the server's first response. This parser assumes the supported_versions extension is fully present in the first arrived UDP datagram.

It does not process DTLS fragmentation. Specifically:

  1. It does not check fragment_offset.
  2. If ext_end > body.len() (truncation due to fragmentation), it instantly aborts the search and falls back to DTLS 1.2.

Because the subsequent fragments belong to a 1.3 handshake, the resulting Client12 state machine throws a parse error and drops the connection.

Likelihood & Impact:

  • Standard Cryptography: Likelihood is near 0%. Standard ServerHello and HelloRetryRequest messages are tiny (~80-120 bytes) and will never fragment.
  • Post-Quantum Cryptography: Likelihood is moderate on constrained networks. An ML-KEM-768 key share puts the ServerHello at roughly ~1168 bytes. While this fits standard IPv6 minimum MTUs (1280), if a client is on a network with a heavily constrained MTU (e.g., under 1150 bytes due to VPNs or IPsec tunnels), the router/server will fragment the ServerHello.

In these specific low-MTU + PQ scenarios, client auto-detect will consistently fail the handshake.

Why it's a gap (and hard to fix):
Unlike the server-side fix, we cannot easily delegate this to the Client13 engine because the client hasn't decided which version state machine to initialize. Fixing this would require building a stateful DTLS fragment reassembly buffer directly inside ClientPending just to read the extensions block.

Potential Mitigations:

  1. Accept the limitation: Let the handshake predictably fail on ultra-low MTU networks when PQ cryptography is involved.
  2. Stateful Sniffer: Add a small reassembly buffer to ClientPending that waits for all fragments of message_seq == 0 before making the version decision.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions