Skip to content

fix: handle fragmented ClientHello in auto-sense server#87

Merged
algesten merged 3 commits intomainfrom
fix/dtls13-fragmented-client-hello
Mar 9, 2026
Merged

fix: handle fragmented ClientHello in auto-sense server#87
algesten merged 3 commits intomainfrom
fix/dtls13-fragmented-client-hello

Conversation

@xnorpx
Copy link
Collaborator

@xnorpx xnorpx commented Mar 8, 2026

Firefox 148+ sends X25519MLKEM768 as its primary key_share, making the ClientHello too large for a single DTLS record. The supported_versions extension ends up in a later fragment, causing the auto-sense server to miss it and fall back to DTLS 1.2.

Replace the lightweight packet-sniffing version detection (ServerPending) with a design that uses Server13 directly in auto mode. The DTLS 1.3 engine handles fragment reassembly natively. After reassembling the complete ClientHello:

  • If supported_versions contains DTLS 1.3: proceed as 1.3 (zero overhead)
  • If not: return Error::Dtls12Fallback, create Server12, replay packets

Changes:

  • Add Error::Dtls12Fallback variant
  • Add auto_mode + auto_packets to Server13
  • Remove ServerPending from Inner enum and detect.rs
  • new_auto() creates Server13::new_auto(), falls back to Server12 on Dtls12Fallback or ParseError
  • Server13 buffers raw packets during AwaitClientHello, frees on commit
  • 47 new tests: server_fallback (15) + cross_matrix (32) covering all version combinations at normal/fragmented/heavy MTU

@xnorpx
Copy link
Collaborator Author

xnorpx commented Mar 8, 2026

Fixes #88

@algesten algesten force-pushed the fix/dtls13-fragmented-client-hello branch from 501cb22 to e09a4bc Compare March 9, 2026 08:54
xnorpx and others added 3 commits March 9, 2026 09:58
Firefox 148+ sends X25519MLKEM768 as its primary key_share, making the
ClientHello too large for a single DTLS record. The supported_versions
extension ends up in a later fragment, causing the auto-sense server to
miss it and fall back to DTLS 1.2.

Replace the lightweight packet-sniffing version detection (ServerPending)
with a design that uses Server13 directly in auto mode. The DTLS 1.3
engine handles fragment reassembly natively. After reassembling the
complete ClientHello:

- If supported_versions contains DTLS 1.3: proceed as 1.3 (zero overhead)
- If not: return Error::Dtls12Fallback, create Server12, replay packets

Changes:
- Add Error::Dtls12Fallback variant
- Add auto_mode + auto_packets to Server13
- Remove ServerPending from Inner enum and detect.rs
- new_auto() creates Server13::new_auto(), falls back to Server12 on
  Dtls12Fallback or ParseError
- Server13 buffers raw packets during AwaitClientHello, frees on commit
- 47 new tests: server_fallback (15) + cross_matrix (32) covering all
  version combinations at normal/fragmented/heavy MTU
@algesten algesten force-pushed the fix/dtls13-fragmented-client-hello branch from e09a4bc to 6e819be Compare March 9, 2026 09:00
@algesten algesten merged commit 55a2f47 into main Mar 9, 2026
46 checks passed
@algesten algesten deleted the fix/dtls13-fragmented-client-hello branch March 9, 2026 09:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants