feat(dtls): verify peer certificate fingerprint (RFC 5763) + no-downgrade gate

[16KnighT]

Peer-certificate fingerprint verification (RFC 5763) + no-downgrade gate

Security fix. DTLS-SRTP channels previously accepted any peer certificate: insecure_skip_verify: true was set and the SDP a=fingerprint was parsed into RemoteDtls.fingerprint but never used. A man-in-the-middle on the media path could therefore complete the DTLS handshake with its own certificate and read/modify the media. This PR authenticates the peer against the SDP fingerprint and refuses to fall back to plaintext when that fails.

Stacking: branched off feat/tier2-rtcp (PR #163, itself stacked on Tier 1 #162) because the RTCP send path (rtcp_tx::send_compound) is one of the plaintext sites the no-downgrade gate must cover. Merge after #162 and #163; until they land the diff here also shows those commits.

Verification

• New dtls_session::PeerFingerprint: parse accepts the full "<algorithm> <hex>" SDP form or a bare colon-hex string (sha-256 assumed, matching what the JS layer forwards today); matches_der computes the SHA-256 of the peer's leaf certificate DER and compares it (case-insensitive) to the expected value. Unknown algorithms fail closed (only sha-256 is honoured — projectrtp's and modern WebRTC's hash).
• spawn_handshake now takes Option<PeerFingerprint>. When present it installs a verify_peer_certificate callback that aborts the handshake (BadCertificate alert) on mismatch. insecure_skip_verify stays true — self-signed WebRTC certs have no CA, so the fingerprint is the authentication.
• Sets client_auth = RequireAnyClientCert so the passive/server role sends a CertificateRequest and actually receives the peer cert — without it the server would never see a certificate to fingerprint.
• Policy: verify when a fingerprint is supplied (all real WebRTC/SIP traffic carries one); a channel that opts into DTLS with an empty fingerprint stays unverified, as before — backward-compatible, and it keeps the fingerprint-less internal handshake test working.

No-downgrade gate (the part that gives verification teeth)

Without this, a rejected handshake silently fell back to plaintext RTP/RTCP — so a MITM whose cert failed the check would still receive the media in the clear, making the fingerprint check cosmetic.

• ChannelState::secure_not_ready() = "DTLS negotiated but no SRTP keys yet". It guards every plaintext send site — both send_rtp paths (tick.rs, mixer.rs), the mixer send_leg inline send, and rtcp_tx::send_compound — so a failed/incomplete handshake withholds media instead of downgrading. No effect on non-DTLS (plain SIP) channels: secure_not_ready() is always false there.

Tests / verification

• Rust (dtls_session::tests): fingerprint parse (bare hex / algorithm-prefixed / empty→None), matches_der (own cert, case-insensitive, flipped digit rejected, unknown algorithm fails closed), and two full loopback handshakes — matching fingerprints complete, a mismatch aborts both sides. cargo test --lib 115 passing; clippy --all-targets + rustfmt clean.
• JS (test/interface/projectrtpdtls.js): new "rejects a peer whose certificate fingerprint does not match SDP" — feeds a corrupted fingerprint and asserts stats.in.count === 0 on both channels (the gate withholds all media). The existing DTLS/secure-RTCP tests (which feed the correct fingerprint) now prove the happy path end-to-end, since fingerprints are actually enforced.
• Full npm test: 150 passing / 6 pending / 0 failing.

Out of scope / follow-ups

• Inbound plaintext injection isn't explicitly rejected (a DTLS channel with no srtp_decrypt simply fails to decrypt SRTP, so injected cleartext is dropped at parse — but there's no explicit reject/close).
• insecure_verification stays default; only sha-256 fingerprints are honoured.

🤖 Generated with Claude Code