Unbounded message length on network receive (MessageHandler::recv in libutil)

[jussiohag]

Summary

On the receive path in libutil (libs/network/src/MessageHandler.cpp, MessageHandler::recv), the message length taken from the wire is only checked for being negative — there is no upper bound. The 64 KB cap that exists on the send side is not mirrored on receive. (Filing here since libutil has issues disabled.)

external/libutil/libs/network/src/MessageHandler.cpp, around lines 135 / 143 / 152 / 174:

• L135: if(header.msgLen < 0) throw ...; — only rejects negative, nothing caps the maximum.
• L174: sock.Recv(ser.GetDataWritable(header.msgLen), header.msgLen); — allocates up to ~2 GB from a 4-byte field before any data arrives.
• L143 / L152: static_cast<int>(header.msgLen + sizeof(header)) — msgLen is summed with a size_t then narrowed to int, so a large msgLen truncates/overflows and corrupts the "enough bytes waiting" guard.

Trigger

A peer (or anything speaking the protocol) sends a valid 6-byte header claiming a very large msgLen. The receiver then attempts a huge allocation (memory-exhaustion / bad_alloc), and the truncating cast makes the length bookkeeping unreliable. This is reachable before any authentication of the payload.

Suggested fix

Reject msgLen > 64*1024 on receive, mirroring the existing send-side cap, before calling GetDataWritable. Also compute the "bytes waiting" comparison in a width that can't truncate (compare as size_t / against msgLen directly) rather than casting the sum to int.

This is a defensive-hardening report (denial-of-service / robustness on malformed input), not a demonstrated remote code execution. Happy to send a PR.

[Flamefire]

I guess a check at https://github.com/Return-To-The-Roots/libutil/blob/bb045875c9a5444948cc3af29895e27eb7272557/libs/network/src/MessageHandler.cpp#L135 would be sufficient: It should ensure that header.msgLen + sizeof(header) doesn't overflow or use unsigned arithmetic using size_t for read >= static_cast<int>(header.msgLen + sizeof(header) as read is known to be non-negative.

But capping the msg size at say 1GB (with an assertion that int can hold that) should be fine too together with setting an error