[Phase 9] Design and implement Binary Links Notation protocol

[konard]

Summary

Design and implement a high-performance binary encoding of Links Notation for efficient data exchange.

Background

The text-based Links Notation is human-readable but not optimal for high-throughput scenarios. Binary notation aims for 5-10x smaller message sizes and zero-copy parsing where possible.

Requirements

Wire Format Design

┌────────┬────────┬────────┬─────────────────────────────────┐
│ Magic  │Version │ Length │           Payload                │
│ 4 bytes│ 2 bytes│ 4 bytes│         Variable                 │
└────────┴────────┴────────┴─────────────────────────────────┘

Link Encoding

┌────────┬────────────┬────────────┐
│  Type  │   Source   │   Target   │
│ 1 byte │  Variable  │  Variable  │
└────────┴────────────┴────────────┘

Type Flags:
• 0x01 - Source is link ID (vs literal)
• 0x02 - Target is link ID (vs literal)
• 0x04 - Self-referencing (named link)
• 0x08 - Has additional metadata

Features

• Versioning: Protocol version for backward compatibility
• Compression: Optional zstd/lz4 compression
• Forward/Backward Compatibility: New fields ignored by old clients
• Zero-Copy Parsing: Direct memory access where possible
• Buffer Pooling: Reuse allocated buffers

Implementation

import { BinaryNotation } from 'links-queue';

// Encode
const buffer = BinaryNotation.encode(links, {
  compress: true,
  version: 1
});

// Decode
const links = BinaryNotation.decode(buffer);

// Stream encoding
const encoder = new BinaryNotation.StreamEncoder();
encoder.write(link1);
encoder.write(link2);
const buffer = encoder.finish();

Protocol Selection

const client = new LinksQueueClient({
  protocol: 'auto',  // Negotiate best available
  // or: 'binary', 'text'
});

Performance Goals

• 5-10x reduction in message size vs text notation
• Sub-microsecond encoding/decoding for simple links
• No memory allocation for decode (where possible)

Deliverables

1. Binary format specification document
2. JavaScript encoder/decoder
3. Rust encoder/decoder
4. Protocol negotiation mechanism
5. Benchmark comparison with text notation
6. Migration guide from text to binary

Dependencies

• [Phase 4] Integrate links-notation for parsing and serialization #16 - Text Links Notation (for fallback)

References

• ARCHITECTURE.md - Binary Links Notation section
• REQUIREMENTS.md - REQ-PROTO-010 through REQ-PROTO-013
• ROADMAP.md - Phase 9: Binary Protocol