Persistent dedupe sidecar: per-segment .idx for fast recovery (closes last Phase A item)

[pbudzik]

Summary

Closes the last open item from Phase A (the external reviewer's "correctness contract"): persistent dedupe index. Each raw / compacted segment now writes a sidecar .idx file with (event_id_hash, payload_hash, ingested_at_ms) triples in segment row order, and recovery uses it to rebuild the hot dedupe cache without re-decoding events.

Non-breaking

The sidecar is an optimization, not a source of truth. Recovery:

1. Tries raw_<uuid>.idx first → loads triples directly into dedupe.
2. On missing or corrupt sidecar → falls back to RawSegmentReader::new + per-event hashing (the existing path).

Segments from before this PR have no sidecar; they continue to recover via the fallback. No format version bump.

File format

magic     b"UDBIDX01"  (8 bytes)
count     u32 LE
entries   bincode Vec<(u128, u128, i64)>
checksum  u64 LE       (blake3 of everything above)

Validated on read: magic + checksum + count consistency. Any failure causes the caller to fall back to the segment scan.

Win

Currently recovery scans every event in every segment whose max_timestamp_ms is within the dedupe TTL window (7 days). For a 1M-event segment with realistic payloads that's a bincode + zstd decode of every event. The sidecar avoids that — it's a single read_to_end + bincode deserialize of 32 bytes/event.

Recovery logs sidecar hit / miss counts so operators can spot incomplete coverage (e.g., after rolling back to a build that didn't write sidecars).

Plumbing

• New src/storage/dedupe_index.rs with write_dedupe_index + read_dedupe_index + index_path helpers.
• flusher.write_bucket writes the sidecar after the segment, before the manifest commit. Failure is non-fatal — log + remove partial.
• compact/worker.run_one_plan does the same for compacted outputs.
• runtime/recovery tries the sidecar first; falls back on Ok(None) or Err.

Tests

8 new tests in tests/dedupe_index.rs:

• Round-trip with u128::MAX and i64::MAX boundary values
• Ok(None) on missing file (fall-back signal)
• Corruption rejection: byte-flip checksum, truncation, missing magic
• Recovery picks up dedupe via the fast path
• Recovery falls back to segment scan without a sidecar
• DedupeEntry type alias compiles to (EventHash, EventHash, i64)

Test plan

• cargo build --all-targets clean with -D warnings
• cargo test --all-targets — 122 tests pass (was 114; +8)
• CI green
• Bump PR Bump Cargo version to 0.4.0 #20 merged first (Cargo version is 0.3.0 on this branch; will rebase or auto-conflict on Cargo.toml line)

🤖 Generated with Claude Code