Harden durability, backups, and synchronous replication

[script3r]

Summary

• add atomic cluster-wide checkpoint export with binary SHA-256 integrity manifests and verified retention
• require explicit non-overlapping checkpoint storage for persistent shard binaries and update every shipped deployment path
• add synchronous primary-to-passive shard replication for every durable mutation with authenticated forwarding and mTLS by default
• serialize realistic ingest batches deterministically so primary and replica assign identical durable IDs
• fail closed after ambiguous replication or local durable-write failures, while validating imports before mutation
• complete admitted unary and streamed ingests in bounded owned tasks so client cancellation cannot strand a WAL or latch a healthy shard closed
• bound gRPC messages and per-connection concurrency, enable load shedding, and verify the production Containerfile in CI
• paginate dirty boundary signatures, process filtered metadata in bounded chunks, and globally canonicalize candidate merges without skipping entity resolution
• bump the distributed protocol to v5 so mixed reconciliation contracts fail closed
• document manual fenced failover, backup/restore drills, recovery complexity, cancellation semantics, and operational limitations
• verify acknowledged writes through process SIGKILL, restart, primary-volume deletion, replica promotion, cancellation, and idempotent retry

Verification

• cargo test --locked --all-features
• cargo clippy --locked --all-targets --all-features -- -D warnings
• cargo +1.88.0 check --locked --all-targets --all-features
• cargo check --locked --all-targets --all-features
• cargo fmt -- --check
• cargo package --locked --allow-dirty --no-verify
• bash -n scripts/*.sh
• docker compose -f compose.yaml config --quiet
• cargo bench --bench bench_quick (persistent ingest about 151K records/sec)
• clean five-shard persistent run: 1,000,000/1,000,000 records acknowledged, zero ingest errors, full reconciliation, 46.3K records/sec including reconciliation

Capacity finding

The historical ~410K records/sec baseline was measured while reconciliation polling could fail or exceed the gRPC message limit. After bounding metadata retention and preserving global merge/conflict canonicalization, this machine sustained 46.3K records/sec at 1M records. Summed per-process peak RSS fell from about 8.6 GiB to 6.6 GiB, with the router peaking at 321 MiB; roughly 6.3 GiB remains in the five correctness-critical shard working sets. The 10M-record target is therefore still not production-qualified on a 32 GiB host. Target-hardware capacity and restart-time soaks remain required before claiming that scale.