Harden persistent ingest and prepare 0.2.0 release

[script3r]

Summary

• remove ingest paths that bypassed entity resolution or acknowledged data held only in in-memory partition stores
• make persistent ingest acknowledgements power-loss durable by syncing both the external request WAL and the RocksDB WAL in the correct order
• frame ingest WALs with version, length, and CRC32 validation; fail startup closed while preserving corrupt recovery evidence
• persist cross-shard merge redirects, make reset clear all resolution state atomically, and sync checkpoints before creation
• propagate persistence and linking failures, make staged batches idempotent, and fix reset/interner recovery edge cases
• modernize to Rust 2024 with MSRV 1.88, update dependencies, track Cargo.lock, and tighten CI/release gates
• correct load-test accounting and publish a durability-correct persistent baseline

Critical correctness findings addressed

Persistent shards previously routed large batches into independent in-memory partition stores. The request was acknowledged, but the primary PersistentStore used by queries and restart recovery never received those records. The partitioned module also exposed an ultra-fast API that skipped both storage and entity resolution. This PR removes that API and confines the current partition implementation to explicitly non-persistent shards.

The external ingest WAL was previously removed after asynchronous RocksDB writes, leaving a power-loss window where an acknowledged request could disappear from both recovery layers. Persistent ingest now writes and fsyncs a checksummed binary request WAL, resolves and persists the batch, synchronously flushes the RocksDB WAL, then removes and directory-syncs the request WAL before acknowledging.

Corrupt or truncated request WALs now fail shard startup with a data-loss error and are quarantined for operator recovery. Cross-shard merge redirects survive restart, checkpoints sync the WAL, and persistent reset clears record, DSU, tiered-index, and linker state in one RocksDB write batch.

Compatibility

• package version: 0.2.0
• edition: Rust 2024
• MSRV: Rust 1.88
• legacy unframed ingest WALs remain replayable during upgrade
• partitioned batch APIs now return Result so linking/storage failures cannot be represented as successful cluster-zero assignments
• standalone in-memory and single-node examples were removed; the maintained example uses the supported distributed deployment

Verification

• cargo +1.88.0 check --locked --all-features --lib
• cargo +1.97.0 test --locked --all-features
• cargo +1.97.0 clippy --locked --all-targets --all-features -- -D warnings
• cargo +1.97.0 fmt --all -- --check
• focused WAL corruption, idempotent replay, restart, reset, sync-failure, and cross-shard merge recovery tests
• five persistent shards, 10,000,000 records, 16 streams, batch 5000, 10% overlap: 50,598 records/sec, all 10,000,000 acknowledged, zero stream errors

The prior ~410K records/sec figure came from the non-durable partition path and is intentionally no longer presented as a production baseline. The earlier 70,539 records/sec audit did not synchronously flush the RocksDB WAL before acknowledgement and has been replaced.

Reconciliation and operator safety

• manual and adaptive reconciliation now require complete, valid metadata from every shard and propagate merge/conflict RPC failures instead of reporting partial success
• cross-shard conflicts are deduplicated, synchronously persisted, restored after shard restart, and reflected in restart-stable stats
• reapplying the active ontology is nondestructive; replacing it on a nonempty cluster fails before any shard changes
• mutation gates prevent ontology replacement or reset from interleaving with ingest, import, queries, checkpoints, or reconciliation
• regression coverage verifies rejected ontology replacement preserves records, incomplete reconciliation fails closed, and cross-shard conflicts survive restart