plan-D5 wiring: dispatch v7-vs-v8 dir-index in EncryptedClient flush + load

[ehsan6sha]

Summary

Follow-up to #6. The v8 envelope (EncryptedDirectoryIndexV8) is implemented and tested at the fula-crypto level. This issue tracks the SDK orchestration layer that actually dispatches v7-vs-v8 in production write/read paths.

Goal

Lift the v7 dir-index 1 MiB cliff for real users by wiring EncryptedDirectoryIndexV8::encrypt_sharded / ::decrypt_sharded into EncryptedClient flush + load paths, while preserving full backward-compat for legacy v7 buckets.

Scope (each is a sub-task with its own design + review)

1. ManifestRoot schema extension — add dir_index_v8_etags: Option<Vec<String>> and dir_index_v8_cids: Option<Vec<Cid>> (with #[serde(default)] for backward compat). Add a LegacyManifestRootPreD5 round-trip test mirroring the W.9.3-A LegacyManifestRoot test pattern.

2. derive_dir_index_v8_shard_key(forest_dek, bucket, shard_idx) helper in private_forest.rs — distinct path namespace from v7's single-blob storage key.

3. Save-path dispatch at crates/fula-client/src/encryption.rs:4172 (Phase 1.6 PUT site):

   • Pre-flight: serialize the v7 plaintext, check size. If ≥ 80% of MAX_MANIFEST_BLOCK_SIZE, dispatch to v8.
   • v8 path: 16 conditional PUTs, each with its own If-Match against the per-shard etag from the manifest.
   • Partial-412 coordination: if any shard 412s, evict the entire forest cache and propagate ConcurrentModification; do NOT accept partial application.
   • Walkable-v8 CID stamping per shard via verify_etag_against_expected_cid.
   • WAL DirIndexWrote variant must encode shard_idx + version=8 so crash-recovery can resume mid-flush.

4. Load-path detection in load_directory_index:

   • Try v8 first: if manifest.root.dir_index_v8_etags.is_some(), fetch all 16 shards in parallel.
   • Else: fall back to v7 (current single-blob path).
   • Each shard fetch uses walkable-v8 cid-hint when available.

5. Transition flush — first time a bucket crosses the threshold:

   • Write all 16 v8 shards.
   • Delete the orphan v7 blob (best-effort; orphan persists if delete fails, doesn't block correctness).
   • Subsequent flushes are pure-v8.

6. Test matrix:

   • 1k regular: still v7 (no behavior change).
   • 30k cliff: dispatches to v8; round-trip works.
   • Transition: bucket grows from 25k (v7) to 35k (v8) — first flush at 35k writes v8, deletes v7.
   • 412 retry: simulate one shard 412; whole cycle retries.
   • Crash mid-flush: WAL replay resumes correctly.
   • Old SDK on v8 bucket: refuses with WireVersionUnsupported(8) typed error (already added in F-batch).
   • 100k / 1M as #[ignore] (W.9.7 pattern).

7. Cross-platform regen — audit FRB/wasm surfaces for new types or error variants; regen if needed.

Backward compat constraints (load-bearing)

• Pre-plan-D5: shard EncryptedDirectoryIndex into 16 hash-prefix blobs to lift 1 MiB cliff #6 buckets must continue to read identically. v8 readers handle BOTH v7 (legacy) and v8 (new).
• Old SDK on v8 bucket: typed WireVersionUnsupported error (already in error.rs). User-actionable.
• No data migration required for buckets below the threshold; they keep using v7 indefinitely.
• Transition is one-way: a bucket that grows past the threshold becomes v8 and stays v8 (new entry could keep it just-under-threshold, but we don't downgrade for stability).

Estimated effort

~700-1000 LOC, similar in shape to walkable-v8's W.9.3+W.9.4 phases. 2-3 dual-review iterations expected.

Files to modify

• crates/fula-crypto/src/private_forest.rs — derive_dir_index_v8_shard_key, helper for size threshold.
• crates/fula-client/src/encryption.rs — Phase 1.6 dispatch, load_directory_index extension.
• crates/fula-client/src/wal.rs — DirIndexWrote variant.
• crates/fula-client/src/types.rs (if needed for new error variants surfaced through bindings).
• crates/fula-flutter/src/api/* and crates/fula-js/src/lib.rs — regen if needed.

Tracking

Depends on: #6 (envelope) — landed.
Blocks: real-user benefit from plan-D5.

[ehsan6sha]

Note: production cliff status

Until this issue ships, real users still hit the v7 dir-index cliff at ~30k+ directories per bucket. The v8 envelope (#6) landed at the fula-crypto layer but is NOT wired into any production code path — see the "Production status" section in #6's comments.

This issue is the load-bearing piece for the actual user benefit. Without it:

• New writes to a 30k-dir bucket still fail with the D1 error message.
• Existing data still remains readable (no regression).
• Production users see zero behavior change from plan-D5: shard EncryptedDirectoryIndex into 16 hash-prefix blobs to lift 1 MiB cliff #6.

Estimated effort and sub-tasks below remain accurate.