feat(mem-wal): ShardWriter::abort + ShardStatus(Sealed) manifest fence for drop-table

[hamersaw]

What

The mem-wal primitives sophon's drop-table two-phase commit needs:

• ShardWriter::abort(&self) — shut down the background flush tasks (task_executor.shutdown_all()) without flushing, discarding buffered memtable state. Unlike close(self) it takes &self (so it's callable through the Arc<ShardWriter> callers hold) and does no object-store IO. The caller must quiesce writes first (documented). Idempotent. Acked data is not lost — it's durable in the WAL log and replays on the next claim, which is what makes the drop's prepare phase reversible.
• ShardStatus { Active | Sealed } on ShardManifest — a durable, reversible lifecycle marker (proto + struct + serde). claim_epoch refuses a Sealed manifest with a distinguishable error instead of minting a new epoch, so a shard mid-drop can't be re-claimed — even by a caller that skips its own status check — and a reader can tell an in-doubt drop apart from an ordinary epoch fence. Set/cleared through the existing epoch-guarded commit_update CAS; carried across claims via ..base, so only the genuinely-fresh constructions default it to Active.

Why

A WAL-enabled table's drop spans two durable resources — the owning pod's fresh-tier state and the catalog/object-store data — so sophon's teardown is a two-phase commit. Before the dataset directory is removed, the owning pod must:

• abort the writer so its background flush task can't re-create _mem_wal/ under the just-deleted directory (a graceful close() would flush it back), and
• durably mark the shard Sealed so the drop is in-doubt across a pod crash or a Maglev rehome — which the in-memory fence flag cannot survive. The seal is reversible (rollback clears it back to Active), making the prepare phase abortable without data loss.

Both build on existing machinery (shutdown_all, the manifest CAS) — thin exposure, not new infrastructure.

Changed since first draft: this PR previously also added Session::invalidate_dataset; it has been dropped. Base-table read freshness rides the recreate's new object-store e_tag (the QN→PE and lance metadata caches are e_tag-keyed and miss the stale entry), so cache invalidation isn't load-bearing — only abort and the Sealed marker remain.

Tests

• test_abort_discards_without_flushing_and_is_idempotent — abort leaves no new L0 generation (contrast with close), idempotent on a second call.
• test_claim_epoch_refuses_sealed_manifest — a Sealed manifest is refused with the distinguishable error and left untouched (no epoch minted); rolling the status back to Active makes the shard claimable again (reversibility).

🤖 Generated with Claude Code

[github-actions]

ACTION NEEDED
Lance follows the Conventional Commits specification for release automation.

The PR title and description are used as the merge commit message. Please update your PR title and description to match the specification.

For details on the error please inspect the "PR Title Check" action.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

[github-actions]

Important

This PR touches the Lance format specification.

Substantive changes to the format specification — the .proto definitions
and the spec docs under docs/src/format/ — require a PMC vote before merge.
Minor edits such as typo fixes, wording, or formatting are excluded; use your
judgment.

If this is a meaningful format change:

• Start a vote following the Lance community voting process.
  Format specification modifications need 3 binding +1 votes (excluding the
  proposer), held on GitHub Discussions, with a minimum voting period of 1 week.
• Once the vote passes, link the completed vote in this PR. It should not be
  merged until the vote is linked.