Resharding: S3-mediated space migration between tree nodes#227
Open
cheggaaa wants to merge 17 commits into
Open
Resharding: S3-mediated space migration between tree nodes#227cheggaaa wants to merge 17 commits into
cheggaaa wants to merge 17 commits into
Conversation
.diskgen marker in the storage root; DiskGen().Fresh() reports a node started on an empty root (new node or replaced disk) so resharding machinery can treat local absence as 'missing', never 'deleted'. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KDgytoFFikB7tY4oRreWom
…er, AdoptArchive RPC The data plane for resharding: spaces move between tree nodes as archive snapshots inside a shared S3 bucket, never streamed node-to-node. - archivestore: Exists (HEAD), Key, CopyFrom (server-side CopyObject), Shared config flag gating the migration machinery - archive.ForceArchive: snapshot a live space to S3 without deleting the local DB or flipping its status (uses DumpStorage, works on open spaces) - archive.Restore keeps the S3 object (copy, not move): idempotent restores; stale objects are cleaned by the next archive cycle / archive sweeper - archive.QueueRestore + background worker: eager restore of adopted spaces - AdoptArchive RPC (nodesyncproto): receiver copies the offered snapshot into its own prefix, verifies with HEAD, registers Archived+heads in its index and nodehead; node-only, rejects tombstoned spaces, repairs lost objects - indexstorage.MarkArchivedRemote: register a never-opened space as archived Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KDgytoFFikB7tY4oRreWom
The draining side of resharding, gated on the shared archive store: - resharder component: on nodeconf change (and periodically), finds local spaces this node is no longer responsible for and hands each off through the shared bucket: archived spaces via CopyObject+AdoptArchive with zero restores; live spaces via ForceArchive snapshot + eager adoption, with the post-snapshot delta converged by tree sync - deletion safety: local copy removed only after >=2 current owners durably hold the same heads (adopt Ok / AlreadyHaveSame); diverged owners trigger hotsync convergence and the space parks for the next cycle; heads recheck before deletion catches writes landed mid-handoff - AdoptArchive responses are head-aware (AlreadyHaveSame vs Diverged) so a stale owner copy can never justify deleting the newest replica - SpaceStatusMoved terminal status; drained spaces leave ldiff/nodehead - spacedeleter: archived spaces delete their S3 object directly instead of restoring first; live deletions clean up leftover archive objects Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KDgytoFFikB7tY4oRreWom
- archivestore.List: paginated iteration of this node's prefix - daily archive sweep: deletes objects that no longer back an Archived index entry (restore leftovers, finished handoffs, deleted spaces, wiped-disk leaks); a 7-day age guard protects in-flight handoffs and fresh restores - resharder reconciles space dirs without index entries before each drain cycle so legacy orphans become drainable Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KDgytoFFikB7tY4oRreWom
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KDgytoFFikB7tY4oRreWom
…+ real-S3 tests Verified against Google Cloud Storage S3 interop: HEAD responses can be stale for seconds after a DELETE (observed 200 for a just-deleted object while ListObjectsV2 already showed it gone). Exists backs the durable-ACK check of the drain handoff, where a stale positive could acknowledge an object that no longer exists — reimplemented over ListObjectsV2 with an exact-key match, which proved read-after-write consistent. - TestRealS3 (archivestore): opt-in conformance test for Put/Get/Exists/ CopyFrom/List/Delete against a real bucket (ARCHIVE_TEST_S3_* env vars) - the drain integration test now runs against the real bucket too when the same env vars are set; verified against GCS Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KDgytoFFikB7tY4oRreWom
Findings from a high-effort multi-agent review, all confirmed and fixed: - drain handoff: heads are now read from the snapshot itself (ForceArchive returns them) so ACKed hashes exactly describe the uploaded object; the drain parks when the index moves past the snapshot, and the pre-delete recheck compares against the snapshot heads - adopter: RemovePrepare (cancellable) no longer reads as 'deleted' — new ErrSpacePendingDeletion parks the sender instead of dropping its copy; only the final Remove status returns ErrSpaceDeleted - adopter: receivers reject AdoptArchive for spaces they are not responsible for (ErrNotResponsible) so a stale/misbehaving sender can't count a non-owner as a durable ACK - resharder: empty owner list parks instead of deleting with zero handoffs (a force-applied broken config could otherwise wipe drained partitions) - resharder: the archived-but-object-missing repair path flips the status to Ok before snapshotting — previously it triggered a doomed restore whose error path deleted the only remaining local db - archive.Restore: no longer removes the space dir when the archive object is missing (nothing was extracted; the dir may hold a live db) - sweeper: serialized with the archiver via a mutex, decisions re-read the index under the lock; Ok-status objects are only collected when the local db actually exists; unknown objects are kept until the storage root is 30 days old (fresh-disk protection); Moved/Remove are collected as before - spacedeleter: deletion-cancel keeps archived spaces Archived (previously flipped to Ok, presenting the only copy as sweeper garbage); final deletion also removes lingering space dirs of archived spaces - coordinator: unique partial index on epoch + duplicate-key retry makes epoch assignment safe under concurrent confapply runs Verified: full unit suites + drain integration test against real GCS. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KDgytoFFikB7tY4oRreWom
- adopter recognizes senders that were tree nodes in recently retained config epochs (nodeconf history): a node removed from the configuration can finish draining after the removal — receivers no longer reject it as a non-node - scenario tests over real chash rings (resharder/scenarios_test.go): TestScenario_AddNode — epoch bump adds a node; losers hand affected spaces off with 2-owner ACKs and delete their copies; unaffected spaces stay put. TestScenario_RemoveNode — removed node drains everything to the current owners (authenticated via config history) and ends up empty; every space remains available on all remaining owners - docs/resharding-operations.md: operator guide for the two supported operations (prerequisites, steps, metrics to watch, safety properties); linked from README Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KDgytoFFikB7tY4oRreWom
- node_resharder_state (0 disabled / 1 idle / 2 draining), node_resharder_epoch (config epoch acted on — shows propagation), node_resharder_last_cycle_unix, node_resharder_errors - adopter gets its own metrics: node_adopter_adopted / already_have_same / already_have_diverged / rejected (the archive-level 'adopted' gauge was registered but never incremented — removed) - operator guide updated with the full metric list and idle-state 'done' definition Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KDgytoFFikB7tY4oRreWom
…fore merge) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KDgytoFFikB7tY4oRreWom
|
Review the following changes in direct dependencies. Learn more about Socket for GitHub.
|
|
Warning Review the following alerts detected in dependencies. According to your organization's Security Policy, it is recommended to resolve "Warn" alerts. Learn more about Socket for GitHub.
|
Coverage provided by https://github.com/seriousben/go-patch-cover-action |
… client code) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KDgytoFFikB7tY4oRreWom
…fig changes A config publish that only changes addresses or non-tree nodes produces an identical chash ring — no space changes hands. The change observer now compares tree member sets and skips the (guaranteed-empty) full index scan for such epochs; the periodic cycle remains as a backstop. Correctness was already given (zero candidates); this removes the wasted work. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KDgytoFFikB7tY4oRreWom
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KDgytoFFikB7tY4oRreWom
An Error entry needs an operator: the local db may hold data that failed to archive. Adopting a snapshot over it would let the later restore overwrite that data, and ACKing the sender could justify deleting the freshest copy. Error owners now answer 'diverged' — no adoption, no ACK; the two healthy owners carry the handoff. Operator guide gains an Error-state playbook. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KDgytoFFikB7tY4oRreWom
Periodic per-node repair (default hourly) for spaces in Error status that the node is responsible for: - repair in place when the db opens fine (transient errors like a failed archive upload) — no data touched - otherwise quarantine the corrupted db under <root>/.quarantine (preserved for the operator) and pull a valid copy from a responsible neighbor via the regular coldsync (works without a shared bucket), validating it before the Error status clears - nodestorage.QuarantineSpace + IndexStorage.ReadSpacesByStatus - metrics: node_repairer_errored/repaired/repaired_in_place/quarantined - operator guide updated: manual playbook shrinks to the cases the repairer deliberately leaves alone Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KDgytoFFikB7tY4oRreWom
…anywhere Tracing the all-replicas-broken scenarios surfaced a bricking bug: an Error entry blocks space creation (checkStatus), so an orphaned entry of an uncommitted space push — no data on any owner, no heads ever recorded — would keep the space id unusable forever and alert forever. - coldsync already returns coded ErrSpaceMissing when the source definitively lacks the space; the repairer now distinguishes that from unreachable/broken owners - when every owner definitively lacks the space, we hold nothing and the entry has no recorded heads, the entry is deleted outright: the id becomes usable again and the alert clears (node_repairer_dropped_entries) - if heads WERE recorded, the entry stays Error: that is a real data-loss signal that must keep alerting Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KDgytoFFikB7tY4oRreWom
Two real node stacks with an in-process drpc pair; node B holds a real GenStorage space, node A repairs through the actual protocol: - corrupted db: quarantined under .quarantine, valid copy pulled from B, status Ok, heads registered from the pulled state, storage opens as a valid space - missing db file: pulled from B without quarantine - space missing everywhere: B answers coded ErrSpaceMissing over the wire and the heads-less entry is garbage-collected Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KDgytoFFikB7tY4oRreWom
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Implements resharding — safely adding and removing tree nodes on a running network. Postponed for years; researched and designed in docs/resharding-plan.md, operated per docs/resharding-operations.md. Depends on anyproto/any-sync#721 and pairs with anyproto/any-sync-coordinator#132.
Design in one paragraph
Placement stays a pure function of the chash ring; the coordinator versions configs with a monotonic epoch and refuses transitions that would leave a partition without a live replica. On a config change each node drains the spaces it no longer owns through the shared archive bucket ("force-archive + rename"): archived spaces are handed over by server-side
CopyObject+ anAdoptArchivepointer RPC (zero restores); live spaces ship as a snapshot with the delta converging via normal tree sync. A local copy is deleted only after ≥2 current owners durably confirm exactly the advertised heads (read from the snapshot itself). The machinery is dormant unlesss3Store.shared: true— self-hosted networks are unaffected.Main pieces
archivestore:Exists(list-based — GCS interop serves stale HEAD after DELETE),CopyFrom(server-side copy),List,sharedflagarchive:ForceArchive(snapshot without touching the live db, returns snapshot heads), restore-as-copy, eager-restore queue, daily prefix sweeper (age + fresh-disk guards)archive/adopter: receiver side ofAdoptArchive— head-aware responses (Ok/AlreadyHaveSame/AlreadyHaveDiverged), responsibility + node-peer checks (with config-history fallback so removed nodes can finish draining), tombstone awarenessresharder: the drain state machine — candidates on config change + periodic cycle, metadata-first verification, hotsync convergence for diverged owners, heads recheck before deletion,SpaceStatusMoved, legacy-orphan reconciliationspacedeleter: archived spaces delete their S3 object directly; cancelled deletions keep archived statusnodestorage:.diskgenfresh-storage markernode_resharder_state/epoch/draining/moved/parked/errors/last_cycle_unix,node_adopter_*Review
A high-effort multi-agent review ran over the branch; 9 confirmed data-loss findings were fixed in dedicated commits (snapshot-derived handoff heads, cancellable-deletion handling, sweeper/cancel interactions, fresh-disk sweeper guard, repair-path restore trap, empty-owner-list guard, non-owner adoption, epoch races).
Testing
TestScenario_AddNode,TestScenario_RemoveNode(resharder/scenarios_test.go)ARCHIVE_TEST_S3_*env vars are set — verified against Google Cloud Storage S3 interopNotes for merge
go.modpoints at the any-syncreshardingbranch pseudo-version so CI builds; swap for the tagged release once nodeconf: configuration epochs, change observers, config history store any-sync#721 ships.s3Store.shared: true.🤖 Generated with Claude Code
https://claude.ai/code/session_01KDgytoFFikB7tY4oRreWom