feat(pubsub): wire pubsub relay into the node#230
Open
requilence wants to merge 4 commits into
Open
Conversation
Add nodespace/pubsubrelay, a component that drives the any-sync commonspace/pubsub engine as a relay: - Relay: IsResponsible / IsResponsibleNode via nodeconf, OtherResponsiblePeers dials the other responsible nodes (excluding self) through the pool. - MembershipChecker: authorizes subscribe/publish against the hosted space's ACL (Permissions at current head via nodespace.Service.GetSpace + Acl()). - Crypto/Peers left nil: the node relays ciphertext it cannot read and routes via Relay, not a client peer provider. - Registers the PubSub DRPC service and owns the engine's Init/Run/Close. - Evicts removed members: nodespace gains SetAclObserver, invoked from nodeSpace.AddConsensusRecords (after releasing the acl lock) so the relay re-checks subscribers via engine.RevalidateMembers on every ACL change. Registered in the node bootstrap after nodespace. go.mod/go.sum bumps come from building against the any-sync pubsub branch.
The ACL-change observer fires from the consensusclient stream reader while it holds the consensusclient mutex. The previous synchronous onAclUpdate called GetSpace (space ocache) inline, which deadlocks: a concurrent space close does UnWatch (same mutex), and GetSpace waits on the closing entry — ABBA — or, on an ocache miss, reloads on the same goroutine and re-enters Watch on the held mutex (self-deadlock). Either bricks consensus delivery node-wide. - onAclUpdate now only enqueues the spaceId (non-blocking, deduping); a relay-owned worker drains the queue and revalidates with a bounded timeout. - Use PickSpace (never loads) so a miss is a no-op and an evicted space is not resurrected; members are re-checked on the next subscribe/publish anyway. - Fire the observer even when AddRawRecords errors (records apply one by one, so a partial batch may have changed membership). - Guard SetAclObserver(nil) against a nil-func panic and unregister the observer in relay.Close before tearing down the worker and engine. Adds regression tests: onAclUpdate is non-blocking and never touches the space cache synchronously; a PickSpace miss is a no-op; the worker drains the queue.
Replace the local-path dev replace with a pseudo-version pinning the any-sync feat/pubsub commit (f2bca56) that ships commonspace/pubsub. Bump to the tagged any-sync release once it is cut.
Picks up the WithMetric fix (f05d5319) so the node's private pubsub pool no longer clobbers the app sync streampool's OutgoingMsg telemetry.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Wires the any-sync
commonspace/pubsubengine into the node as a relay, so clients can publish/subscribe ephemeral topics within a space through their responsible sync nodes.Depends on anyproto/any-sync#726 (pinned via pseudo-version until that's released and tagged).
How
New
nodespace/pubsubrelaycomponent that drives the engine's lifecycle and supplies its node-role dependencies:IsResponsible/IsResponsibleNodefrom nodeconf;OtherResponsiblePeersdials the other responsible nodes (excluding self) via the pool, so a client publish is forwarded exactly once per the one-hop rule.GetSpace→Acl().RLock()→ current-headPermissions). This is enforcement the sync path never had.Relay, not a client peer provider. Metric wired through when present.PubSubDRPC service alongsideSpaceSync.Active member eviction (DESIGN §6.4):
nodespace.ServicegainsSetAclObserver, fired fromnodeSpace.AddConsensusRecords. The relay hooks it to re-check subscribers viaRevalidateMemberson every ACL change, cutting off removed members' subscriptions the moment their removal record lands.Concurrency note
The ACL observer fires from the consensus stream reader while it holds the consensusclient mutex. Doing the space-cache lookup inline there deadlocks against
Watch/UnWatch(which need the same mutex) — a node-bricking ABBA. SoonAclUpdateonly enqueues; a relay-owned worker drains the queue and revalidates withPickSpace(never loads → no space resurrection) under a bounded timeout. This was caught in review and has a dedicated regression test (TestOnAclUpdateNonBlocking).Testing
Unit tests for the relay's routing (self-exclusion, responsible-node membership, undialable-peer skipping, single-node networks) and the async-revalidation invariants (non-blocking enqueue, miss-is-noop, worker drains off the consensus goroutine). Race-clean; full node suite green.