feat: mint LiveKit island rooms from Pulse cluster feed (Archipelago ⇒ Pulse, iteration 1)

[mikhail-dcl]

Iteration 1 of the Archipelago ⇒ Pulse migration, gatekeeper side.

Pulse now owns peer clustering and publishes each peer's assignment to NATS. The one hop it cannot own — minting the LiveKit connection string, which needs LiveKit credentials and the ban authority — lands here. This subscribes to that feed, mints, and re-emits the existing IslandChangedMessage, so WS Connector and every client stay untouched.

Ships behind a feature flag, default off. Enabling it per environment is the cutover switch.

What changed vs archipelago-core

	archipelago-core (before)	This PR (after)
Trigger	peer.*.heartbeat, flushed every 2 s	peer.{addr}.cluster_change, per assignment change
Clustering	greedy split/merge, O(n·m) island-pair tests	Pulse: union-find over AoI cells, O(N + cells)
Island size	hard cap 100 (LIVEKIT_ISLAND_SIZE) — connected crowds fragmented into merge-order-dependent rooms	uncapped; sizing is entirely Pulse's, this service never subdivides a cluster
Island IDs	I{base36}, sequential; split minorities always got a new ID	Pulse C{n}, sticky IDs + 3-pass dwell debounce
Room name	island ID verbatim	island-{clusterId} — one cluster, one room
Ban check	HTTP GET /users/:address/bans to this service, 1 s timeout, 20-way concurrency cap	in-process: wallet + recorded device + deny list, 30 s cache
peers map	island members' positions	empty — unity-explorer reads only connStr
Publishes per event	one per peer per 2 s flush, last-write-wins	one per assignment change

Deliberately identical, so nothing user-visible drifts:

• The LiveKit grant: roomJoin, roomList: false, canPublish, canSubscribe, canPublishData, canUpdateOwnMetadata, canPublishSources: [microphone], 300 s TTL. Pinned by an integration test that verifies the JWT signature, not just its shape.
• connStr format livekit:{host}?access_token={jwt}.
• The published subject engine.peer.{addr}.island_changed and its message shape.
• A banned wallet gets no publish at all — not a publish with an empty connStr.

What's new here

• src/adapters/nats.ts — first NATS client in this service. Not @well-known-components/nats-component, which the archipelago services use: it requireString('NATS_URL') at construction (throws at boot when unset), process.exit(1)s when the connection closes, and exposes no metrics. A broker blip must not take down scene tokens, voice chat, or scene bans.
• src/logic/cluster-subscriber/ — the pipeline, plus room naming and per-wallet state.
• src/adapters/livekit.ts — the island- branch in getRoomMetadataFromRoomName moved ahead of the scene/world branches. Those match on configurable prefixes that are empty by default, and startsWith('') is always true, so an island room could be reported to SNS as a scene room with a bogus realm. No other room shape can begin with the literal island-.
• First custom metrics in this repo, named to mirror Pulse's dcl_pulse_nats_* so both sides line up on one dashboard.

Only cluster_change is consumed. Pulse's engine.islands topology snapshot is not — this service has no use for it now that it does no sizing of its own; that feed continues to serve archipelago-stats. peer.*.heartbeat and peer.*.disconnect also survive iteration 1 untouched and are deliberately unused here, since both retire in iteration 2.

Ops

Config — all default to off or to safe values; nothing changes until you set the first two.

Key	Default	Notes
CLUSTER_SUBSCRIBER_ENABLED	unset = off	the cutover switch
NATS_URL	unset = off	same secret name as the archipelago services; unset disables independently
NATS_SUBJECT_PREFIX	''	must match Pulse's Nats:SubjectPrefix
NATS_QUEUE_GROUP	comms-gatekeeper-cluster	so N replicas don't each mint for every event
CLUSTER_PEER_STATE_TTL_MS / _MAX	3600000 / 20000	feeds fromIslandId
CLUSTER_BAN_CACHE_TTL_MS	30000

Metrics: dcl_gatekeeper_cluster_{events_received,tokens_minted,published,publish_failed,banned_skipped}_total and dcl_gatekeeper_nats_connected. In steady state publish_failed should sit at 0.

Cutover sequence:

1. Merge this with the flag off — behaviour is byte-identical to today (verified: with a broker reachable on 4222 and the flag off, no connection is attempted).
2. Set NATS_URL + CLUSTER_SUBSCRIBER_ENABLED=true per environment, with Pulse publishing.
3. Confirm events_received_total > 0 and clients joining island-C{n} rooms.
4. Only then may archipelago-workers' feat!: remove archipelago-core deploy.

Rollback: disable the flag here first — it is the durable half. Otherwise a CI redeploy re-injects Pulse's secret and you get dual-publish flap.

Watch for: a NATS_SUBJECT_PREFIX mismatch with Pulse fails silently — the connected gauge reads 1, every counter stays 0, no error, no log. events_received_total > 0 is the only real check.

Capacity note for Pulse, not this service: with no sharding here, a fully percolated realm maps to one LiveKit room. LiveKit rooms are single-node-bound (~low thousands of audio participants, 100 audio subscriptions per participant), so keeping clusters within that bound is Pulse's responsibility.

⚠️ Blocking before merge

package.json pins @dcl/protocol to a CDN tarball built from the still-open protocol#454, which generates the PeerClusterChange TypeScript this branch imports. Dockerfile runs yarn install --frozen-lockfile, so every image build resolves that URL. yarn.lock carries its sha1, so the risk is availability and contract drift rather than integrity — but this must be re-pinned to a published version once #454 merges.

Testing

• 93 suites, 1219 passing, 0 failing. yarn typecheck clean.
• Integration test runs the real pipeline against a live NATS broker and Postgres — real protobuf, a DB-backed ban, and HS256 verification of the minted JWT.
• CI gap: the shared apps-with-db-build workflow provisions Postgres but not NATS, so that spec skips there with a warning. Adding a broker to the shared workflow is a follow-up.

Follow-ups (not this PR)

• Re-pin @dcl/protocol after #454 merges.
• Add NATS to the shared CI workflow so the integration test actually runs.
• Warn if no cluster_change arrives within N seconds of connecting, to make a prefix mismatch loud.
• Confirm WORLD_ROOM_PREFIX is non-empty in every environment: a world literally named island-*.dcl.eth would now classify ISLAND rather than WORLD.
• Pre-existing: getRoomMetadataFromRoomName's scene branch is unguarded against an empty SCENE_ROOM_PREFIX, which would misclassify voice and private-message rooms as scene rooms.
• Add .idea/ to .gitignore.

🤖 Generated with Claude Code

[coveralls]

Coverage Report for CI Build 30655110208

Coverage increased (+0.8%) to 86.19%

Details

• Coverage increased (+0.8%) from the base build.
• Patch coverage: 232 of 232 lines across 15 files are fully covered (100%).
• No coverage regressions found.

Uncovered Changes

No uncovered changes found.

Coverage Regressions

No coverage regressions found.

---

Coverage Stats

Relevant Lines:	3709
Covered Lines:	3304
Line Coverage:	89.08%
Relevant Branches:	1512
Covered Branches:	1196
Branch Coverage:	79.1%
Branches in Coverage %:	Yes
Coverage Strength:	76.73 hits per line

---

💛 - Coveralls

[LautaroPetaccio]

Standards pass over this branch + four fixes it surfaced

Pushed as 70d748c. One commit — the structural moves and the fixes land in the same files (the NATS adapter is a new file containing both), so splitting them would have meant intermediate commits that don't build.

Why

The feature worked, but it deviated from the WKC standards in ways that would cost us later rather than now:

• Both new components used the deprecated start/stop instead of the lifecycle symbols, so the service logged a deprecation warning per component on every boot.
• The NATS adapter was a flat file while src/adapters/lands/ and every src/logic/* component use the folder layout.
• The peer state store lived in logic/ and was constructed inline, so it couldn't be mocked or swapped.
• The ban + deny-list gate now existed in three places (comms-scene-handler, private-messages/get-token-handler, and the new subscriber) — three copies of a security check is how they drift.
• The island room-name builder sat outside the livekit adapter that owns every other room-name builder and the island- constant it depends on.

Reviewing that work then turned up four real defects, which is the more important half of this PR.

How

Structural (no behaviour change intended)

Change	Detail
Lifecycle	[START_COMPONENT] / [STOP_COMPONENT], matching logic/cron-job
src/adapters/nats/	component / types / index; deletes src/types/nats.type.ts
src/adapters/peer-state/	in-memory state is an adapter; now injected, not built inline
src/logic/access-gate/	the shared lookup, used by all three call sites
livekit.getIslandRoomName	joins its inverse in the adapter that owns the prefix
getErrorMessage()	replaces 9 copies of the same ternary in the new code

The access gate deliberately shares only the lookup. The three callers genuinely differ and all three behaviours are preserved:

	comms-scene	private-messages	cluster-subscriber
deviceId from	signed-fetch metadata	signed-fetch metadata	playerConnectionDb
connection upsert	yes, best-effort	yes, best-effort	never (read-only by design)
fail-open	ban lookup only	neither	whole gate
precedence	ban → deny list	deny list → ban	n/a (skip + metric)

That ban-only fail-open is why the component owns one policy flag rather than none: it has to happen inside the concurrent pair, or a ban-store outage would silently drop deny-list enforcement along with it.

Fixes

1. The NATS status feed was never closed. nats.js hands every status() caller its own iterator and only ever pushes into it — it never closes them. I confirmed against a live broker: after drain() + closed(), the loop is still parked on next() and the listener is still registered. So it held the dead connection reachable, and every reconnect stranded another one. resetConnection() now ends it, and both floating promises .catch(), so an iterator fault can't take the process down as an unhandled rejection.

2. A dropped publish was counted as a success. publish() no-ops when the connection is gone rather than throwing. If the connection dropped during the mint, we incremented published_total, left publish_failed_total at zero, and stored an assignment the peer never received — pointing the next fromIslandId at a room it was never told to join. The metrics lied exactly when the feed was broken. publish() now returns delivery status and the subscriber acts on it. (isConnected() is deliberately not the check — the adapter keeps connection set through a blip so nats.js can buffer.)

3. ?? on cache bounds let a configured 0 through. lru-cache reads max: 0 as unbounded and ttl: 0 as never-expires, so a zero meant as "disable this" silently removed the bound. A never-expiring ban cache keeps serving a wallet cached as allowed after it's banned. Guarded via positiveNumberOr.

4. Shadowed isBanned in comms-scene-handler. My refactor introduced it; the scene-scoped isBanned is declared further down. Platform bans throw before that point, so the outer is always false there — remove the inner declaration and scene bans silently stop being enforced. Renamed to isPlatformBanned.

Testing

100% statements/branches/functions/lines on all four components; 1329 passing across 94 suites.

Where it mattered I checked the tests can actually fail: deleting the queue-group option, and deleting the reconnect re-subscription loop, both produce the expected failures. That second one is worth calling out — the old reconnect test asserted a new connection was opened but never that subscriptions were re-established on it, which is the nastiest failure mode here: connected, gauge at 1, health checks green, and completely deaf.

New integration coverage targets what unit tests structurally cannot reach — chiefly that the queue group really does stop N replicas minting N tokens for one assignment.

Known, unchanged

• This integration suite self-skips in CI. The shared apps-with-db-build.yml provisions Postgres but not NATS, so those 9 tests report as passed having asserted nothing. Fixing the misleading count means switching to opt-in via an env var (Jest can't decide skip-ness after beforeAll probes) — happy to, if preferred.
• Ban-cache staleness window. A wallet cached as allowed keeps getting tokens for up to CLUSTER_BAN_CACHE_TTL_MS (30s) after being banned. banPlayer clears live rooms at ban time, but a mint inside that window lets them into a new one. Existing documented tradeoff.
• No concurrency limit on event processing. On a mass reconnect, N peers produce N simultaneous DB reads before the ban cache warms. Degrades rather than breaks (the pool queues, lookups fail open), but mint latency would spike.
• banned_skipped metric/log doesn't distinguish deny-listed from banned.
• test/setup-env.ts now supplies LiveKit fixtures globally, so tests no longer pick up a developer's real .env values. Deterministic, but a wider blast radius than the original file-scoped version.

[decentraland-bot]

Thanks for the detailed PR and tests. I found a few issues that should be addressed before merging.

Findings:

• P1 — Cluster-token access checks fail open too broadly: a connection-info, ban-store, or deny-list lookup error currently allows token minting instead of preserving deny-list enforcement.
• P1 — Broker connection logging may expose credentials if credentials are embedded in the configured URL.
• P1 — Runtime protocol dependency is pinned to a branch CDN tarball; please re-pin to a released immutable package before merge as noted in the PR.

Notes:

• CI is passing: validations, test, quay build, and title check passed; one deploy-related check is skipped.
• Security review found the access-gate fail-open and URL logging issues above.
• Consumer/API impact: the client-facing engine.peer.{addr}.island_changed subject and payload remain backward-compatible; I did not find a downstream break from the preserved message shape.
• I could not run tests locally because dependencies are not installed in the checkout, so I relied on CI and code review.

---

Reviewed by Jarvis 🤖 · Requested by Lautaro Petaccio (<@U025WCHLMN3>) via Slack

[decentraland-bot]

[P1] This catch fails open for the whole access check, including deny-list failures and even an unrelated connection-info DB failure before accessGate runs. That means a deny-listed wallet can still receive an island LiveKit token during those outages. Please keep the intended ban lookup fail-open behavior, but continue enforcing the deny list; for example, fetch the device ID best-effort, then call accessGate.getAccessState({ address: wallet, deviceId }, { failOpenOnBanLookupError: true }) and let deny-list lookup failures stop this mint instead of returning false.

[decentraland-bot]

[P1] Avoid logging the full broker URL. These URLs often support embedded credentials, so this can write secrets into application logs. Please log only a redacted/sanitized endpoint (for example hostnames without userinfo) or a generic connection success message. The failure log below has the same issue.

[decentraland-bot]

[P1] This runtime dependency is pinned to a branch CDN tarball. Even with the lockfile hash, production builds now depend on a non-release artifact and can drift from the final protocol contract. Please re-pin to a published immutable @dcl/protocol version before merge, as the PR description already calls out.

[LautaroPetaccio]

Re: the three P1s

Thanks for these. One is a deliberate design decision I've now documented in code (0557e15); the other two I don't think are issues here. Details below.

---

1. Access checks fail open — intended, now documented in code

This is a deliberate product decision rather than an oversight, and we're keeping it.

If any of the three lookups (connection-info read, platform ban store, deny list) fails and we fail closed, the peer gets no island token and therefore cannot get into voice at all. An outage in any single one of those dependencies would take down comms for everyone it touches, not just for moderated accounts. We judge platform availability the more important property here: a moderation gate that is briefly permissive is recoverable, a world nobody can connect to is not.

Two things bound the exposure, and I'd rather state them than imply the trade-off is free — while a lookup is failing, a banned or deny-listed wallet can be minted an island token:

• Banning already removes the participant from every live room at ban time, so this only affects a new room the wallet joins during the outage window.
• The fail-open result is deliberately not written to banCache, so the next event for that wallet retries the lookup rather than the process staying wrong for the full 30s TTL.

On the specific suggestion of "preserve deny-list enforcement even when the other lookups fail" — that's a real distinction and it is how the signed-fetch HTTP path behaves: there only the ban lookup fails open, and a deny-list error still rejects the request. The two paths differ on purpose. The HTTP path is a synchronous, user-initiated request that can surface an error and be retried by the client. The cluster subscriber is a background feed with no caller to report to, so dropping the event just leaves the peer silently roomless with nothing to trigger a retry.

0557e15 records all of the above in src/logic/cluster-subscriber/component.ts at the catch, including the asymmetry with the HTTP path, so the next reader (or reviewer) hits the reasoning before the code.

2. Broker URL logging — NATS_URL carries no credentials

NATS_URL is not a secret in this deployment. It's a host/seed list (comma-separated), and the adapter never parses credentials out of it or attaches any — the full set of options we pass is:

natsConnect({
  servers: url.split(',').map((s) => s.trim()),
  name: 'comms-gatekeeper',
  maxReconnectAttempts: -1,
  reconnectTimeWait: RECONNECT_DELAY_MS
})

No user, pass, token, or authenticator. There is no credential in the value to leak, so logging the host we connected to (and the host we failed to connect to, which is the useful half when debugging a broker outage) is fine.

If we ever move to authenticated NATS, the credential would arrive as a separate option rather than inside this URL — and at that point redacting here still wouldn't be the control that matters.

3. @dcl/protocol pin — already immutable and commit-addressed

The tarball is pinned to a specific commit, not to a moving branch tag:

@dcl/protocol/branch//dcl-protocol-1.0.0-30550755753.commit-b0705a3.tgz

commit-b0705a3 makes the artifact immutable — the CDN publishes one tarball per commit and never rewrites it. yarn.lock additionally pins the content hash:

resolved "…commit-b0705a3.tgz#5933d298d5597424c04c3ea07a90aa82ea6184b3"

so an install is content-verified and cannot silently drift even if the URL were re-pointed. This is the standard flow across DCL services while a protocol change is still in flight; it gets re-pinned to the released version as part of the normal protocol release, not as a blocker on this PR.

[decentraland-bot]

Re-review after the latest commit.

Verdict: approved. The previously raised blockers have either been documented as an intentional product/availability tradeoff or clarified sufficiently for this PR:

• The cluster-subscriber fail-open behavior is now documented at the call site, including the moderation exposure and why it differs from the synchronous HTTP token path.
• The current NATS configuration is passed as broker host/seed URLs only, not as embedded credentials.
• The temporary @dcl/protocol tarball is commit-addressed and lockfile-pinned; please still re-pin to the released package during the normal protocol release follow-up.

Non-blocking follow-ups I recommend keeping tracked:

• Add an explicit metric/alert for cluster access-gate fail-open events, ideally with reason labels.
• Make the NATS-backed integration suite run in CI with a broker instead of self-skipping.
• Consider validating the subject wallet and clusterId charset/length before minting, even if Pulse/NATS are trusted producers.
• Consider an operational signal for CLUSTER_SUBSCRIBER_ENABLED=true but no events received, to catch prefix mismatches or a deaf subscriber.

CI is passing: validations, tests, quay build, and title check passed; the deploy-related check is skipped.

Consumer/API impact: the client-facing engine.peer.{addr}.island_changed subject and payload remain backward-compatible; no downstream break found from the preserved message shape.

I did not run tests locally because dependencies are not installed in this checkout; I relied on CI and code review.

---

Reviewed by Jarvis 🤖 · Requested by Lautaro Petaccio (<@U025WCHLMN3>) via Slack

[LautaroPetaccio]

Great work!

[LautaroPetaccio]

Would you mind creating a PR in the definitions repo to fill these configs?

[LautaroPetaccio]

We should get the deviceId from the nats message. This would work for now, but Pulse should send it if possible.