[Phase 6] Implement multi-node clustering (JavaScript)

[konard]

Summary

Implement multi-node clustering for JavaScript, enabling distributed queue operation across multiple nodes.

Background

Clustering enables horizontal scaling by distributing queues across multiple nodes. Supports both `multi-memory` (ephemeral) and `multi-stored` (persistent) modes.

Requirements

Node Discovery

• Static node list configuration
• Health checking between nodes using heartbeat
• Failure detection with timeout
• Node join/leave handling

Cluster Coordination

• Leader election for coordination tasks
• Distributed queue registry
• Cross-node routing for queue operations

Queue Partitioning

• Consistent hashing for queue distribution
• Partition assignment to nodes
• Rebalancing on topology changes
• Partition-aware consumers

Multi-Node Modes

• `multi-memory`: Distributed in-memory, no persistence
• `multi-stored`: Distributed with link-cli persistence per node

Implementation

import { LinksQueue, ClusterConfig } from 'links-queue';

const queue = new LinksQueue({
  mode: 'multi-stored',
  backend: { type: 'link-cli', path: './data/db.links' },
  cluster: {
    nodes: ['node1:5000', 'node2:5000', 'node3:5000'],
    discovery: 'static',
    replication: {
      factor: 2,
      sync: true
    }
  }
});

await queue.connect();

// Operations are routed to correct partition
await queue.enqueue('tasks', { source: 'job', target: 'data' });

// Listen for cluster events
queue.cluster.on('node-joined', (node) => {
  console.log('Node joined:', node.address);
});

queue.cluster.on('rebalancing', () => {
  console.log('Cluster rebalancing...');
});

Deliverables

1. src/cluster/coordinator.ts - Cluster coordination
2. src/cluster/discovery.ts - Node discovery
3. src/cluster/partition.ts - Partitioning logic
4. src/cluster/replication.ts - Data replication
5. src/cluster/gossip.ts - Gossip protocol for membership
6. Unit tests with simulated cluster
7. Integration tests with real multi-node setup

Dependencies

• [Phase 1] Define Link and LinkStore interfaces/traits (API Contract) #7 - Link interfaces
• [Phase 2] Define Queue and QueueManager interfaces/traits (API Contract) #10 - Queue interfaces
• [Phase 2] Implement basic Queue and QueueManager (JavaScript) #11 - Queue implementation
• [Phase 4] Integrate links-notation for parsing and serialization #16 - Links Notation protocol
• [Phase 5] Implement TCP server mode (JavaScript) #17 - Server mode (for inter-node communication)
• [Phase 6] Define Cluster and Node interfaces/traits (API Contract) #19 - Cluster interfaces

Can Be Worked In Parallel With

• Rust clustering implementation (separate codebase)

References

• ARCHITECTURE.md - Cluster Coordinator, Multi-Node Modes
• REQUIREMENTS.md - REQ-MODE-020 through REQ-MODE-033
• ROADMAP.md - Phase 6: Multi-Node Clustering