[Phase 7] Implement advanced queue features (scheduling, rate limiting, routing)

[konard]

Summary

Implement advanced queue features for production parity with competitors: delayed messages, rate limiting, topic-based routing, and pub/sub patterns.

Background

These features are required to compete with established solutions like BullMQ, RabbitMQ, and Celery.

Requirements

Scheduling

• Delayed Messages: Enqueue for future delivery
• Cron Jobs: Scheduled recurring tasks
• TTL: Time-to-live for messages
• Expiration: Automatic removal of expired messages

// Delayed message
await queue.enqueue(link, { delay: 30000 }); // 30 seconds

// Scheduled job
queue.schedule('0 */5 * * *', async () => {
  await queue.enqueue({ source: 'cron', target: 'cleanup' });
});

// TTL
await queue.enqueue(link, { ttl: 60000 }); // Expires in 1 minute

Rate Limiting

• Per-queue rate limits
• Per-consumer rate limits
• Sliding window algorithm
• Backpressure signaling

const queue = new LinksQueue({
  name: 'api-calls',
  rateLimit: {
    max: 100,       // Maximum 100 messages
    window: 60000,  // Per minute
  }
});

Routing

• Topic-based routing with wildcards
• Exchange types (direct, topic, fanout)
• Pattern matching on link content
• Binding management

// Topic routing
queue.bindTopic('logs.*', logsQueue);
queue.bindTopic('logs.error', errorQueue);

// Fanout
queue.fanout('notifications', [queue1, queue2, queue3]);

Pub/Sub

• Topic creation/deletion
• Subscribe/unsubscribe
• Fan-out delivery
• Message filtering

// Publisher
await queue.publish('events', { source: 'user', target: 'created' });

// Subscriber
queue.subscribe('events', async (link) => {
  console.log('Event received:', link);
});

Deliverables

JavaScript

1. src/features/scheduler.ts - Delayed messages and cron
2. src/features/rate-limiter.ts - Rate limiting
3. src/features/router.ts - Topic routing
4. src/features/pubsub.ts - Pub/sub patterns
5. Unit and integration tests

Rust

1. src/features/scheduler.rs
2. src/features/rate_limiter.rs
3. src/features/router.rs
4. src/features/pubsub.rs
5. Unit and integration tests

Dependencies

• [Phase 2] Implement basic Queue and QueueManager (JavaScript) #11, [Phase 2] Implement basic Queue and QueueManager (Rust) #12 - Base queue implementation

References

• REQUIREMENTS.md - Feature Parity with Competitors
• ROADMAP.md - Phase 7: Advanced Features
• COMPARISON.md - Feature comparison table