[Phase 8] Implement observability (metrics, logging, health checks)

[konard]

Summary

Implement comprehensive observability features including metrics, structured logging, and health check endpoints.

Background

Observability is critical for production deployments. Following REQUIREMENTS.md REQ-OBS-001 through REQ-OBS-032.

Requirements

Metrics

• Queue depth metrics
• Throughput metrics (messages/second)
• Latency histograms (enqueue, dequeue, processing)
• Consumer lag
• Error rates
• Resource usage (memory, connections)
• Prometheus export format for integration with monitoring tools

// Get metrics
const metrics = queue.getMetrics();
console.log(metrics.depth);
console.log(metrics.throughput.enqueued);
console.log(metrics.latency.p99);

// Prometheus endpoint
server.expose('/metrics', PrometheusExporter.format(queue.getMetrics()));

Logging

• Structured logging (JSON format)
• Configurable log levels (debug, info, warn, error)
• Correlation IDs for request tracing
• Log rotation support
• Context-aware logging

const queue = new LinksQueue({
  logging: {
    level: 'info',
    format: 'json',
    destination: './logs/queue.log',
    rotate: { maxSize: '10MB', maxFiles: 5 }
  }
});

Health Checks

• Liveness endpoint: Process is running
• Readiness endpoint: Ready to accept requests
• Component health details (backend, cluster, etc.)
• Kubernetes-compatible format

// Health check endpoints
server.get('/health/live', () => queue.isAlive());
server.get('/health/ready', () => queue.isReady());
server.get('/health/details', () => queue.getHealthDetails());

// Response
{
  "status": "healthy",
  "components": {
    "backend": { "status": "healthy", "latency": 5 },
    "cluster": { "status": "healthy", "nodes": 3 }
  }
}

Management UI (Optional)

• Web-based dashboard
• Queue visualization
• Message inspection
• Basic operations (purge, move)

Deliverables

JavaScript

1. src/observability/metrics.ts - Metrics collection
2. src/observability/prometheus.ts - Prometheus exporter
3. src/observability/logger.ts - Structured logging
4. src/observability/health.ts - Health checks
5. Unit tests

Rust

1. src/observability/metrics.rs
2. src/observability/prometheus.rs
3. src/observability/logger.rs
4. src/observability/health.rs
5. Unit tests

Optional

• Basic management UI (separate package)

Dependencies

• [Phase 2] Implement basic Queue and QueueManager (JavaScript) #11, [Phase 2] Implement basic Queue and QueueManager (Rust) #12 - Base queue implementation
• [Phase 5] Implement TCP server mode (JavaScript) #17, [Phase 5] Implement TCP server mode (Rust) #18 - Server mode (for endpoints)

References

• REQUIREMENTS.md - REQ-OBS-001 through REQ-OBS-032
• ROADMAP.md - Phase 8: Observability