Add Prometheus metrics, OpenTelemetry tracing, and Grafana dashboards for observability

[memplethee-lab]

Summary

What: Add a lightweight observability and performance monitoring integration (Prometheus + Grafana metrics + request tracing) and an API to export metrics.
Why: Improve visibility into API performance, errors, and resource usage to help with debugging and capacity planning.
Which part: Server bootstrap, middleware, monitoring folder, deployment/infra configurations.

Problem Statement

Current limitation

• No built-in metrics/exported telemetry for request latency, error rates, or resource usage.

Pain point

• Hard to correlate incidents, measure performance regressions, or plan scaling.

Why it matters

• Observability is critical for production readiness; contributors need signals to prioritize fixes and measure impact.

Proposed Solution

How it should work

• Expose Prometheus-compatible /metrics endpoint with counters, histograms (request durations), and Gauges for uptime and resource usage.
• Integrate with a tracing solution (OpenTelemetry) to capture request traces and integrate with Jaeger/OTLP.
• Add Grafana dashboard JSON templates under monitoring/grafana/dashboards.
• Add env-based toggles to enable/disable telemetry and control sampling rates.

Expected behavior

• Running the server with MONITORING=true exposes /metrics and starts OTLP exporter when configured.
• Prometheus can scrap /metrics; Grafana dashboard displays latency, error rate, throughput, and memory usage.

Important notes

• Do not export sensitive data in traces. Use sampling to limit overhead.

Implementation Guidance

Files/folders likely affected

• src/main.ts
• src/monitoring/*
• .github/workflows/deploy.yml (if deployment uses monitoring)
• docker-compose.yml (add monitoring services for local dev)

Modules/Services involved

• Global middleware, interceptors, monitoring service

Dependencies/prerequisites

• prom-client, @opentelemetry/api, @opentelemetry/sdk-node, jaeger-client or OTLP exporter

Suggested steps

1. Add prom-client and configure default metrics and an express/Nest middleware to track requests.
2. Expose /metrics route and secure with auth/token if in production.
3. Add OpenTelemetry setup and optional jaeger exporter.
4. Provide Grafana dashboard JSON and a docker-compose monitoring stack for local dev.
5. Document setup in docs/monitoring.md

Acceptance Criteria

Acceptance Criteria

• /metrics endpoint exposes Prometheus metrics
• Request latency histogram and error counters implemented
• OpenTelemetry traces exported when enabled
• Grafana dashboard template included in monitoring/grafana/
• Documentation added to docs/monitoring.md
• Unit/integration tests for metric collection where applicable
• Feature passes linting and existing CI checks

Technical Notes

API changes

• Adds /metrics endpoint; consider auth for production
  Database changes
• None
  Security
• Avoid PII in traces; secure /metrics in production
  Performance
• Use sampling for traces; keep default metrics low overhead
  Accessibility
• N/A
  Edge cases
• High cardinality labels can cause Prometheus memory issues; limit labels

Definition of Done

• All acceptance criteria met
• Docs and docker-compose monitoring stack available

Labels

• enhancement, backend, monitoring, documentation

Priority

• High

Estimated Difficulty

• Advanced

Estimated Effort

• L