Skip to content

feat: add local telemetry playground and monitoring docs#286

Draft
udsmicrosoft wants to merge 1 commit intodocumentdb:mainfrom
udsmicrosoft:users/urismiley/telemetry-playground
Draft

feat: add local telemetry playground and monitoring docs#286
udsmicrosoft wants to merge 1 commit intodocumentdb:mainfrom
udsmicrosoft:users/urismiley/telemetry-playground

Conversation

@udsmicrosoft
Copy link
Collaborator

Summary

Adds a complete local telemetry demo for Kind clusters and monitoring documentation.

Local Telemetry Playground (documentdb-playground/telemetry/local/)

  • 3-instance DocumentDB HA cluster (1 primary + 2 streaming replicas) on Kind
  • Full observability stack: OTel Collector, Prometheus, Tempo, Loki, Grafana
  • Gateway metrics, traces, and logs via OTLP push
  • PostgreSQL metrics via the OTel postgresql receiver
  • System resource metrics via the kubeletstats receiver
  • Pre-built Grafana dashboard with Gateway, PostgreSQL, and System Resources sections
  • Traffic generator Job for demo workload
  • Kind cluster setup script with local Docker registry

Sidecar Injector Changes

  • Add POD_NAME env var (from K8s downward API) and OTEL_RESOURCE_ATTRIBUTES with service.instance.id=$(POD_NAME) for per-instance metric attribution in multi-replica deployments

Monitoring Documentation (docs/.../monitoring/)

  • Architecture overview showing OTLP push flow from gateway, collector deployment modes, and verification steps
  • Metrics reference with actual gateway metrics (db_client_operations_total, gateway_client_connections_active, pool metrics, etc.) and PromQL examples
  • Updated mkdocs.yml navigation

Related

Add a complete local telemetry demo for Kind clusters with:
- 3-instance DocumentDB HA cluster (1 primary + 2 streaming replicas)
- Full observability stack: OTel Collector, Prometheus, Tempo, Loki, Grafana
- Gateway metrics, traces, and logs via OTLP push
- PostgreSQL metrics via the OTel postgresql receiver
- System resource metrics via the kubeletstats receiver
- Pre-built Grafana dashboard with Gateway, PostgreSQL, and System Resources
- Traffic generator Job for demo workload
- Kind cluster setup script with local Docker registry

Sidecar injector changes:
- Add POD_NAME env var (from downward API) and OTEL_RESOURCE_ATTRIBUTES
  with service.instance.id for per-instance metric attribution

Monitoring documentation:
- Architecture overview with OTLP push flow and collector deployment modes
- Metrics reference with actual gateway metrics and PromQL examples
- Updated mkdocs.yml navigation

Signed-off-by: urismiley <urismiley@microsoft.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant