feat: Add native OpenTelemetry (OTLP) support for metrics and traces

[knechtionscoding]

What type of PR is this?

/kind feature

What this PR does / why we need it:

Adds native OpenTelemetry (OTLP) support for metrics and traces to the Kelos
controller and spawner. Previously the only observability surface was the
Prometheus /metrics pull endpoint, and there was no distributed tracing at
all — so operators running a mixed-signal stack (OTEL Collector →
Grafana/Tempo/Honeycomb/Datadog/etc.) had to bolt on a Prometheus-scrape bridge
and had no way to see the spawner → Task → agent-pod pipeline as a single trace.

Everything here is opt-in and backward-compatible: with no OTLP endpoint
configured, behavior is identical to today (Prometheus /metrics only). Setting
OTEL_EXPORTER_OTLP_ENDPOINT (or the Helm observability.otlp.endpoint value)
turns on OTLP export.

This PR folds in the distributed-tracing scope from #797 alongside the
metrics/traces scope of #1003, per maintainer agreement to consolidate the two
into one feature.

What it adds:

• Foundation (internal/observability): Setup() installs global
  Tracer/Meter providers with OTLP exporters when an endpoint is configured, and
  returns a graceful shutdown. It honors the standard OTEL environment variables
  (OTEL_EXPORTER_OTLP_ENDPOINT, ..._PROTOCOL, ..._HEADERS,
  OTEL_RESOURCE_ATTRIBUTES, OTEL_SERVICE_NAME) and selects grpc or
  http/protobuf. It fails fast if an endpoint is set but an exporter cannot
  be built, rather than silently degrading to no export.
• Metrics via the OTEL Prometheus bridge: rather than re-instrumenting every
  metric, the existing controller-runtime metrics.Registry (which both
  binaries already register on) is bridged into an OTLP PeriodicReader. Every
  metric already scraped from /metrics is also pushed over OTLP, and the pull
  endpoint keeps working unchanged.
• Traces: each reconciler emits a reconcile.<Kind> span (all nine
  reconcilers instrumented uniformly). The spawner emits a spawner.discover
  span per discovery cycle with a child spawner.create_task span per Task.
• End-to-end propagation: the spawner stamps W3C trace context onto each
  Task's kelos.dev/traceparent annotation; the Task controller extracts it so
  its reconcile joins the same trace; and the agent pod receives it as the
  standard TRACEPARENT env var so OTEL-aware agents can continue the trace.
  The controller also mirrors its OTLP env onto the spawner Deployments it
  launches so they export to the same collector.

Which issue(s) this PR is related to:

Fixes #1003

Folds in and supersedes #797 (OpenTelemetry distributed tracing) — the tracing
design from #797 is implemented here.

Special notes for your reviewer:

• Opt-in / zero migration: Setup() and all propagation helpers are no-ops
  when no OTLP endpoint is configured; the Prometheus endpoint is untouched. The
  Helm chart renders no OTEL env unless observability.otlp.endpoint is set
  (verified via helm template).

• Prometheus bridge, not re-instrumentation: metrics export reads the shared
  sigs.k8s.io/controller-runtime/pkg/metrics registry via
  go.opentelemetry.io/contrib/bridges/prometheus, so no metric call sites
  changed and there is no risk of metric drift between the two surfaces.

• Reconcile spans are added as a uniform two-line insert per reconciler
  (startReconcileSpan in internal/controller/tracing.go). Spans resolve
  through the global TracerProvider, so they are cheap no-ops until Setup
  installs a provider.

• Dependency churn: adding OpenTelemetry v1.44 pulls forward several
  transitive deps (grpc, golang.org/x/*, genproto, protobuf). go mod tidy
  is clean and make verify passes.

• Scope / follow-ups: OTLP logs export (issue item 4) is intentionally
  left as a follow-up. Per-step spans inside a reconcile (e.g.
  task.check_dependencies, task.build_job) can be layered on later; this PR
  establishes the per-reconcile span and the cross-resource trace so those are
  additive.

• In-pod tracing is scoped to one-shot Task pods (important — follow-up
  planned): The TRACEPARENT env injection assumes the identity one pod =
  one Task = one trace, which only holds for the one-shot agent Job created per
  Task. It does not cover long-lived, multiplexed pods:

  • Session pods are built through JobBuilder.Build from a synthetic Task
    that carries no trace annotation, so no TRACEPARENT is injected. (This is
    the correct outcome — a session pod handles many turns, so a single static
    traceparent would incorrectly fuse unrelated turns into one unbounded
    trace.)
  • Worker pods are built via WorkerPoolReconciler.buildStatefulSet and
    never go through JobBuilder.Build, so they receive no TRACEPARENT at
    all.

  Controller-side coverage is unaffected: Session/SessionSpawner/WorkerPool
  reconciles still emit their own reconcile.<Kind> spans (one per reconcile),
  and metrics are aggregated on the controller/spawner registries independent of
  pod lifetime. Only in-pod trace continuation is limited to the Task path.

  The correct model for long-lived pods is per-operation runtime propagation —
  each session turn / each task pickup carries and continues its own trace
  context (via the Session/Task record or dispatch payload) rather than a
  process-wide env var — and requires instrumenting the session/worker runtime
  itself. This is deliberately deferred to a follow-up PR.

• Tests: added coverage for Enabled(), the disabled no-op path,
  traceparent inject/extract round-trip, TRACEPARENT pod-env injection
  (present and absent), and controller→spawner OTLP env propagation (on and
  off). Docs added under docs/reference.md.

• Ran make verify and make test — both pass.

Does this PR introduce a user-facing change?

Kelos now supports native OpenTelemetry (OTLP) export for metrics and traces. Set OTEL_EXPORTER_OTLP_ENDPOINT (or the Helm observability.otlp values) to push controller/spawner metrics and end-to-end spawner→Task→agent-pod traces to an OTLP collector, in addition to the existing Prometheus /metrics endpoint. Export is opt-in; unset means unchanged Prometheus-only behavior.

---

Summary by cubic

Adds native OpenTelemetry (OTLP) export for metrics and traces across the controller and spawner. It’s opt-in and keeps the Prometheus /metrics endpoint unchanged by default, with secure Secret-based header propagation to spawner pods.

• New Features

  • internal/observability.Setup() installs global OTEL Tracer/Meter when an endpoint is set; honors standard OTEL env; supports gRPC and HTTP/protobuf; fails fast on misconfig.
  • Metrics are pushed via the Prometheus bridge (go.opentelemetry.io/contrib/bridges/prometheus) while the /metrics endpoint stays.
  • Traces: per-reconcile spans (reconcile.<Kind>), plus spawner spans (spawner.discover, spawner.create_task).
  • End-to-end propagation: spawner writes W3C context to kelos.dev/traceparent; Task reconcile continues it; agent Jobs get TRACEPARENT env.
  • Controller mirrors OTLP config to spawner Deployments. Supports Secret-based OTLP headers: set observability.otlp.headersSecretName (and optional headersSecretKey) so the controller and spawner read OTEL_EXPORTER_OTLP_HEADERS via secretKeyRef without copying tokens in plaintext.
  • Helm: new observability.otlp values (endpoint, protocol, headers, headersSecretName, headersSecretKey, resourceAttributes). Docs and tests added.

• Migration

  • Default behavior is unchanged. No action needed unless enabling OTLP.
  • To enable: set OTEL_EXPORTER_OTLP_ENDPOINT or Helm observability.otlp.endpoint. For bearer tokens, use observability.otlp.headersSecretName instead of plaintext headers.

^{Written for commit 4310b85. Summary will update on new commits.}

[cubic-dev-ai]

All reported issues were addressed across 23 files

<sub>Reply with feedback, questions, or to request a fix.

Re-trigger cubic</sub>