feat(collector): derive request metrics with trace exemplars from spans

[jordan-simonovski]

First of five PRs replacing #2536, which grew to 85 files and four rounds of review.

Why this is on its own

This changes what the collector runs, and it has its own rollout ordering. A round-1 review on the original PR made the point directly: a config naming a component type the binary does not register fails to decode as a whole, and the shipped docker/otel-collector/config.yaml defines no pipelines of its own — so a bad type name leaves the collector with nothing to run and stops all ingestion, not just this feature. That is not something to ship in the same revert unit as a flag-gated UI overlay.

Merge and roll out the collector image before enabling the flag.

What it does

Adds spanmetricsconnector to the collector build and wires it into the generated OpAMP config behind ENABLE_SPAN_METRICS, off by default. The connector consumes the traces pipeline and feeds a dedicated metrics pipeline, so traces.span.metrics.* reach ClickHouse with Exemplars.* pointing back at the spans they were measured from.

Buckets are exponential rather than a fixed ladder. An explicit ladder puts everything slow into one wide top bucket, so a high quantile interpolates well past the slowest real request — and then no exemplar can sit on the plotted line, which defeats the point of the metrics.

ENABLE_SPAN_METRICS_PROM_RW additionally remote-writes the derived metrics to a Prometheus endpoint, for exercising Prometheus's native exemplar path. The endpoint is resolved API-side and inlined into the generated config, so the collector container does not need it in its own environment; the flag stays off unless the endpoint is set.

Note for the reviewer

The connector key must be exactly spanmetrics. This was a P0 on the original PR — the config said span_metrics, which the binary does not register. There is now a test that reads builder-config.yaml and pins every generated component id against what the build actually registers, including pipeline processor references. It fails if the name regresses.

Verification

make ci-lint and make ci-unit pass. The OpAMP config tests cover the flag off, the flag on, the remote-write variant, and the component-type pinning.

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated (UTC)
hyperdx-oss	Ready	Preview	Aug 5, 2026 10:00pm
hyperdx-storybook	Ready	Preview	Aug 5, 2026 10:00pm

[changeset-bot]

🦋 Changeset detected

Latest commit: a46bf56

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 3 packages

Name	Type
@hyperdx/otel-collector	Minor
@hyperdx/api	Minor
@hyperdx/app	Minor

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

[github-actions]

🔴 Tier 4 — Critical

Touches authentication, tenancy data models, the public API or shipped database config — or substantially changes background tasks, the OTel pipeline, image build, or release CI.

Why this tier:

• Critical-path files (1) — tenancy, public API, or shipped database config:
  • packages/api/src/config.ts

Additional context: touches background tasks or the delivery pipeline lightly (3 lines, under the 30-line bar for Tier 4)

Review process: Deep review from a domain expert. Synchronous walkthrough may be required.
SLA: Schedule synchronous review within 2 business days.

Stats

• Production files changed: 4
• Production lines changed: 199 (+ 334 in test files, excluded from tier calculation)
• Critical-path lines changed: 40
• Branch: jordansimonovski/collector-spanmetrics
• Author: jordan-simonovski

To override this classification, remove the review/tier-4 label and apply a different review/tier-* label. Manual overrides are preserved on subsequent pushes.

[greptile-apps]

Greptile Summary

The PR adds opt-in span-derived request metrics with trace exemplars and an optional Prometheus remote-write sink.

• Registers the spanmetrics connector in the custom collector build.
• Generates a bounded spanmetrics pipeline with exponential histograms and ClickHouse export.
• Validates the optional remote-write endpoint and keeps HTTPS certificate verification enabled.
• Adds tests covering feature flags, pipeline wiring, component registration, cardinality controls, and missing endpoint behavior.

Confidence Score: 5/5

The PR appears safe to merge.

No blocking failure remains.

Important Files Changed

Filename	Overview
packages/api/src/config.ts	Adds opt-in spanmetrics flags and validates the optional remote-write URL before exposing it to generated collector configuration.
packages/api/src/opamp/controllers/opampController.ts	Generates the spanmetrics connector and metrics pipeline with bounded aggregation, exponential histograms, exemplars, and optional verified remote write.
packages/api/src/opamp/controllers/tests/opampController.test.ts	Covers disabled and enabled configurations, exporter wiring, component registration, bootstrap compatibility, cardinality controls, and endpoint absence.
packages/api/src/tests/config.test.ts	Verifies acceptance of HTTP(S) endpoints and rejection of URL userinfo, unsupported schemes, malformed values, and missing configuration.
packages/otel-collector/builder-config.yaml	Registers the published spanmetrics connector module in the custom collector binary.
packages/otel-collector/README.md	Documents rollout ordering, cardinality controls, exemplars, and optional Prometheus remote write.
.changeset/spanmetrics-connector.md	Records the API and collector behavior change as minor releases.
.gitignore	Excludes Stryker mutation-testing output from future changes.

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart LR
  Spans[OTLP spans] --> Traces[Traces pipeline]
  Traces --> ClickHouseTraces[ClickHouse trace export]
  Traces --> SpanMetrics[spanmetrics connector]
  SpanMetrics --> MetricsPipeline[metrics/spanmetrics pipeline]
  MetricsPipeline --> ClickHouseMetrics[ClickHouse metrics export]
  MetricsPipeline -. optional .-> Prometheus[Prometheus remote write]
  SpanMetrics -. trace and span IDs .-> Exemplars[Metric exemplars]

Loading

_{Reviews (4): Last reviewed commit: "Merge remote-tracking branch 'origin/mai..." | Re-trigger Greptile}

[github-actions]

E2E Test Results

✅ All tests passed • 270 passed • 1 skipped • 965s

Status	Count
✅ Passed	270
❌ Failed	0
⚠️ Flaky	0
⏭️ Skipped	1

Tests ran across 4 shards in parallel.

View full report →

[github-actions]

<!-- deep-review -->

Deep Review

✅ No critical issues found. Nothing here breaks on the happy path: both flags default off, so merging this diff is a no-op at runtime. builder-config.yaml correctly registers spanmetricsconnector, the generated connector key is spanmetrics (matching the registered type), and both collector Dockerfiles — standalone (docker/otel-collector/Dockerfile:21) and all-in-one (docker/hyperdx/Dockerfile:31) — build from the same manifest, so the all-in-one image cannot suffer version skew. The component-pinning test does pass against the current generated ids.

The P2 items below are worth resolving before enabling the flag anywhere.

🟡 P2 -- recommended

• packages/api/src/opamp/controllers/opampController.ts:385 -- Nothing gates emission of the spanmetrics connector on the collector's actual version, so enabling the flag before the new image is rolled out makes every agent reject the whole remote config, and because docker/otel-collector/config.yaml:174 declares only pipeline processors: and no receivers or exporters, a rejected config leaves the collector with no pipelines at all rather than just no span metrics.
  • Fix: Gate the connector block on the agent's reported collector version, which handleOpampMessage already reads at line 589, or add receiver/exporter fallbacks to the bootstrap config so a rejected remote config cannot zero out ingestion.
• packages/api/src/opamp/controllers/opampController.ts:423 -- The new metrics/spanmetrics pipeline sets processors: ['memory_limiter', 'batch'], which the comment at lines 302-310 and packages/otel-collector/README.md:276 both state the remote config deliberately never does because a remote processors: list overrides the bootstrap-plus-custom merge.
  • Fix: Declare the metrics/spanmetrics processors list in docker/otel-collector/config.yaml and omit processors: from the generated pipeline so CUSTOM_OTELCOL_CONFIG_FILE overrides still apply.
• packages/api/src/opamp/controllers/opampController.ts:396 -- The dimensions list combines app.tenant_id with http.route while aggregation_cardinality_limit is unset and aggregation_temporality is left at its cumulative default, so series are retained indefinitely and re-emitted to ClickHouse on every 15s flush with no upper bound on count.
  • Fix: Set an explicit aggregation_cardinality_limit and drop or bound the app.tenant_id dimension.
• packages/api/src/opamp/controllers/opampController.ts:396 -- Three of the five dimensions will not populate for spans from current SDKs: http.method and http.status_code are the pre-stable names for http.request.method and http.response.status_code, and host.region is not a semantic convention at all.
  • Fix: Switch to the stable semconv attribute names and replace host.region with cloud.region or remove it.
• packages/api/src/config.ts:60 -- The comment states the feature is "enabled in dev", but packages/api/.env.development:28 sets only ENABLE_PROMQL=true and docker-compose.dev.yml never sets ENABLE_SPAN_METRICS, so no default path in the repo exercises this code at runtime.
  • Fix: Add ENABLE_SPAN_METRICS=true to the dev env file, or correct the comment to state the flag is off everywhere by default.
• packages/api/src/opamp/controllers/__tests__/opampController.test.ts:228 -- The registers() helper substring-matches the manifest and ORs across all four component kinds, so an id registered as one kind satisfies the assertion for another; declaring an exporters.prometheus entry would pass because /prometheusreceiver appears in the manifest even though no Prometheus exporter is built.
  • Fix: Match only the component kind matching the section the id was declared in, rather than any of the four.
• packages/api/src/opamp/controllers/__tests__/opampController.test.ts:15 -- The mock exposes IS_SPAN_METRICS_PROM_RW_ENABLED and SPAN_METRICS_PROM_RW_ENDPOINT as independent getters, so the real derivation at packages/api/src/config.ts:72 that makes the non-null assertion at line 417 safe is never exercised; removing that guard would drop the exporter's endpoint key from the serialized config with no test failing.
  • Fix: Add a case asserting prometheusremotewrite/spanmetrics is omitted when the remote-write flag is on but the endpoint is unset.

🔵 P3 nitpicks (6)

• packages/api/src/opamp/controllers/opampController.ts:113 -- The namespace?: string field on the connector type is never assigned anywhere, leaving dead type surface that implies a knob the code does not set.
  • Fix: Remove the field or set it explicitly rather than relying on the connector's default namespace.
• packages/api/src/opamp/controllers/opampController.ts:156 -- prometheusremotewrite and prometheusremotewrite/spanmetrics declare byte-identical object shapes in the CollectorConfig type.
  • Fix: Extract a single named type and reference it from both keys.
• packages/api/src/opamp/controllers/opampController.ts:418 -- tls: { insecure: true } is carried over from the in-cluster ClickHouse exporter, but for an HTTP exporter insecure disables TLS rather than skipping verification, so it is inert and misleading against an https:// operator-supplied endpoint.
  • Fix: Drop the tls block, or use the field that actually expresses the intended behavior for this endpoint.
• packages/otel-collector/README.md:103 -- The Connectors inventory table, which the package README maintains as the reference for what the build includes, was not updated for spanmetrics.
  • Fix: Add a spanmetrics row noting it is used by the OpAMP controller behind the opt-in flag.
• packages/api/src/opamp/controllers/opampController.ts:366 -- The otelCollectorConfig.connectors && otelCollectorConfig.exporters conjuncts can never be false, since both keys are assigned object literals at construction.
  • Fix: Drop the two truthiness checks and keep only the flag condition.
• packages/api/src/opamp/controllers/opampController.ts:364 -- The span-metrics block sits inside the apiKeys.length > 0 guard, so the flag silently does nothing on a deployment with no team API keys, which is not documented alongside the flag.
  • Fix: Note the API-key precondition in the flag comment in config.ts.

---

Reviewers (1): orchestrator direct analysis. Seven personas (correctness, testing, adversarial, maintainability, project-standards, performance, security) were dispatched but had not returned before output was required, so every finding above was verified by the orchestrator directly against the cited lines; none are attributed to agent output. Environment caveat: the Bash tool failed on every invocation (bwrap sandbox init) and no Grep/Glob tools were available, so scope was reconstructed by reading exact paths rather than from git diff; a changed file outside the reconstructed set could have been missed.

Testing gaps:

• No test covers the remote-write flag on with the endpoint unset — the guard protecting the non-null assertion.
• No test asserts the processors list on the new metrics/spanmetrics pipeline.
• No test covers span metrics enabled with zero team API keys, where the feature silently no-ops.
• The component-pinning test leaves ENABLE_DATADOG_RECEIVER off, so the datadog receiver id is never validated against the manifest.
• No test or check asserts the emitted dimensions correspond to attributes real spans carry.

[github-actions]

Deep Review

⚠️ Degraded run. The multi-agent fan-out could not execute: Bash fails on every command in this environment (bwrap: Can't create file at /home/.mcp.json: Permission denied), and Glob/Grep are unavailable — confirmed for sub-agents too via a probe. With no git diff, reviewer sub-agents would have had no scope and no search. Findings below come from a single-pass manual review reading the touched files at exact paths. See Coverage limits at the end.

✅ No critical issues found. No P0/P1: nothing here is a guaranteed happy-path break, and the whole feature is gated off by default (ENABLE_SPAN_METRICS unset ⇒ IS_SPAN_METRICS_ENABLED === false, packages/api/src/config.ts:69).

🟡 P2 -- recommended

• packages/api/src/opamp/controllers/opampController.ts:410 -- exemplars: { enabled: true } sets no max_per_data_point, and the connector's default is unlimited, so every span in each 15s flush window attaches an exemplar to its series.
  • Fix: Add max_per_data_point (a small single-digit value) to the exemplars block so exemplar volume per data point is bounded.
• packages/api/src/opamp/controllers/opampController.ts:460 -- resource_to_telemetry_conversion: { enabled: false } only stops resource attributes being copied onto metric labels; the prometheusremotewrite exporter still emits a target_info series carrying resource attributes, so host.name/k8s.pod.name/k8s.namespace.name do reach the third-party endpoint despite the adjacent comment claiming they do not.
  • Fix: Disable target_info on the prometheusremotewrite/spanmetrics exporter (or strip resource attributes ahead of it) and correct the comment to match actual behavior.
• packages/api/src/opamp/controllers/opampController.ts:429 -- aggregation_cardinality_limit: 10_000 applies per resource-cache entry, so with resource_metrics_cache_size left at its default the ceiling is roughly 1000 × 10 000 cumulative series, each rewritten to ClickHouse on every 15s flush because cumulative series are never evicted.
  • Fix: Set resource_metrics_cache_size explicitly and lower aggregation_cardinality_limit so the worst-case series count and write volume are a stated, bounded number.
• packages/api/src/opamp/controllers/opampController.ts:453 -- the remote-write exporter block sets only endpoint and resource_to_telemetry_conversion, while the connector emits exponential (native) histograms exclusively; if the pinned collector build does not send native histograms over remote write by default, the endpoint receives traces.span.metrics.calls but never the latency histogram the flag exists to exercise.
  • Fix: Verify against collector 0.155.0 that exponential histograms are remote-written, and set the exporter's native-histogram option explicitly rather than relying on the default.
• packages/api/src/opamp/controllers/__tests__/opampController.test.ts:288 -- the component-pinning loop walks only each pipeline's receivers and exporters, so the new metrics/spanmetrics pipeline's processors: ['memory_limiter', 'batch'] and service.extensions are pinned against nothing, leaving the exact class of typo the test exists to catch uncovered for those keys.
  • Fix: Extend the loop to assert every processors entry resolves to a processor declared in docker/otel-collector/config.yaml and every service.extensions entry resolves to a declared extension.
• packages/api/src/opamp/controllers/opampController.ts:364 -- the span-metrics block is nested inside if (apiKeys && apiKeys.length > 0) even though it depends on no API key, so with ENABLE_DATADOG_RECEIVER=true and no team keys the datadog receiver still feeds the traces pipeline at line 509 while ENABLE_SPAN_METRICS=true is silently ignored.
  • Fix: Hoist the span-metrics block out of the API-key branch so it is driven only by its own flag.
• packages/api/src/config.ts:84 -- an invalid SPAN_METRICS_PROM_RW_ENDPOINT (unparseable, non-HTTP scheme, or credential-bearing) is silently coerced to undefined, so a typo disables the feature with no log line and no startup signal anywhere.
  • Fix: Log a warning naming the rejection reason whenever a non-empty value is discarded.
• packages/otel-collector/README.md:197 -- the remote-write section does not state that the target Prometheus must run with --web.enable-remote-write-receiver and --enable-feature=exemplar-storage, and the repo's own dev Prometheus at docker-compose.dev.yml:162 overrides no command flags, so following the docs produces a 404 loop from the exporter and silently discarded exemplars.
  • Fix: Document both required Prometheus flags and add them to the dev prometheus service command.

🔵 P3 nitpicks (5)

• packages/api/src/opamp/controllers/__tests__/opampController.test.ts:262 -- CORE_COMPONENTS excludes nop, debug, memory_limiter and batch on the stated grounds that they are not declared as gomods, but nopreceiver, nopexporter and debugexporter are declared in packages/otel-collector/builder-config.yaml, and memory_limiter/batch never enter declaredIds at all, so the list only weakens the pin for nop and debug.
  • Fix: Delete the CORE_COMPONENTS skip list and its comment.
• packages/api/src/config.ts:60 -- the rollout-ordering and cardinality rationale is written out three times in near-identical prose across config.ts:60-83, opampController.ts:364-435, and packages/otel-collector/README.md:161-206, giving three places to drift.
  • Fix: Keep the long-form rationale in the README and reduce the two code comments to a one-line pointer.
• docker/otel-collector/config.standalone.yaml:1 -- the header declares this file the standalone mirror of buildOtelCollectorConfig() and asks that it stay in sync, but there is no span-metrics equivalent, so the feature is unreachable in standalone mode.
  • Fix: Record the intentional OpAMP-only scope in that header, or add an env-gated fragment mirroring config.standalone.promql.yaml.
• packages/api/src/config.ts:94 -- the credential check inspects only url.username/url.password, so an endpoint carrying a token in its query string is still inlined into the config served from the unauthenticated OpAMP endpoint.
  • Fix: Reject endpoints that carry a query string, or strip it before inlining.
• packages/api/src/opamp/controllers/opampController.ts:187 -- buildOtelCollectorConfig and its inline CollectorConfig type now dominate a 771-line controller, well past the 300-line guidance in AGENTS.md.
  • Fix: Move buildOtelCollectorConfig and CollectorConfig into their own module under packages/api/src/opamp/.

---

Reviewers (1): manual single-pass review by the orchestrator — the plugin's persona sub-agents (correctness, testing, maintainability, project-standards, plus cross-cutting) could not be dispatched usefully because Bash, Glob, Grep and WebFetch are all unavailable in this environment, leaving sub-agents with no diff and no search.

Coverage limits:

• No git access, so the changed-file set was inferred from the PR's stated scope and then read directly; files reviewed were packages/api/src/opamp/controllers/opampController.ts, packages/api/src/opamp/controllers/__tests__/opampController.test.ts, packages/api/src/config.ts, packages/api/src/__tests__/config.test.ts, packages/otel-collector/builder-config.yaml, packages/otel-collector/README.md, docker/otel-collector/config.yaml, and supporting Docker/compose files. Any other file this PR touches was not seen.
• Base-vs-head comparison was impossible, so pre-existing versus newly-introduced lines could not be separated with certainty; findings are anchored to code that is load-bearing for this feature.
• .changeset/ could not be enumerated, so whether the required changeset for @hyperdx/api / @hyperdx/otel-collector (AGENTS.md:205-211) exists is unverified — not reported as a finding.

Testing gaps:

• Pipeline processors: and service.extensions references are not pinned against declared components, unlike receivers/exporters.
• No case covers IS_SPAN_METRICS_ENABLED = true with zero team API keys, which is exactly the path where the feature silently no-ops.
• The bootstrapPipelineNames helper scans for the literal line pipelines: at a fixed indent, so any reformatting of docker/otel-collector/config.yaml turns the pipeline-coverage assertions vacuous rather than failing loudly.

[jordan-simonovski]

Closing this.

The connector is not required for the exemplars feature. Nothing in #2806–#2809 depends on it — the only reference anywhere in the stack is the string traces.span.metrics.duration used as test data in a chart-config test. It was mostly a way to generate exemplar-carrying metrics for local testing, and in exchange it adds a feature to the OpAMP config surface that is not otherwise supported.

The branch jordansimonovski/collector-spanmetrics is preserved. Four things in it are not about the connector, if any turn out to be worth picking up separately:

• .gitignore entries for Stryker output (packages/*/reports/, packages/*/.stryker-tmp/).
• A test asserting every component id the OpAMP controller generates matches something builder-config.yaml actually builds. A config naming an unregistered component type fails to decode as a whole, so a typo takes ingestion down rather than disabling one feature.
• A test asserting every pipeline declared in docker/otel-collector/config.yaml is filled in by the generated config. The supervisor merges that file unconditionally and the collector rejects a pipeline with no receivers or exporters, so a bootstrap-declared pipeline the generated config omits stops the agent starting at all.
• A README row for the pre-existing prometheusremotewrite exporter behind ENABLE_PROMQL, which the Exporters inventory never listed.