feat(api): add /v1/prometheus/query_exemplars and harden the proxy

[jordan-simonovski]

Second of five PRs replacing #2536. Independent of #2805 — either can merge first.

What it does

Adds /v1/prometheus/query_exemplars, proxying to Prometheus's native endpoint for Prometheus-backed connections. ClickHouse-backed connections get an empty success: there is no table function for exemplars, and those charts read them from the metric table client-side instead.

The window is narrowed rather than rejected when it exceeds the cap. A 30-day dashboard range is an ordinary request, and Prometheus keeps exemplars in a small recent circular buffer, so the older part of a wide window has nothing to return — rejecting it would surface as a chart-level error on a perfectly healthy chart. Genuinely invalid or inverted bounds still 400.

Three fixes to the shared proxy

This adds a fourth entry point into proxyToPrometheus, so it fixes what was already there:

Security (pre-existing on main, from 973d120). The proxy forwarded the upstream content-type verbatim with no nosniff. The connection host is member-configured, /api/* is same-origin-proxied by the app, and the session cookie is sameSite: lax — so a text/html body from a hostile host would render as script on our own origin. Now sends nosniff unconditionally and passes the content-type through only for JSON media types. Not introduced by this work, but this widens the surface, so it is fixed here.

Error accounting. proxyToPrometheus handles its own failures by writing 400/502/504 and returning normally, so the callers' catch never executed. All four proxied endpoints reported zero errors while still recording duration. Now counted — on 5xx only, since an upstream 4xx is usually a user's malformed PromQL and counting those makes the metric track typos rather than backend health.

Test typing. The integration-test upstream fixture is typed as Response, dropping the as any at its call sites. Incidentally this brings @hyperdx/api back under its eslint warning ceiling, which the original PR had exceeded.

Verification

make ci-lint and make ci-unit pass. resolveExemplarWindow was extracted as a pure function so the bounding logic has unit tests that do not need Docker — 7 cases covering narrowing, the inverted range, missing and unparseable bounds, and ISO timestamps. The header hardening has integration tests (Docker-gated, typechecked but not executed here).

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated (UTC)
hyperdx-oss	Ready	Preview	Aug 6, 2026 6:20am
hyperdx-storybook	Ready	Preview	Aug 6, 2026 6:20am

[changeset-bot]

🦋 Changeset detected

Latest commit: 533c4c5

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 3 packages

Name	Type
@hyperdx/api	Minor
@hyperdx/app	Minor
@hyperdx/otel-collector	Minor

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

[github-actions]

🟡 Tier 3 — Standard

Introduces new logic, modifies core functionality, or touches areas with non-trivial risk.

Why this tier:

• Diff size: 262 production lines changed (Tier 2 max: < 250)
• Touches API routes or data models — hidden complexity risk

Review process: Full human review — logic, architecture, edge cases.
SLA: First-pass feedback within 1 business day.

Stats

• Production files changed: 1
• Production lines changed: 262 (+ 455 in test files, excluded from tier calculation)
• Branch: jordansimonovski/prometheus-query-exemplars
• Author: jordan-simonovski

To override this classification, remove the review/tier-3 label and apply a different review/tier-* label. Manual overrides are preserved on subsequent pushes.

[greptile-apps]

Greptile Summary

This PR adds a Prometheus exemplar-query endpoint and strengthens behavior shared by the Prometheus proxy.

• Proxies exemplar requests for Prometheus-backed connections while returning an empty success for ClickHouse-backed connections.
• Bounds wide exemplar windows by moving the start time forward while preserving the requested end.
• Relabels streamed responses as JSON, adds nosniff, redacts URL credentials in proxy-generated messages, and records server-side proxy failures.
• Adds unit and integration coverage for routing, window handling, response headers, error accounting, and stream-disconnect classification.

Confidence Score: 5/5

The PR appears safe to merge.

No blocking failure remains.

Important Files Changed

Filename	Overview
packages/api/src/routers/api/prometheus.ts	Adds the exemplar route, bounded-window resolution, proxy response hardening, credential-safe target rendering, and status-based error accounting.
packages/api/src/routers/api/tests/prometheus.int.test.ts	Adds integration coverage for exemplar routing, team scoping, backend branching, outgoing window bounds, hardened headers, and credential redaction.
packages/api/src/routers/api/tests/prometheus.test.ts	Adds focused unit coverage for exemplar-window resolution, proxy outcome accounting, and disconnect classification.
.changeset/query-exemplars-route.md	Documents the new endpoint and shared proxy behavior changes for the API package release.

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart TD
  A[Authenticated query_exemplars request] --> B{Connection exists for team?}
  B -- No --> C[404 bad_data]
  B -- Yes --> D{Prometheus-backed?}
  D -- No --> E[Return success with empty data]
  D -- Yes --> F[Parse and bound start/end]
  F --> G{Valid window?}
  G -- No --> H[400 bad_data]
  G -- Yes --> I[Proxy to Prometheus query_exemplars]
  I --> J[Stream response as application/json with nosniff]
  I --> K[Record server-side proxy failures]

Loading

_{Reviews (4): Last reviewed commit: "Merge branch 'main' into jordansimonovsk..." | Re-trigger Greptile}

[github-actions]

E2E Test Results

✅ All tests passed • 271 passed • 1 skipped • 1041s

Status	Count
✅ Passed	271
❌ Failed	0
⚠️ Flaky	1
⏭️ Skipped	1

Tests ran across 4 shards in parallel.

View full report →

[github-actions]

            <!-- deep-review -->

Deep Review

Scope note: Bash was non-functional in this environment (bwrap failure on every invocation, sandboxed and unsandboxed) and network fetch was unavailable, so no git diff could be computed. The review surface was reconstructed by reading the working tree directly. Findings marked pre-existing are classified from code reading rather than from blame, so that classification is less certain than the findings themselves.

✅ No critical issues found. Nothing in this diff introduces data loss, an auth bypass, an injection, a leaked secret, or a happy-path crash. The nosniff + relabel hardening holds up under scrutiny: CORS is pinned to FRONTEND_URL, no global helmet sets the header, and application/json; charset=utf-8 + nosniff blocks the HTML-rendering and JSON-hijacking paths the change targets. The recordProxyOutcome fix correctly closes the swallowed-error gap it describes. What follows is mostly about blind spots the new 5xx-only gate leaves open, and coverage that does not run.

🟡 P2 -- recommended

• packages/api/src/routers/api/prometheus.ts:226 -- Pre-existing: any ordinary team member can create a connection whose host is an arbitrary URL and have this proxy fetch it and stream the full body back, reaching loopback, RFC1918, and 169.254.169.254 metadata targets; the new endpoint inherits that surface rather than widening it.
  • Fix: Require an http(s) scheme and reject loopback, link-local, and RFC1918 resolved targets before fetch, and pass redirect: 'manual' so a permitted host cannot redirect onto an internal one.
  • _{security, adversarial}
• packages/api/src/routers/api/clickhouseProxy.ts:281 -- The untrusted-member-configured-upstream threat model this PR acts on applies equally to the ClickHouse proxy, whose proxyRes hook rewrites only CORS headers and forwards the upstream content-type verbatim with no nosniff, leaving the same hole open on the same origin.
  • Fix: Apply the same unconditional content-type relabel and x-content-type-options: nosniff in the proxyRes hook of clickhouseProxy.ts.
  • _{learnings-researcher, security}
• packages/api/src/routers/api/prometheus.ts:316 -- The 5xx-only gate means a connection host that is a permanently unparseable URL returns 400 on every request forever while hyperdx.prometheus.query_errors stays at zero, so an alert on that counter cannot distinguish a fully misconfigured connection from a healthy one even though the user sees a dead chart.
  • Fix: Emit a distinct label or counter for the invalid-host branch at line 201 so a persistent server-side misconfiguration is visible to alerting instead of being bucketed with PromQL typos.
  • _{reliability, correctness, adversarial}
• packages/api/src/routers/api/prometheus.ts:695 -- A malformed connectionId such as ?connectionId=abc makes getConnectionById throw a Mongoose CastError into the handler's catch, which increments prometheusQueryErrors and returns 400 -- exactly the client-error-inflates-backend-health outcome the 5xx-only gate was introduced to prevent.
  • Fix: Validate connectionId as an ObjectId and return 400 before the database call so a malformed id never reaches the counter.
  • _{adversarial, security}
• packages/api/src/routers/api/prometheus.ts:199 -- Pre-existing: new URL(path, upstreamHost) resolves a root-absolute path against the host, silently discarding any subpath in the connection host, so a Prometheus served under --web.external-url at http://host/prometheus receives /api/v1/query_exemplars and returns a 404 HTML page that the new relabel now hands to the client as application/json.
  • Fix: Join the upstream path onto the host's existing pathname instead of resolving a root-absolute path against the base.
  • _{correctness, adversarial}
• packages/api/src/routers/api/__tests__/prometheus.int.test.ts:128 -- All five header-hardening tests sit inside describe('GET /v1/prometheus/query_range') but every one of them requests /v1/prometheus/query_exemplars, so the relabel and nosniff behavior is asserted on exactly one of the four routes the shared helper changed and query_range's own proxy responses are never checked.
  • Fix: Move the tests at lines 128-267 into the query_exemplars describe block and add one assertion per remaining route that content-type is relabelled and nosniff is present.
  • _{testing, correctness}
• packages/api/src/routers/api/prometheus.ts:274 -- The content-type relabel is the security fix at the centre of this change and its only coverage lives in the Docker-gated integration suite, which was typechecked but not executed for this change, so a regression here ships undetected by any job that actually runs.
  • Fix: Extract the relabel decision into an exported pure function so the unit suite that runs on every CI job can assert it.
• packages/api/src/routers/api/prometheus.ts:311 -- No test asserts that recordProxyOutcome is actually wired at any of its four call sites, so deleting any one of them leaves the suite green and silently reintroduces the zero-errors-reported bug this PR exists to fix.
  • Fix: Assert the mocked counter increments from a route-level 502 for each of the four proxied endpoints.
  • _{correctness, maintainability, reliability, testing}
• packages/api/src/routers/api/prometheus.ts:710 -- router.post('/query_exemplars') is registered but no test issues a POST to it, leaving the urlencoded-body-merged-over-query-params path through getParams completely unexercised on a route registration this diff adds.
  • Fix: Add an integration test that POSTs a form-encoded body to /v1/prometheus/query_exemplars and asserts the same outgoing URL as the GET case.
  • _{testing, api-contract}

🔵 P3 nitpicks (13)

• packages/api/src/routers/api/prometheus.ts:587 -- A whitespace-only bound passes the v === '' guard and Number(' ') returns 0, so ?start=%20 is accepted as epoch 0 and narrowed to a valid 7-day window instead of returning the 400 an unparseable bound gets.
  • Fix: Change the guard to if (v == null || v.trim() === '') return NaN;.
• packages/api/src/routers/api/prometheus.ts:204 -- Pre-existing: the Invalid URL 400 echoes JSON.stringify(upstreamHost), so a scheme-less host carrying basic-auth credentials returns the password in a response body the browser displays, on the one path that runs before the redactedTarget redaction exists.
  • Fix: Strip any //user:pass@ segment from upstreamHost before interpolating it into this error body.
• packages/api/src/routers/api/prometheus.ts:681 -- String(window.start) yields exponential notation for values at or above 1e21, so ?start=1e21&end=1e21 forwards 1e+21 and 400s upstream -- the precise class of accepted-here-rejected-there value the surrounding comment claims re-serializing both bounds prevents.
  • Fix: Reject resolved bounds outside a sane epoch range in resolveExemplarWindow.
• packages/api/src/routers/api/prometheus.ts:158 -- The 7-day cap bounds time, not exemplar volume, which is driven by matched-series cardinality; a wide regex selector returns a large body inside a one-second window, so the stated size protection does not hold and the real bounds are the stream plus the 90s timeout.
  • Fix: Correct the rationale comment and bound the streamed body by bytes if a size limit is actually wanted.
  • _{adversarial, performance}
• packages/api/src/routers/api/prometheus.ts:87 -- Pre-existing: getParams is typed Record<string, string> but express's qs parsing yields arrays and nested objects, so ?start[]=1700000000 coerces through Number() and is accepted as a valid bound while the declared type hides that any value may not be a string.
  • Fix: Type the return as string | string[] | ParsedQs per key and normalize or reject non-scalar values before use.
  • _{kieran-typescript, adversarial}
• packages/api/src/routers/api/prometheus.ts:585 -- resolveExemplarWindow returns an untagged {start,end} | {error} union, so narrowing depends on string keys and a typo like 'eror' in window compiles and silently evaluates false.
  • Fix: Return a discriminated union keyed on ok and narrow on that instead of in.
  • _{kieran-typescript, maintainability}
• packages/api/src/routers/api/prometheus.ts:313 -- recordProxyOutcome takes endpoint: string despite the comment at line 16 asserting it is a bounded enum, so a typo'd label at a fifth call site compiles cleanly and only shows up as a stray series on a dashboard.
  • Fix: Introduce a PrometheusEndpoint union type and use it for the endpoint parameter and all four inline metric call sites.
• packages/api/src/routers/api/__tests__/prometheus.test.ts:194 -- The ISO-timestamp case asserts only 'start' in result and never the computed value, so it would still pass if the seconds-versus-milliseconds conversion were wrong by a factor of 1000.
  • Fix: Assert the numeric start and end values rather than the presence of the key.
• packages/api/src/routers/api/prometheus.ts:663 -- An over-wide window is narrowed with nothing in the response indicating it happened, so a caller that asked for 30 days and received 7 cannot distinguish "no exemplars in 30 days" from "only 7 days were queried".
  • Fix: Echo the effective start and end actually proxied in the success response.
  • _{api-contract, agent-native}
• packages/api/src/routers/api/prometheus.ts:693 -- The ClickHouse branch's {status:'success',data:[]} is byte-identical to a genuine empty Prometheus result, so a caller cannot tell "this backend does not serve exemplars here" from "queried and found nothing".
  • Fix: Add a warnings entry naming that exemplars for ClickHouse-backed connections come from the metric table rather than this endpoint.
  • _{api-contract, agent-native}
• packages/api/src/routers/api/__tests__/prometheus.int.test.ts:33 -- Declaring fakeUpstreamResponse(): Response via as unknown as Response removes as any from call sites but relocates the unsoundness to the definition, where it is invisible to readers and will keep compiling if proxyToPrometheus starts reading a Response field the fixture never implements.
  • Fix: Narrow proxyToPrometheus to a structural type covering only status, headers.get, and body so the fixture satisfies it without a cast.
  • _{kieran-typescript, testing}
• packages/api/src/routers/api/prometheus.ts:826 -- agent_docs/code_style.md states "File Size: Max 300 lines - refactor when approaching limit", echoed in root CLAUDE.md key principle 4, and this diff adds roughly 200 lines to a file already at 826.
  • Fix: Move the exemplar-window and proxy-outcome helpers into a separate module alongside the router.
• packages/api/src/api-app.ts:112 -- Pre-existing: the whole /v1/prometheus router is gated on isUserAuthenticated, which accepts only a passport session cookie, so an agent authenticating the way this repo's own MCP server expects via validateUserAccessKey bearer tokens cannot reach the new endpoint at all, and no MCP tool exposes it.
  • Fix: Add the access-key strategy to this router or surface exemplar lookup as an MCP tool alongside the existing trace tools.

---

Reviewers (12): correctness, security, adversarial, reliability, testing, maintainability, api-contract, performance, kieran-typescript, project-standards, agent-native, learnings-researcher.

Testing gaps:

• No test drives a pipeline rejection through a route, so neither the genuine mid-stream fault (502, counted) nor the client-disconnect path (upstream status, uncounted) is exercised; isClientDisconnect is only called directly with hand-built error objects, leaving the assumption about which error Node actually raises untested.
• The 504 TimeoutError branch and the !upstreamResp.body branch have no coverage at any level.
• resolveExemplarWindow has no case for start === end, negative or zero epochs, a future end, whitespace-only bounds, or qs-produced array values.
• fakeUpstreamResponse always closes its stream cleanly -- there is no errored, truncated, stalled, or large-body fixture, so the streaming and backpressure claims are unexercised.
• No test covers a connection host carrying a path prefix, a non-http scheme, or a redirecting upstream (fetch defaults to redirect: 'follow').
• agent_docs/observability.md:249 cites this file as its reference example for swallowed-error counters; the citation still holds after this change, so no doc update is needed.