Skip to content

app/vtselect/traces/tempo: return matched spans in /api/search instead of root span#174

Open
yusufcemalcelebi wants to merge 1 commit into
VictoriaMetrics:masterfrom
yusufcemalcelebi:fix/tempo-search-matched-spans
Open

app/vtselect/traces/tempo: return matched spans in /api/search instead of root span#174
yusufcemalcelebi wants to merge 1 commit into
VictoriaMetrics:masterfrom
yusufcemalcelebi:fix/tempo-search-matched-spans

Conversation

@yusufcemalcelebi

@yusufcemalcelebi yusufcemalcelebi commented Jun 4, 2026

Copy link
Copy Markdown

Problem

The Tempo-compatible /select/tempo/api/search endpoint populated spanSets[].spans[] with the trace root span (carrying a synthesized nestedSetParent=-1 and only service.name) instead of the spans that actually satisfied the TraceQL filter.

Grafana's Tempo datasource deep-links to spanSets[].spans[].spanID, so clicking a result row always focused the root span rather than the matched child span (real Tempo focuses the matched span). matched was also coupled to root-span presence, returning 0 when the root had not yet been ingested even though child spans matched.

Fix

Retain the spans that matched the filter and project them into the response:

  • query: mark the matched spans within the existing all-spans fetch (query 2) via a | format if (<filter>) "1" as _vt_matched pipe — no extra storage query/round-trip. Non-matching spans are still returned (carrying no marker), since trace-level metadata (duration, per-service stats) is derived from all spans of the trace.
  • summary: build per-trace matchedSpans with their real spanID, name, start/duration and the query-referenced attributes (mirroring Tempo — kind/status rendered as names, scope prefixes stripped; name emitted only when referenced). matched reports the true count regardless of root presence, capped by spss. Root span is used only for trace-level metadata.
  • parser: expose Query.ReferencedFields() and KindCodeToName for the projection.
  • response: emit a serviceStats block (per-service span/error counts) and drop the synthesized root span / nestedSetParent=-1.
  • parse the spss (spansPerSpanSet) query param (default 3, clamped to [1, 1000]).

Out of scope (follow-ups): accepting kind enum names like kind=consumer on input, and the top-level metrics block.

Testing

  • Unit test (tempo_test.go): decisive case (root present but non-matching -> matched child returned, not root; the unmarked root is not counted) and the root-absent case (count still correct).
  • End-to-end against a local single-node build (OTLP ingest of a non-matching root + matching child span, then real /api/search calls):
    • q={ span.http.status_code = "500" } -> returns the child spanID with only the query-referenced attribute (http.status_code: "500"), matched: 1, no nestedSetParent, trace metadata from the root, and a populated serviceStats (service-b -> errorCount: 1).
    • q={} -> both spans marked, matched: 2.
  • make fmt, make vet, go build ./..., and tempo + traceql package tests pass.

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

1 issue found across 14 files

Reply with feedback, questions, or to request a fix.

Fix all with cubic | Re-trigger cubic

Comment thread docs/victoriatraces/changelog/CHANGELOG.md Outdated
@yusufcemalcelebi yusufcemalcelebi force-pushed the fix/tempo-search-matched-spans branch 2 times, most recently from f1f0a39 to 15f5e98 Compare June 4, 2026 19:33
@jiekun jiekun self-assigned this Jun 5, 2026
spanSets[].spans[] now contains the spans that satisfied the TraceQL filter
(real spanID, name and query-referenced attributes), so Grafana deep-links to
the matched span instead of the trace root. matched reports the true count
regardless of root-span presence, and serviceStats is included.

Matched spans are detected within the existing all-spans scan via a
'format if (<filter>) "1" as _vt_matched' pipe, so no extra storage query is
needed.

Signed-off-by: yusufcemalcelebi <yusuf@mailerlite.com>
@yusufcemalcelebi yusufcemalcelebi force-pushed the fix/tempo-search-matched-spans branch from 15f5e98 to 548e5f0 Compare June 11, 2026 13:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants