fix(metrics): preserve exact sum/avg when converting histogram buckets to DDSketch by vladimir-dd · Pull Request #25752 · vectordotdev/vector

vladimir-dd · 2026-07-06T14:42:38Z

Summary

AgentDDSketch::transform_to_sketch converts an AggregatedHistogram into a DDSketch for sinks like datadog_metrics. It rebuilds sum/avg purely by interpolating within each bucket's boundaries, discarding the exact sum/count that Histogram::record already tracked as each observation came in.

This breaks down badly when a bucket's true values sit far from its edges — most notably the unbounded first/last bucket, which insert_interpolate_buckets collapses to a point mass at its single finite edge. If the real observations are much smaller (first bucket) or larger (last bucket) than that edge, the reconstructed sum/avg can be off by multiple orders of magnitude, even though count survives interpolation exactly (interpolation always redistributes the exact input count, never losing or gaining any).

Example

Two real observations of source_send_latency_seconds, each ~18µs:

true sum = 0.000036183s (36.183µs total)
true count = 2
true avg = 0.0000180915s (18.09µs)

The histogram's buckets for this window:

bucket 1: upper_limit = 2⁻¹² (244.14µs), count = 2 — both observations land here, since it's the smallest bucket
bucket 2: upper_limit = +Inf, count = 0

reconstructed_sum ≈ upper_limit * count
                   = 2⁻¹² * 2
                   = 0.00048828125s   (488.28µs)

reconstructed_avg ≈ 0.000244140625s   (244.14µs)

relative error    = (reconstructed_sum - true_sum) / true_sum
                   = (0.00048828125 - 0.000036183) / 0.000036183
                   ≈ +1249%

source_send_latency_seconds and similar sub-millisecond timers routinely record values well under 244µs, so in practice their entire mass — count and all — piles into this one unbounded-below bucket, which is exactly the case this bug hits hardest.

So avg was reported as ~244µs when the true average was ~18µs — off by >13x.

This exact scenario (two observations, both far below the first bucket's edge) is what test_transform_to_sketch_preserves_exact_sum_for_unbounded_first_bucket reproduces.

Fix

Override sum/avg with the source histogram's own exact sum/count. quantile() only reads the sketch's bins, not sum/count, so this doesn't affect percentile correctness — it only fixes sum/avg.

Additionally, based on review feedback:

Undercounted buckets (Prometheus): Some sources (Prometheus, in particular) hand us fewer buckets than the histogram's true total count — it always drops its cumulative +Inf bucket when converting to deltas. In that corner case, we skip exact computation of sum/avg, because taking avg=sum/sketch.count() would inflate avg, and taking avg=sum/true_count would break the invariant avg=sum/count in the DD sketch.

Side effect

Unknown sum (OTLP): OTLP histograms may legitimately omit sum (for histograms that may record negative values). Vector histograms do not support this and cast null sums to 0. With this change, a DDSketch will now also report sum=0 and avg=0 in that case – the alternative would be to cast null sums to a NaN to distinguish them from a 0, which might have unexpected effects in other sinks.

Test plan

Added test_transform_to_sketch_preserves_exact_sum_for_unbounded_first_bucket, reproducing the worst case (two observations, both far below the first bucket's finite edge) and asserting the sketch's sum/avg match the source histogram exactly instead of the ~1260x-inflated bucket-derived value.
Existing ddsketch test suite passes unchanged.

Change Type

Is this a breaking change?

Yes
No

Does this PR include user facing changes?

Yes. Please add a changelog fragment based on our guidelines.
No. A maintainer will apply the no-changelog label to this PR.

References

Notes

Please read our Vector contributor resources.
Do not hesitate to use @vectordotdev/vector to reach out to us regarding this PR.
Some CI checks run only after we manually approve them.
- We recommend adding a pre-push hook, please see this template.
- Alternatively, we recommend running the following locally before pushing to the remote branch:
  - make fmt
  - make check-clippy (if there are failures it's possible some of them can be fixed with make clippy-fix)
  - make test
After a review is requested, please avoid force pushes to help us review incrementally.
- Feel free to push as many commits as you want. They will be squashed into one before merging.
- For example, you can run git merge origin master and git push.
If this PR introduces changes Vector dependencies (modifies Cargo.lock), please
run make build-licenses to regenerate the license inventory and commit the changes (if any). More details on the dd-rust-license-tool.

github-actions · 2026-07-06T14:42:53Z

All contributors have signed the CLA ✍️ ✅
_{Posted by the CLA Assistant Lite bot.}

datadog-vectordotdev · 2026-07-06T14:43:03Z

Tests

🎉 All green!

🧪 All tests passed
❄️ No new flaky tests detected

_{This comment will be updated automatically if new data arrives.

🔗 Commit SHA: 21060e6 | Docs | Give us feedback!}

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 7e06e7d26b

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

vladimir-dd · 2026-07-07T14:30:21Z

recheck

vladimir-dd · 2026-07-07T14:31:12Z

I have read the CLA Document and I hereby sign the CLA

…s to DDSketch AggregatedHistogram already tracks an exact running sum/count as each observation is recorded (see Histogram::record). transform_to_sketch discarded both and rebuilt approximate versions purely from bucket-boundary interpolation, which is a poor approximation when a bucket's true values sit far from its edges (e.g. nearly all mass in an unbounded first/last bucket, where interpolation collapses the bucket to a point mass at its single finite edge). count already survives interpolation exactly (interpolation always redistributes the exact input count, never losing or gaining any), so only override sum/avg with the values the source histogram already had. quantile() is unaffected since it only reads the sketch bins, not sum/count. Signed-off-by: Vladimir Zhuk <vladimir.zhuk@datadoghq.com>

- Use the histogram's own exact count (not the sketch's bucket-derived count) as the divisor for avg, since sources like Prometheus drop their cumulative +Inf bucket when converting to deltas, which can make the bucket-derived count undercount relative to the true total. - Represent OTLP's optional histogram sum as NaN instead of defaulting to an exact 0.0, so the sum/avg override is skipped (falling back to the bucket-derived estimate) rather than corrupting non-empty histograms that legitimately omit sum. - Extend min/max to keep them consistent with the now-exact avg, since interpolation can otherwise report an impossible avg outside [min, max] (most notably for the unbounded first/last bucket).

gwenaskell · 2026-07-07T17:06:05Z

@codex review

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: f8aea1e2ae

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

bruceg

Makes sense to me

Co-authored-by: Bruce Guenter <bruce.guenter@datadoghq.com>

github-actions Bot added the domain: core Anything related to core crates i.e. vector-core, core-common, etc label Jul 6, 2026

gwenaskell self-assigned this Jul 7, 2026

gwenaskell marked this pull request as ready for review July 7, 2026 14:08

gwenaskell requested a review from a team as a code owner July 7, 2026 14:08

chatgpt-codex-connector Bot reviewed Jul 7, 2026

View reviewed changes

Comment thread lib/vector-core/src/metrics/ddsketch.rs Outdated

Comment thread lib/vector-core/src/metrics/ddsketch.rs Outdated

Comment thread lib/vector-core/src/metrics/ddsketch.rs Outdated

vladimir-dd force-pushed the ddsketch-exact-sum-fix branch from 7e06e7d to 82f2774 Compare July 7, 2026 14:27

vladimir-dd and others added 3 commits July 7, 2026 16:34

add a changelog

cb3b454

fix clippy

8f55262

vladimir-dd force-pushed the ddsketch-exact-sum-fix branch from 82f2774 to 012d914 Compare July 7, 2026 14:36

gwenaskell added 2 commits July 7, 2026 19:05

simplify changelog and comment

625bb96

chatgpt-codex-connector Bot reviewed Jul 7, 2026

View reviewed changes

Comment thread lib/opentelemetry-proto/src/metrics.rs Outdated

Comment thread lib/vector-core/src/metrics/ddsketch.rs Outdated

revert use of NaN and skip exact computation if counts mismatch

5ab4c00

gwenaskell force-pushed the ddsketch-exact-sum-fix branch from f8aea1e to 5ab4c00 Compare July 8, 2026 08:53

gwenaskell requested a review from bruceg July 8, 2026 09:48

bruceg approved these changes Jul 8, 2026

View reviewed changes

Comment thread lib/vector-core/src/metrics/ddsketch.rs Outdated

gwenaskell and others added 2 commits July 9, 2026 09:52

apply suggestion

a7270b9

Co-authored-by: Bruce Guenter <bruce.guenter@datadoghq.com>

fmt

21060e6

gwenaskell mentioned this pull request Jul 9, 2026

fix(metrics): calculate ddsketch averages from histogram sum & count fields #24379

Closed

8 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

fix(metrics): preserve exact sum/avg when converting histogram buckets to DDSketch#25752

fix(metrics): preserve exact sum/avg when converting histogram buckets to DDSketch#25752
vladimir-dd wants to merge 8 commits into
masterfrom
ddsketch-exact-sum-fix

vladimir-dd commented Jul 6, 2026 •

edited by gwenaskell

Loading

Uh oh!

github-actions Bot commented Jul 6, 2026 •

edited

Loading

Uh oh!

datadog-vectordotdev Bot commented Jul 6, 2026 •

edited

Loading

Uh oh!

chatgpt-codex-connector Bot left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

vladimir-dd commented Jul 7, 2026

Uh oh!

vladimir-dd commented Jul 7, 2026

Uh oh!

gwenaskell commented Jul 7, 2026

Uh oh!

chatgpt-codex-connector Bot left a comment

Uh oh!

Uh oh!

Uh oh!

bruceg left a comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

vladimir-dd commented Jul 6, 2026 • edited by gwenaskell Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Example

Fix

Side effect

Test plan

Change Type

Is this a breaking change?

Does this PR include user facing changes?

References

Notes

Uh oh!

github-actions Bot commented Jul 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

datadog-vectordotdev Bot commented Jul 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

Uh oh!

Uh oh!

vladimir-dd commented Jul 7, 2026

Uh oh!

vladimir-dd commented Jul 7, 2026

Uh oh!

gwenaskell commented Jul 7, 2026

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

Uh oh!

bruceg left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

vladimir-dd commented Jul 6, 2026 •

edited by gwenaskell

Loading

github-actions Bot commented Jul 6, 2026 •

edited

Loading

datadog-vectordotdev Bot commented Jul 6, 2026 •

edited

Loading