Return percentiles metrics for batch size and event count within 1, 5, 15 minutes windows. #18503

andsel · 2025-12-17T16:29:26Z

Release notes

TODO

What does this PR do?

Introduce a new user metric designed to capture histograms and expose various percentiles over defined time windows. This implementation utilizes HdrHistogram and leverages the existing window policies and capture lists established during the refactoring in #18502.
The HdrHistogramFlowMetric class utilizes multiple windows, each containing a HistogramSnapshotsWindow. These individual HistogramSnapshotsWindow instances are responsible for gathering histogram snapshots at intervals determined by the window policy's defined resolution.

Each window maintains an HdrHistogram's Recorder to generate incremental histogram snapshots. When values are recorded by HistogramSnapshotsWindow.recordValue, the code checks if a new snapshot needs to be created, potentially collecting it within a RetentionWindow. This value recording process is designed for parallelism.

On the read side, the HdrHistogramFlowMetric.getValue method processes the collected histogram snapshots to create an aggregated histogram for each time window.

Why is it important/What is the impact to the user?

This PR enrich the response of batch metrics provided in _node/stats API, giving a statistical view in terms of percentiles and for various time windows regarding batch size and length.

Checklist

My code follows the style guidelines of this project
I have commented my code, particularly in hard-to-understand areas
~~[ ] I have made corresponding changes to the documentation~~ follow up PR on this.
~~[ ] I have made corresponding change to the default configuration files (and/or docker env variables)~~
I have added tests that prove my fix is effective or that my feature works

Author's Checklist

update licenses
create histogram flow also for batch event count
cover with test, in particular check 1m window after a minute with no events should go down to 0
reshape the response in logstash-core/lib/logstash/api/commands/stats.rb

How to test this PR locally

The test consists in changing the sampling mode pipeline.batch.metrics.sampling_mode, executing a pipeline that accepts some traffic, monitoring with curl the _node/stats endpoint and use a traffic generator to feed the pipeline.

Change sampling mode

Edit config/logstash.yml to have batch sampling mode to full:

pipeline.batch.metrics.sampling_mode: full

Run Logstash with a pipeline

Run Logstash with a sample pipeline like:

bin/logstash -e "input{ tcp {port => 3333} } output{ sink{} }"

Monitor the API

Execute to monitor API response, and check the various percentiles for batch metrics:

while true; do curl http://localhost:9600/_node/stats | jq .pipelines.main.batch; sleep 3; clear; done

Execute a traffic load script

Use jbang to run the Java feeding script, check https://www.jbang.dev/documentation/jbang/latest/installation.html to install.
Run the script in the git https://gist.github.com/andsel/d3b372b90bd66e0db98a6acc1ac32c80 with:

./TrafficSimulator.java -p 3333

Related issues

Use cases

Screenshots

Logs

…ed to compute a flow metric.

…tored inside the RetentionWindow

…apshots of HdrHistograms

github-actions · 2025-12-17T16:29:40Z

🤖 GitHub comments

Just comment with:

run docs-build : Re-trigger the docs validation. (use unformatted text in the comment!)
/run exhaustive tests : Run the exhaustive tests Buildkite pipeline.

mergify · 2025-12-17T16:30:05Z

This pull request does not have a backport label. Could you fix it @andsel? 🙏
To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

backport-8./d is the label to automatically backport to the 8./d branch. /d is the digit.
If no backport is necessary, please add the backport-skip label

…s.<ppl_name>.batch to respect the specification (batch byte_size part)

Copilot

Pull request overview

This PR introduces HdrHistogram-based metrics to capture statistical information about batch sizes and event counts over time windows. The implementation provides percentile metrics (p50, p90) for 1, 5, and 15-minute intervals, enriching the existing batch metrics available through the _node/stats API. The PR also includes a refactoring that extracts common capture and retention window functionality into reusable base classes (DatapointCapture and RetentionWindow) to support both the existing flow metrics and the new histogram metrics.

Key changes:

Introduces HdrHistogramFlowMetric class that maintains histogram snapshots across multiple time windows
Refactors FlowCapture to extend new DatapointCapture base class and extracts RetentionWindow from ExtendedFlowMetric
Integrates histogram recording into QueueReadClientBatchMetrics for batch byte size and event count

Reviewed changes

Copilot reviewed 15 out of 15 changed files in this pull request and generated 16 comments.

Show a summary per file

File	Description
logstash-core/build.gradle	Adds HdrHistogram dependency (version 2.2.2)
tools/dependencies-report/src/main/resources/licenseMapping.csv	Maps HdrHistogram to BSD-2-Clause license
tools/dependencies-report/src/main/resources/notices/org.hdrhistogram!HdrHistogram-NOTICE.txt	Adds BSD-2-Clause license notice for HdrHistogram
logstash-core/src/main/java/org/logstash/instrument/metrics/DatapointCapture.java	New abstract base class for time-stamped data captures
logstash-core/src/main/java/org/logstash/instrument/metrics/RetentionWindow.java	Extracted class for managing capture retention across time windows
logstash-core/src/main/java/org/logstash/instrument/metrics/HistogramCapture.java	New capture implementation for histogram snapshots
logstash-core/src/main/java/org/logstash/instrument/metrics/HistogramMetricData.java	Data class holding percentile values from histograms
logstash-core/src/main/java/org/logstash/instrument/metrics/HistogramFlowMetric.java	Interface definition for histogram-based flow metrics
logstash-core/src/main/java/org/logstash/instrument/metrics/HdrHistogramFlowMetric.java	Implementation using HdrHistogram for percentile tracking
logstash-core/src/main/java/org/logstash/instrument/metrics/FlowCapture.java	Refactored to extend DatapointCapture base class
logstash-core/src/main/java/org/logstash/instrument/metrics/ExtendedFlowMetric.java	Refactored to use extracted RetentionWindow class
logstash-core/src/main/java/org/logstash/instrument/metrics/FlowMetricRetentionPolicy.java	Adds samplesCount() method for memory estimation
logstash-core/src/main/java/org/logstash/execution/QueueReadClientBatchMetrics.java	Integrates histogram metrics for batch byte size and event count
logstash-core/lib/logstash/api/commands/stats.rb	Adds histogram percentile data to batch metrics response
logstash-core/src/test/java/org/logstash/instrument/metrics/HdrHistogramFlowMetricTest.java	Test coverage for histogram metric behavior across time windows

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot · 2025-12-23T14:33:43Z

logstash-core/src/main/java/org/logstash/instrument/metrics/HdrHistogramFlowMetric.java

+            // Is this a problem? Probably not, because it concur to the calculation of the percentiles for a fraction, and
+            // giving that's a statistical approximation, doesn't really matter having exact numbers.


Grammatical error in comment: "it concur" should be "it contributes". Also, "giving that's" should be "given that it's".

Suggested change

// Is this a problem? Probably not, because it concur to the calculation of the percentiles for a fraction, and

// giving that's a statistical approximation, doesn't really matter having exact numbers.

// Is this a problem? Probably not, because it contributes to the calculation of the percentiles for a fraction, and

// given that it's a statistical approximation, doesn't really matter having exact numbers.

Copilot · 2025-12-23T14:33:44Z

logstash-core/src/main/java/org/logstash/instrument/metrics/HdrHistogramFlowMetric.java

+            // If two threads race to update lastRecordTimeNanos, only one updates the variable. If two threads reads
+            // currentTimeNanos that are different but really close to the other, then only one succeed in updating the atomic long,
+            // and will satisfy updatedLast == currentTimeNanos, so will create the snapshot. The other will read a different
+            // updatedLast and the condition will fail.
+            // If two threads reads exactly the same nanosecond time, also in this case only one will update the atomic long, but both


Grammatical error in comment: "If two threads reads" should be "If two threads read" (without the 's').

Suggested change

// If two threads race to update lastRecordTimeNanos, only one updates the variable. If two threads reads

// currentTimeNanos that are different but really close to the other, then only one succeed in updating the atomic long,

// and will satisfy updatedLast == currentTimeNanos, so will create the snapshot. The other will read a different

// updatedLast and the condition will fail.

// If two threads reads exactly the same nanosecond time, also in this case only one will update the atomic long, but both

// If two threads race to update lastRecordTimeNanos, only one updates the variable. If two threads read

// currentTimeNanos that are different but really close to the other, then only one succeed in updating the atomic long,

// and will satisfy updatedLast == currentTimeNanos, so will create the snapshot. The other will read a different

// updatedLast and the condition will fail.

// If two threads read exactly the same nanosecond time, also in this case only one will update the atomic long, but both

Copilot · 2025-12-23T14:33:44Z

logstash-core/src/test/java/org/logstash/instrument/metrics/HdrHistogramFlowMetricTest.java

+        for(int i = 0; i < 45; i++) {
+            sut.recordValue(100);
+            referenceHistogram.recordValue(100);
+            clock.advance(Duration.ofSeconds(1));
+        }
+        for(int i = 0; i < 15; i++) {


Missing whitespace after 'for' keyword. Should be 'for (int i = 0; i < 15; i++)' for consistency with Java coding standards.

Suggested change

for(int i = 0; i < 45; i++) {

sut.recordValue(100);

referenceHistogram.recordValue(100);

clock.advance(Duration.ofSeconds(1));

}

for(int i = 0; i < 15; i++) {

for (int i = 0; i < 45; i++) {

sut.recordValue(100);

referenceHistogram.recordValue(100);

clock.advance(Duration.ofSeconds(1));

}

for (int i = 0; i < 15; i++) {

logstash-core/src/test/java/org/logstash/instrument/metrics/HdrHistogramFlowMetricTest.java

logstash-core/src/main/java/org/logstash/instrument/metrics/RetentionWindow.java

Copilot · 2025-12-23T14:33:46Z

logstash-core/lib/logstash/api/commands/stats.rb

+          def reshape_histogram_percentiles_for_window(target_field, histogram_metric, window, result)
+            result[target_field][:p50] = {} if result[target_field][:p50].nil?
+            result[target_field][:p90] = {} if result[target_field][:p90].nil?
+
+            result[target_field][:p50][window] = histogram_metric.value[window.to_s].get50Percentile.round
+            result[target_field][:p90][window] = histogram_metric.value[window.to_s].get90Percentile.round
+          end


Potential code duplication. The histogram_metric.value[window.to_s] is called twice for each window - once in the condition check and once inside the method. Consider storing the result in a local variable to avoid redundant method calls.

Copilot · 2025-12-23T14:33:46Z

logstash-core/src/test/java/org/logstash/instrument/metrics/HdrHistogramFlowMetricTest.java

+        for(int i = 0; i < 45; i++) {
+            sut.recordValue(100);
+            referenceHistogram.recordValue(100);
+            clock.advance(Duration.ofSeconds(1));
+        }
+        for(int i = 0; i < 15; i++) {


Missing whitespace after 'for' keyword. Should be 'for (int i = 0; i < 45; i++)' for consistency with Java coding standards.

Suggested change

for(int i = 0; i < 45; i++) {

sut.recordValue(100);

referenceHistogram.recordValue(100);

clock.advance(Duration.ofSeconds(1));

}

for(int i = 0; i < 15; i++) {

for (int i = 0; i < 45; i++) {

sut.recordValue(100);

referenceHistogram.recordValue(100);

clock.advance(Duration.ofSeconds(1));

}

for (int i = 0; i < 15; i++) {

logstash-core/src/test/java/org/logstash/instrument/metrics/HdrHistogramFlowMetricTest.java

logstash-core/src/main/java/org/logstash/instrument/metrics/RetentionWindow.java

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

elasticmachine · 2025-12-23T15:24:19Z

💛 Build succeeded, but was flaky

Buildkite Build
Commit: 2a5807f

Failed CI Steps

🥼 IT Persistent Queues / part 1-of-6

History

💚 Build #4031 succeeded c8b1e93
💔 Build #4030 failed 2c77bec
💛 Build #4017 was flaky c9a61db
💚 Build #4007 succeeded 297f328
💔 Build #4005 failed 5a3bf7e
💔 Build #4003 failed 0881ea1

cc @andsel

andsel added 6 commits December 17, 2025 14:25

Introduced common base class to group all kind of data that can be us…

ae0eed1

…ed to compute a flow metric.

Extracted flow metric's RetentionWindow in its own class

683e7c5

Moved selectNewerCapture to newly created DatapointCapture

ad3f56f

Added dependency on HdrHistogram and created HistogramCapture to be s…

5153ce3

…tored inside the RetentionWindow

Implemented HistogramFlowMetric leveraging RetentionWindow to keep sn…

fbef946

…apshots of HdrHistograms

Exposed user metric for histrogam flow batch byte size

ca6e76f

andsel self-assigned this Dec 19, 2025

Added license for HdrHistogram library

5a3bf7e

andsel force-pushed the feature/create_histogram_flowmetrics branch from 0881ea1 to 5a3bf7e Compare December 19, 2025 13:17

andsel added 5 commits December 19, 2025 16:46

Reshaped the histogram part of the API _node/stats reposne, .pipeline…

297f328

…s.<ppl_name>.batch to respect the specification (batch byte_size part)

Added histogram for batch's event count

0fa6ad7

Added log printing the HdrHistrogram memory consumption

c9a61db

[Test] Added some tests to cover the flow histogram metrics

bda3dd3

[Test] Removed console prints of histrograms during tests

ae3ce7c

andsel changed the title ~~Feature/create histogram flowmetrics~~ Present percentiles of batch size and event count within 1, 5, 15 minutes windows. Dec 22, 2025

andsel changed the title ~~Present percentiles of batch size and event count within 1, 5, 15 minutes windows.~~ Return percentiles metrics for batch size and event count within 1, 5, 15 minutes windows. Dec 22, 2025

andsel added 3 commits December 23, 2025 09:36

Minor, removed commented code and added license header.

4a22d99

Switched 75th percentile to 50th.

2c77bec

Fixed failing test

c8b1e93

andsel requested a review from Copilot December 23, 2025 14:27

Copilot started reviewing on behalf of andsel December 23, 2025 14:27 View session

Copilot AI reviewed Dec 23, 2025

View reviewed changes

andsel and others added 2 commits December 23, 2025 15:50

Apply suggestion from @Copilot

1741701

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

Applied some suggestions from Copilot

2a5807f

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Return percentiles metrics for batch size and event count within 1, 5, 15 minutes windows. #18503

Return percentiles metrics for batch size and event count within 1, 5, 15 minutes windows. #18503

andsel commented Dec 17, 2025 •

edited

Loading

Uh oh!

github-actions bot commented Dec 17, 2025

Uh oh!

mergify bot commented Dec 17, 2025

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI Dec 23, 2025

Uh oh!

Copilot AI Dec 23, 2025

Uh oh!

Copilot AI Dec 23, 2025

Uh oh!

Uh oh!

Uh oh!

Copilot AI Dec 23, 2025

Uh oh!

Copilot AI Dec 23, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

elasticmachine commented Dec 23, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

		// Is this a problem? Probably not, because it concur to the calculation of the percentiles for a fraction, and
		// giving that's a statistical approximation, doesn't really matter having exact numbers.

Return percentiles metrics for batch size and event count within 1, 5, 15 minutes windows. #18503

Are you sure you want to change the base?

Return percentiles metrics for batch size and event count within 1, 5, 15 minutes windows. #18503

Conversation

andsel commented Dec 17, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Release notes

What does this PR do?

Why is it important/What is the impact to the user?

Checklist

Author's Checklist

How to test this PR locally

Change sampling mode

Run Logstash with a pipeline

Monitor the API

Execute a traffic load script

Related issues

Use cases

Screenshots

Logs

Uh oh!

github-actions bot commented Dec 17, 2025

🤖 GitHub comments

Uh oh!

mergify bot commented Dec 17, 2025

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Copilot AI Dec 23, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Dec 23, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Dec 23, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Copilot AI Dec 23, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Dec 23, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

elasticmachine commented Dec 23, 2025

💛 Build succeeded, but was flaky

Failed CI Steps

History

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

andsel commented Dec 17, 2025 •

edited

Loading