Skip to content

enhancement(vector source, vector sink): enable HTTP/2 adaptive flow-control windows#25782

Open
Evador wants to merge 1 commit into
vectordotdev:masterfrom
Evador:vector-h2-adaptive-window
Open

enhancement(vector source, vector sink): enable HTTP/2 adaptive flow-control windows#25782
Evador wants to merge 1 commit into
vectordotdev:masterfrom
Evador:vector-h2-adaptive-window

Conversation

@Evador

@Evador Evador commented Jul 8, 2026

Copy link
Copy Markdown

Summary

Enables HTTP/2 adaptive flow-control windows on the vector sink's hyper client and on the shared gRPC server builder used by the vector source.

Today neither peer sets any window options, so both run with the RFC 7540 default 65,535-byte stream/connection windows. With acknowledgements enabled the server only completes a PushEvents request (and the client only gets its WINDOW_UPDATE credit back) after the events have been accepted by the downstream components, so a single vector-to-vector connection is capped at roughly 64 KiB / (network RTT + ack latency) no matter how batch.* or request.concurrency are tuned.

We hit this in production on 0.44.0: a single connection between an agent and an aggregator plateaus at ~5 MB/s of event bytes (~7k events/s at our event sizes) on a 2 ms LAN, with CPU nearly idle on both ends. The client builder is unchanged on master (src/sinks/vector/config.rs, hyper::Client::builder().http2_only(true)), same for the server side, so the cap should reproduce on any recent version.

Evidence that it's flow control and not processing:

  • doubling batch.max_bytes (4 MiB -> 8 MiB) exactly doubled per-request latency, throughput unchanged
  • request.concurrency 1 vs 8 vs 32 made no difference over the LAN (all requests share one connection window)
  • N parallel connections scale linearly (~20 connections reached ~45k events/s aggregate while each single one stays at ~7k)
  • on localhost, where WINDOW_UPDATEs return in microseconds, the same 0.44.0 binaries and configs sustain 65-74k events/s over a single connection, and request.concurrency: 8 then does help (163k events/s) — so the server pipeline itself is nowhere near the limit, the window pacing over a real network is

Letting hyper/tonic size the windows from the observed bandwidth-delay product (http2_adaptive_window) removes the ceiling without adding a config knob.

Vector configuration

Producer:

sources:
  logs:
    type: file
    include: ["/tmp/bench/synthetic.log"]   # 15M lines, ~180 bytes each
sinks:
  out:
    type: vector
    inputs: ["logs"]
    address: "aggregator:9000"
    compression: true
    acknowledgements:
      enabled: true
    batch:
      max_events: 20000
      max_bytes: 8388608
      timeout_secs: 2
    request:
      concurrency: 1

Aggregator:

sources:
  in:
    type: vector
    address: "0.0.0.0:9000"
sinks:
  sink:
    type: blackhole
    inputs: ["in"]
    acknowledgements:
      enabled: true

(Production topology has a remap + route fan-out into multiple file sinks with disk buffers behind the source; the plateau is the same, the ack latency is just a bit longer.)

How did you test this PR?

  • Diagnosed the ceiling in production (0.44.0, agent fleet -> central aggregator over a 2 ms LAN with TLS) with the observations listed above, then isolated it with local single-host runs: same binaries and configs over localhost reach 65-74k events/s per connection with acks on, gzip on/off within 6%, an 8-way fan-out topology within 12% — confirming the network-side window pacing is the limiting factor, not the pipeline.
  • cargo check --no-default-features --features "sources-vector,sinks-vector" passes on this branch; the change only touches the two transport builders.
  • I don't have an A/B of the patched build under the production workload yet; happy to run one on our fleet and post numbers here if useful.

Change Type

  • Bug fix
  • New feature
  • Dependencies
  • Non-functional (chore, refactoring, docs)
  • Performance

Is this a breaking change?

  • Yes
  • No

Does this PR include user facing changes?

  • Yes. Please add a changelog fragment based on our guidelines.
  • No. A maintainer will apply the no-changelog label to this PR.

References

None — happy to open an issue with the full analysis first if that's preferred.

@Evador Evador requested review from a team as code owners July 8, 2026 19:24
@github-actions github-actions Bot added the docs review on hold The documentation team reviews PRs only after a PR is approved by the COSE team. label Jul 8, 2026
@github-actions

github-actions Bot commented Jul 8, 2026

Copy link
Copy Markdown
Contributor

All contributors have signed the CLA ✍️ ✅
Posted by the CLA Assistant Lite bot.

@github-actions github-actions Bot added domain: sources Anything related to the Vector's sources domain: sinks Anything related to the Vector's sinks labels Jul 8, 2026
@datadog-vectordotdev

This comment has been minimized.

@Evador

Evador commented Jul 8, 2026

Copy link
Copy Markdown
Author

I have read the CLA Document and I hereby sign the CLA

@Evador Evador force-pushed the vector-h2-adaptive-window branch from e39272d to dfb3690 Compare July 8, 2026 19:39
@Evador

Evador commented Jul 9, 2026

Copy link
Copy Markdown
Author

Results from my production setup:
stock client: 1% CPU, 64KiB-per-RTT trickle; patched: saturates its CPU budget; host converged to zero lag

@Evador

Evador commented Jul 9, 2026

Copy link
Copy Markdown
Author

Opened #25792 with the full analysis. @vectordotdev/vector ready for a look — happy to switch to a config option instead of always-on adaptive windows if that's preferred.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

docs review on hold The documentation team reviews PRs only after a PR is approved by the COSE team. domain: sinks Anything related to the Vector's sinks domain: sources Anything related to the Vector's sources

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant