Skip to content

Add opt-in drainProducerRate to throttle producers during the backlog drain#151

Merged
oleiman merged 2 commits into
mainfrom
drain-producer-rate
Jul 1, 2026
Merged

Add opt-in drainProducerRate to throttle producers during the backlog drain#151
oleiman merged 2 commits into
mainfrom
drain-producer-rate

Conversation

@oleiman

@oleiman oleiman commented Jul 1, 2026

Copy link
Copy Markdown
Member

buildAndDrainBacklog() only pauses/resumes consumers — producers keep publishing
at producerRate through the drain. That makes it impossible to measure max
consume throughput on a cold backlog (e.g. one aged onto tiered storage) without
producers competing.

This adds an opt-in Workload.drainProducerRate (int, default -1 = unset).
When ≥ 0, buildAndDrainBacklog() calls the existing worker.adjustPublishRate()
just before resuming consumers:

  • unset / -1 → producers run through the drain (unchanged)
  • N → throttled to N msg/s; 0 → ~1 msg/s via the existing <1 floor (≈ stopped)

It reuses the internal lever findMaximumSustainableRate already uses (fans out to
distributed workers) — no new Worker method or endpoint. Default -1 skips the
call, so existing workloads are byte-identical.

Tested: WorkloadTest covers the default + deserialization; a backlog run with
drainProducerRate: 0 drops producers to ~1 msg/s at "Start draining backlog".

Requires https://github.com/redpanda-data/vtools/pull/4353 to use in benchrunner

@oleiman oleiman self-assigned this Jul 1, 2026
@oleiman oleiman requested a review from a team as a code owner July 1, 2026 17:40
@oleiman oleiman requested review from ballard26 and removed request for a team July 1, 2026 17:40
oleiman and others added 2 commits July 1, 2026 14:05
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Oren Leiman <oren.leiman@redpanda.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Oren Leiman <oren.leiman@redpanda.com>
@oleiman oleiman force-pushed the drain-producer-rate branch from 70817bf to 95f0ac4 Compare July 1, 2026 21:05

@ballard26 ballard26 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@oleiman oleiman merged commit 5296ad0 into main Jul 1, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants