SDSTOR-22783: priority scheduling with dual watermarks and cross-scan quota#436
Merged
Conversation
xiaoxichen
commented
Jun 17, 2026
xiaoxichen
commented
Collaborator
- Sort eligible chunks by garbage ratio (desc) before submission so the most garbage-heavy chunks are always GC'd first
- Add gc_garbage_rate_threshold_low (default 30%) as a low watermark; chunks between the two watermarks consume at most half the quota
- Track m_pending_normal_gc_task_count in pdev_gc_actor to reflect tasks queued or running in m_gc_executor across scan cycles; previous code only capped submissions per scan, allowing unbounded queue growth
- scan_chunks_for_gc now skips a pdev entirely when already at quota, and derives low_tier_cap proportionally from remaining_capacity
- Add ADR docs/adr/gc-priority-scheduling.md
|
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## stable/v4.x #436 +/- ##
==============================================
Coverage ? 54.41%
==============================================
Files ? 36
Lines ? 5419
Branches ? 685
==============================================
Hits ? 2949
Misses ? 2164
Partials ? 306 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
| eligible.push_back({chunk_id, ratio_pct}); | ||
| } | ||
|
|
||
| // Sort eligible chunks by garbage ratio descending so the most garbage-heavy chunks are GC'd first. |
Collaborator
There was a problem hiding this comment.
this is a topK problem.
suggest to use a heap with a capacity of remaining_capacity (priority_queue), so that we can hold at most remaining_capacity ChunkGCInfo in memory. and for any new ChunkGCInfo, we only need to compare it with the top of the heap.
Collaborator
Author
There was a problem hiding this comment.
IMO It doesnt worth those lines of code , considering the maximum chunk is 32K.
Collaborator
There was a problem hiding this comment.
better to have it since it is not complicated and will also make code more simple, for example, the loop on the entire chunk collection will not be involved.
Collaborator
Author
|
@JacksonYao287 addressed |
JacksonYao287
previously approved these changes
Jun 24, 2026
Sort eligible chunks by garbage ratio using a bounded top-K min-heap (O(K) per pdev) so the most garbage-heavy chunks GC first. Add gc_garbage_rate_threshold_low (default 30%) as a low watermark; chunks between the two watermarks consume at most half the quota. Track m_pending_normal_gc_task_count across scan cycles to cap queued+running tasks (not just per-scan submissions); skip pdevs already at quota. Fix counter underflow by incrementing before m_gc_executor->add(), and fix quota leak by decrementing on the early-return path in process_gc_task. Add ADR docs/adr/gc-priority-scheduling.md. Signed-off-by: Xiaoxi Chen <xiaoxchen@ebay.com>
xiaoxichen
changed the title
JacksonYao287
approved these changes
Jun 24, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.