Skip to content

fix(vlog): upgrade discardStats to RWMutex and secure Iterate read path (#19431)#2309

Open
HarshalPatel1972 wants to merge 1 commit into
dgraph-io:mainfrom
HarshalPatel1972:fix/vlog-gc-race
Open

fix(vlog): upgrade discardStats to RWMutex and secure Iterate read path (#19431)#2309
HarshalPatel1972 wants to merge 1 commit into
dgraph-io:mainfrom
HarshalPatel1972:fix/vlog-gc-race

Conversation

@HarshalPatel1972

Copy link
Copy Markdown

Description

Fixes #19431

This PR resolves a high-frequency data race and associated memory log truncation leak within the Value Log garbage collection subsystem (discardStats).

Technical Root Cause

The discardStats struct embeds a standard sync.Mutex guarding internal state tracking arrays and maps. While execution boundaries like Update and MaxDiscard correctly acquire exclusive ownership via lf.Lock(), the exported Iterate method completely lacked synchronization.

Under intense concurrent write workloads combined with background telemetry, metrics collection, or CLI debugging cycles, external calls to Iterate would traverse lf.nextEmptySlot and slice elements concurrently while a garbage collection thread or compaction task was executing an un-synchronized Update (which triggers slice mutation and sorting operations via sort.Sort). This led to runtime data races, stale memory tracking offsets, and occasionally missing log truncation windows, leaving old value log (vlog) files stranded on disk.

Resolution Strategy

  • Upgraded Locking Posture: Swapped the embedded sync.Mutex inside discardStats out for a high-performance sync.RWMutex.
  • Read-Gate Optimization: Wrapped the full range-loop traversal inside func (lf *discardStats) Iterate(...) with shared read-locks (lf.RLock() / defer lf.RUnlock()). This allows infinite concurrent telemetry readers to safely scan statistics without blocking each other or starving the system.
  • Exclusive Write Boundary: Maintained exclusive lf.Lock() / defer lf.Unlock() boundaries inside Update to cleanly isolate mutations and sorting operations.

Testing & Regression

  • Added TestDiscardStats_ConcurrentRace within discard_test.go.
  • The test mounts parallel high-frequency loops: 20 worker goroutines continuously firing randomized statistical Update frames, alongside 20 worker goroutines executing overlapping Iterate read scans.
  • Validated using the Go standard race detector (go test -race -v ./...), returning 100% clean thread safety boundaries.

Checklist

  • Code compiles correctly and linting passes locally
  • Tests added for new functionality, or regression tests for bug fixes added as applicable

@HarshalPatel1972 HarshalPatel1972 requested a review from a team as a code owner June 22, 2026 19:47
Copilot AI review requested due to automatic review settings June 22, 2026 19:47
@CLAassistant

Copy link
Copy Markdown

CLA assistant check
Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
You have signed the CLA already but the status is still pending? Let us recheck it.

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

@matthewmcneely matthewmcneely self-requested a review June 23, 2026 21:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Development

Successfully merging this pull request may close these issues.

3 participants