Skip to content

[MeshSync] Fix data race on informer stores map - #592

Open
singhharsh1708 wants to merge 3 commits into
meshery:masterfrom
singhharsh1708:fix/meshsync-stores-data-race
Open

[MeshSync] Fix data race on informer stores map#592
singhharsh1708 wants to merge 3 commits into
meshery:masterfrom
singhharsh1708:fix/meshsync-stores-data-race

Conversation

@singhharsh1708

@singhharsh1708 singhharsh1708 commented Jul 16, 2026

Copy link
Copy Markdown

What's going on

Handler.stores — the per-GVR informer store set — gets replaced wholesale by startDiscovery on the discovery goroutine every time (re)discovery runs, and it's read on a different goroutine by handleInformerStoreRequestlistStoreObjects whenever Meshery Server asks for an informer-store. Those two run at the same time with nothing synchronizing them, so the field is read and written with no happens-before between them — a data race, and go test -race flags it. It's the same shape as the bug already fixed for the exec/log-stream sessions map.

What I changed

I put an RWMutex (storesMu) around the stores field. The wholesale swap now goes through replaceStores under the write lock, and reads go through snapshotStores, which copies the current store values under the read lock and hands those back — so the lock is never held across cache.Store.List(). listStoreObjects iterates snapshotStores() instead of the raw map.

Nothing exported changes — no API, wire format, CRD, or config. Access is read-mostly, which is why RWMutex, and the set of objects returned is exactly what it was before.

Testing

Added meshsync/stores_test.go. TestStoresConcurrentAccess runs 16 goroutines racing replace against snapshot; it passes under -race, and it reports the race if you remove the guard, so it actually catches the regression. There are also completeness and nil-map cases. go test --short ./... -race is green across all packages, and go vet / gofmt are clean.

Why send it on its own

It's a prerequisite for the periodic-reconciliation work (docs/design/fd5-periodic-reconciliation.md), which adds a timer-driven reconcile loop that would be the second concurrent reader of h.stores alongside the informer-store path. I'm keeping it separate so that feature builds on a race-free store instead of bundling the fix into it.

Summary by CodeRabbit

  • Bug Fixes

    • Improved reliability when discovery updates resource stores while requests are being handled.
    • Prevented potential concurrency issues and failures when retrieving stored resources, including when no stores are available.
  • Tests

    • Added coverage for concurrent store updates, complete store snapshots, and empty-store scenarios.

startDiscovery replaces h.stores wholesale on the discovery goroutine while handleInformerStoreRequest ranges it on the request-listener goroutine, an unsynchronized read/write of the map field. Guard it with storesMu (RWMutex) and route both accesses through replaceStores and snapshotStores, mirroring the sessions-map fix. snapshotStores copies the store slice under the read lock and releases it before any store List(), so a store read never blocks the next discovery swap.

Signed-off-by: Harsh Singh <hs1663531@gmail.com>
@github-actions github-actions Bot added the language/go Golang related label Jul 16, 2026

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces thread-safe access to the 'stores' map in the 'Handler' struct by adding a 'sync.RWMutex' ('storesMu') and implementing 'replaceStores' and 'snapshotStores' helper methods. These changes prevent race conditions between the discovery goroutine, which updates the stores, and the request listener goroutine, which reads them. Additionally, a new test file 'meshsync/stores_test.go' has been added to verify concurrent access and correct snapshot behavior. I have no further feedback to provide.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

@coderabbitai

coderabbitai Bot commented Jul 22, 2026

Copy link
Copy Markdown

Review Change Stack

📝 Walkthrough

Walkthrough

The handler now protects discovery store replacement with an RWMutex, snapshots stores before listing objects, and adds tests for concurrent access, complete snapshots, and nil store maps.

Changes

Store concurrency

Layer / File(s) Summary
Store swap and snapshot path
meshsync/meshsync.go, meshsync/discovery.go, meshsync/handlers.go
Handler adds storesMu; discovery replaces stores under a write lock, and list handling reads a copied store snapshot under a read lock.
Concurrency and snapshot validation
meshsync/stores_test.go
Tests cover concurrent replacement and reads, complete snapshots, and nil-map snapshots.

Estimated code review effort: 3 (Moderate) | ~20 minutes

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly and concisely describes the main change: fixing a data race on the informer stores map.
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Comment @coderabbitai help to get the list of available commands.

@singhharsh1708

Copy link
Copy Markdown
Author

Friendly ping on this — it's been open a couple of weeks and CI is green (DCO, CodeRabbit, triage all passing). It's a small change: guards the informer stores map with an RWMutex and adds a test that reproduces the original race and passes under -race. Happy to rebase or adjust anything if a maintainer has feedback whenever it reaches the queue. Thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

language/go Golang related

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant