Skip to content

fix: stabilize membersGlobalActivityCount mv refresh (CM-1302)#4309

Merged
skwowet merged 2 commits into
mainfrom
bugfix/CM-1302
Jul 4, 2026
Merged

fix: stabilize membersGlobalActivityCount mv refresh (CM-1302)#4309
skwowet merged 2 commits into
mainfrom
bugfix/CM-1302

Conversation

@skwowet

@skwowet skwowet commented Jul 4, 2026

Copy link
Copy Markdown
Collaborator

Summary

The refreshMemberEnrichmentMaterializedViews workflow was failing because REFRESH MATERIALIZED VIEW "membersGlobalActivityCount" exceeded the 10-minute activity timeout. As memberSegmentsAgg grew (~239M rows), refresh duration crossed that limit. Unlimited activity retries left zombie PG sessions holding exclusive locks, which also blocked member list queries.

Changes

  • Increase activity startToCloseTimeout from 10 to 45 minutes
  • Cap activity retries at 3 attempts with backoff
  • Use REFRESH MATERIALIZED VIEW CONCURRENTLY so reads on the MV are not blocked during refresh
  • Remove ORDER BY from the membersGlobalActivityCount MV definition (enrichment already sorts in its own query; removes an extra sort over ~635K rows on each refresh)

Signed-off-by: Yeganathan S <63534555+skwowet@users.noreply.github.com>
@skwowet skwowet self-assigned this Jul 4, 2026
Copilot AI review requested due to automatic review settings July 4, 2026 17:23

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR addresses Temporal workflow failures and query blocking caused by long-running refreshes of the membersGlobalActivityCount materialized view as the underlying memberSegmentsAgg table grows. It increases the activity timeout, limits retries to prevent lingering Postgres sessions/locks, and makes the refresh non-blocking for readers by switching to a concurrent refresh while also removing an unnecessary ORDER BY from the MV definition.

Changes:

  • Increased the Temporal activity startToCloseTimeout for MV refresh from 10 to 45 minutes and added a capped retry policy (max 3 attempts with backoff).
  • Switched MV refresh to REFRESH MATERIALIZED VIEW CONCURRENTLY for membersGlobalActivityCount.
  • Removed ORDER BY from the membersGlobalActivityCount MV definition to avoid extra sorting work during refresh.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated no comments.

File Description
services/apps/members_enrichment_worker/src/workflows/refreshMemberEnrichmentMaterializedViews.ts Increases activity timeout and adds bounded retries to stabilize the MV refresh workflow.
services/apps/members_enrichment_worker/src/activities/enrichment.ts Enables concurrent MV refresh in the activity implementation to avoid blocking readers.
backend/src/database/migrations/R__memberEnrichmentMaterializedViews.sql Removes unnecessary ORDER BY from the MV definition to reduce refresh cost.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot AI review requested due to automatic review settings July 4, 2026 17:25

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated no new comments.

@skwowet skwowet merged commit 78abab5 into main Jul 4, 2026
12 checks passed
@skwowet skwowet deleted the bugfix/CM-1302 branch July 4, 2026 19:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants