dbi 604 handle replication slot creation getting stuck by masterashu · Pull Request #4439 · PeerDB-io/peerdb

masterashu · 2026-06-17T16:54:24Z

Handle status terminating during resync-drop-flow.
Add user facing logs for waiting replication slot creation warning.
- Streaming replication protocol doesn't support timeouts, so best we can do is user warning

claude · 2026-06-17T16:58:25Z

Code review

No issues found. Checked for bugs and CLAUDE.md compliance.

claude · 2026-06-17T16:58:38Z

Code review

No issues found. Checked for bugs and CLAUDE.md compliance.

github-actions · 2026-06-17T17:16:15Z

🔄 Flaky Test Detected

Analysis: The actual e2e tests passed; failures are isolated to the "Ingest tests results for analysis" CI infrastructure step, where the analytics ClickHouse endpoint returned HTTP 500 "Timeout error" — a network/infra issue, not a code bug.
Confidence: 0.97

✅ Automatically retrying the workflow

View workflow run

github-actions · 2026-06-17T17:35:49Z

🔄 Flaky Test Detected

Analysis: The actual test step passed; only the "Ingest tests results for analysis" telemetry step failed because the stresshouse ClickHouse metrics instance rejected the upload with a "Too many simultaneous queries (1007/1000)" rate-limit error after exhausting retries — an infrastructure issue, not a code bug.
Confidence: 0.97

✅ Automatically retrying the workflow

View workflow run

github-actions · 2026-06-17T17:57:10Z

🔄 Flaky Test Detected

Analysis: The actual tests passed; the only failure is the CI "Ingest tests results for analysis" step hitting repeated HTTP 500 "Timeout error" responses from the external analytics-upload endpoint, an infrastructure/network issue unrelated to the test code.
Confidence: 0.97

✅ Automatically retrying the workflow

View workflow run

codecov · 2026-06-22T11:38:56Z

❌ 1 Tests Failed:

Tests completed	Failed	Passed	Skipped
1552	1	1551	11

View the full list of 1 ❄️ flaky test(s)

::TestMain
Flake rate in main: 100.00% (Passed 0 times, Failed 89 times)
Stack Traces | 0s run time
FAIL	github..../peerdb/flow/e2e [build failed]

To view more test analytics, go to the Test Analytics Dashboard
_{📋 Got 3 mins? Take this short survey to help us improve Test Analytics.}

github-actions · 2026-06-22T11:42:26Z

❌ Test Failure

Analysis: The new TestApiMongo/TestDropRunningFlow fails deterministically with an "unsupported operation" error at drop_flow_test.go:18 across multiple matrix configs, indicating a real bug in the PR's new drop-flow code path (no timeout/race/network/intermittent characteristics).
Confidence: 0.9

⚠️ This appears to be a real bug - manual intervention needed

View workflow run

ilidemi · 2026-06-23T06:05:33Z


 	logger.Info("waiting for slot to be created...")
+	slotCreateStart := time.Now()
+	stopSlotCreateWarning := common.Interval(ctx, 5*time.Minute, func() {


Let's move this into postgres connector. We've only seen the problem there and the logged advice only applies there

ilidemi · 2026-06-23T06:14:56Z

 	}

-	if input.Resync {
+	if input.Resync && activeSignal != model.TerminateSignal {


If terminate signal arrived during RemoveFlowDetailsFromCatalog activity, in the end the catalog row is not removed and stays as RESYNC. Can introduce finalDrop bool, starting as false and set to true when the terminate signal comes as part of the handler here, and run RemoveFlowDetailsFromCatalogRequest again if it was resync and now a termination

ilidemi · 2026-06-23T06:26:18Z

+			var onFlowSignal func(model.CDCFlowSignal, bool)
+			onFlowSignal = func(val model.CDCFlowSignal, _ bool) {
+				activeSignal = model.FlowSignalHandler(activeSignal, val, logger)
+				flowSignalChan.AddToSelector(sigSelector, onFlowSignal)


I don't think we needs a second AddToSelector inside the callback (same for onStateChange)? The callback can be inlined then

ilidemi · 2026-06-23T06:27:10Z

+			var onStateChange func(*protos.FlowStateChangeRequest, bool)
+			onStateChange = func(req *protos.FlowStateChangeRequest, _ bool) {
+				if req.RequestedFlowState == protos.FlowStatus_STATUS_TERMINATING {
+					activeSignal = model.TerminateSignal


Propagate req.SkipDestinationDrop and req.DropMirrorStats as well

github-actions · 2026-06-23T12:13:24Z

❌ Test Failure

Analysis: Deterministic compile/build failure — e2e/postgres_qrep_test.go:157 calls s.conn.SetupReplication with too few arguments after a signature change, causing the e2e package to fail to build (not a flaky test).
Confidence: 0.99

⚠️ This appears to be a real bug - manual intervention needed

View workflow run

github-actions · 2026-06-23T12:14:20Z

❌ Test Failure

Analysis: A deterministic Go compilation error ("not enough arguments in call to s.conn.SetupReplication" in e2e/postgres_qrep_test.go:157) caused the e2e package to fail to build, which is a real code bug, not a flaky test.
Confidence: 0.99

⚠️ This appears to be a real bug - manual intervention needed

View workflow run

ilidemi

Awesome job

…ation-slot-creation-getting-stuck

masterashu requested a review from a team as a code owner June 17, 2026 16:54

masterashu self-assigned this Jun 17, 2026

masterashu requested a review from ilidemi June 23, 2026 04:43

ilidemi reviewed Jun 23, 2026

View reviewed changes

masterashu and others added 6 commits June 23, 2026 16:42

fix: handle status terminating during resync-drop-flow.

ac5b5a2

feat: add user facing logs for waiting replication slot creation

e1c5533

fix

351232e

Add test for drop flow changes

2c8439a

Update drop_flow_test.go

307613b

move replication slot warning to postgres connector

4b9278f

masterashu force-pushed the dbi-604-handle-replication-slot-creation-getting-stuck branch from 6169b21 to 4b9278f Compare June 23, 2026 12:06

masterashu requested a review from a team as a code owner June 23, 2026 12:06

move replication slot warning to postgres connector

7773986

masterashu added 2 commits June 23, 2026 18:10

fix test missing arg

bf51959

resolve comments

3197cd2

masterashu requested a review from a team as a code owner June 23, 2026 13:01

fix lint error

91b8663

ilidemi approved these changes Jun 24, 2026

View reviewed changes

Merge remote-tracking branch 'origin/main' into dbi-604-handle-replic…

b6ac32d

…ation-slot-creation-getting-stuck

masterashu enabled auto-merge (squash) June 25, 2026 05:21

masterashu merged commit e490cee into main Jun 25, 2026
16 checks passed

masterashu deleted the dbi-604-handle-replication-slot-creation-getting-stuck branch June 25, 2026 05:38

Uh oh!

Conversation

masterashu commented Jun 17, 2026

Uh oh!

claude Bot commented Jun 17, 2026

Code review

Uh oh!

claude Bot commented Jun 17, 2026

Code review

Uh oh!

github-actions Bot commented Jun 17, 2026

🔄 Flaky Test Detected

Uh oh!

github-actions Bot commented Jun 17, 2026

🔄 Flaky Test Detected

Uh oh!

github-actions Bot commented Jun 17, 2026

🔄 Flaky Test Detected

Uh oh!

codecov Bot commented Jun 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

❌ 1 Tests Failed:

Uh oh!

github-actions Bot commented Jun 22, 2026

❌ Test Failure

Uh oh!

ilidemi Jun 23, 2026

Choose a reason for hiding this comment

Uh oh!

ilidemi Jun 23, 2026

Choose a reason for hiding this comment

Uh oh!

ilidemi Jun 23, 2026

Choose a reason for hiding this comment

Uh oh!

ilidemi Jun 23, 2026

Choose a reason for hiding this comment

Uh oh!

github-actions Bot commented Jun 23, 2026

❌ Test Failure

Uh oh!

github-actions Bot commented Jun 23, 2026

❌ Test Failure

Uh oh!

ilidemi left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

codecov Bot commented Jun 22, 2026 •

edited

Loading