[release-3.5] enhance `etcdutl check v2store` to check both v2 snapshot and WAL records by silentred · Pull Request #21889 · etcd-io/etcd

silentred · 2026-06-03T07:01:07Z

I've given it a try to resolve #21849, but haven't added tests yet. Could you take a look and see if the fix is heading in the right direction?
cc @ahrtr

ahrtr

Overall looks good, thx

Note we also need to update the v3.5 -> v3.6 upgrade guide, https://etcd.io/docs/v3.6/upgrades/upgrade_3_6/

ahrtr · 2026-06-03T09:13:36Z

+	if err := assertNoV2StoreContent(st); err != nil {
+		return err
+	}


We need to continue to check WAL files. We need to provide output something like below,

case 1: neither v2 snapshot nor WAL file have custom v2 content

No custom content found in v2store.

case 2: only v2 snapshot file has custom v2 content

detected custom content in v2store

In this case, users can either remove the v2 content from v2store, or just set --v2-deprecation=write-only-skip-check. Refer to #21850

I think we need to backport #21848 to release-3.5 as well. In this case, users can set --v2-deprecation=write-only-skip-check when they are still on v3.5, and then upgrade to v3.6; otherwise they can only add this option after moving to v3.6, but the upgrade may have already failed.

cc @fuweid @sahilpatel09

case 3: only WAL files have custom v2 content

detected custom v2 content in WAL records

In this case, users can perform the workaround as documented in the first section "v2 store" in https://etcd.io/docs/v3.6/upgrades/upgrade_3_6/.

case 4: both v2 snapshot and WAL record have custom v2 content

detected custom v2 content in both v2store and WAL records

In this case, In this case, users need to perform the same workaround as case 2 mentioned above, and also the workaround as documented in the first section "v2 store" in https://etcd.io/docs/v3.6/upgrades/upgrade_3_6/

The error message for case 1 should be something like No custom content found in both v2store and WAL records.

Noted. I will revise the code according to your doc.

ahrtr · 2026-06-03T19:02:15Z

This isn't correct. When there is no snapshot, we load all WAL records.

Updated. Trying to fix e2e tests.

ahrtr · 2026-06-04T09:21:57Z

LGTM with a minor comment.

Can you please also manually verify all the scenarios and show us the results?

Signed-off-by: shenmu.wy <shenmu.wy@antfin.com>

silentred · 2026-06-04T10:01:23Z

manually verify all the scenarios and show us the results

I have added two e2e tests to cover the 2nd and 3rd cases. Combined with the existing TestV2DeprecationCheckCustomContentOffline test, covering the 1st and 4th cases, I think this provides equivalent coverage to manual verification.

ahrtr · 2026-06-04T10:10:52Z

+	for _, m := range matches {
+		require.NoError(t, os.Remove(m))
+		t.Logf("Removed v2 snapshot: %s", m)
+	}


do we need to remove the v2snapshot files? There should be no v2 snapshot files at all?

There should be no .snap files at all. I removed this block of code.

ahrtr · 2026-06-04T10:34:28Z

+	assert.NoError(t, err)
+}
+
+// TestCtlV2CustomContentOnlyV2Store exercises case 2: a v2 snapshot with


Other contributors do not know what's case 2 or case 3, pls either add a link (below) to the comment or remove it.

#21889 (comment)

Updated the comment.

ahrtr · 2026-06-04T10:34:37Z

 	assert.NoError(t, err)
 }
+
+// TestCtlV2CustomContentOnlyWAL exercises case 3: a v2 PUT is recorded in


silentred · 2026-06-04T11:18:21Z

TestCtlV2CustomContentWithDedicatedWALDir seems to be flaky.

etcd/tests/e2e/v2store_deprecation_test.go

Lines 185 to 186 in 9fa3224

    
           _, err = proc.Expect("detected custom content in v2store") 
        
           assert.NoError(t, err)

this error may be "read /dev/ptmx: input/output error"

Examples:

silentred · 2026-06-04T11:19:19Z

/retest

ahrtr · 2026-06-04T11:38:32Z

Both errors seem related to the test case itself. I see that the SnapshotCount is 1 in TestCtlV2CustomContentWithDedicatedWALDir. So it's expected that there is no v2 custom data in WAL records.

    v2store_deprecation_test.go:186: 
        	Error Trace:	/home/prow/go/src/github.com/etcd-io/etcd/tests/e2e/v2store_deprecation_test.go:186
        	Error:      	Received unexpected error:
        	            	match not found. Set EXPECT_DEBUG for more info Err: read /dev/ptmx: input/output error, last lines:
        	            	Error: detected custom content in v2store
        	Test:       	TestCtlV2CustomContentWithDedicatedWALDir

Signed-off-by: shenmu.wy <shenmu.wy@antfin.com>

ahrtr

LGTM & thx

cc @fuweid

k8s-ci-robot · 2026-06-04T13:12:28Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: ahrtr, silentred

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

~~OWNERS~~ [ahrtr]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

ahrtr · 2026-06-05T19:08:51Z

ping @fuweid

k8s-ci-robot added the size/M label Jun 3, 2026

ahrtr reviewed Jun 3, 2026

View reviewed changes

silentred force-pushed the fix-21849-check-v2store-wal branch from 74319af to a28de3b Compare June 3, 2026 14:06

k8s-ci-robot added size/L and removed size/M labels Jun 3, 2026

sahilpatel09 mentioned this pull request Jun 3, 2026

[release-3.5] Add write-only-skip-check option for --v2-deprecation to bypass the v2 content check #21897

Merged

ahrtr reviewed Jun 3, 2026

View reviewed changes

ahrtr mentioned this pull request Jun 3, 2026

etcdutl check v2store should check WAL records as well #21849

Open

silentred force-pushed the fix-21849-check-v2store-wal branch 2 times, most recently from 43f69e5 to 7893963 Compare June 4, 2026 03:26

silentred changed the title ~~[Draft] etcdutl: validate WAL records in check v2store~~ etcdutl: validate WAL records in check v2store Jun 4, 2026

ahrtr reviewed Jun 4, 2026

View reviewed changes

Comment thread etcdutl/etcdutl/check_command.go Outdated

ahrtr changed the title ~~etcdutl: validate WAL records in check v2store~~ [release-3.5] enhance etcdutl check v2store to check both v2 snapshot and WAL records Jun 4, 2026

ahrtr mentioned this pull request Jun 4, 2026

etcdutl: check WAL records in check v2store command #21901

Closed

silentred added 2 commits June 4, 2026 17:41

etcdutl: validate WAL records in check v2store

8273628

Signed-off-by: shenmu.wy <shenmu.wy@antfin.com>

fix e2e tests in v2store_deprecation_test.go

867f6d7

Signed-off-by: shenmu.wy <shenmu.wy@antfin.com>

silentred force-pushed the fix-21849-check-v2store-wal branch from 19bfec4 to 64612fb Compare June 4, 2026 09:41

ahrtr reviewed Jun 4, 2026

View reviewed changes

silentred force-pushed the fix-21849-check-v2store-wal branch from 64612fb to 11f580e Compare June 4, 2026 10:41

add e2e tests for checking v2 custom content in v2store or WAL

eebec78

Signed-off-by: shenmu.wy <shenmu.wy@antfin.com>

silentred force-pushed the fix-21849-check-v2store-wal branch from 11f580e to eebec78 Compare June 4, 2026 11:42

ahrtr approved these changes Jun 4, 2026

View reviewed changes

k8s-ci-robot added the approved label Jun 4, 2026

ahrtr requested a review from fuweid June 4, 2026 13:12

Conversation

silentred commented Jun 3, 2026

Uh oh!

ahrtr left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

case 1: neither v2 snapshot nor WAL file have custom v2 content

case 2: only v2 snapshot file has custom v2 content

case 3: only WAL files have custom v2 content

case 4: both v2 snapshot and WAL record have custom v2 content

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

ahrtr commented Jun 4, 2026

Uh oh!

silentred commented Jun 4, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

silentred commented Jun 4, 2026

Uh oh!

silentred commented Jun 4, 2026

Uh oh!

ahrtr commented Jun 4, 2026

Uh oh!

ahrtr left a comment

Choose a reason for hiding this comment

Uh oh!

k8s-ci-robot commented Jun 4, 2026

Uh oh!

ahrtr commented Jun 5, 2026

Uh oh!

Reviewers

Assignees

Labels

Milestone

Development

Uh oh!

3 participants