server, e2e: auto-promote learners that have caught up with the leader#21744
Draft
john7doe wants to merge 10 commits into
Draft
server, e2e: auto-promote learners that have caught up with the leader#21744john7doe wants to merge 10 commits into
john7doe wants to merge 10 commits into
Conversation
|
Hi @john7doe. Thanks for your PR. I'm waiting for a etcd-io member to verify that this patch is reasonable to test. If it is, they should reply with Regular contributors should join the org to skip this step. Once the patch is verified, the new status will be reflected by the I understand the commands that are listed here. DetailsInstructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. |
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: john7doe The full list of commands accepted by this bot can be found here. DetailsNeeds approval from an approver in each of these files:Approvers can indicate their approval by writing |
john7doe
force-pushed
the
auto-promote-learners
branch
2 times, most recently
from
ac7ab62 to
23a750e
Compare
john7doe
force-pushed
the
auto-promote-learners
branch
from
23a750e to
5f708ae
Compare
Introduce a new alpha feature gate that, when enabled, will let the leader automatically promote a learner member once it has caught up with the leader's log. This commit only registers the gate; the behaviour is wired up in a follow-up commit. Refs: etcd-io#21611 Signed-off-by: Simon Scharf <john7doe@gmail.com>
Add a leader-side monitor goroutine that periodically scans cluster members for learners and promotes any that have caught up. Readiness is decided by the existing isLearnerReady check (>=90% Match against the leader), so behaviour matches a manual 'etcdctl member promote' issued at the leader. The loop is gated by the AutoPromoteLearners feature gate (alpha, default off, added in the previous commit) and reuses the existing promoteMember -> mayPromoteMember -> configure pipeline, including StrictReconfigCheck quorum-safety and the learnerPromoteSucceed / learnerPromoteFailed metrics. The internal call uses authStore.WithRoot to bypass the user-level admin check for the server's own goroutine, mirroring the existing lease-revocation pattern. Trigger cadence reuses monitorVersionInterval and additionally wakes on firstCommitInTerm so a freshly-elected leader runs the check promptly. Expected-pending errors (learner not ready, quorum unsafe, race with manual promote, leadership lost) are logged at DEBUG to avoid log spam; unexpected errors are surfaced at WARN. Refs: etcd-io#21611 Signed-off-by: Simon Scharf <john7doe@gmail.com>
Add TestMonitorAutoPromoteLearners_ExitsCleanly which verifies that
monitorAutoPromoteLearners:
- wakes on firstCommitInTerm without touching cluster / authStore
when the local member is not the leader (i.e. the leader gate
short-circuits before tryAutoPromoteLearners runs);
- exits promptly when the stopping channel is closed.
A regression here would either deadlock the goroutine or surface a
nil-pointer dereference once the leader gate is removed, so this is
a cheap guard for the contract enforced by the loop's scaffolding.
Refs: etcd-io#21611
Signed-off-by: Simon Scharf <john7doe@gmail.com>
Add an integration test that verifies a learner is auto-promoted to a voting member without an explicit MemberPromote call when the AutoPromoteLearners feature gate is enabled on every cluster member. To enable the feature gate from integration tests, plumb a new EnableAutoPromoteLearners field through ClusterConfig and MemberConfig, mirroring the existing EnableLeaseCheckpoint pattern in MustNewMember (it is appended to the per-member feature-gate string). The test mirrors the setup of TestMemberPromote (add learner, launch it) but instead of calling MemberPromote it polls MemberList until the learner transitions to a voter; the auto-promote loop ticks at monitorVersionInterval (~4s), so a 30s deadline gives several iterations of headroom on slow runners. Refs: etcd-io#21611 Signed-off-by: Simon Scharf <john7doe@gmail.com>
Suppress transient errors that promoteMember can return during normal
operation under the leader-side auto-promote loop, so the Warn log
fires only on truly unexpected failures:
- ErrLeaderChanged: leadership flips between isLeader() and the
raft proposal.
- ErrTimeout / ErrTimeoutWaitAppliedIndex: backpressure or proposal
timeout; the next monitor tick retries.
- ErrStopped / ErrCanceled: server is shutting down; the monitor's
stopping select exits on the next iteration.
The learnerPromoteFailed metric is no longer incremented for these
transient states.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Signed-off-by: Simon Scharf <john7doe@gmail.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Signed-off-by: Simon Scharf <john7doe@gmail.com>
Clarify behavior:
- Readiness reuses the same isLearnerReady check as the manual
etcdctl member promote path.
- Only the leader scans; followers short-circuit.
- The gate must be set on every member that may become leader for
auto-promotion to take effect cluster-wide.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Signed-off-by: Simon Scharf <john7doe@gmail.com>
End-to-end coverage for the AutoPromoteLearners feature gate using a real 3-node etcd process cluster. The test starts the cluster with --feature-gates=AutoPromoteLearners=true, adds a fourth member as a learner, and asserts the leader auto-promotes it to a voting member without an explicit etcdctl member promote invocation. The deadline gives ~10 ticks of the auto-promote monitor (monitorVersionInterval, ~4s) to keep the test reliable on slow runners. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Signed-off-by: Simon Scharf <john7doe@gmail.com>
Regenerated via 'make fix' to drop the stale github.com/antithesishq/antithesis-sdk-go entry so 'make verify-bom' passes locally. No source dependency changes. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Signed-off-by: Simon Scharf <john7doe@gmail.com>
monitorAutoPromoteLearners/tryAutoPromoteLearners adjust comment remove test Signed-off-by: Simon Scharf <john7doe@gmail.com>
john7doe
force-pushed
the
auto-promote-learners
branch
from
5f708ae to
b3a94fd
Compare
serathius
reviewed
May 12, 2026
| // owner: @john7doe | ||
| // alpha: v3.7 | ||
| // issue: https://github.com/etcd-io/etcd/issues/21611 | ||
| AutoPromoteLearners featuregate.Feature = "AutoPromoteLearners" |
Member
There was a problem hiding this comment.
This might be a good candidate for first cluster level flag, what do you think @siyuanfoundation ? Context: kubernetes/enhancements#5189
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #21611.
Motivation
Adding a member as a learner is the recommended way to grow an etcd cluster, but turning that learner into a voting member still requires an operator to issue
etcdctl member promote(or callMemberPromote). That manual step is the only thing standing between "learner is ready" and "learner is a voting member", and the readiness check already lives in the server — it is the same one the manual path uses.This change adds an opt-in feature gate,
AutoPromoteLearners, that makes the leader perform the promotion automatically once the learner has caught up.Design
AutoPromoteLearners(Alpha, defaultfalse). With the gate off, behavior is unchanged.monitorAutoPromoteLearnersgoroutine is launched fromEtcdServer.Startonly when the gate is enabled. It wakes onfirstCommitInTerm(so a newly elected leader picks up any pending learners promptly) or everymonitorVersionInterval(~4s), short-circuits on non-leaders, and delegates totryAutoPromoteLearners.tryAutoPromoteLearnersiteratess.cluster.Members(), filters for learners, and calls the existingpromoteMember. That goes throughisLearnerReady(the same check used byetcdctl member promote) and the standard raft conf-change proposal, so the WAL audit trail is identical to a manual promotion.ErrLearnerNotReady,ErrNotEnoughStartedMembers,ErrNotLeader,ErrLeaderChanged,ErrTimeout,ErrTimeoutWaitAppliedIndex,ErrStopped,ErrCanceled,ErrMemberNotLearner,ErrIDNotFound) are logged at Debug; only unexpected errors are logged at Warn and bumped on thelearnerPromoteFailedmetric.learnerPromoteSucceed/learnerPromoteFailedcounters.Behavior change
--feature-gates=AutoPromoteLearners=trueon the prospective leader, a learner is promoted automatically once it has caught up.The gate must be enabled on every member that may become leader for auto-promotion to take effect cluster-wide.
Tests
TestMonitorAutoPromoteLearners_ExitsCleanlyverifies the monitor wakes onfirstCommitInTerm, short-circuits when the local member is not the leader, and exits promptly onstopping.TestAutoPromoteLearnerintests/integration/clientv3brings up a 3-node cluster with the gate enabled, adds a learner via the client API, and asserts thatMemberListreports the learner promoted within 30s.TestAutoPromoteLearnerintests/e2ebrings up a full 3-node etcd process cluster with--feature-gates=AutoPromoteLearners=true, adds a learner viaetcdctl member add --learner, and asserts auto-promotion within 45s plus that the new member is healthy.TestMemberPromote*tests continue to pass.Verification
make verify— green (lint, dep, shellcheck, mod-tidy, shellws, proto-annotations, genproto, gomodguard, go-workspace, go-versions, grpc-experimental, bom).make build— green.-race -count=1.Open questions for reviewers
bill-of-materials.jsonrefresh somake verify-bomis green). Happy to squash, or drop the BOM refresh if a parallel fix is already in flight.CHANGELOG
CHANGELOG/CHANGELOG-3.7.mdupdated under "etcd server".