feat(nodetask): add DiscoverPeers + RestartPod SeiNodeTask kinds

[bdchatham]

What

Two new composable SeiNodeTask kinds to refresh a running node's persistent-peers without baking peer-drift logic into the core SeiNode reconcile:

• DiscoverPeers (sidecar-backed) — re-resolves the target SeiNode's spec.peers and writes persistent-peers into config.toml on the PVC via the existing sidecar discover-peers task. Scalar merge → preserves the rest of the config, so no config-apply needed. Disk-only: the running seid picks it up on its next restart.
• RestartPod (controller-side) — restarts the target's pod so seid re-reads config.toml.

Sequenced DiscoverPeers → RestartPod applies fresh peers to a live node (the arctic-1 EC2→K8s validator migration use case), composable into Chaos Mesh Workflows.

Why

Migrated K8s validators need their persistent-peers updated after a spec.peers change (e.g. adding the proposer cohort). We established that:

• peers is not in the controller's rollout templateHash, so a peers edit doesn't trigger a plan;
• the NodeUpdate (image) plan runs neither discover-peers nor config-apply, and mark-ready is a no-op — so an image bump restarts seid onto the stale config.toml;
• there is no working hot-reload of persistent-peers into a live seid (seictl config-reload's seid-signal is a TODO).

Rather than enshrine a peer-drift trigger in the core reconcile (a heavier, harder-to-reverse change with fleet-wide blast radius), this delivers the capability as opt-in, composable tasks. Promote to an automatic controller trigger later if it proves worth it (tracked separately).

Design notes

• RestartPod completion is content-addressed by pod UID. Execute deletes only the captured-UID pod; Status completes when an owned Ready pod has a different UID. The UID is captured at synthesis and persisted on status.task (survives reconcile reconstruction). This is immune to the creationTimestamp 1s-resolution race a wall-clock epoch would have. It does not reuse replace-pod (which keys on StatefulSet revision and no-ops when the spec is unchanged).
• Bounded by default. RestartPod defaults to a 10m timeout, DiscoverPeers 2m (explicit spec.timeoutSeconds always wins; sidecar gov/await kinds stay unbounded). A hand-authored RestartPod whose pod never goes Ready transitions to Failed(reason=Timeout) rather than requeuing forever.
• Fail-fast on empty peers. A target with no spec.peers — including a label source whose status.resolvedPeers is empty — fails with ParamsBuildFailed rather than submitting a zero-source discover-peers that would silently wipe persistent-peers.
• Double-sign safe. RestartPod is the same single-replica + OnDelete + RWO-PVC stop-then-start as the existing replace-pod; multi-replica is a terminal guard.
• Atomicity caveat (documented on the kind): DiscoverPeers succeeds then RestartPod fails → fresh config on disk + old seid still running; transient, observable, self-healing on the next successful RestartPod.

Tests

• Unit: taskParamsForKind for both kinds (ec2Tags/label/static/mixed-empty-label/empty-peers/nil-target; RestartPod params + UID threading), restart-pod execution (UID match/no-op/missing/terminating/multi-replica guard, same-second replacement regression), CEL union (reject zero/multiple payloads, kind immutability).
• Reconcile (fake client): both kinds happy-path, empty-peers→Failed, label-empty-resolved→Failed, pod-missing→waits→completes, never-Ready→Timeout.
• envtest: CEL validation against a real apiserver.
• make test + make test-integration green; make lint 0 new findings vs main.

Follow-up (deferred)

• Promote peer-drift to an automatic controller trigger once proven.
• seid pod has no explicit terminationGracePeriodSeconds (inherits 30s) — if seid needs longer to flush on shutdown, worth setting in the StatefulSet generator. Independent of this PR.

🤖 Generated with Claude Code

[cursor]

PR Summary

Medium Risk
Touches live validator pods (delete/restart) and on-disk peer config; mistakes could affect connectivity or cause brief downtime, though single-replica/RWO guards and fail-fast peer validation limit blast radius.

Overview
Adds two SeiNodeTask kinds for opt-in peer refresh on running nodes: DiscoverPeers (sidecar writes persistent-peers to disk from the target’s spec.peers) and RestartPod (deletes a caller-supplied pod UID so seid reloads config.toml). CRD/API gains union validation, executionStartedAt / restartedPodUID status fields, and clarified timeout semantics (execution budget starts after requirePhase, with per-kind defaults for the new kinds).

The nodetask reconciler delegates param building to task.SeiNodeTaskParamsFor, uses FailureReason for stable fail reasons, stamps execution metadata at synthesis, and gains pod delete RBAC. restart-pod is a new controller task (UID-based completion, shared podCycle with refactored replace-pod). DiscoverPeers fails fast when peer sources are empty/unresolved (unlike the planner init path).

Tests: broad unit/reconcile coverage, new nodetask envtest for CEL rules, and make test-integration extended to that package.

^{Reviewed by Cursor Bugbot for commit 467ea31. Bugbot is set up for automated code reviews on this repo. Configure here.}

[bdchatham]

Pushed 2a91ca9 addressing all review feedback:

• Cursor Bugbot (HIGH, timeout includes target wait): execution budget now measured from a new status.task.executionStartedAt (stamped once the target is ready), so the requirePhase wait is no longer charged against the per-kind timeout. spec.timeoutSeconds is now execution-only (documented).
• CI lint (goconst): repeated test literals hoisted to consts.
• Your 4 comments: consts → top of file; ErrUnsupportedKind → task package; shared peer-source builder (planner de-duped); taskParamsForKind → thin dispatcher with per-kind constructors in the task package; verbose doc comments trimmed.

Two expert cross-review rounds + a regression pass signed off on the refactor. Note: spec.timeoutSeconds semantics changed from total→execution-only (additive executionStartedAt status field, back-compat).

[bdchatham]

Pushed 1aaaefa:

• Bugbot MEDIUM (init plan empty discover-peers): fixed — new controller-side discover-peers-init task re-derives peer sources from the live node each reconcile; empty/unresolved sources defer via the retry budget (bounded) instead of submitting a zero-source write that could wipe persistent-peers. Converges once status.resolvedPeers populates.
• Bugbot HIGH (timeout includes target wait): already resolved in 2a91ca9 (execution budget from executionStartedAt); the re-anchored inline was the original-commit comment.
• DRY (maintainer follow-up): extracted a shared podCycle helper so restart-pod reuses replace-pod's pod-deletion plumbing — both behavior-preserving.

Regression cross-review signed off on both. CI + Bugbot re-running.

[bdchatham]

Ready for human review — automated bar met (f477e34)

All CI checks green and Cursor Bugbot passes with no new findings:

Cursor Bugbot   pass     lint   pass     test   pass     test-integration   pass     verify-generated   pass

Review trail (each fix had a regression cross-review before push):

Finding	Resolution
Bugbot HIGH — timeout includes requirePhase wait	executionStartedAt (exec-only budget) — 2a91ca9
CI lint goconst	hoisted test literals to consts — 2a91ca9
Maintainer ×4 (consts→top, sentinel→task pkg, shared peer builder, thin dispatcher)	2a91ca9
Bugbot MEDIUM — init zero-source discover-peers	discover-peers-init defers via retry budget — 1aaaefa
Maintainer — DRY restart/replace	shared podCycle helper — 1aaaefa
Bugbot MEDIUM — empty-UID silent skip	synthesis-defer + task backstop — f477e34

The three inline comments still shown are all from earlier commits and resolved above (the HIGH @387 is the original first-commit comment re-anchored to the diff; Bugbot's per-commit check is green).

Note: spec.timeoutSeconds is now execution-only (additive status.task.executionStartedAt, back-compat). Deferred follow-up (separate): promote peer-drift to an automatic controller trigger once proven.

Over to you for the final look, @bdchatham.

[bdchatham]

Pushed 39bd88c — unwound the over-built complexity per your call + addressed the two new comments:

• Unwind: deleted discover-peers-init; reverted the planner's init discover-peers to main (byte-identical — git diff origin/main -- internal/planner/planner.go is empty); the nodetask DiscoverPeers now builds its own peer sources with its own empty/label-unresolved fail-fast (slight, deliberate duplication — no shared interface to conflict). The nodetask still can't submit a zero-source write that would wipe persistent-peers.
• :149 — ErrUnsupportedKind is now a typed error carrying the kind (errors.As).
• :159 — RestartPod no-pod-at-synthesis → fail fast (RestartTargetPodNotFound), dropping the awaitRestartTarget defer; uncached read avoids cache-lag false negatives; task-side backstop kept.

Net: the init path is untouched vs main, and the two new tasks are self-contained. Expert regression cross-review signed off. CI + Bugbot re-running.

[cursor]

Cursor Bugbot has reviewed your changes using default effort and found 2 potential issues.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit 39bd88c. Configure here.}

[bdchatham]

Pushed ec2b591. Status of the Bugbot findings on the latest commit:

• In-flight tasks lose timeout (HIGH) — fixed: driveTask lazy-stamps executionStartedAt=now when a populated status.task has a nil anchor (a task synthesized by a pre-this-change controller), so in-flight legacy tasks get a bounded fresh budget on upgrade instead of running unbounded. Idempotent; not anchored to startedAt (would back-date the deadline).
• Task timeout includes target wait (HIGH) — stale/re-anchored, resolved in 2a91ca9: the budget measures from executionStartedAt (stamped after the target is ready), not startedAt.
• Empty pod UID skips restart (MEDIUM) — resolved in 39bd88c: no owned pod at synthesis → fail fast (RestartTargetPodNotFound); task-side backstop keeps an empty UID in Running (never completes). The defer machinery is gone.

CI + Bugbot re-running on ec2b591.