Set state flag when rebalance is interrupted by eric-higgins-ai · Pull Request #292 · NVIDIA/aistore

eric-higgins-ai · 2026-05-01T21:21:57Z

I would like to use the RebalanceInterrupted bit of the ais_target_state_flags Prometheus metric to monitor rebalance stability. However, in the current code it's never set to 1.

alex-aizman · 2026-05-14T01:09:48Z

I can see where RebalanceInterrupted flag gets set, but I don’t see where it gets cleared...

I guess, the point is to mirror the respective persistent marker (fname.RebalanceMarker).

need to find out how does that work with existing code:

https://github.com/NVIDIA/aistore/blob/main/ais/htrun.go#L216

etc.

and why that's not enough.

eric-higgins-ai · 2026-05-14T17:35:34Z

@alex-aizman the flag gets cleared in RemoveMarker, for example here

I don't have a good enough mental model of the code to know for sure why https://github.com/NVIDIA/aistore/blob/main/ais/htrun.go#L216 isn't sufficient, but my understanding is just that that's never synced to the ais_target_state_flags prometheus metric

alex-aizman · 2026-07-03T22:35:25Z

I would like to use the RebalanceInterrupted bit of the ais_target_state_flags Prometheus metric to monitor rebalance stability. However, in the current code it's never set to 1.

No, that's not true

$ git grep RebalanceInterrupted | grep "\bSet"
ais/htcommon.go:580:            nsti.Flags = nsti.Flags.Set(cos.RebalanceInterrupted)
ais/htrun.go:233:                       cm.Flags = cm.Flags.Set(cos.RebalanceInterrupted)
$

Closing.

eric-higgins-ai · 2026-07-08T20:28:45Z

@alex-aizman those calls don't get propagated to the ais_target_state_flags prometheus metric. The prometheus metric is only set via SetFlag/ClrFlag in stats/common.go. If you look for usages of those functions, you'll see that ClrFlag is called with RebalanceInterrupted here but there's no corresponding SetFlag call

eric-higgins-ai · 2026-07-08T20:31:42Z

You can reproduce this by running ais start rebalance and then immediately running ais stop rebalance to interrupt. If you observe the value of the flag in prometheus with this promql query floor(ais_target_state_flags / 16) % 2 == 1 then you can see that the flag isn't set when using the code from the main branch, and is set when this PR is applied

alex-aizman · 2026-07-08T20:50:17Z

how about ais show cluster - much more direct way to see all those alerts:

 11 type NodeStateFlags uint64 // same as cos.BitFlags
 12 
 13 const NodeAlerts = "state.flags"
 14 
 15 const noAlerts = "ok"
 16 
 17 const (
 18         VoteInProgress       = NodeStateFlags(1 << iota) // warning
 19         ClusterStarted                                   // info: (primary: cluster-started | all other nodes: joined-cluster)
 20         NodeStarted                                      // info: (started; possibly, not joined yet)
 21         Rebalancing                                      // warning
 22         RebalanceInterrupted                             // warning
 23         Resilvering                                      // warning
 24         ResilverInterrupted                              // warning
 25         NodeRestarted                                    // warning (powercycle, crash)
 26         OOS                                              // node out of space; red alert (see IsRed below)
 27         OOM                                              // out of memory; red alert
 28         MaintenanceMode                                  // warning
 29         LowCapacity                                      // node (used > high); warning: OOS possible soon..
 30         LowMemory                                        // ditto OOM
 31         DiskFault                                        // red
 32         NumGoroutines                                    // yellow
 33         HighNumGoroutines                                // red
 34         CertWillSoonExpire                               // warning X.509
 35         CertificateExpired                               // red --/--
 36         CertificateInvalid                               // red --/--
 37         KeepAliveErrors                                  // warning (new keep-alive errors during the last 5m)
 38         OOCPU                                            // out of CPU; red
 39         LowCPU                                           // warning
 40         DiskOOS                                          // disk out of space
 41         DiskLowCapacity                                  // warning
 42 )

does it reproduce with ais show cluster?

eric-higgins-ai · 2026-07-08T20:51:56Z

Yeah ais show cluster works - I can see the rebalance-interrupted flag there. I wanted to access the information in Prometheus though, so we can integrate it with our on-call alerting setup.

alex-aizman · 2026-07-08T20:53:29Z

@gaikwadabhishek take a look

reopening this PR...

gaikwadabhishek

Hey @eric-higgins-ai , thanks for contributing. Few edge scenarios are not handled or needs exploration. In re: prometheus, the alert already exists - AISRebalanceInterrupted @ ais-k8s/monitoring/kube-prom/alert-rules/templates/native-rules.yaml.

I think if you have time this change can also be applied to resilver interrupted scenario.

gaikwadabhishek · 2026-07-08T21:56:57Z

-	if !ok {
-		nlog.Warningln(rargs.logHdr, "ended streams when curr. stage:", stages[curStage])
+	} else {
+		tstats.SetFlag(cos.NodeAlerts, cos.RebalanceInterrupted)


This does not cover cleanup-mode (--cleanup in ais start rebalance) rebalances. RunCleanup creates the same RebalanceMarker, but finiCleanup retains it on abort without setting RebalanceInterrupted. I reproduced this by aborting cleanup mode: the marker remained and cluster status reported interruption, while ais_target_state_flags stayed clear. Please update finiCleanup

@eric-higgins-ai you marked this as resolved but no changes were implemented? any issue?

ah sorry, I resolved the comments for my own bookkeeping but I haven't pushed changes yet as I'm still testing them. I can unresolve the comments

gaikwadabhishek · 2026-07-08T22:00:57Z

Please handle a false return from RemoveMarker. It currently clears RebalanceInterrupted unconditionally even when marker deletion fails. I reproduced ok=false, markerExists=true, and the metric bit cleared. This leaves cluster status and Prometheus inconsistent. The bit should only be cleared after confirmed marker removal, or restored when removal fails

the comment is for the below line

gaikwadabhishek · 2026-07-08T22:05:39Z

Could a stop request arrive after Quiesce() returns but before xreb.Finish()? The marker is already removed here, but the xaction is still running, so the stop is accepted. Finish() then records it as aborted while the marker and Prometheus flag indicate success. Can we either reject late stops or update the marker after the final xaction state is known?

this scenario is pre-existing, technically not in your scope as well. so skip this for now. but needs fixing so add a TODO atleast

Signed-off-by: eric-higgins-ai <erichiggins@applied.co>

eric-higgins-ai · 2026-07-10T18:38:23Z

ok @gaikwadabhishek I addressed your comments. I also found a few other bugs while testing, which I also fixed:

When a new rebalance starts, the rebalance-interrupted flag gets cleared in the output of ais show cluster but doesn't get cleared in Prometheus until the rebalance finishes successfully
When launching a rebalance with ais start rebalance --cleanup, there were 2 issues:
- The rebalance-interrupted flag doesn't get cleared in ais show cluster or in prometheus. This is inconsistent with the behavior without the "cleanup" flag
- The rebalancing flag doesn't get set in prometheus

eric-higgins-ai requested a review from alex-aizman as a code owner May 1, 2026 21:21

alex-aizman closed this Jul 3, 2026

alex-aizman reopened this Jul 8, 2026

alex-aizman assigned gaikwadabhishek Jul 8, 2026

gaikwadabhishek requested changes Jul 8, 2026

View reviewed changes

Set state flag when rebalance is interrupted

98b2b3b

Signed-off-by: eric-higgins-ai <erichiggins@applied.co>

eric-higgins-ai force-pushed the set-rebalance-interrupted branch from 692e142 to 98b2b3b Compare July 10, 2026 18:29

eric-higgins-ai requested a review from gaikwadabhishek July 10, 2026 18:38

Uh oh!

Conversation

eric-higgins-ai commented May 1, 2026

Uh oh!

alex-aizman commented May 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

eric-higgins-ai commented May 14, 2026

Uh oh!

alex-aizman commented Jul 3, 2026

Uh oh!

eric-higgins-ai commented Jul 8, 2026

Uh oh!

eric-higgins-ai commented Jul 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

alex-aizman commented Jul 8, 2026

Uh oh!

eric-higgins-ai commented Jul 8, 2026

Uh oh!

alex-aizman commented Jul 8, 2026

Uh oh!

gaikwadabhishek left a comment

Choose a reason for hiding this comment

Uh oh!

gaikwadabhishek Jul 8, 2026

Choose a reason for hiding this comment

Uh oh!

gaikwadabhishek Jul 10, 2026

Choose a reason for hiding this comment

Uh oh!

eric-higgins-ai Jul 10, 2026

Choose a reason for hiding this comment

Uh oh!

gaikwadabhishek Jul 8, 2026

Choose a reason for hiding this comment

Uh oh!

gaikwadabhishek Jul 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

gaikwadabhishek Jul 8, 2026

Choose a reason for hiding this comment

Uh oh!

gaikwadabhishek Jul 8, 2026

Choose a reason for hiding this comment

Uh oh!

eric-higgins-ai commented Jul 10, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

alex-aizman commented May 14, 2026 •

edited

Loading

eric-higgins-ai commented Jul 8, 2026 •

edited

Loading

gaikwadabhishek Jul 8, 2026 •

edited

Loading