Operator should add graceful shutdown to the pods

[himadrisingh]

The ideal graceful flow would be:

1. SYSTEM STOP MERGES -- no NEW merges start
2. SYSTEM STOP FETCHES -- no NEW fetches (replication) start
3. Wait for in-flight merges/fetches to finish
   (query system.merges / system.replicated_fetches to check)
4. SYSTEM SHUTDOWN -- now safe to kill, nothing in progress

What the operator does today:

1. SYSTEM SHUTDOWN -- everything in progress gets killed

Upgrades can cause some race condition with various setups.

[AdheipSingh]

+1

The current restart flow lives in

clickhouse-operator/pkg/controller/chi/worker-reconciler-chi.go

Lines 446 to 521 in 3440eac

	func (w *worker) hostSoftwareRestart(ctx context.Context, host *api.Host) error {
	w.a.V(1).M(host).F().Info("Host software restart start. Host: %s", host.GetName())
	// Get restart counters - they'll be used to check restart success
	restartCounters, err := w.c.kube.Pod().(interfaces.IKubePodEx).GetRestartCounters(ctx, host)
	if err != nil {
	w.a.V(1).M(host).F().Info("Host software restart abort 1. Host: %s err: %v", host.GetName(), err)
	return err
	}
	// Issue SQL shutdown command - expect host to shutdown and pod be restarted by k8s
	err = w.ensureClusterSchemer(host).HostShutdown(ctx, host)
	if err != nil {
	w.a.V(1).M(host).F().Info("Host software restart abort 2. Host: %s err: %v", host.GetName(), err)
	return err
	}
	w.a.V(1).M(host).F().Info("Host software shutdown ok. Host: %s", host.GetName())
	// Wait for restart counters to change
	err = w.waitHostRestart(ctx, host, restartCounters)
	if err != nil {
	w.a.V(1).M(host).F().Info("Host software restart abort 3. Host: %s err: %v", host.GetName(), err)
	return err
	}
	w.a.V(1).M(host).F().Info("Host software restart ok. Host: %s", host.GetName())
	// Wait for pod to start
	err = w.waitHostIsStarted(ctx, host)
	if err != nil {
	w.a.V(1).M(host).F().Info("Host software restart abort 4. Host: %s is not started", host.GetName())
	return fmt.Errorf("host is not started")
	}
	w.a.V(1).M(host).F().Info("Host software pod is started. Host: %s ", host.GetName())
	// Ensure pod is running
	err = w.waitHostIsRunning(ctx, host)
	if err != nil {
	w.a.V(1).M(host).F().Info("Host software restart abort 5. Host: %s is not running", host.GetName())
	return fmt.Errorf("host is not running")
	}
	w.a.V(1).M(host).F().Info("Host software pod is running. Host: %s ", host.GetName())
	// Ensure pod is ready
	err = w.waitHostIsReady(ctx, host)
	if err != nil {
	w.a.V(1).M(host).F().Info("Host software restart abort 6. Host: %s is not ready", host.GetName())
	return fmt.Errorf("host is not ready")
	}
	w.a.V(1).M(host).F().Info("Host software pod is ready. Host: %s ", host.GetName())
	// At this stage we'd expect to have software up and able to respond
	err = w.isHostSoftwareAbleToRespond(ctx, host)
	if err != nil {
	w.a.V(1).M(host).F().Info("Host software restart abort 7. Host: %s err: %v", host.GetName(), err)
	return err
	}
	w.a.V(1).M(host).F().Info("Host software version ok. Host: %s ", host.GetName())
	// However, some containers within the pod may still have flapping problems and be in CrashLoopBackOff
	if w.isPodCrushed(ctx, host) {
	w.a.V(1).M(host).F().Info("Host software restart abort 8. Host: %s is crushed", host.GetName())
	return fmt.Errorf("host is crushed")
	}
	w.a.V(1).M(host).F().Info("Host software is not crushed. Host: %s ", host.GetName())
	// Check whole pod health
	if !w.isPodOK(ctx, host) {
	w.a.V(1).M(host).F().Info("Host software restart abort 9. Host: %s is not ok", host.GetName())
	return fmt.Errorf("host is not ok")
	}
	w.a.V(1).M(host).F().Info("Host software pod is ok. Host: %s ", host.GetName())
	w.a.V(1).M(host).F().Info("Host software restart success. Host: %s", host.GetName())
	return nil
	}

which calls

clickhouse-operator/pkg/model/chi/schemer/schemer.go

Lines 180 to 185 in 3440eac

	// HostShutdown shutdown a host
	func (s *ClusterSchemer) HostShutdown(ctx context.Context, host *api.Host) error {
	log.V(1).M(host).F().Info("Host shutdown: %s", host.GetName())
	return s.ExecHost(ctx, host, s.sqlShutDown(),
	clickhouse.NewQueryOptions().SetRetry(false).SetLogQueries(true))
	}

, issuing only a

clickhouse-operator/pkg/model/chi/schemer/sql.go

Lines 276 to 278 in 3440eac

	func (s *ClusterSchemer) sqlShutDown() []string {
	return []string{"SYSTEM SHUTDOWN"}
	}

.

@Slach - would be willing to PR this.

Add new SQL queries in pkg/model/chi/schemer/sql.go:
- SYSTEM STOP MERGES
- SYSTEM STOP FETCHES

[alex-zaitsev]

In-flight merges may take hours. So stopping/waiting merges does not make a lot of sense. It is also ok to interrupt merge in any moment, no data is lost in any case. There could be some garbage left, but ClickHouse in general leaves a lot of garbage in detached parts.

Same for fetches. Interrupting a fetch leaves some garbage but it does not lead to any data lost.

Waiting for both merges and fetches to complete may be even worse than now, since it will increase replication lag before shutting down the pod.

Do you have any example of a race condition? We have not seen any in years of operation.

[himadrisingh]

We are facing a weird issue where data gets corrupted during upgrades on a clickhouse cluster backed by S3.
We are seeing objects not getting pushed to S3 in a distributed cluster, resulting into failures after restart and corrupted cluster.

Code: 499. DB::Exception: The specified key does not exist. This error happened for S3 disk: while reading key: replicas/<>/zku/jcovpdgjbsryxfrkznbcehelfeoka, from bucket: <bucket>: Cache info: Buffer path: replicas/<>/zku/jcovpdgjbsryxfrkznbcehelfeoka, hash key: 11e2e7cb909012e20a1d8169183d1446, file_offset_of_buffer_end: 0, read_until_position: 96, internal buffer end: 0, read_type: REMOTE_FS_READ_BYPASS_CACHE

Not sure if fsync settings are causing thisbut still figuring whats causing them to fail.

[alex-zaitsev]

@himadrisingh , do you use zero copy replication or just a normal S3?

[himadrisingh]

@himadrisingh , do you use zero copy replication or just a normal S3?

We did had zero copy and had a lot of issues then. But recently we moved away but still seeing these issues thats why wondering if its unclean shutdown causing this.