Making scheduling cycle optionally run asynchronously

[JamesMurkin]

This PR makes it so the scheduler can run the scheduling loop in 2 modes:

• Synchronous
  • The existing approach, scheduling algo is called directly by the main loop, which blocks until schedule is complete)
• Asynchronous
  • The new approach, where scheduling algo runs on a separate go routine and the main loop simply triggers it and retrieves the result when it is ready

Motivation

The reason for this PR is that most job state transitions run through the scheduler (pending/running/succeeded/failed/preempted/cancelled etc), but they are getting delayed by the scheduling algo taking a long time, resulting in poor UX.

By splitting the scheduling_algo into a separate routine that gets its result merged in when ready, we can have a far more responsive system

• Improves UX (time between cancel request -> cancel showing up in events + lookout can be reduced from 30-60s to ideally 1-2s
• Allows downstream systems to process next steps faster - as they get job state updates in a more timely manner

Caveats

When the scheduling algo runs, it maintains a snapshot of the state at the time it started. As time passes it inevitably gets more and more out of date

Meaning we can end up scheduling a job that has been cancelled on the main loop.

There is reconciliation code to remove decisions on jobs that have since finished

[greptile-apps]

Greptile Summary

This PR introduces an optional asynchronous scheduling mode that decouples the scheduling algorithm from the main scheduler loop, allowing job state transitions (cancel, fail, preempt) to flow through without waiting for the scheduling algo to complete. When async mode is enabled, the background goroutine operates on a DryRunTxn snapshot and results are reconciled against current state before being applied.

• AsyncSchedulingRunner implements a clear state machine (Idle → RunRequested → Running → ResultReady → Idle) with mutex-protected state, a buffered wake channel, and a Reset() that cancels in-flight runs and blocks until completion — ensuring leadership failovers produce a clean slate.
• scheduler.go drives the async lifecycle correctly: Reset() on every become-leader transition, Trigger() only after a clean committed cycle, and result consumption gated on shouldSchedule to avoid hammering GetSchedulerResult.
• Reconciliation in async.go filters stale scheduled/preempted decisions (including whole-gang drops) against the current txn, and a new schedulingDuration histogram tracks the algo's wall-clock time independently from the main cycle time.

Confidence Score: 5/5

The async scheduling mode is safe to merge; the state machine, cancellation path, and reconciliation logic are all well-reasoned and well-tested.

The core state machine in AsyncSchedulingRunner is correct and race-free. Reset() blocks until any in-flight scheduling run has finished before returning, ensuring the next Trigger always schedules against committed state. The two findings in the reconcile path are defensive-coding concerns against non-standard SchedulingAlgo implementations; the production FairSchedulingAlgo invariants prevent them from firing today. Test coverage for the async lifecycle, leadership transitions, and reconciliation logic is thorough.

internal/scheduler/scheduling/runner/async.go — the reconcile helpers updatePreemptedJobs and the UpdateFairShares call in reconcile() assume AdditionalSchedulingInfo/EvictorResult/SchedulingContext are non-nil; worth adding nil guards for future-proofing.

Important Files Changed

Filename	Overview
internal/scheduler/scheduling/runner/async.go	New async runner with a well-structured state machine (Idle→RunRequested→Running→ResultReady→Idle); Reset() correctly cancels in-flight runs and blocks until they complete. Two missing nil guards in the reconcile path could panic with non-standard SchedulingAlgo implementations.
internal/scheduler/scheduler.go	Refactors cycle() to return schedulingAttempted flag; Trigger() correctly fires only after a clean committed cycle as leader; Reset() on leadership transitions correctly discards stale async state.
internal/scheduler/scheduler_test.go	Good coverage added: TestScheduler_AsyncRunner exercises the two-cycle consume pattern, TestRun_AsyncRunnerResetOnLeadershipChange verifies Reset fires on become-leader transitions. testSchedulingAlgo refactored to use atomic.Int64 for thread safety.
internal/scheduler/scheduling/runner/async_test.go	Thorough unit tests covering state transitions, Reset on in-flight runs, gang reconciliation, and context cancellation.
internal/scheduler/metrics/cycle_metrics.go	Adds schedulingDuration histogram properly wired into describe/collect. New metric measures algo wall-clock time independently from the main-loop cycle time.

Sequence Diagram

sequenceDiagram
    participant ML as Main Loop
    participant AR as AsyncRunner
    participant BG as Background Goroutine
    participant DB as JobDb

    ML->>AR: Reset() [on become-leader]
    AR-->>ML: (state → Idle)

    ML->>AR: "GetSchedulerResult() [shouldSchedule=true, no result]"
    AR-->>ML: "nil, nil (state=Idle)"
    ML->>AR: Trigger() [after clean commit]
    AR->>BG: wake (state → RunRequested → Running)

    BG->>DB: DryRunTxn()
    BG->>BG: schedulingAlgo.Schedule(runCtx, txn)
    BG-->>AR: finishRun(result) [state → ResultReady]

    ML->>AR: GetSchedulerResult() [next shouldSchedule cycle]
    AR->>AR: reconcile(result, currentTxn)
    AR->>AR: upsertSchedulerResult(txn, result)
    AR-->>ML: "*SchedulerResult (state → Idle)"
    ML->>ML: Publish events, commit txn
    ML->>AR: Trigger() [after clean commit]
    AR->>BG: wake (state → RunRequested → Running)

    Note over ML,AR: On error or leadership loss
    ML->>AR: Reset() [become-leader / error path]
    AR->>BG: cancel() [if Running]
    BG-->>AR: done closed
    AR-->>ML: (state → Idle, result discarded)

Loading

_{Reviews (3): Last reviewed commit: "Merge branch 'master' into async_schedul..." | Re-trigger Greptile}