[SPARK-58263][CORE] Concurrently schedule pipelined-shuffle stage groups in the DAGScheduler by jerrypeng · Pull Request #57341 · apache/spark

[SPARK-XXXXX][CORE] Concurrently schedule stages connected by a pipelined shuffle

jerrypeng

committed

[SPARK-XXXXX][CORE] Fail fast when a pipelined group cannot fit in cluster capacity

jerrypeng

committed

[SPARK-XXXXX][CORE] Admit pipelined groups against free slots (spec S4.1)

jerrypeng

committed

[SPARK-XXXXX][CORE] Pipelined group admission: count enqueued demand, add a slot-check disable flag

jerrypeng

committed

[SPARK-XXXXX][CORE] Defer completion of pipelined-group members until the group finishes

jerrypeng

committed

[SPARK-XXXXX][CORE] Fix pipelined-group admission: skip zombie attempts, reject dynamic allocation

jerrypeng

committed

[SPARK-XXXXX][CORE] Document that isPipelinedGroupMember is consumed by a later PR in the stack

jerrypeng

committed

[SPARK-XXXXX][CORE] Fix stale free-slots doc wording: outstanding = running plus enqueued

jerrypeng

committed

[SPARK-XXXXX][CORE] Restrict pipelined jobs to all-regular-or-all-PG; admit the group up front

jerrypeng

committed

[SPARK-XXXXX][CORE] Align pipelined deferred-completion naming with ConcurrentStageDAGScheduler

jerrypeng

committed

[SPARK-XXXXX][CORE] Review round 1: add dependentStageMap to the leak check; fix a vacuous admission test

jerrypeng

committed

[SPARK-XXXXX][CORE] Pipelined consumer deferral: defer only completion, run per-task effects inline (S5.1)

jerrypeng

committed

[SPARK-XXXXX][CORE] Fix isPipelinedGroupMember scaladoc: drop stale "reads as unused" and a phantom symbol

jerrypeng

committed

[SPARK-XXXXX][CORE] Tighten outstandingTasksForOtherWorkInProfile to private[scheduler]

jerrypeng

committed

[SPARK-XXXXX][CORE] Clarify pipelined admission vs. barrier: contrast the retry behavior

jerrypeng

committed

[SPARK-XXXXX][CORE] Don't re-post TaskEnd for a finishOnly replay that lands on a torn-down stage

jerrypeng

committed

[SPARK-XXXXX][CORE] Clarify the deferral drop-path comment: inline accumulator deltas are not un-merged

jerrypeng

committed

[SPARK-XXXXX][CORE] Remove internal jargon from pipelined-shuffle comments

jerrypeng

committed

[SPARK-XXXXX][CORE] Reject a pipelined job whose member uses a non-default resource profile

jerrypeng

committed

[SPARK-XXXXX][CORE] Simplify pipelined-consumer deferral: defer the whole completion event, remove finishOnly

jerrypeng

committed

[SPARK-XXXXX][CORE] Flush a buffered consumer's TaskEnds on job teardown, not just on the release path

jerrypeng

committed

[SPARK-58263][CORE] Address review nits: correct two scheduler comments

jerrypeng

committed

[SPARK-XXXXX][CORE] Fix scalastyle line-length in the teardown-flush test assert

jerrypeng

committed

[SPARK-58263][CORE] Clarify that buffering the whole event defers all per-task side effects

jerrypeng

committed

[SPARK-58263][CORE] Drop a fan-in consumer's deferral as soon as any producer fails

jerrypeng

committed

[SPARK-58263][CORE] Document the barrier-resubmit edge for the fan-in drop path

jerrypeng

committed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[SPARK-58263][CORE] Concurrently schedule pipelined-shuffle stage groups in the DAGScheduler#57341

[SPARK-58263][CORE] Concurrently schedule pipelined-shuffle stage groups in the DAGScheduler#57341
jerrypeng wants to merge 26 commits into
apache:masterfrom
jerrypeng:stack/pipelined-shuffle-pr3-scheduling

Commits on Jul 25, 2026

[SPARK-XXXXX][CORE] Concurrently schedule stages connected by a pipelined shuffle

[SPARK-XXXXX][CORE] Fail fast when a pipelined group cannot fit in cluster capacity

[SPARK-XXXXX][CORE] Admit pipelined groups against free slots (spec S4.1)

[SPARK-XXXXX][CORE] Pipelined group admission: count enqueued demand, add a slot-check disable flag

[SPARK-XXXXX][CORE] Defer completion of pipelined-group members until the group finishes

[SPARK-XXXXX][CORE] Fix pipelined-group admission: skip zombie attempts, reject dynamic allocation

[SPARK-XXXXX][CORE] Document that isPipelinedGroupMember is consumed by a later PR in the stack

[SPARK-XXXXX][CORE] Fix stale free-slots doc wording: outstanding = running plus enqueued

[SPARK-XXXXX][CORE] Restrict pipelined jobs to all-regular-or-all-PG; admit the group up front

[SPARK-XXXXX][CORE] Align pipelined deferred-completion naming with ConcurrentStageDAGScheduler

[SPARK-XXXXX][CORE] Review round 1: add dependentStageMap to the leak check; fix a vacuous admission test

[SPARK-XXXXX][CORE] Pipelined consumer deferral: defer only completion, run per-task effects inline (S5.1)

[SPARK-XXXXX][CORE] Fix isPipelinedGroupMember scaladoc: drop stale "reads as unused" and a phantom symbol

[SPARK-XXXXX][CORE] Tighten outstandingTasksForOtherWorkInProfile to private[scheduler]

[SPARK-XXXXX][CORE] Clarify pipelined admission vs. barrier: contrast the retry behavior

[SPARK-XXXXX][CORE] Don't re-post TaskEnd for a finishOnly replay that lands on a torn-down stage

[SPARK-XXXXX][CORE] Clarify the deferral drop-path comment: inline accumulator deltas are not un-merged

[SPARK-XXXXX][CORE] Remove internal jargon from pipelined-shuffle comments

[SPARK-XXXXX][CORE] Reject a pipelined job whose member uses a non-default resource profile

[SPARK-XXXXX][CORE] Simplify pipelined-consumer deferral: defer the whole completion event, remove finishOnly

[SPARK-XXXXX][CORE] Flush a buffered consumer's TaskEnds on job teardown, not just on the release path

[SPARK-58263][CORE] Address review nits: correct two scheduler comments

[SPARK-XXXXX][CORE] Fix scalastyle line-length in the teardown-flush test assert

[SPARK-58263][CORE] Clarify that buffering the whole event defers all per-task side effects

[SPARK-58263][CORE] Drop a fan-in consumer's deferral as soon as any producer fails

[SPARK-58263][CORE] Document the barrier-resubmit edge for the fan-in drop path