Fix rewind deadlock when Task.all had multiple failed siblings (#299)

[YunchuWang]

Summary

Fixes a rewind deadlock where a nested orchestration whose Task.all fan-out contained more than one failed activity never completes after a server-side (Azure Storage / DurableTask.Core) rewind. The worker re-dispatched only one previously-failed sibling and left the other as a permanently-unresolved pending task, so the child's Task.all never resolved, no SubOrchestrationInstanceCompleted was emitted, and the parent/root hung until the 30s timeout.

Closes #299

Root cause (verified)

On the Azure Storage backend, rewind is computed server-side by DurableTask.Core — this is a plain replay of a backend-rewritten history, not the EXECUTIONREWOUND / buildRewindResult path from #296 (0 ExecutionRewound events observed).

The rewind (DurableTask.AzureStorage RewindHistoryAsync / DurableTask.Core ProcessRewindOrchestrationDecision) removes/morphs a failed activity's TaskScheduled only when a TaskFailed event exists for it. Because Task.all fails fast, the child completes-failed after the first sibling failure, so later siblings' TaskFailed events are never committed (late completion for a terminal instance is dropped). After rewind:

• Sibling A (TaskFailed present): its TaskScheduled is morphed away → its pending action survives replay → re-dispatched cleanly. ✅
• Sibling B (TaskFailed missing): its TaskScheduled is left bare (no terminal) → handleTaskScheduled deletes its pending action assuming a completion will arrive, but after a rewind none ever does → orphaned pending task → Task.all deadlocks. ❌

Observed worker trace (single child instance): Waiting for 2 task(s) … Returning 1 action(s) (one sibling re-dispatched, runs, succeeds) then Waiting for 1 task(s) … Returning 0 action(s) = deadlock.

Fix (worker executor)

1. Detect a rewind revival from the NEW events. Azure Storage wakes a rewound instance with a GenericEvent (RewindTaskOrchestrationAsync sends new GenericEvent(-1, reason)); the DTS jump-start path uses ExecutionRewound. Gating on new events scopes reconciliation to the revival work item, so ordinary post-rewind replays (whose new events are normal completions) never re-dispatch genuinely in-flight tasks.
2. Re-dispatch orphaned activities. During replay, archive each consumed scheduleTask action. At the end of a revival replay, for every still-pending task whose scheduleTask action was consumed by a bare TaskScheduled but has no live pending action, re-add the archived action. General for N failed siblings. Sub-orchestrations are intentionally excluded (their retained SubOrchestrationInstanceCreated is revived separately by the backend).
3. Make duplicate TaskScheduled idempotent. DurableTask.Core ProcessScheduleTaskDecision always appends a fresh TaskScheduled per action (no dedup), so re-dispatching the bare orphan produces a second TaskScheduled with the same id. A repeat for an already-handled id is now treated as idempotent instead of raising NonDeterminismError.

57 lines of source across 2 files (orchestration-executor.ts, runtime-orchestration-context.ts).

Test evidence

New deterministic executor-level regression test test/rewind-multi-sibling-deadlock.spec.ts drives the executor through episodes modelling the classic Azure Storage backend (append-only TaskScheduled, revival GenericEvent trigger):

• re-dispatches ALL previously-failed Task.all siblings after a rewind and completes — red without the fix (deadlock), green with it.
• re-dispatches N (=3) failed siblings — generality check.
• does NOT re-dispatch an in-flight bare TaskScheduled during ordinary (non-rewind) replay — no-regression guard (passes on main and with the fix).

Red/green proof (identical test code):

# source reverted (main behavior):
× re-dispatches ALL previously-failed Task.all siblings ...   (deadlocked: expected false, got true)
× re-dispatches N (=3) failed siblings ...
√ does NOT re-dispatch an in-flight bare TaskScheduled ...
Tests: 2 failed, 1 passed

# with fix:
√ re-dispatches ALL previously-failed Task.all siblings ...
√ re-dispatches N (=3) failed siblings ...
√ does NOT re-dispatch an in-flight bare TaskScheduled ...
Tests: 3 passed

Sister-SDK parity (corroboration)

The same defect exists in durabletask-python (independently verified against microsoft/durabletask-python@e7b5f15), which confirms this is a core, cross-SDK replay-contract issue rather than a JS-specific quirk:

• Its taskScheduled handler (worker.py ~L2196-2205) pops the pending action on a bare TaskScheduled and raises a non-determinism error if the action is already gone — action = ctx._pending_actions.pop(task_id, None); if not action: raise _get_non_determinism_error(...) — while only reading (.get) the pending task. So a rewound-away sibling leaves an orphaned pending task, exactly as in JS.
• get_actions() (worker.py ~L1549) returns list(self._pending_actions.values()), so a popped orphan is never re-dispatched → identical Task.all deadlock after a multi-failed-sibling rewind.
• Python's rewind support (_build_rewind_result) only short-circuits the DTS jump-start path (an executionRewound event in the new events). A classic Azure Storage server-side rewind (no executionRewound) falls through to normal replay — the same path that deadlocks here — and there is no _consumed / rewind-revival re-dispatch and no idempotent duplicate-TaskScheduled guard. The durabletask-python functions-support PR Andystaples/add functions support durabletask-python#155 only added a genericEvent ignore, not re-dispatch, so the deadlock remains unfixed there.

This independently corroborates two things about this PR:

1. The root cause is at the DurableTask replay-contract level, not a JS bug.
2. The idempotent handled-id guard is load-bearing. Python's if not action: raise NonDeterminism on a repeat TaskScheduled is exactly what a naive re-dispatch would trip — because re-dispatching the orphan makes the backend append a second TaskScheduled for the same id. Re-dispatch without the idempotency guard would convert the deadlock into a NonDeterminismError, which is why this PR pairs re-dispatch with idempotent duplicate-TaskScheduled handling.

A durabletask-python parity fix is therefore warranted and should be tracked separately.

E2E status on real Azure Storage (verified)

Run on a live Azure Storage backend (fix logic ported onto the #282 gRPC-worker core; fix anchors byte-identical; callEntities=false, single target framework to force one deterministic instance):

• The NonDeterminism / duplicate-TaskScheduled assumption is CLOSED — zero NonDeterminismError across numFailures 1 and 2.
• On the path DurableTask.Core actually rewinds, the signaled child logs Waiting for 2 → Returning 2 (both failed activity siblings re-dispatched) and completes, whereas the un-fixed baseline logs Returning 1 and deadlocks — the decisive proof this fix is real and necessary.

This fix is necessary but not sufficient to turn RewindOrchestratorTests.RewindFailedOrchestration_ShouldSucceed(1|2) green, because that E2E also exercises a distinct, deeper gap: the root Task.all fails fast on the first sub-orchestration failure, so the second failing sub-orch's failure is dropped (late completion for a now-terminal parent). Server-side rewind therefore signals only the committed-failed subtree; the other failing subtree is re-driven with no rewind signal, so its grandchild hits the identical #299 orphan that the (correctly gate-guarded) reconcile cannot repair without a signal. Which subtree is dropped is a race. That unsignaled-sibling gap is tracked separately in #301.

Gates

• npm run build — clean (all workspaces)
• npm run lint — clean
• npm run test:unit — 1283 passed, 0 failed (1122 core incl. the new spec and the Add rewind support #296 rewind-e2e suite, 106 azuremanaged, 55 export-history)

Notes / honesty

• Root cause and fix were verified against the real DurableTask.Core / DurableTask.AzureStorage rewind source, the failing-run worker log, and the issue Rewind of nested orchestration deadlocks when a Task.all had multiple failed activities (only one failed sibling re-dispatched on replay) #299 A/B evidence.
• The shipped correctness proof is the executor-level regression test, which faithfully models the append-only backend and the duplicate-TaskScheduled behavior the idempotent guard handles. The full extension E2E was additionally run live on Azure Storage (see E2E status above): on the path DTFx actually rewinds it confirms the Rewind of nested orchestration deadlocks when a Task.all had multiple failed activities (only one failed sibling re-dispatched on replay) #299 fix — the signaled grandchild logs Waiting for 2 → Returning 2 (both failed siblings re-dispatched) and completes, versus the NOFIX baseline's Returning 1 deadlock — with zero NonDeterminismError across every run, so the idempotent handled-id guard holds on the real backend. However, clean single-target-framework runs of RewindFailedOrchestration_ShouldSucceed(1|2) still fail at RewindOrchestratorTests.cs:50 for a distinct, deeper reason — an unsignaled re-driven sibling subtree, tracked in Rewind after a multi-failed-sibling WhenAll deadlocks the UNSIGNALED sibling subtree (emit-once replay can't reconcile a re-driven orphan with no rewind signal) #301 — and an earlier apparent pass was a multi-TFM (net8.0;net10.0) concurrency artifact, not a reliable green. Fix rewind deadlock when Task.all had multiple failed siblings (#299) #300 stays correctly scoped to the Rewind of nested orchestration deadlocks when a Task.all had multiple failed activities (only one failed sibling re-dispatched on replay) #299 signaled-path fix.

Co-authored-by: Copilot App 223556219+Copilot@users.noreply.github.com

[Copilot]

Pull request overview

Fixes a worker-side replay deadlock after server-side rewinds (Azure Storage / DurableTask.Core) when a nested orchestration’s Task.all fan-out had multiple failed activity siblings, leaving one or more siblings as bare TaskScheduled events with no terminal completion and causing the child (and therefore parent) orchestration to hang.

Changes:

• Adds rewind-revival reconciliation that re-dispatches “orphaned” activity schedule actions whose TaskScheduled was consumed during replay but never receives a terminal event post-rewind.
• Makes duplicate TaskScheduled events for the same task ID idempotent during replay to tolerate Azure Storage’s append-only scheduling behavior.
• Adds deterministic executor-level regression tests modeling the Azure Storage backend rewind shape and duplicate TaskScheduled behavior.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 2 comments.

File	Description
packages/durabletask-js/src/worker/orchestration-executor.ts	Rewind-revival detection + orphaned activity re-dispatch reconciliation; idempotent handling for duplicate TaskScheduled.
packages/durabletask-js/src/worker/runtime-orchestration-context.ts	Adds executor bookkeeping to remember consumed activity actions and handled taskScheduled IDs.
packages/durabletask-js/test/rewind-multi-sibling-deadlock.spec.ts	New executor-level regression suite reproducing and guarding the multi-failed-sibling rewind deadlock scenario.