Hayhooks V2: durable execution for pipelines and A2A agents

[mpangrazzi]

Scope: Haystack 3 only.

Stack

Merge in order: #253, then #258, then #259. PR #258 adds durable A2A trace correlation; PR #259 adds its dashboard classification and presentation.

Durable execution and managed A2A Agents

This PR adds durable execution for Haystack 3 Pipelines and Agents. It accepts typed work, persists it before returning, runs it outside the request, and recovers it from safe checkpoints after a worker or process disappears.

Highlights

• Crash recovery for native Haystack 3 Pipelines and Agents: Redis records plus explicit Pipeline or Agent checkpoints resume work after a worker or process disappears.
• Safe at-least-once delivery: fenced renewable leases keep one active owner while abandoned work is recovered automatically.
• Checkpoint-aware retries: bounded retries continue from the latest checkpoint instead of repeating completed upstream Pipeline work.
• Built-in interaction: typed inspection, progress, wait/resume, terminal results, idempotent submission, and cooperative cancellation.
• Revision-safe rollouts: an exact durable_revision gate prevents incompatible queued or waiting work from resuming against changed code.
• First-class long-running A2A Agents: durable task projection covers progress, input-required continuation, completion, failure, cancellation, and Redis-backed recovery.
• Focused operational footprint: purpose-built for moderate-scale durable Haystack workloads without introducing a separate general-purpose workflow platform.

See Hayhooks durable engine and Temporal for the capability and tradeoff comparison.

Portability

The durable engine can now be embedded through hayhooks.durable without depending on Hayhooks server startup. The public surface exports DurableRuntime, ExecutionStore, ExecutionStoreProvider, and the built-in memory and Redis providers while keeping Redis optional at import time.

Standalone runtimes own and start only their attached deployments; they do not inspect the process-global pipeline registry or require private server-loader method flags. Runtime, deployment, provider, and store use one settings snapshot, conflicting settings fail before deployment, and provider replacement is locked once a provider or deployment candidate exists. The runtime owns provider shutdown.

See Embedding the runtime for the public lifecycle and configuration example.

Diff breakdown for reviewers

The PR is broad because it ships the engine together with its recovery tests, runnable examples, operations documentation, and A2A integration. Of the 13,109 additions, 6,013 (about 46%) are tests, examples, or documentation.

Area	Files	Diff	What to focus on
Durable engine core	12	+4,225 / -0	State reducer, persistence contract, Redis atomicity/fencing, worker manager, and Haystack checkpoint adapters
A2A integration	12	+2,063 / -404	Durable task projection, owner isolation, recovery CAS, and protocol lifecycle; the deletions include the 399-line legacy a2a_utils.py replaced by focused A2A modules
REST/deploy/runtime integration	12	+750 / -64	Typed routes, deployment revision/lifecycle safety, settings, health, and tracing
Tests	24	+4,415 / -111	Reducer/store contracts, races, crash recovery, Redis integration, A2A recovery, and deployment lifecycle
Runnable examples	8	+898 / -4	End-to-end Pipeline, Agent, A2A, retry, approval, and process-recovery usage
Documentation	14	+700 / -108	Public API, operating boundaries, configuration, and the Temporal comparison
CI and packaging	5	+58 / -3	Haystack 2/3 matrix, Redis service, durable extra, and docs checks
Total	87	+13,109 / -694

Suggested review order:

1. src/hayhooks/durable/engine.py, backend.py, store.py, redis.py, and reference.py for lifecycle and storage correctness.
2. manager.py, context.py, adapters.py, runtime.py, and server/durable/routes.py for execution and Haystack boundaries.
3. server/a2a/durable_executor.py and redis_task_store.py for the A2A-specific projection and recovery layer.
4. Tests for the corresponding contract and failure cases; examples and docs are supporting material.

Supported features

Durable REST execution

• An ordinary BasePipelineWrapper implements run_durable() or run_durable_async(). Hayhooks invokes that method for each durable execution with a DurableContext and typed Pydantic request.
• The wrapper calls context.run_pipeline() / context.run_pipeline_async() for checkpointed Pipeline work, or context.run_agent() / context.run_agent_async() for Agent work.
• Typed wrappers expose:
  • POST /{pipeline}/run-durable
  • GET /{pipeline}/executions/{id}
  • POST /{pipeline}/executions/{id}/cancel
  • POST /{pipeline}/executions/{id}/resume
• Submission validates and persists input before returning. Idempotency-Key replays the same operation safely and rejects reuse with different input.
• Executions provide bounded public progress, wait information, terminal results, and safe error details.
• The engine supports bounded retries, cooperative cancellation, typed wait/resume, terminal retention, and optional owner-isolated access through a trusted upstream owner header.
• Durable wrappers declare durable_revision, allowing queued and waiting work to be checked against checkpoint-relevant deployment code.

Pipeline checkpoints

Pipeline wrappers opt in through context.run_pipeline(..., checkpoint_at=[...]) or context.run_pipeline_async(..., checkpoint_at=[...]).

For every named boundary, Hayhooks asks Haystack to stop at a public Breakpoint and immediately persists the returned PipelineSnapshot before that component executes. If Haystack exposes a snapshot with a PipelineRuntimeError, Hayhooks persists that snapshot too.

On recovery, Hayhooks rebuilds the PipelineSnapshot, passes it back to the Pipeline, and starts from the saved scheduler state. Haystack skips the already-completed upstream component visits; work after the last checkpoint is replayed.

Agent checkpoints

Managed Agents use the public Haystack hook surface. Hayhooks installs synchronous and asynchronous versions of these hooks for the active durable execution:

Hook	Durable behavior
before_run	Restores the persisted serializable State while keeping fresh per-run tools and hook context.
before_llm	Checks for cooperative cancellation before every model call.
after_tool	Saves an Agent state checkpoint after a tool-result batch when the Agent will continue to another model step, then checks cancellation.
on_exit	Saves application-adjusted state when an application exit hook requests continue_run.
after_run	Saves a final checkpoint. A recovered execution returns this saved Agent result without another model call.

The serialized Agent checkpoint excludes live tools and hook_context; the current deployment recreates them for the recovered run. Checkpoints and progress are saved together as a durable execution transition.

Redis recovery

• Redis is the default durable store; the in-memory store provides the same contract for local development and tests.
• Each deployment keeps its own controls, opaque payloads, and two indexes: runnable for queued work and lease-expiry for active claims.
• Workers claim due work with monotonic fences and renewable leases. Redis TIME, optimistic transactions, and fenced transitions keep ownership safe across replicas.
• Lease maintenance requeues abandoned work. Redis TTL retains terminal records and idempotency bindings for the configured window.
• Health reports durable nonterminal, runnable, and lease_expiry counts.

Managed long-running A2A Agents

• A Haystack 3 Agent in A2APipelineWrapper is exposed as a durable A2A Agent. Hayhooks supplies the durable worker, queue, execution record, checkpointing, progress projection, and Redis integration.
• The durable execution record supplies A2A progress, input-required, completion, failure, and cancellation state. A follow-up A2A message resumes work while the execution is waiting for input.
• Redis-backed A2A task storage persists task-to-execution bindings and task snapshots. Startup repairs active snapshots under a per-task lease and compare-and-set version; GetTask and list requests project the latest durable state directly.
• A2A task retention and durable execution retention are independently configurable.

Execution model

The lifecycle is a pure reducer over a compact execution control record:

queued -> running -> completed | failed | canceled
              |\
              | -> queued  (retry or expired lease)
              | -> waiting -> queued  (resume)

The store atomically persists each reducer plan and its derived Redis indexes. The durable manager owns worker polling, lease heartbeats, retries, and shutdown draining; Haystack adapters own the Pipeline snapshot and Agent hook integration.

See the durable operations guide.

Validation

• Focused Haystack 3 durable/A2A suite after portability changes: 113 passed.
• Strict documentation build: passed.
• All PR checks pass across Python 3.10–3.14 and the Haystack 2/3 test matrix.

[socket-security]

Review the following changes in direct dependencies. Learn more about Socket for GitHub.

Diff	Package	Supply Chain Security	Vulnerability	Quality	Maintenance	License
	pypi/​redis@​8.1.0

View full report

[socket-security]

All alerts resolved. Learn more about Socket for GitHub.

This PR previously contained dependency changes with security issues that have been resolved, removed, or ignored.

View full report