Skip to content

fix(queue): make Redis delivery crash-recoverable#89

Closed
abnegate wants to merge 5 commits into
mainfrom
fix/dat-1993-durable-redis-delivery
Closed

fix(queue): make Redis delivery crash-recoverable#89
abnegate wants to merge 5 commits into
mainfrom
fix/dat-1993-durable-redis-delivery

Conversation

@abnegate

Copy link
Copy Markdown
Member

What changed

  • add an idempotent publisher capability with caller-stable message IDs, typed Enqueued / Existing acknowledgements, and a typed conflict for mismatched reuse
  • claim pending Redis messages and register processing state atomically with Lua
  • add opt-in visibility leases, explicit renewal, timeout reclaim, and receipt-fenced acknowledgements
  • keep Redis Cluster keys in the pending queue's hash slot and preserve ordinary enqueue, priority, failed retry, reconnect polling, and legacy in-flight acknowledgement paths
  • replace the single-architecture Redis Cluster test image with an official multi-architecture Redis fixture

Why

The previous Redis receive path removed an envelope from the pending list before storing its job and processing records. A worker or connection failure in that gap could strand the message. Producers also could not safely retry a publish after losing its response because the broker generated a new ID for every enqueue.

enqueueOnce() now lets a producer retry the same canonical envelope safely. Atomic claims remove the pending-to-processing loss window. Visibility reclaim remains disabled by default for compatibility and is enabled only when the caller chooses a processing deadline; long-running workers can renew that lease explicitly.

Validation

  • bin/monorepo validate
  • XDEBUG_MODE=off bin/monorepo check queue
  • XDEBUG_MODE=off bin/monorepo test queue — 35 unit tests / 98 assertions; 33 e2e tests / 169 assertions
  • real standalone Redis and three-node Redis Cluster coverage for producer retry, conflict, priority, response loss, acknowledgement loss, claim reclaim, lease renewal, reconnect, retry, concurrency, and cluster hash-slot compatibility
  • Vale 3.15.1 on packages/queue/README.md

Linear: DAT-1993

@abnegate

Copy link
Copy Markdown
Member Author

@greptile review

@greptile-apps

greptile-apps Bot commented Jul 24, 2026

Copy link
Copy Markdown
Contributor

Greptile Summary

Makes Redis queue delivery crash-recoverable.

  • Adds idempotent publication with stable message IDs and conflict detection.
  • Moves message claims, acknowledgements, rejection, renewal, and retry operations into atomic Lua scripts.
  • Adds visibility leases, receipt fencing, timeout reclaim, and Redis Cluster-compatible key derivation.
  • Extends pooled and locking connections with the new publishing, leasing, and Lua capabilities.
  • Adds standalone and clustered Redis durability coverage and updates the test fixture.

Confidence Score: 5/5

The PR appears safe to merge because no additional blocking failure eligible for this follow-up review was identified.

No blocking failure remains within the scope of the previous review threads.

Important Files Changed

Filename Overview
packages/queue/src/Queue/Broker/Redis.php Implements atomic Redis queue claims, receipt-fenced acknowledgements, visibility leases, idempotent publication, and retry compatibility.
packages/queue/src/Queue/Broker/Redis/Keys.php Derives queue-specific Redis keys that remain colocated within a Redis Cluster hash slot.
packages/queue/src/Queue/Connection/Redis.php Adds Lua-script execution support to the standalone Redis connection.
packages/queue/src/Queue/Connection/RedisCluster.php Adds cluster-aware Lua-script execution support.
packages/queue/src/Queue/Connection/Locking.php Serializes Lua evaluation when Redis connections are shared between coroutines.
packages/queue/src/Queue/Message.php Adds delivery receipts to message serialization for acknowledgement fencing.
packages/queue/docker-compose.yml Replaces the previous Redis Cluster image with a three-node official Redis test fixture.
packages/queue/tests/Queue/E2E/Adapter/RedisDurabilityTest.php Adds standalone Redis coverage for publication and delivery durability behavior.
packages/queue/tests/Queue/E2E/Adapter/RedisClusterDurabilityTest.php Adds Redis Cluster durability and hash-slot compatibility coverage.

Reviews (4): Last reviewed commit: "fix(queue): make legacy retries idempote..." | Re-trigger Greptile

Comment thread packages/queue/src/Queue/Broker/Redis.php Outdated
Comment thread packages/queue/src/Queue/Broker/Redis/Keys.php
@abnegate

Copy link
Copy Markdown
Member Author

@greptile review

@abnegate

Copy link
Copy Markdown
Member Author

@greptile review

Comment thread packages/queue/src/Queue/Broker/Redis.php Outdated
@abnegate

Copy link
Copy Markdown
Member Author

@greptile review

Copy link
Copy Markdown
Member Author

Closing as campaign scope cleanup. Crash-recoverable queue delivery primitives are broad shared infrastructure, not a narrowly evidenced dedicated-database fix. Nothing from this draft was merged or released. A future Queue proposal needs its own independently approved scope and consumers.

@abnegate abnegate closed this Jul 24, 2026
@abnegate
abnegate deleted the fix/dat-1993-durable-redis-delivery branch July 24, 2026 11:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant