Skip to content

fix(rpc): stop multicall cache from replaying transient errors forever - #82

Merged
Danswar merged 2 commits into
developfrom
fix/multicall-transient-error-poisons-cache
Jul 21, 2026
Merged

fix(rpc): stop multicall cache from replaying transient errors forever#82
Danswar merged 2 commits into
developfrom
fix/multicall-transient-error-poisons-cache

Conversation

@Danswar

@Danswar Danswar commented Jul 21, 2026

Copy link
Copy Markdown
Contributor

Incident

Since today ~09:05 UTC both monitoring instances (DEV and PRD) failed every 5-minute cycle with server response 500 Internal Server Error (RPC call failed after 4 attempts), counters climbing past 25 consecutive failures with no recovery. Position/challenge/collateral/minter/dEURO state sync was stale the whole time; event collection and the block cursor kept advancing.

Root cause

ethers-multicall-provider (6.5.0, latest) batches contract calls through a DataLoader created with cache: true. Two behaviors combine into a permanent-failure trap:

  1. When an aggregate eth_call fails, the wrapper assigns the same Error instance to every call of the batch — and DataLoader explicitly caches Error results for individual keys.
  2. The wrapper clears cache entries for latest-block calls only in a .then(onFulfilled) — i.e. only when the load succeeds. On rejection the poisoned entry stays forever.

So one transient HTTP 500 from the RPC provider poisons the cache keys (to+data+blockTag) of the whole batch. Every later identical call — including all withRetry attempts and every subsequent monitoring cycle — is served the original cached error without any RPC request being sent. Only a process restart recovers. This also explains why isConnectionError deliberately skipping provider recycling for 5xx ("the retry reaches the server anyway") did not help: the retry never reached the server.

Verified by replaying the production call set (98 positions, identical chunking/ABI/versions) against the same RPC endpoint — from a fresh process it succeeds every time, including from inside the failing container's own network; and a fail-once fake provider reproduces the permanent failure with cache: true and recovery with cache: false.

Fix

Pass cache = false to MulticallWrapper.wrap. Per-tick batching (the reason the wrapper is used) is unaffected; cross-cycle caching never existed on the success path anyway, because latest-block entries were cleared after each successful load. With the cache gone, withRetry genuinely re-sends the RPC and recovers from transient errors.

  • jest moduleNameMapper for src/... imports added so the new spec (which imports ProviderService) resolves.
  • Regression spec fails on the previous code (both tests) and passes with the fix.

Deploying

Rolling this out restarts the currently stuck instances, which also clears their poisoned in-memory caches.

Danswar added 2 commits July 21, 2026 10:18
ethers-multicall-provider's internal DataLoader (cache enabled by default)
caches per-call Error results and only clears entries when a load fulfills.
A single transient RPC failure (e.g. one HTTP 500) on a multicall batch is
therefore replayed from the cache for every future identical call
(same to+data+blockTag): retries and later monitoring cycles never reach
the RPC again and the service stays down until a process restart.

Wrap with cache=false: per-tick call batching is unaffected, and since
latest-block entries were cleared after every successful load anyway, no
effective caching is lost. withRetry becomes able to actually recover from
a transient multicall failure.
…g batch

Otherwise the intentionally failed first batch also takes down the lazy
eth_chainId network detection, which logs noise and leaves a 1s retry
timer that can trigger jest's force-exit warning.
@Danswar

Danswar commented Jul 21, 2026

Copy link
Copy Markdown
Contributor Author

2 review passes until zero findings. Pass 1 flagged that the spec's intentionally failed first batch also dragged down the provider's lazy eth_chainId network detection (log noise plus a lingering 1s retry timer that could trigger jest's force-exit warning); fixed by completing the network bootstrap against the healthy fake before arming the failure. Pass 2 was clean. Verified along the way: the regression spec fails on the previous code with the exact incident symptom and passes with the fix, and the full suite runs green repeatedly with no warnings.

@Danswar
Danswar marked this pull request as ready for review July 21, 2026 13:36
@Danswar
Danswar merged commit f30e58f into develop Jul 21, 2026
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant