fix(rpc): stop multicall cache from replaying transient errors forever - #82
Merged
Merged
Conversation
ethers-multicall-provider's internal DataLoader (cache enabled by default) caches per-call Error results and only clears entries when a load fulfills. A single transient RPC failure (e.g. one HTTP 500) on a multicall batch is therefore replayed from the cache for every future identical call (same to+data+blockTag): retries and later monitoring cycles never reach the RPC again and the service stays down until a process restart. Wrap with cache=false: per-tick call batching is unaffected, and since latest-block entries were cleared after every successful load anyway, no effective caching is lost. withRetry becomes able to actually recover from a transient multicall failure.
…g batch Otherwise the intentionally failed first batch also takes down the lazy eth_chainId network detection, which logs noise and leaves a 1s retry timer that can trigger jest's force-exit warning.
Contributor
Author
|
2 review passes until zero findings. Pass 1 flagged that the spec's intentionally failed first batch also dragged down the provider's lazy eth_chainId network detection (log noise plus a lingering 1s retry timer that could trigger jest's force-exit warning); fixed by completing the network bootstrap against the healthy fake before arming the failure. Pass 2 was clean. Verified along the way: the regression spec fails on the previous code with the exact incident symptom and passes with the fix, and the full suite runs green repeatedly with no warnings. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Incident
Since today ~09:05 UTC both monitoring instances (DEV and PRD) failed every 5-minute cycle with
server response 500 Internal Server Error(RPC call failed after 4 attempts), counters climbing past 25 consecutive failures with no recovery. Position/challenge/collateral/minter/dEURO state sync was stale the whole time; event collection and the block cursor kept advancing.Root cause
ethers-multicall-provider(6.5.0, latest) batches contract calls through a DataLoader created withcache: true. Two behaviors combine into a permanent-failure trap:eth_callfails, the wrapper assigns the same Error instance to every call of the batch — and DataLoader explicitly caches Error results for individual keys..then(onFulfilled)— i.e. only when the load succeeds. On rejection the poisoned entry stays forever.So one transient HTTP 500 from the RPC provider poisons the cache keys (
to+data+blockTag) of the whole batch. Every later identical call — including allwithRetryattempts and every subsequent monitoring cycle — is served the original cached error without any RPC request being sent. Only a process restart recovers. This also explains whyisConnectionErrordeliberately skipping provider recycling for 5xx ("the retry reaches the server anyway") did not help: the retry never reached the server.Verified by replaying the production call set (98 positions, identical chunking/ABI/versions) against the same RPC endpoint — from a fresh process it succeeds every time, including from inside the failing container's own network; and a fail-once fake provider reproduces the permanent failure with
cache: trueand recovery withcache: false.Fix
Pass
cache = falsetoMulticallWrapper.wrap. Per-tick batching (the reason the wrapper is used) is unaffected; cross-cycle caching never existed on the success path anyway, because latest-block entries were cleared after each successful load. With the cache gone,withRetrygenuinely re-sends the RPC and recovers from transient errors.jestmoduleNameMapper forsrc/...imports added so the new spec (which importsProviderService) resolves.Deploying
Rolling this out restarts the currently stuck instances, which also clears their poisoned in-memory caches.