DF-23489 multinode: tighten polling criteria for liveness

[vlfig]

Description

Having ship-shape RPCs is crucial for keeping the odds of missing a transmit as low as possible, which is itself crucial for SVR. It is our suspicion (and telemetry concurs) that our nodes are too lenient on RPCs with respect to their polling failures and that there are gains to be had in booting those out of the alive pool. In particular, our nodes do not detect grey-failures — when RPCs fail polling intermittently under a certain rate.

This changes the behaviour in both directions between the unreachable and alive states:

1. In the alive loop, instead of resetting the failure counter, successful polls now only decrement it. This way, we keep the same behavior for error rates above 5/6=0.8333, where an RPC gets booted after 5 probes (on average), but we now eventually declare unreachable nodes that sustain error rates between that and 1:1, which we'd previously tolerate;
2. In the unreachable loop, we add a polling phase to the journey back into Alive, mirroring the other direction and ensuring a node doesn't make it back into alive in the same condition that got it kicked out in the first place. However, instead of following the same "decaying" counter we opted for a stricter flow where a single poll failure resets the node back to square one (dialing). This new phase/step is governed by a new config property PollSuccessThreshold that, defaulting to 0 matches current behavior and allows for progressive, explicit rollout.

Implements DF-23489.

Requires Dependencies

None.

Resolves Dependencies

Waiting for tag:

• DF-23489 Bump chainlink-framework for multinode & metrics chainlink-evm#432
• DF-23489 WIP bump multinode, retain behaviour chainlink-solana#1533

[dimriou]

Is there a config check that prevents the pollInterval from being 0? If not, in the case where PollSuccessThreshold != 0 && PollInterval == 0 this will enter an infinite failure loop.

[vlfig]

There isn't indeed. Added it to the guard condition.

[dimriou]

I think the pollFailures change is too focused on throttled RPCs. I'm wondering if we're slowing down an RPC that might have a few transient errors during polling by decreasing the counter. I guess one explanation is that it's better to be slower at detecting healthy RPCs than producing false positives. Let's see how this change performs. Perhaps a middle ground solution could be to decrease the counter by 2 instead of one, in order to be more optimistic.

[vlfig]

@dhaidashenko @DylanTinianov What do you think?