Skip to content

mock_chain_validation: swallowed validation failures are metric-only — log the height and hash pair #3752

Description

@bdchatham

Problem

When a mock_chain_validation (or mock_block_validation) build swallows a consensus-validation failure, ConsensusPolicy.HandleError calls recordUnsafeValidationSkipped(err) and returns nil. That helper only increments sei_unsafe_validation_skipped_total{validation_error=...} — the error object itself is discarded (sei-tendermint/types/policy_metrics.go, consensus_policy_mock_chain_validation.go).

The discarded error is the fully-wrapped one carrying the diagnostics an investigation actually needs: the height context and the expected %X, got %X hash pair from validateBlock. Because the swallow happens inside validateBlock, the caller's detailed "LastResultsHash mismatch detected" log at internal/state/execution.go:170 is also never reached — the caller sees nil. Net: a replayer that runs through N diverging blocks records only an unlabeled running count, with no record of which heights diverged or with what hashes.

Impact

The point of the mock_chain_validation build is to run through consensus divergences so they can be studied (its own doc comment defers correctness to an external comparator). Discarding the per-occurrence evidence at the moment of swallowing undermines that: an investigation that doesn't already know the divergence height from an external source (e.g. a production crash log) has to reconstruct the timeline by querying block_results height-by-height after the fact.

Concrete case: the arctic-1 v2/giga LastResultsHash divergence (CON-368). Reproduction with a mock_chain_validation replayer works because production nodes handed us the exact height by panicking — a divergence first observed on a mock build would not come with that gift.

Proposed approach

In the mock policies' HandleError, log the swallowed error at Warn with its wrapped context before returning nil, e.g.

if errors.Is(err, e) {
    recordUnsafeValidationSkipped(err)
    log.Warn("swallowed halting validation failure", "err", err)
    return nil
}

The wrapped error already carries the hash pair; if height isn't in the wrap chain at this call site, plumbing it in (or logging at the validateBlock call sites where height is in scope) would complete the picture. Behavior of the default (production) policy is unchanged — this only touches the mock-build files.

Out of scope

  • Any change to production halting semantics or the default policy.
  • Changing the metric (the counter stays; the log complements it).

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions