[one-shot implementation for discussion, not for merging]: feat(midnight-evm): add dispatcher-based partitioned verifier flow for Poseidon and IVC proofs

[jtcoolen]

[Only the last 9 commits apply, the first commits come from #1 and #2]

Summary

This PR introduces and hardens a dispatcher-based partitioned (referred to as unrolled-sharded below) verifier path for Midnight EVM examples, so we can run benchmarks even when a single unrolled verifier contract would exceed EVM deployment/runtime size limits.

• Adds unrolled-sharded verifier artifact generation (dispatcher + shard contracts + manifest + sizing metadata).
• Wires unrolled-sharded deployment/call benchmarking through REVM (deploy shards → deploy dispatcher → verify via dispatcher call).
• Updates Poseidon and IVC example flows to emit and benchmark sharded artifacts.
• Fixes truncated-challenges compatibility by restoring truncate_scalar_to_128 in the EVM loader.
• For truncated-challenges runs, explicitly skips compact/hybrid generation (unsupported path).
• For oversized monolithic unrolled IVC verifier code, skips unrolled REVM benchmark with a clear EIP-170/EIP-3860 reason instead of failing with a misleading revert message.
• Restores EvmLoader::code_mut() accessor required by transcript/decider code paths after history cleanup.

Design / implementation details

What changed (core)

• New sharded artifact types were added so codegen can return dispatcher + shards + sizing metadata:

  • snark-verifier/src/loader/evm/code.rs:8
  • snark-verifier/src/loader/evm/code.rs:31

• SolidityAssemblyCode started tracking emitted runtime blocks (runtime_blocks) in addition to the flat runtime string, enabling splitting:

  • snark-verifier/src/loader/evm/code.rs:54
  • snark-verifier/src/loader/evm/code.rs:89

Sharding algorithm

• Implemented in EvmLoader::unrolled_sharded_verifier_artifacts():

  • snark-verifier/src/loader/evm/loader.rs:393

• Enforces EVM limits (24,576 runtime via EIP-170, 49,152 initcode via EIP-3860) and starts from coarse 64-statement groups:

  • snark-verifier/src/loader/evm/loader.rs:34
  • snark-verifier/src/loader/evm/loader.rs:36

Flow

1. Compile a candidate statement group.
2. If too large or uncompilable, bisect recursively until each group is individually deployable.
3. Binary-search the largest contiguous window that still fits to maximize shard size.
4. Compile final shard contracts + dispatcher and produce a manifest.

Key functions

• candidate compile: snark-verifier/src/loader/evm/loader.rs:267
• recursive split: snark-verifier/src/loader/evm/loader.rs:283
• window binary search: snark-verifier/src/loader/evm/loader.rs:324
• final shard build: snark-verifier/src/loader/evm/loader.rs:357

Contract architecture

• Each shard is generated as its own Solidity fallback contract that executes only its statement range and returns a 32-byte success flag:

  • snark-verifier/src/loader/evm/loader.rs:107

• A dispatcher contract stores shard addresses and loops through them with delegatecall, bubbling revert data and checking returned success:

  • snark-verifier/src/loader/evm/loader.rs:136

Deployment / execution path (REVM)

• New REVM helper deploys all shard contracts, ABI-encodes address[] constructor args for dispatcher, deploys dispatcher, then calls it with verifier calldata:

  • snark-verifier/src/loader/evm/util/executor.rs:123
  • constructor arg encoding: snark-verifier/src/loader/evm/util/executor.rs:227

• compile_solidity_runtime was added so runtime size can be checked during partitioning:

  • snark-verifier/src/loader/evm/util.rs:112

Public API exposure

• Re-exports added for sharded artifacts and deploy helper:

  • snark-verifier/src/loader/evm.rs:7
  • snark-verifier/src/loader/evm.rs:14

• SDK added:

  • artifact generation API: snark-verifier-sdk/src/midnight_adapter/evm.rs:35
  • revm sharded verification API: snark-verifier-sdk/src/midnight_adapter/evm.rs:66

Validation / adoption via examples

• Poseidon example switched to writing dispatcher+shard artifacts and using sharded REVM verification:

  • snark-verifier-sdk/examples/midnight_poseidon_evm.rs:30
  • snark-verifier-sdk/examples/midnight_poseidon_evm.rs:235

• New IVC example was added to stress large verifier code and compare unrolled vs sharded deployability against EIP limits:

  • snark-verifier-sdk/examples/midnight_ivc_evm.rs:403
  • snark-verifier-sdk/examples/midnight_ivc_evm.rs:558

Why

Monolithic unrolled IVC verifier bytecode exceeds EVM deploy limits, so on-chain-compatible benchmarking must use sharding.
Unrolled-sharded keeps dispatcher/shards deployable while preserving verifier semantics and giving realistic gas measurements.

Testing

1) Poseidon

cd snark-verifier-clean

RUN_REVM=1 \
MIDNIGHT_EVM_MODE=unrolled-sharded \
MIDNIGHT_ARTIFACT_DIR=/tmp/midnight_poseidon_unrolled_sharded \
cargo +1.88.0 run --release \
  --example midnight_poseidon_evm \
  --features midnight,loader_evm,revm \
  -p snark-verifier-sdk

proof bytes: 3968
verifier mode: unrolled-sharded
calldata bytes: 4000
wrote /tmp/midnight_poseidon_unrolled_sharded/midnight_poseidon.calldata
unrolled-sharded dispatcher runtime bytes: 196
unrolled-sharded dispatcher initcode bytes: 961
unrolled-sharded shard runtime sizes (bytes): [19570]
unrolled-sharded shard initcode sizes (bytes): [19598]
wrote /tmp/midnight_poseidon_unrolled_sharded/MidnightPoseidonVerifierUnrolledShardedDispatcher.sol
wrote /tmp/midnight_poseidon_unrolled_sharded/midnight_poseidon_unrolled_sharded_dispatcher.bytecode
wrote /tmp/midnight_poseidon_unrolled_sharded/midnight_poseidon_unrolled_sharded_shards.bytecode
wrote /tmp/midnight_poseidon_unrolled_sharded/midnight_poseidon_unrolled_sharded_manifest.txt
revm gas (unrolled-sharded): 637514

2) IVC

Note that the verifier contract does not check the final accumulator yet.

cd snark-verifier-clean

RUN_REVM=1 \
MIDNIGHT_EVM_OUT_DIR=/tmp/midnight_ivc_unrolled_sharded \
cargo +1.88.0 run --release \
  --example midnight_ivc_evm \
  --features midnight,loader_evm,revm,truncated-challenges \
  -p snark-verifier-sdk

computed vk and pk in 17.486349666s
collapsed accumulator (step 0): 77 field elements (2464 bytes)
generated IVC proof step 0 (Poseidon) in 36.890915625s
collapsed accumulator (step 1): 77 field elements (2464 bytes)
generated IVC proof step 1 (EVM) in 35.189767959s
proof bytes: 8544
unrolled deployment code bytes: 102348
unrolled runtime code bytes: 102318
unrolled runtime deployable (EIP-170 <= 24576): false
unrolled initcode deployable (EIP-3860 <= 49152): false
unrolled-sharded dispatcher runtime bytes: 196
unrolled-sharded dispatcher initcode bytes: 961
unrolled-sharded shard runtime sizes (bytes): [23646, 24443, 23623, 23769, 7357]
unrolled-sharded shard initcode sizes (bytes): [23674, 24471, 23651, 23797, 7385]
unrolled-sharded dispatcher runtime deployable (EIP-170 <= 24576): true
unrolled-sharded dispatcher initcode deployable (EIP-3860 <= 49152): true
unrolled-sharded shard runtime deployable (all): true
unrolled-sharded shard initcode deployable (all): true
calldata bytes: 11072
wrote /tmp/midnight_ivc_unrolled_sharded/MidnightIvcVerifier.sol
wrote /tmp/midnight_ivc_unrolled_sharded/midnight_ivc.bytecode
wrote /tmp/midnight_ivc_unrolled_sharded/midnight_ivc.calldata
wrote /tmp/midnight_ivc_unrolled_sharded/MidnightIvcVerifierUnrolledShardedDispatcher.sol
wrote /tmp/midnight_ivc_unrolled_sharded/midnight_ivc_unrolled_sharded_dispatcher.bytecode
wrote /tmp/midnight_ivc_unrolled_sharded/midnight_ivc_unrolled_sharded_shards.bytecode
wrote /tmp/midnight_ivc_unrolled_sharded/midnight_ivc_unrolled_sharded_manifest.txt
=== REVM deployment + verification gas benchmarks ===
skipping unrolled revm benchmark: verifier exceeds EIP deploy limits (runtime=102318 > 24576, initcode=102348 > 49152)
revm gas (unrolled-sharded): 1359482