[one-shot implementation for discussion, not for merging]: feat(midnight): add first-class Midnight proof support (native + EVM) via protocol/model extensions and SDK adapter

[jtcoolen]

[Only the last 10 commits apply, the first 21 commits come from #1]

Context / Motivation

Midnight proofs are not wire-compatible with vanilla Halo2 in several places:

1. Committed instance prefix: some leading instance columns are commitments (EC points), not scalar vectors.
2. Instance-length hashing: transcript absorption can hash instance-column lengths before instance values.
3. Trailing challenges: there can be extra challenges not tied to any witness phase.
4. Extra commitments: there can be additional commitments after challenges are squeezed.
5. Quotient recombination base: quotient chunk recombination uses (z^n / z) (vs Halo2’s z^n), with a fixed chunk count derived from Midnight domain configuration.

You can understand this branch as:

Make the generic verifier model express Midnight semantics, then translate Midnight VK/CS into that model.

Once the protocol language can encode the 5 differences above, the bulk of verification logic (native + EVM codegen) stays generic.

---

High-Level Design

• Extend core PLONK protocol/proof structs to represent transcript/layout variants (the “protocol language”).

• Keep legacy behavior as defaults so existing Halo2 flows remain unchanged.

• Add a Midnight-specific KZG multiopen implementation (KzgAs<_, Midnight>).

• Add a feature-gated SDK bridge that:

  • converts Midnight artifacts into halo2-axiom types,
  • builds a Midnight-correct PlonkProtocol,
  • reuses the same generic verifier path for both native verification and EVM codegen/runtime.

---

Detailed Changes

1) Core protocol model extensions

File: snark-verifier/src/verifier/plonk/protocol.rs

Protocol metadata added

• committed_instance_count
• hash_instance_lengths
• trailing_challenges
• extra_commitments

These fields are the main abstraction pivot: Midnight is represented as protocol metadata rather than a forked verifier.

Quotient model extended

• QuotientChunkBase::{Zn, ZnMinusOne} where ZnMinusOne corresponds to (z^n / z)
• num_chunk_override: Option<usize>

Behavior:

• num_chunk() honors the explicit override when present.
• Metadata is preserved across conversions (loaded, loaded_preprocessed_as_witness).

Legacy behavior preserved

• Vanilla Halo2 compile path initializes neutral defaults so existing flows stay unchanged.

---

2) Proof parsing / verification updates

File: snark-verifier/src/verifier/plonk/proof.rs

Committed instance handling (absorption order)

• A committed-instance-aware flow now:

  1. absorbs committed-instance EC points first (if configured),
  2. then absorbs scalar instances for non-committed columns,
  3. optionally absorbs each non-committed column length first when hash_instance_lengths=true.

New parser entrypoint

• read_with_committed_instances(...) supports caller-provided committed-instance commitments.

Trailing challenges + extra commitments

• After phase-driven witness/challenge parsing, the parser appends:

  • trailing_challenges
  • extra_commitments

Commitment indexing alignment

• Committed-instance placeholders are appended to align commitment indexing with num_instance column layout.

Quotient recombination supports both bases

• Verifier recombines quotient chunks using either:

  • Halo2 base: z^n (Zn)
  • Midnight base: (z^n / z) (ZnMinusOne)

Tests added

• committed prefix ordering + scalar absorption order
• deterministic errors for layout mismatches
• committed-instance placeholder extension behavior

---

3) Midnight KZG PCS flavor (multiopen)

Files:

• snark-verifier/src/pcs/kzg.rs
• snark-verifier/src/pcs/kzg/multiopen.rs
• snark-verifier/src/pcs/kzg/multiopen/midnight.rs

Added:

• Midnight multiopen scheme and MidnightProof struct.

Transcript layout

• Parsed as: x1, x2, f_com, x3, q_evals_on_x3, x4, pi

Verification routine

• groups queries by point-set structure
• reconstructs interpolation terms
• performs batched inversion
• maps Midnight pairing equation into snark-verifier accumulator convention

Optional: truncated challenges

• truncated-challenges feature applies 128-bit truncation policy for challenge powers (native + EVM loaders).

---

4) EVM transcript/runtime correctness alignment

Files:

• snark-verifier/src/system/halo2/transcript/evm.rs
• snark-verifier/src/loader/evm/loader.rs
• snark-verifier/src/loader/evm/code.rs

Fixes/hardening:

• transcript absorption handles disjoint memory by copying into a contiguous transcript buffer
• identity point encoding/decoding aligned to EVM sentinel (0,0)
• constant EC point loader emits (0,0) for identity
• scalar truncation helper for truncated-challenge mode
• batched MSM precompile path for multi-scalar multiplications
• zero-safe deterministic batch inversion path
• generated Solidity pragma updated to 0.8.30

---

5) SDK Midnight adapter (translation layer)

Files:

• snark-verifier-sdk/src/midnight_adapter/*

Added:

• MidnightProofBundle for constructing verification/codegen inputs from Midnight params/VK/proof/instances
• strict normalization/validation of committed-instance shape
• Midnight→halo conversions for fields and curve points
• MidnightProtocolBuilder translating Midnight VK/CS into PlonkProtocol

MidnightProtocolBuilder guarantees:

• query/evaluation ordering matches Midnight transcript expectations

• committed-instance semantics reflected in protocol metadata

• inclusion of trash constraints + trash witness columns

• quotient config:

  • chunk_base = ZnMinusOne
  • num_chunk_override = vk quotient degree (fixed chunk count from Midnight domain config)

EVM parity design

• build_evm_verifier_loader() replays the same generic proof read + verify flow over EvmLoader,
  so emitted runtime code is derived from verifier execution (not a hand-written special-case path).

---

6) Feature plumbing

Files:

• snark-verifier-sdk/Cargo.toml
• snark-verifier-sdk/src/lib.rs
• snark-verifier/Cargo.toml

Added:

• SDK midnight feature gating Midnight deps + adapter module
• truncated-challenges passthrough from SDK to core
• example registration for Midnight EVM transcript flow

---

7) End-to-end example

Files:

• snark-verifier-sdk/examples/midnight_poseidon_evm.rs
• snark-verifier-sdk/examples/support/midnight_evm_transcript.rs

Example demonstrates:

• build Poseidon proof with Midnight EVM transcript hash
• verify natively
• generate Solidity verifier + bytecode + calldata
• optionally run local revm verification for gas/result sanity

---

Backward Compatibility

• Existing non-Midnight flows remain unchanged.
• New protocol fields are serde-defaulted and initialized to legacy values in standard Halo2 compile path.
• Midnight behavior is opt-in via feature flags and adapter usage.

---

Risks / Review Focus

• Query ordering and polynomial index offsets in MidnightProtocolBuilder.
• Correct absorption order for committed vs non-committed instances (including optional length hashing).
• Quotient chunk recombination base (ZnMinusOne) and fixed chunk count override.
• KZG Midnight multiopen transcript layout compatibility with prover output.
• EVM identity encoding consistency ((0,0) sentinel) across read/write/codegen.

---

Testing

• cargo check -p snark-verifier-sdk --features midnight,loader_evm
• cargo +1.88.0 check -p snark-verifier-sdk --features midnight,loader_evm,revm
• cargo +1.88.0 check -p snark-verifier-sdk --features midnight,loader_evm,revm,truncated-challenges
• cargo run --example midnight_poseidon --features midnight -p snark-verifier-sdk (if present in local tree)
• RUN_REVM=1 cargo +1.88.0 run --example midnight_poseidon_evm --features midnight,loader_evm,revm -p snark-verifier-sdk

proof bytes: 3968
deployment code bytes: 22817
calldata bytes: 4000
wrote .../target/midnight/MidnightPoseidonVerifier.sol
wrote .../target/midnight/midnight_poseidon.bytecode
wrote .../target/midnight/midnight_poseidon.calldata
revm gas: 597142

[jtcoolen]

Only the last 10 commits apply, the first 21 commits come from #1