Upgrade evmone and use full transaction execution - #16916
Draft
chfast wants to merge 5 commits into
Draft
Conversation
Contributor
|
The key interesting piece, beyond easier upgrades, is that precompiles come fully implemented by evmone, and no need to manually add precompile expection cases in EVMHost. |
Refresh the vendored EVMC headers from evmone 0.23.0 (ABI 13 -> 18) and adapt the test host to the new interface: - The VM now computes the CREATE/CREATE2 address and passes it in msg.recipient, so EVMHost no longer derives it and evmc_result::create_address is gone. The RLP/keccak derivation moves to EVMHost::computeCreateAddress, which callers use for top-level creations where no VM frame exists to compute it. - The Host exposes get_nonce(), which the VM reads live to derive that address. Created accounts therefore store nonce 1 (EIP-161) and test accounts start at nonce 1, which keeps every derived address byte-identical to the previous behaviour. destination.nonce is set before execution, not after: evmone reads get_nonce(sender) during a nested CREATE, so setting it later would give a contract creating from inside its own constructor a stale nonce of 0. - VM capability queries no longer exist in EVMC; drop the EVM1 checks. - EVMC_CONSTANTINOPLE was removed. Map constantinople to Petersburg, its closest surviving revision. Note this is an approximation: Petersburg uses the legacy SSTORE schedule where Constantinople used EIP-1283 net metering, and no fixture asserts gas at that version, so the difference is untested. solc's own code generation is unaffected, being driven by langutil::EVMVersion rather than evmc_revision. One behaviour change, deliberate: m_contractAddress is now set for every creation attempt, where previously evmc_result::create_address was left as the zero address on the early failure paths (tx-gas underflow, insufficient balance, CREATE2 collision) and on code-deposit out-of-gas. The address is computed before the call and is observable regardless of outcome. Also fixes an RLP length-prefix bug carried over from the old derivation: a two-byte nonce needs the 0x82 short-string prefix, not 0xb9. Unreachable in practice, but the helper is now shared. EVMHostTest covers the encoder against evmone's own compute_create_address vectors, including nonces above 255. (cherry picked from commit 4c6783c)
Update every pinned reference to evmone so the tree matches the headers the test host now compiles against: - .circleci/osx_install_dependencies.sh, with a new sha256 for the 0.23.0 darwin-arm64 asset. - scripts/install_evmone.ps1 and the download links in test/Common.h. - The four buildpack-deps Dockerfiles. Two download the release tarball (new sha256 for the linux-x86_64 asset); two build from the git tag. These are not separable from the source bump the way the digest re-pin is: leaving them at 0.22.0 would have the next image rebuild ship an ABI-13 library against ABI-18 headers. - Their LABEL version values, incremented as scripts/docker/buildpack-deps requires. Without this, scripts/ci/docker_upgrade.sh hard-errors on a Dockerfile diff carrying no version bump and no image is built at all. test/evmc/README.md now records that the headers come from evmone 0.23.0, and that loader.c/loader.h can no longer be refreshed from upstream: as of 0.23.0 evmone's evmc/lib builds only the evmc, evmc_cpp and mocked_host targets, so the EVMC loader is gone. The copies kept here are ABI-agnostic -- they read only vm->abi_version -- so they need no per-version maintenance. Also drops a long-stale instruction to delete tooling.hpp and instructions.h, which have not shipped in evmc/include/evmc for several versions. REQUIRED FOLLOW-UP, not doable from a source checkout: the buildpack-deps images must be rebuilt and republished, and their digests re-pinned in .circleci/config.yml (lines 17, 21, 25 and 29). Linux CI cannot pass until then, because those images still carry evmone 0.22.0. (cherry picked from commit 8b3e8a9)
Fetch evmone 0.23.0 and intx 0.15.0 from hash-pinned release archives and take EVMC from evmone's own tree, so the version is pinned by the build system rather than by documentation. cmake/EvmoneDependency.cmake owns this. evmone resolves intx through Hunter, which cannot run from a CMake subproject: HunterGate() injects CMAKE_TOOLCHAIN_FILE, which CMake reads only at the first top-level project() call, and it actively refuses to run once a project is already named. Disabling Hunter makes hunter_add_package() a no-op instead of an error, and EVMONE_INTX_DIR -- evmone's own escape hatch -- supplies intx via add_subdirectory(). test/evmc/ keeps only loader.c and loader.h, because evmone 0.23.0 ships no EVMC loader and EVMHost::getVM needs evmc_load_and_configure for the user-facing --vm flag and ETH_EVMONE. Every other header is deleted; exactly one <test/evmc/...> include survives. With evmone linked in, an empty VM path now selects the built-in evmone, so --vm and ETH_EVMONE remain available for running against any other EVMC VM. Solidity's local tweak to the vendored mocked_host.hpp -- MockedAccount::storage as std::map, for deterministic EVMHostPrinter output (PR argotorg#11094) -- cannot survive taking that header from upstream. EVMHostPrinter now sorts storage at the point of output instead, so upstream's unordered_map is safe to consume, and EVMHostTest covers the ordering directly. Building evmone in-tree rather than downloading it needs three accommodations: - Solidity's PEDANTIC block adds -Wall -Wextra -Werror -pedantic at top level and those propagate into evmone's sources, while evmone's own compensating options never run: they sit inside cable_configure_compiler(), which returns early when PROJECT_IS_TOP_LEVEL is false. Warnings are disabled for the fetched targets, and their interface include directories are marked SYSTEM so consumers get -isystem. The latter matters because Solidity compiles test/evmc/loader.c itself, and that C translation unit includes evmone's <evmc/evmc.h>, where EVMC 18 declares `enum evmc_access_status : bool` -- C23 syntax, which is a hard error under the C17 default most toolchains still use. Solidity's own targets keep -Werror. - BUILD_SHARED_LIBS and HUNTER_ENABLED are generic, third-party-owned names, so forcing them into the cache overwrote the user's setting permanently: a configure with -DBUILD_SHARED_LIBS=ON came out of the cache as OFF, and any later reconfigure that did not re-pass it silently turned every Solidity library static. They are shadowed with plain set() inside block(SCOPE_FOR VARIABLES), which evmone still reads but which writes nothing to the cache. The evmone-specific EVMONE_* names stay forced, having no other consumer. - The CMake features this needs -- CMAKE_FIND_PACKAGE_REDIRECTS_DIR and OVERRIDE_FIND_PACKAGE (3.24), block(SCOPE_FOR VARIABLES) (3.25) -- raise the floor, but only for building the tests. The requirement therefore lives in test/CMakeLists.txt rather than at the top level, so `cmake -DTESTS=0` still builds with the project's own 3.13 floor. That matters for the Emscripten build, whose image ships CMake 3.16 and which never configures the test directory. scripts/ci/install_and_check_minimum_requirements.sh installs 3.25 accordingly; note its download URL also needed the lowercase -linux-x86_64 spelling, which CMake switched to in 3.20. Because the 3.28 EXCLUDE_FROM_ALL keyword for FetchContent_Declare is above that floor, evmone is populated and added with the documented pre-3.28 equivalent, add_subdirectory(... EXCLUDE_FROM_ALL), keeping evmone's tools out of the default build. cmake_policy(SET CMP0169 OLD) is guarded with if(POLICY CMP0169), since CMP0169 is a 3.30 addition and setting an unknown policy is a hard configure error rather than a no-op.
evmone's state library lives in test/state and is built only under EVMONE_TOOLS, which also pulls in test/utils and tools/. Because Hunter is disabled here, hunter_add_package() is a no-op and each following find_package(... CONFIG REQUIRED) must be satisfied some other way. There are two, and they need opposite treatments: - nlohmann_json is already provided by Solidity itself, via add_subdirectory(deps/nlohmann-json). Declaring it through FetchContent with OVERRIDE_FIND_PACKAGE would add that same source tree a second time, which is a duplicate-target error. An empty config in CMAKE_FIND_PACKAGE_REDIRECTS_DIR satisfies find_package() without creating anything, since the target exists. - CLI11, required by tools/evmone, has no counterpart in Solidity, so OVERRIDE_FIND_PACKAGE is exactly right. Pinned to v2.5.0, matching evmone's own Hunter config. Nothing under tools/ joins the default build; it is nested inside evmone's EXCLUDE_FROM_ALL subtree. (cherry picked from commit 2131dc3)
Add a prototype execution path that runs semantic tests through evmone's state library instead of Solidity's hand-written mock host, so tests can be exercised against real Mainnet semantics: genuine intrinsic gas, EIP-161/2929/6780, and evmone's own precompiles rather than the mock's hard-coded answers. EVMState provides an in-memory evmone::state::StateView and BlockHashes over an ordered account map; EVMTransactionDriver builds an evmone::state::Transaction, calls validate_transaction() and transition(), and applies the returned StateDiff back. Selected by --use-evmone-state, which defaults off: the flag is read in exactly one place, choosing a backend and nothing else, so the EVMHost path is untouched. isoltest compiles ExecutionFramework.cpp too, so it links evmone::state and builds the driver's sources alongside soltest's. Details worth knowing: - Transaction::sender is a plain field, so no signing is needed, and ecrecover works without libsecp256k1 via evmone's built-in implementation. - EIP-7825 caps transaction gas at 0x1000000, far below this framework's InitialGas. The cap is applied only from Osaka onward, matching evmone's own revision check, so pre-Osaka runs are not silently over-constrained. - TransactionReceipt carries no return data, so output is captured by replaying evmone's pre-call setup. That replica is cross-checked against the official run's status, gas used and gas refund on every call; a divergence in output alone cannot be detected, and the comment says so rather than implying otherwise. - applyDiff erases storage slots whose new value is zero, matching evmone's own reference applier. Retaining them would leave an all-zero account reporting has_storage, which feeds Account::has_initial_storage and makes is_create_collision() reject a later CREATE2 to that address. - Code-deposit gas is reconstructed from the StateDiff by summing the size of each changed code entry. This deliberately yields zero for a failed creation, where the mock host instead charges for result.output_size -- revert-reason bytes, not deployed code -- and accumulates it into a counter it never rolls back. evmone's behaviour is the correct one here. This is a prototype, not a replacement: with the flag on, a substantial number of fixtures still differ, dominated by EIP-7623's calldata floor, EIP-170's deposited-code limit and the EIP-7825 cap -- all real semantics the mock host never modelled. Deleting EVMHost is follow-up work this unblocks, not part of this change.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Upgrades the test suite to evmone 0.23.0 / EVMC ABI 18, starts consuming evmone from the build system instead of vendoring its headers, and adds an opt-in path that runs semantic tests through
evmone::state::transition()— evmone's real state library — instead of the hand-written mock host intest/EVMHost.cpp.The first two parts are meant to land as-is. The third is a prototype behind a flag that is off by default; it is included so the remaining gap to real Mainnet semantics is visible and measurable rather than hypothetical.
1. EVMC 13 → 18 (
test:,ci:commits)evmone 0.23.0 reports EVMC ABI 18, which the previous vendored headers (ABI 13) refuse to load, so this is a required upgrade rather than an optional one. The breaking changes and how they are handled:
msg.recipientEVMHostno longer derives it; derivation moves toEVMHost::computeCreateAddressfor top-level creationsevmc_result::create_addressremovedevmc_message::create2_saltremovedget_nonce()added to the host interfaceMockedHost; the VM reads it live to derive CREATE addressesEVMC_CONSTANTINOPLEremovedconstantinoplemaps to Petersburg (see caveat below)Because the VM now derives CREATE addresses from
get_nonce(sender), created accounts store nonce 1 (EIP-161) and test accounts start at nonce 1. That keeps every derived contract address byte-identical to before.Verification: the ported suite reproduces the pre-port baseline exactly across nine EVM versions (
osaka,osaka --optimize, andsemanticTestsatconstantinople,petersburg,berlin,london,cancun,prague,amsterdam) — same test-case counts, same assertion counts, zero failures, with--enforce-gas-costenabled at the default version. No test expectation was changed anywhere in this PR.2. evmone via FetchContent (
build:commits)cmake/EvmoneDependency.cmakefetches evmone 0.23.0 and intx 0.15.0 from hash-pinned release archives, so the version is pinned by the build system rather than by documentation. EVMC headers now come from evmone's own tree; the eight vendored copies undertest/evmc/are deleted.test/evmc/keeps exactly two files:loader.candloader.h. As of 0.23.0 evmone'sevmc/libbuilds only theevmc,evmc_cppandmocked_hosttargets — the EVMC loader is gone upstream — andEVMHost::getVMneedsevmc_load_and_configurefor the user-facing--vmflag. Those two files are ABI-agnostic (they read onlyvm->abi_version), so they need no per-version maintenance.--vmandETH_EVMONEcontinue to work against an external EVMC VM; with no path given, the linked-in evmone is used.Two notes for reviewers:
cmake_minimum_requiredrises to 3.25, the version this design needs (3.24 forCMAKE_FIND_PACKAGE_REDIRECTS_DIR/OVERRIDE_FIND_PACKAGE, 3.25 forblock(SCOPE_FOR VARIABLES)).scripts/ci/install_and_check_minimum_requirements.shanddocs/installing-solidity.rstare updated in step. This drops Ubuntu 22.04, which ships CMake 3.22.-Werrorpropagates into evmone's sources while evmone's own-Wno-attributes=clang::/msvc::suppressions do not, because they sit insidecable_configure_compiler()'sif(PROJECT_IS_TOP_LEVEL)guard. Without this a default (PEDANTIC=ON) configure fails outright. Solidity's own targets keep-Werror.3.
--use-evmone-state(prototype, off by default)With the flag set,
ExecutionFrameworkbuilds anevmone::state::Transactionand callsvalidate_transaction()+transition(), applying the returnedStateDiffback to an in-memoryevmone::state::StateView. That replaces the mock host, its simplified state model, and its ~1000 lines of hand-written precompiles with evmone's real ones. The flag is read in exactly one place and selects a backend — nothing else changes.What works
--enforce-gas-cost.What is expected to fail
Excluding
SolidityAuctionRegistrar(see below):--use-evmone-state--use-evmone-state --enforce-gas-cost(The 8 skipped are version-gated fixtures that do not run at the default EVM version; they are skipped on the default path too.)
The 28 break down as:
GasMeterTests/GasCostTests— these assert that solc's static gas estimate is an upper bound on actual gas. It no longer is, because the estimator does not model EIP-7623's calldata floor.blobhash(builtinFunctions,inlineAssembly,state,state/uncalled) — the driver does not populatetx.blob_hashes.EXTCODEHASH/balance on precompile accounts (various/codehash,codehash_assembly,codebalance_assembly) — the mock host pre-seeds precompile accounts with a balance and codehash; evmone does not. Precompile computation is unaffected.state/tx_origin—evmone::state::Transactionhas no separate origin field; origin comes fromtx.sender.operators/userDefined/all_possible_user_defined_value_types_with_operators) — deposits ~38 KB of code, over the 0x6000 limit that evmone enforces and the mock host never checked.validate_transactionrejection (isoltestTesting/account,INSUFFICIENT_ACCOUNT_FUNDS) — gas is really debited from the sender's balance here.array/array_storage_index_access,array_storage_push_pop,storageLayoutSpecifier/dynamic_array_storage_end,mapping_storage_end,variables/transient_state_address_variable_members).The extra 58 under
--enforce-gas-costare gas-value expectations: 88 fixtures carry a// gas … code:line, and real gas differs for the EIP-7623/7825/170 reasons above.Every one of these is a case where the state path is applying semantics the mock host never modelled. None was worked around, and no expectation was regenerated.
Known caveats
--use-evmone-staterun aborts.SolidityAuctionRegistrar/auction_simplereaches directly intom_evmcHost, which is null on the state path, and dies with a memory access violation. This is Solidity test code, not evmone. Until it is adapted, exclude it:-t '!SolidityAuctionRegistrar'.--use-evmone-statewith pre-Osaka--evm-versionyet. The driver's block gas limit (20M) is below the uncappedInitialGas(100M), soGAS_ALLOWANCE_EXCEEDEDfollows. Correct block-level validation, but the prototype does not yet lower gas to suit.TransactionReceiptcarries none. The replica is cross-checked against the official run's status, gas used and gas refund on every call; a divergence in output alone would not be detected, and the code says so.Required follow-up before CI can pass
The four
buildpack-depsimages must be rebuilt and republished, and their digests re-pinned in.circleci/config.yml(lines 17, 21, 25, 29). Those images still carry evmone 0.22.0, so Linux jobs would load an ABI-13 library against ABI-18 headers. The Dockerfiles and theirLABEL versionvalues are bumped in this PR, which is what triggers the rebuild; the digest update cannot be done from a source checkout.Caveat worth flagging explicitly
constantinoplenow maps toEVMC_PETERSBURG. Petersburg uses the legacy SSTORE schedule where Constantinople used EIP-1283 net metering, so this is an approximation. No fixture asserts gas at that version, so the difference is untested. solc's own code generation is unaffected — it is driven bylangutil::EVMVersion, notevmc_revision.The ossfuzz targets were updated for the new API but could not be compiled in the development environment (no libprotobuf-mutator, no clang++); they were syntax-checked only.