test: Add the evmone test command by chfast · Pull Request #1685 · ipsilon/evmone

chfast · 2026-08-31T12:54:13Z

One evmone test command runs both state and blockchain fixtures,
deciding which format each test case is when it runs it:
_info.fixture-format names it, and a case without one is recognised
by the shape only that format has.

What the tool cannot run is treated three ways, because a fixture
directory holds more than fixtures. A format it does not run, engine
fixtures above all, is skipped. A file holding no test at all is skipped
whole, which is what EEST's shared pre-allocation is: 35715 files in one
release. Anything else it cannot recognise, in a file whose other cases
are tests, is a fault in the fixture and fails.

Collection reads no fixture, so a directory costs a walk.

evmone-statetest and evmone-blockchaintest are untouched and still
pass CI; a later PR moves CI to evmone test and deletes them.

Stacked on #1692 and #1693: only the last commit is this PR.

Verified against them on three fixture releases, verdict for verdict:
2710 state tests, 2827 blockchain tests with 2 skipped, and 8172 / 8612
on tests@v20.0.1.

codspeed-hq · 2026-08-31T12:58:12Z

Merging this PR will not alter performance

✅ 129 untouched benchmarks

_{Comparing test/evmone-test-cmd (8b12a8f) with master (a203f55)}

codecov · 2026-08-31T13:07:00Z

Codecov Report

❌ Patch coverage is 99.07407% with 1 line in your changes missing coverage. Please review.
✅ Project coverage is 97.99%. Comparing base (a203f55) to head (8b12a8f).

Files with missing lines	Patch %	Lines
test/utils/test_driver.cpp	97.77%	0 Missing and 1 partial ⚠️

Additional details and impacted files

@@           Coverage Diff            @@
##           master    #1685    +/-   ##
========================================
  Coverage   97.98%   97.99%            
========================================
  Files         181      182     +1     
  Lines       16199    16307   +108     
  Branches     3724     3747    +23     
========================================
+ Hits        15873    15980   +107     
  Misses        248      248            
- Partials       78       79     +1

Flag	Coverage Δ
eest-develop	`86.05% <0.00%> (-1.09%)`	⬇️
eest-develop-gmp	`26.01% <0.00%> (-0.17%)`	⬇️
eest-legacy	`16.82% <0.00%> (-0.12%)`	⬇️
eest-libsecp256k1	`28.19% <0.00%> (-0.19%)`	⬇️
eest-stable	`86.01% <0.00%> (-1.09%)`	⬇️
evmone-unittests	`94.28% <99.07%> (+0.03%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Components	Coverage Δ
core	`96.19% <ø> (ø)`
tooling	`94.32% <99.07%> (+0.26%)`	⬆️
tests	`99.81% <ø> (ø)`

Files with missing lines	Coverage Δ
test/utils/test_collector.cpp	`100.00% <100.00%> (ø)`
test/utils/test_collector.hpp	`100.00% <ø> (ø)`
test/utils/test_driver.hpp	`100.00% <100.00%> (ø)`
tools/evmone/main.cpp	`97.04% <100.00%> (+0.69%)`	⬆️
test/utils/test_driver.cpp	`98.47% <97.77%> (-0.37%)`	⬇️

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Copilot

Pull request overview

Adds a unified evmone test command for state and blockchain fixtures.

Changes:

Adds lazy fixture collection, classification, filtering, and execution.
Exposes per-case state and blockchain loaders.
Updates empty/skipped-run exit semantics and integration coverage.

Reviewed changes

Copilot reviewed 16 out of 16 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
`tools/evmone/main.cpp`	Adds the `test` subcommand.
`test/utils/test_driver.hpp`	Defines the nothing-verified exit code.
`test/utils/test_driver.cpp`	Handles fully skipped runs.
`test/utils/test_collector.hpp`	Declares collection settings and API.
`test/utils/test_collector.cpp`	Classifies and runs mixed fixtures.
`test/utils/statetest.hpp`	Exposes single-case loading.
`test/utils/statetest_runner.cpp`	Documents empty-post behavior.
`test/utils/statetest_loader.cpp`	Implements single-case loading.
`test/utils/CMakeLists.txt`	Builds the collector.
`test/utils/blockchaintest.hpp`	Exposes single-case loading.
`test/utils/blockchaintest_loader.cpp`	Publishes the case loader.
`test/unittests/test_driver_test.cpp`	Tests skipped-run semantics.
`test/integration/testcmd/one_unsupported_case.json`	Covers mixed supported and unsupported cases.
`test/integration/testcmd_fault/unrecognised_case.json`	Covers malformed fixture cases.
`test/integration/testcmd_cases/case_after_exception.json`	Covers continuation after exceptions.
`test/integration/CMakeLists.txt`	Adds command-level integration tests.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

+    // A fixture file in which this tool ran nothing it was asked for is skipped, not passed.
+    if (!any_ran && !declined.empty())
+        throw UnsupportedTestFeature{declined.front().second};


+    cmd.add_flag("--trace-summary", opts.settings.trace_summary,
+        "Report each test's execution summary; --trace reports it as well.");


+        auto format = info->find("fixture-format");
+        if (format == info->end())
+            format = info->find("fixture_format");  // The spelling EEST also accepts.


Copilot

Pull request overview

Copilot reviewed 7 out of 7 changed files in this pull request and generated no new comments.

Suppressed comments (1)

test/utils/test_collector.cpp:152

When -k matches no cases in a directory, every collected file returns normally and is counted as passed, so the command exits successfully despite executing no fixture. This also makes filtering behavior depend on whether the same fixture was named as a file (which correctly returns NOTHING_VERIFIED). Mark a file with no runnable selected cases as skipped so an empty directory selection propagates NOTHING_VERIFIED.

    // A fixture file in which this tool ran nothing it was asked for is skipped, not passed.
    // TODO: A file whose cases -k all deselected still passes, as it did before this command
    //   existed, so a filter which matches nothing reports a tree of passing tests. Skip it
    //   instead, and an empty selection reaches NOTHING_VERIFIED on its own.
    if (!any_ran && !declined.empty())
        throw UnsupportedTestFeature{declined.front().second};

Copilot

Pull request overview

Copilot reviewed 14 out of 14 changed files in this pull request and generated 1 comment.

Suppressed comments (1)

test/utils/test_collector.cpp:148

With a directory root, -k matching no case leaves each collected file as passed, so the command exits successfully despite verifying nothing; the same filter on a named file correctly returns NOTHING_VERIFIED. Treat a fixture file where no selected case ran as skipped so an entirely empty selection produces the non-success exit code.

    // A fixture file in which this tool ran nothing it was asked for is skipped, not passed.
    // TODO: A file whose cases -k all deselected still passes, as it did before this command
    //   existed, so a filter which matches nothing reports a tree of passing tests. Skip it
    //   instead, and an empty selection reaches NOTHING_VERIFIED on its own.
    if (!any_ran && !declined.empty())
        throw UnsupportedTestFeature{declined.front().second};

+    // TODO: A test with an empty "post" checks nothing and still passes, so a gutted fixture is
+    //   indistinguishable from a working one. Report a test which verified nothing as a failure.


The module collects the tests a run is given, and #1685 is about to make it collect them from inside a file too, not only find the files. A pure rename, peeled out of #1685 so the real change there stops showing as a 54%-similar rewrite of this file.

Every test which did not pass leaves exactly one note, so the counts follow from the notes: the three counters and the switch feeding them go, and a skip's reason is stored like any other. Peeled out of #1685, where it was a rewrite of #1684 code hiding inside "add the command".

One command runs both fixture formats, deciding which each test case is when it runs it: "_info.fixture-format" names it, and a case without one is recognised by the shape only that format has. A format the tool does not run is skipped, as is a file holding no test at all, which is what EEST's shared pre-allocation is. Anything else it cannot recognise is a fault in the fixture. The two test tools are untouched and keep working. Claude-Session: https://claude.ai/code/session_016UHPAGwcwXMjqhTLpT31K7

An entry the tool did not recognise was a fault or a skip depending on whether the rest of its file holds fixtures, a bool threaded from the collector through a lambda into run_fixture to feed one branch. Loading the file settles it instead: a file with no fixture in it is skipped whole and nothing in it reaches execution, so an unrecognised entry in a fixture file is simply a fault. Named on the command line, such a file is one skipped test rather than one per entry. Claude-Session: https://claude.ai/code/session_016UHPAGwcwXMjqhTLpT31K7

chfast requested a balanced review from Copilot August 31, 2026 13:12

Copilot started reviewing on behalf of chfast August 31, 2026 13:13 View session

Copilot AI reviewed Aug 31, 2026

View reviewed changes

chfast force-pushed the test/evmone-test-cmd branch 4 times, most recently from 2cfb252 to 3460a62 Compare August 31, 2026 20:52

chfast requested a balanced review from Copilot August 31, 2026 20:54

Copilot started reviewing on behalf of chfast August 31, 2026 20:55 View session

Copilot AI reviewed Aug 31, 2026

View reviewed changes

chfast force-pushed the test/evmone-test-cmd branch 4 times, most recently from 0080993 to cdc9e3d Compare September 1, 2026 08:22

chfast requested a balanced review from Copilot September 1, 2026 09:33

Copilot started reviewing on behalf of chfast September 1, 2026 09:33 View session

Copilot AI reviewed Sep 1, 2026

View reviewed changes

Comment thread test/utils/statetest_runner.cpp Outdated

Comment on lines +19 to +20

// TODO: A test with an empty "post" checks nothing and still passes, so a gutted fixture is

// indistinguishable from a working one. Report a test which verified nothing as a failure.

chfast force-pushed the test/evmone-test-cmd branch 6 times, most recently from e9d827d to 1d8a9ca Compare September 1, 2026 13:25

This was referenced Sep 2, 2026

test: Rename test_files to test_collector #1692

Merged

test: Count outcomes from the notes #1693

Merged

chfast force-pushed the test/evmone-test-cmd branch 2 times, most recently from 81b6c18 to 4686ff5 Compare September 2, 2026 05:51

chfast force-pushed the test/evmone-test-cmd branch from 4686ff5 to 894674a Compare September 2, 2026 06:04

chfast force-pushed the test/evmone-test-cmd branch from 894674a to 910136c Compare September 2, 2026 06:49

Navigation Menu

Uh oh!

test: Add the evmone test command - #1685

test: Add the evmone test command#1685
chfast wants to merge 2 commits into
masterfrom
test/evmone-test-cmd

chfast commented Aug 31, 2026 •

edited

Loading

Uh oh!

codspeed-hq Bot commented Aug 31, 2026 •

edited

Loading

Uh oh!

codecov Bot commented Aug 31, 2026 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI left a comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

		cmd.add_flag("--trace-summary", opts.settings.trace_summary,
		"Report each test's execution summary; --trace reports it as well.");

		// TODO: A test with an empty "post" checks nothing and still passes, so a gutted fixture is
		// indistinguishable from a working one. Report a test which verified nothing as a failure.

Uh oh!

Conversation

chfast commented Aug 31, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

codspeed-hq Bot commented Aug 31, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Merging this PR will not alter performance

Uh oh!

codecov Bot commented Aug 31, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

chfast commented Aug 31, 2026 •

edited

Loading

codspeed-hq Bot commented Aug 31, 2026 •

edited

Loading

codecov Bot commented Aug 31, 2026 •

edited

Loading