Skip to content

test: Report a file's fixtures one result each - #1700

Closed
chfast wants to merge 1 commit into
masterfrom
test/results-per-fixture
Closed

test: Report a file's fixtures one result each#1700
chfast wants to merge 1 commit into
masterfrom
test/results-per-fixture

Conversation

@chfast

@chfast chfast commented Sep 4, 2026

Copy link
Copy Markdown
Member

A file collected from a directory was one test recording one outcome, so what happened to the fixtures beside the first had nowhere to go: a file holding two failing fixtures reported one line, and a fixture the loader declined disappeared into the file's verdict.

Running a test now returns one result per fixture, and a file which never gets as far as one — unparsable, or refused by the loader — returns the single result which says so. Collection and the counts are unchanged.

FAILED  statetest/tests1/SuiteA/test2_multi.json::test_case_1 - state root
FAILED  statetest/tests1/SuiteA/test2_multi.json::test_case_2 - state root

Ordered before #1699, which drops the second collection form: naming a file to see its fixtures listed only pays for itself while this is missing.

🤖 Generated with Claude Code

https://claude.ai/code/session_016UHPAGwcwXMjqhTLpT31K7

@codspeed-hq

codspeed-hq Bot commented Sep 4, 2026

Copy link
Copy Markdown
Contributor

Merging this PR will not alter performance

✅ 129 untouched benchmarks


Comparing test/results-per-fixture (56bab94) with master (13ec12d)

Open in CodSpeed

A file collected from a directory was one test recording one outcome,
so what happened to the fixtures beside the first had nowhere to go: a
file with two failing fixtures reported one line, and a fixture the
loader declined disappeared into the file's verdict.

Running a test now returns one result per fixture, and a file which
never gets as far as one returns the single result which says so. The
counts stay as they were, per test as collected.

Claude-Session: https://claude.ai/code/session_016UHPAGwcwXMjqhTLpT31K7
@chfast
chfast force-pushed the test/results-per-fixture branch from 7edd03a to 56bab94 Compare September 4, 2026 22:55
@chfast

chfast commented Sep 5, 2026

Copy link
Copy Markdown
Member Author

Not worth its own PR: the collapse in #1699 does not need it, and evmone test wants per-fixture results in its own run_fixture_file, where the loader can decline one fixture at a time. Measured: this saves 334 lines from #1685 only by moving the same scaffolding upstream, and adds ~180 net.

@chfast chfast closed this Sep 5, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant