Skip to content

fix(bench): fail the run when a benchmark file fails to load - #1304

Merged
Chemaclass merged 2 commits into
mainfrom
fix/bench-exit-code-on-file-failures
Aug 17, 2026
Merged

fix(bench): fail the run when a benchmark file fails to load#1304
Chemaclass merged 2 commits into
mainfrom
fix/bench-exit-code-on-file-failures

Conversation

@Chemaclass

Copy link
Copy Markdown
Member

🤔 Background

Related #1303

bashunit bench printed a file-level failure and then exited 0. Same scenario, both subcommands — one good file plus one whose set_up_before_script fails:

$ bashunit bench benchmarks/
✗ Error: Set up before script
bench_works    1    1    6
EXIT=0            <- bench

$ bashunit tests/
EXIT=1            <- test

A syntax error was worse: every bench_ function after it vanished from the table and nothing was reported as failed at all.

💡 Changes

  • load_bench_files checks the sourcing status and matches the captured stderr for syntax error / unexpected EOF, as the test loop does — bash reports a syntax error and carries on with a zero status, so the status alone does not catch it. case, not grep, to stay fork-free
  • cmd_bench consults the failure counter. It exited non-zero only for "No benchmarks found" and the baseline gate, so the existing set_up_before_script failure path could not affect the verdict — it only looked correct when the failure left nothing to run and "No benchmarks found" fired instead
  • A benchmark that merely returns non-zero still exits 0: a benchmark measures time and has no assertion concept

🔍 How it was found

Applying the pathological-fixture sweep from #1295/#1297/#1299/#1301 to bench, which has its own runner and reports and is far less exercised than test.

bench printed a file-level failure and exited 0, so it reached a human
reading the log and never reached CI.

Two gaps. load_bench_files sourced with no status check, while the test
loop checks the status and matches the captured stderr for 'syntax error'
/ 'unexpected EOF' -- bash reports a syntax error and carries on with a
zero status, so a broken file silently lost every bench_ function after
the error and the run stayed green.

And cmd_bench consulted only 'No benchmarks found' and the baseline gate,
never the failure counter, so the existing set_up_before_script failure
path could not affect the verdict. It only looked right when the failure
left nothing to run and 'No benchmarks found' fired instead.

A benchmark that merely returns non-zero still exits 0: a benchmark
measures time and has no assertion concept.

Closes #1303
@Chemaclass Chemaclass added the bug Something isn't working label Aug 17, 2026
@Chemaclass Chemaclass self-assigned this Aug 17, 2026
@Chemaclass
Chemaclass merged commit 2899c84 into main Aug 17, 2026
37 checks passed
@Chemaclass
Chemaclass deleted the fix/bench-exit-code-on-file-failures branch August 17, 2026 06:35
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant