fix(bench): fail the run when a benchmark file fails to load - #1304
Merged
Conversation
bench printed a file-level failure and exited 0, so it reached a human reading the log and never reached CI. Two gaps. load_bench_files sourced with no status check, while the test loop checks the status and matches the captured stderr for 'syntax error' / 'unexpected EOF' -- bash reports a syntax error and carries on with a zero status, so a broken file silently lost every bench_ function after the error and the run stayed green. And cmd_bench consulted only 'No benchmarks found' and the baseline gate, never the failure counter, so the existing set_up_before_script failure path could not affect the verdict. It only looked right when the failure left nothing to run and 'No benchmarks found' fired instead. A benchmark that merely returns non-zero still exits 0: a benchmark measures time and has no assertion concept. Closes #1303
…n-file-failures # Conflicts: # CHANGELOG.md
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
🤔 Background
Related #1303
bashunit benchprinted a file-level failure and then exited 0. Same scenario, both subcommands — one good file plus one whoseset_up_before_scriptfails:A syntax error was worse: every
bench_function after it vanished from the table and nothing was reported as failed at all.💡 Changes
load_bench_fileschecks the sourcing status and matches the captured stderr forsyntax error/unexpected EOF, as the test loop does — bash reports a syntax error and carries on with a zero status, so the status alone does not catch it.case, notgrep, to stay fork-freecmd_benchconsults the failure counter. It exited non-zero only for "No benchmarks found" and the baseline gate, so the existingset_up_before_scriptfailure path could not affect the verdict — it only looked correct when the failure left nothing to run and "No benchmarks found" fired instead🔍 How it was found
Applying the pathological-fixture sweep from #1295/#1297/#1299/#1301 to
bench, which has its own runner and reports and is far less exercised thantest.