Skip to content

Split Benchee integration tests#483

Closed
Will-thom wants to merge 1 commit into
bencheeorg:mainfrom
Will-thom:improve/benchee-test-split-480
Closed

Split Benchee integration tests#483
Will-thom wants to merge 1 commit into
bencheeorg:mainfrom
Will-thom:improve/benchee-test-split-480

Conversation

@Will-thom

Copy link
Copy Markdown
Contributor

Summary

  • split large Benchee integration coverage into focused test modules
  • move shared integration assertions into Benchee.IntegrationHelpers
  • replace the slow max_sample_size sampling assertions with configuration-level coverage and make the outlier test deterministic under slower runtimes

Fixes #480

Validation

  • mix format --check-formatted test/benchee_test.exs test/benchee/save_load_test.exs test/benchee/hooks_test.exs test/benchee/measurement_integration_test.exs test/benchee/profiling_integration_test.exs test/benchee/function_overhead_and_outliers_test.exs test/benchee/evaluation_warning_test.exs test/benchee/escript_test.exs test/benchee/max_sample_size_test.exs test/support/integration_helpers.ex
  • mix test test/benchee_test.exs test/benchee/save_load_test.exs test/benchee/hooks_test.exs test/benchee/measurement_integration_test.exs test/benchee/profiling_integration_test.exs test/benchee/function_overhead_and_outliers_test.exs test/benchee/evaluation_warning_test.exs test/benchee/escript_test.exs test/benchee/max_sample_size_test.exs --slowest-modules 10
  • mix test --slowest-modules 10

The validation was run in an isolated Docker container with Elixir 1.17.3 / OTP 27.

@Will-thom Will-thom force-pushed the improve/benchee-test-split-480 branch from d033bb5 to 8b4e8e8 Compare June 6, 2026 02:47
@Will-thom

Copy link
Copy Markdown
Contributor Author

Rebased this branch onto the current main and resolved the test split conflict by moving the newer max_sample_size default integration coverage into the dedicated max_sample_size test module.

Local validation ran in an ephemeral Elixir 1.17 Docker container:

mix test test/benchee/max_sample_size_test.exs test/benchee_test.exs

Result: 33 tests, 0 failures, 2 excluded.

@PragTob

PragTob commented Jun 6, 2026

Copy link
Copy Markdown
Member

Heyo, thanks a lot! 💚

I was a bit busy, I'll take a look at this and likely adjust it a little.

IMG_20171117_104248

@PragTob PragTob mentioned this pull request Jun 6, 2026
@PragTob

PragTob commented Jun 6, 2026

Copy link
Copy Markdown
Member

Merged as #487 - thank you very much!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Split/make benchee_test.exs run faster

2 participants