Parallelize the test suite in CI with parallel_tests by maebeale · Pull Request #1572 · rubyforgood/awbw

maebeale · 2026-06-06T18:23:47Z

What is the goal of this PR and why is this important?

The CI build-and-test job runs the full RSpec suite (~280 specs, 27 of them Selenium system specs) in a single process, making it the wall-clock bottleneck (~8 min).
GitHub's ubuntu-latest runners have 4 vCPUs that sat mostly idle.
This splits the suite across 4 parallel processes to use them, which should roughly cut test-execution wall time.

How did you approach the change?

Added the parallel_tests gem.
Appended TEST_ENV_NUMBER to the test database name in config/database.yml so each process gets an isolated DB (awbw_test, awbw_test2, …).
CI build-and-test now runs rake parallel:create parallel:load_schema, then parallel_rspec spec/ with PARALLEL_TEST_PROCESSORS=4.
Each process writes its own JSON results file to tmp/rspec_results/ (a single --out file would be clobbered across processes); the results artifact now uploads that directory.
Moved SimpleCov behind a COVERAGE flag set only by the coverage-badge workflow on main. It no longer runs on every PR — this removes branch-coverage overhead from PR runs and avoids the cross-process result-merge / per-process minimum_coverage problems that parallel execution otherwise creates.

Anything else to add?

Behavior change to flag: PRs no longer enforce the SimpleCov minimum_coverage 20 gate (coverage still tracked on main via the badge job). Happy to keep it on PRs instead by wiring up proper parallel SimpleCov merging if preferred.
Verified locally: 846 model examples across 3 processes, 0 failures; per-process JSON result files written correctly; a non-JS system spec passed under parallel. (JS-driven system specs fail locally with or without this change due to a pre-existing local vite/asset issue — unrelated.)
Follow-up (not in this PR): sharding across a GitHub matrix (multiple runners) would stack on top of this for further gains, especially isolating the slow system specs. Profiling with rspec --profile would target the worst offenders.

🤖 Generated with Claude Code

maebeale · 2026-06-06T18:24:02Z

-  database: awbw_test<%= "_#{ENV['CONDUCTOR_PORT']}" if ENV['CONDUCTOR_PORT'] %>
+  # TEST_ENV_NUMBER is set by parallel_tests to give each parallel process its
+  # own database (blank for the first process, "2", "3", ... for the rest).
+  database: awbw_test<%= "_#{ENV['CONDUCTOR_PORT']}" if ENV['CONDUCTOR_PORT'] %><%= ENV['TEST_ENV_NUMBER'] %>


🤖 From Claude: This TEST_ENV_NUMBER suffix is what gives each parallel process its own DB — blank for process 1 (awbw_test), then awbw_test2, etc. It composes with the existing CONDUCTOR_PORT suffix and is empty for normal single-process/local runs, so nothing changes there.

Just a nit, not sure if "CONDUCTOR_PORT" is hard coded with conductor or something you've added locally but a more generic name would be better if other contributors need to set this port for different "assistants". Or even if you change from using conductor to using claude code for example, a generic name would be more appropriate.

great point. ok, will do that in separate pr

maebeale · 2026-06-06T18:24:03Z

 require "active_storage_validations/matchers"

-if ENV["CI"]
+if ENV["COVERAGE"]


🤖 From Claude: SimpleCov is now gated on COVERAGE (set only by the badge workflow on main) instead of CI. This keeps branch-coverage overhead off PR runs and sidesteps the per-process minimum_coverage check + .resultset.json merge that parallel execution would otherwise break. Trade-off: PRs lose the 20% coverage gate.

maebeale · 2026-06-06T18:24:04Z

+    # 4 parallel_tests processes (each with its own database) to cut wall time.
+    env:
+      PARALLEL_TEST_PROCESSORS: 4
+


🤖 From Claude: PARALLEL_TEST_PROCESSORS=4 matches the 4 vCPUs on ubuntu-latest. Running 4 headless Chrome system specs concurrently is the main flakiness risk to watch — if CI gets flaky, dropping to 3 or sharding system specs into their own job is the lever.

Copilot

Pull request overview

Parallelizes the RSpec suite in GitHub Actions using parallel_tests to reduce CI wall-clock time, while adjusting coverage and test-result reporting to work cleanly with multi-process execution.

Changes:

Add parallel_tests and update CI to run parallel:create/parallel:load_schema + parallel_rspec across 4 processes.
Isolate each parallel process with its own test database via TEST_ENV_NUMBER suffixing in config/database.yml.
Gate SimpleCov behind COVERAGE=true and write per-process RSpec JSON results under tmp/rspec_results/, updating docs and artifacts accordingly.

Reviewed changes

Copilot reviewed 6 out of 7 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
`spec/spec_helper.rb`	Moves SimpleCov behind `COVERAGE` and writes per-process JSON result files in CI.
`Gemfile`	Adds `parallel_tests` gem for parallelized specs.
`Gemfile.lock`	Locks `parallel_tests` dependency.
`config/database.yml`	Appends `TEST_ENV_NUMBER` to test DB name for per-process isolation.
`AGENTS.md`	Documents new coverage flag behavior and parallel run commands.
`.github/workflows/sanity-check-main.yml`	Sets `COVERAGE=true` for the coverage badge workflow run.
`.github/workflows/ci.yml`	Updates CI to create/load parallel DBs and run `parallel_rspec`, uploading a results directory artifact.

maebeale · 2026-06-06T18:32:55Z

@jmilljr24 @lenikadali hello! our test suite on gh is now at about 8min. do you think these changes acceptable?

jmilljr24 · 2026-06-06T18:42:35Z

@jmilljr24 @lenikadali hello! our test suite on gh is now at about 8min. do you think these changes acceptable?

My prior knowledge told me that rspec didn't align with parallel testing but I see the gem added in this PR supports it so I say let give it a go.

I will note that we are getting heavy with system tests which is contributing to the longer times. Something to consider when adding new tests.

jmilljr24 · 2026-06-06T18:48:54Z

-  database: awbw_test<%= "_#{ENV['CONDUCTOR_PORT']}" if ENV['CONDUCTOR_PORT'] %>
+  # TEST_ENV_NUMBER is set by parallel_tests to give each parallel process its
+  # own database (blank for the first process, "2", "3", ... for the rest).
+  database: awbw_test<%= "_#{ENV['CONDUCTOR_PORT']}" if ENV['CONDUCTOR_PORT'] %><%= ENV['TEST_ENV_NUMBER'] %>


Just a nit, not sure if "CONDUCTOR_PORT" is hard coded with conductor or something you've added locally but a more generic name would be better if other contributors need to set this port for different "assistants". Or even if you change from using conductor to using claude code for example, a generic name would be more appropriate.

Copilot

Pull request overview

Copilot reviewed 6 out of 7 changed files in this pull request and generated 1 comment.

+  # TEST_ENV_NUMBER is set by parallel_tests to give each parallel process its
+  # own database (blank for the first process, "2", "3", ... for the rest).
+  database: awbw_test<%= "_#{ENV["CONDUCTOR_PORT"]}" if ENV["CONDUCTOR_PORT"] %><%= ENV["TEST_ENV_NUMBER"] %>


The CI build-and-test job ran the full RSpec suite in a single process, making it the wall-clock bottleneck (~8 min). GitHub's ubuntu-latest runners have 4 vCPUs that sat mostly idle. Split the suite across 4 parallel_tests processes, each with its own database, to use them. - Add parallel_tests; append TEST_ENV_NUMBER to the test DB name so each process gets an isolated database. - CI now runs `parallel:create parallel:load_schema` then `parallel_rspec`. - Each process writes its own JSON results file (a single --out would clobber). - Move SimpleCov behind a COVERAGE flag set only by the coverage-badge workflow on main. It no longer runs on PRs, removing branch-coverage overhead and avoiding cross-process result-merge complexity under parallel. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Copilot AI review requested due to automatic review settings June 6, 2026 18:23

Copilot started reviewing on behalf of maebeale June 6, 2026 18:23 View session

maebeale commented Jun 6, 2026

View reviewed changes

Copilot AI reviewed Jun 6, 2026

View reviewed changes

Comment thread spec/spec_helper.rb

Comment thread config/database.yml Outdated

maebeale marked this pull request as ready for review June 6, 2026 18:30

maebeale requested review from jmilljr24 and lenikadali June 6, 2026 18:32

jmilljr24 approved these changes Jun 6, 2026

View reviewed changes

Copilot AI review requested due to automatic review settings June 6, 2026 21:02

Copilot started reviewing on behalf of maebeale June 6, 2026 21:02 View session

Copilot AI reviewed Jun 6, 2026

View reviewed changes

maebeale force-pushed the maebeale/speed-up-test-suite branch from a61c3cf to 44f88e2 Compare June 6, 2026 22:32

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Parallelize the test suite in CI with parallel_tests#1572

Parallelize the test suite in CI with parallel_tests#1572
maebeale wants to merge 1 commit into
mainfrom
maebeale/speed-up-test-suite

maebeale commented Jun 6, 2026

Uh oh!

maebeale Jun 6, 2026

Uh oh!

jmilljr24 Jun 6, 2026

Uh oh!

maebeale Jun 6, 2026

Uh oh!

maebeale Jun 6, 2026

Uh oh!

maebeale Jun 6, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

maebeale commented Jun 6, 2026

Uh oh!

jmilljr24 commented Jun 6, 2026

Uh oh!

jmilljr24 Jun 6, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

maebeale commented Jun 6, 2026

What is the goal of this PR and why is this important?

How did you approach the change?

Anything else to add?

Uh oh!

maebeale Jun 6, 2026

Choose a reason for hiding this comment

Uh oh!

jmilljr24 Jun 6, 2026

Choose a reason for hiding this comment

Uh oh!

maebeale Jun 6, 2026

Choose a reason for hiding this comment

Uh oh!

maebeale Jun 6, 2026

Choose a reason for hiding this comment

Uh oh!

maebeale Jun 6, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

maebeale commented Jun 6, 2026

Uh oh!

jmilljr24 commented Jun 6, 2026

Uh oh!

jmilljr24 Jun 6, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants