Parallel test runner

[benoit-cty]

Technical changes

Add a system to parallelize tests:

make test-core openfisca_args="--in-parallel"

[⚙] Run openfisca-core tests…
Running 15 test files across 15 workers...
  Worker 0: test_dummy.py
  Worker 1: test_formulas.py
  Worker 2: test_rates.py
  Worker 3: test_entity.py
  Worker 4: test_group_entity.py
  Worker 5: test_role.py
  Worker 6: test_helpers.py
  Worker 7: test_enum.py
  Worker 8: test_enum_array.py
  Worker 9: test_helpers.py
  Worker 10: test_instant.py
  Worker 11: test_period.py
  Worker 12: test_instant.py
  Worker 13: test_parsers.py
  Worker 14: test_period.py

✓ Worker 10: test_instant.py (2.4s).0s elapsed)
✓ Worker 9: test_helpers.py (2.4s)
✓ Worker 8: test_enum_array.py (2.5s)
✓ Worker 0: test_dummy.py (2.6s)
✓ Worker 7: test_enum.py (2.6s)
✓ Worker 3: test_entity.py (2.6s)
✓ Worker 2: test_rates.py (2.6s)
✓ Worker 5: test_role.py (2.8s)
✓ Worker 6: test_helpers.py (2.8s)
✓ Worker 11: test_period.py (2.9s)
✓ Worker 13: test_parsers.py (2.9s)
✓ Worker 12: test_instant.py (2.9s)
✓ Worker 1: test_formulas.py (2.9s)
✓ Worker 14: test_period.py (2.9s)
✓ Worker 4: test_group_entity.py (3.0s)

================================================================================
✓ All tests passed!
  15 test files across 15 workers in 3.00s
================================================================================
[✓] test-core:passed ❤

❯ time make test-core openfisca_args="--in-parallel --num-workers=4"
[⚙] Run openfisca-core tests…
Running 15 test files across 4 workers...
  Worker 0: test_dummy.py, test_group_entity.py, test_enum_array.py (+1 more)
  Worker 1: test_formulas.py, test_role.py, test_helpers.py (+1 more)
  Worker 2: test_rates.py, test_helpers.py, test_instant.py (+1 more)
  Worker 3: test_entity.py, test_enum.py, test_period.py

✓ Worker 0: test_dummy.py, test_group_entity.py, test_enum_array.py (+1 more) (2.2s)
✓ Worker 3: test_entity.py, test_enum.py, test_period.py (2.2s)
✓ Worker 1: test_formulas.py, test_role.py, test_helpers.py (+1 more) (2.3s)
✓ Worker 2: test_rates.py, test_helpers.py, test_instant.py (+1 more) (2.5s)

================================================================================
✓ All tests passed!
  15 test files across 4 workers in 2.51s
================================================================================
[✓] test-core:passed ❤

[MattiSG]

That's very exciting performance gains, thanks @benoit-cty! But I thought we already parallelise tests, at least in CI. How does that parallelisation relate to the CI splitting? Should we also update our CI run command? 🙂

[benoit-cty]

Tests aren't parallelized locally.
For France, we parallelize tests in CI, by using Github Action matrix. My proposal is to allow this locally.
The parameter "--in-parallel" is not activated by default, so CI won't be impacted.

For "core" the tests took longer with parallelization because they are small and quick. I will evaluate it with France where it tooks hours with single thread.

[MattiSG]

Thank you for these details.

Do I understand it correctly that the runner needs to be updated in Core, but you suggest not to activate that option for Core tests themselves, only for large country models?