ci: add scheduled property-test workflow#107
Conversation
| gh issue create \ | ||
| --repo "${{ github.repository }}" \ | ||
| --title "Property tests failed (run #${{ github.run_number }})" \ | ||
| --body-file property-test-failure.md |
There was a problem hiding this comment.
as I typed in special considerations, I'm afraid of doing this as it might generate too much spam issues. we need to see how this works live
|
Thanks. I don't think it'll generate spam issues, because if we trust the properties, then they are valid things we need to fix. You need to update the max_cases of the properties to be configurable by an env var. In CI we'll set it to something really high that can run for a long time. locally we can default it to whatever it currently is. You can see the bigger values that were shorted in this commit: 8128f41. They do range, with the biggest being 2mil and the others being like 500k and 250k or less. maybe just settle for running them all at like 250k in ci. |
|
|
||
| - name: Unit Tests | ||
| run: nix develop --command bash -c "mix test" | ||
| run: nix develop --command bash -c "mix test --exclude property" |
There was a problem hiding this comment.
lets still run the properties here, just run them with the smaller max runs value.
There was a problem hiding this comment.
TBH I disagree - this will keep the havoc with proptests spuriously breaking random PR unit tests
|
Just need to run the formatter i think and then looks good |
done |
Add a dedicated GitHub Actions workflow for property tests so they no longer run in the regular CI test job. The new workflow runs daily at 00:00 UTC. It automatically creates an issue when property tests fail (including run metadata and log tail). Using a random seed so scheduled executions explore different generated inputs over time. Special consideration: automatic issue creation might be noisy. We need to see how this behaves in time. fix elixir-tools#106
Remove timeout setting for proptest job in workflow
Replace hardcoded property-test max_runs values with a shared option that reads SPITFIRE_PROPERTY_MAX_RUNS. Local/default behavior is unchanged at 1000 runs when the env var is not set. Invalid values now fail fast with a clear ArgumentError. Set SPITFIRE_PROPERTY_MAX_RUNS=250000 in the scheduled proptest workflow so CI runs deeper property checks while regular local runs stay fast.
Use String.to_integer/1 for SPITFIRE_PROPERTY_MAX_RUNS and remove additional positivity validation to keep the env parsing path minimal.
Inline SPITFIRE_PROPERTY_MAX_RUNS and default value in a single module-attribute expression using System.get_env/2 and String.to_integer/1.
mhanberg
force-pushed
the
mkaput/ci-proptest-106
branch
from
c055440 to
8c118ce
Compare
2 participants
Add a dedicated GitHub Actions workflow for property tests so they no longer run in the regular CI test job. The new workflow runs daily at 00:00 UTC. It automatically creates an issue when property tests fail (including run metadata and log tail). Using a random seed so scheduled executions explore different generated inputs over time.
Special consideration: automatic issue creation might be noisy. We need to see how this behaves in time.
fix #106