test: run every script in smoke, not a 10-of-15 allowlist - #43
Merged
Conversation
smoke_tests.txt was an allowlist: a script was tested only if someone remembered to add it, so every new tutorial was uncovered from birth. Coverage was 4/26 in HowToGalaxy, 6/40 in HowToLens and 10/15 in HowToFit. That is how a public teaching notebook stayed broken in three places with CI green (HowToGalaxy #56/#57) - no job had ever executed it. The only backstop is PyAutoHeart's workspace-smoke, which runs weekly. Coverage is now opt-OUT. run_smoke.py becomes a thin shim over PyAutoHands' autohands/run_python.py - the same entry point Heart's workspace-validation uses for run_scripts - so the PR gate and the validation runner cannot drift apart, and both read one exclusion list (config/build/no_run.yaml, which the notebook runner already honoured). Discovery, exclusion and env resolution are no longer reimplemented here. --report-dir is passed deliberately, not cosmetically: run_python.py only propagates failures when a report was built, and without it the suite runs to completion and always exits 0. It also switches execute_script from abort-on-first-failure to record-and-continue, matching the old runner's behaviour. A negative control confirms a deliberate failure turns the suite red. Executed-script counts, measured, not assumed: HowToGalaxy 4 -> 26 HowToLens 6 -> 39 (+1 deliberately skipped) HowToFit 10 -> 15 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…ous commit) The previous commit deleted smoke_tests.txt but, due to a stale pathspec in the `git add` invocation, staged none of the accompanying changes - so it removed the allowlist while leaving the old runner that requires it, and CI failed with "ERROR: no smoke_tests.txt". This commit carries the work that should have been in it. run_smoke.py becomes a thin shim over PyAutoHands' autohands/run_python.py, the same entry point Heart's workspace-validation uses for run_scripts, so the PR gate and the validation runner share one code path and one exclusion list (config/build/no_run.yaml). --report-dir is passed deliberately: run_python.py only propagates failures when a report was built, and without it the suite always exits 0. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Replaces the
smoke_tests.txtallowlist with opt-out coverage: smoke now runs every script underscripts/, minus the exclusions inconfig/build/no_run.yaml.Under the allowlist a script was tested only if someone remembered to add it, so every new tutorial was uncovered from birth:
That is how a public teaching notebook stayed broken in three places while CI was green (#56 / #57) — no job had ever executed it. The only backstop is PyAutoHeart's
workspace-smoke, which runs weekly, and the run that caught it was a manualworkflow_dispatch, not the schedule.Design
run_smoke.pybecomes a thin shim over PyAutoHands'autohands/run_python.py— the same entry point PyAutoHeart's workspace-validation uses for itsrun_scriptsjob. Discovery, exclusion and env resolution are no longer reimplemented per-repo, so the PR gate and the validation runner cannot drift apart, and both read one exclusion list (config/build/no_run.yaml, which the notebook runner already honoured).The delegated runner also orders
simulator*first and thenstart_here.py, which is what tutorials depending on simulated datasets need.Depends on
PyAutoLabs/PyAutoHands#225 —
execute_script()silently rewrote any failure to PASS for scripts withinversionin their path. Without it,chapter_4_pixelizations/tutorial_3_inversions.pywould be unconditionally green in this newly-expanded suite, defeating the purpose.Scripts Changed
.github/scripts/run_smoke.py— delegates torun_python.pysmoke_tests.txt— deletedAGENTS.md,.gitignoreconfig/build/no_run.yamlalready exists and parses to an empty list, so there is nothing to exclude here and it is unchanged.Test Plan
EXIT=0start_here.pypasses in ~0.0s — verified prose-only (0 non-docstring statements), legitimate not vacuousGenerated by the PyAutoLabs agent workflow.