Skip to content

test: run every script in smoke, not a 10-of-15 allowlist - #43

Merged
Jammy2211 merged 2 commits into
mainfrom
feature/howto-smoke-all-tutorials
Aug 4, 2026
Merged

test: run every script in smoke, not a 10-of-15 allowlist#43
Jammy2211 merged 2 commits into
mainfrom
feature/howto-smoke-all-tutorials

Conversation

@Jammy2211

Copy link
Copy Markdown
Contributor

Summary

Replaces the smoke_tests.txt allowlist with opt-out coverage: smoke now runs every script under scripts/, minus the exclusions in config/build/no_run.yaml.

Under the allowlist a script was tested only if someone remembered to add it, so every new tutorial was uncovered from birth:

repo covered total
HowToGalaxy 4 26
HowToLens 6 40
HowToFit 10 15

That is how a public teaching notebook stayed broken in three places while CI was green (#56 / #57) — no job had ever executed it. The only backstop is PyAutoHeart's workspace-smoke, which runs weekly, and the run that caught it was a manual workflow_dispatch, not the schedule.

Design

run_smoke.py becomes a thin shim over PyAutoHands' autohands/run_python.py — the same entry point PyAutoHeart's workspace-validation uses for its run_scripts job. Discovery, exclusion and env resolution are no longer reimplemented per-repo, so the PR gate and the validation runner cannot drift apart, and both read one exclusion list (config/build/no_run.yaml, which the notebook runner already honoured).

The delegated runner also orders simulator* first and then start_here.py, which is what tutorials depending on simulated datasets need.

--report-dir is passed deliberately, not cosmetically. run_python.py only propagates failures (sys.exit(1)) when a report was built; without it the suite runs to completion and always exits 0 — a vacuously green gate, strictly worse than the allowlist. It also switches execute_script from abort-on-first-failure to record-and-continue, matching the old runner's behaviour. The negative control below is what proves this.

Depends on

PyAutoLabs/PyAutoHands#225execute_script() silently rewrote any failure to PASS for scripts with inversion in their path. Without it, chapter_4_pixelizations/tutorial_3_inversions.py would be unconditionally green in this newly-expanded suite, defeating the purpose.

Scripts Changed

  • .github/scripts/run_smoke.py — delegates to run_python.py
  • smoke_tests.txtdeleted
  • AGENTS.md, .gitignore

config/build/no_run.yaml already exists and parses to an empty list, so there is nothing to exclude here and it is unchanged.

Test Plan

  • Executed-script count 10 → 15 (report JSON: 15 passed, 0 failed, 0 skipped), EXIT=0
  • start_here.py passes in ~0.0s — verified prose-only (0 non-docstring statements), legitimate not vacuous
  • Smoke tests pass for all affected workspaces

Generated by the PyAutoLabs agent workflow.

smoke_tests.txt was an allowlist: a script was tested only if someone
remembered to add it, so every new tutorial was uncovered from birth.
Coverage was 4/26 in HowToGalaxy, 6/40 in HowToLens and 10/15 in
HowToFit. That is how a public teaching notebook stayed broken in three
places with CI green (HowToGalaxy #56/#57) - no job had ever executed it.
The only backstop is PyAutoHeart's workspace-smoke, which runs weekly.

Coverage is now opt-OUT. run_smoke.py becomes a thin shim over
PyAutoHands' autohands/run_python.py - the same entry point Heart's
workspace-validation uses for run_scripts - so the PR gate and the
validation runner cannot drift apart, and both read one exclusion list
(config/build/no_run.yaml, which the notebook runner already honoured).
Discovery, exclusion and env resolution are no longer reimplemented here.

--report-dir is passed deliberately, not cosmetically: run_python.py only
propagates failures when a report was built, and without it the suite
runs to completion and always exits 0. It also switches execute_script
from abort-on-first-failure to record-and-continue, matching the old
runner's behaviour. A negative control confirms a deliberate failure
turns the suite red.

Executed-script counts, measured, not assumed:
  HowToGalaxy   4 -> 26
  HowToLens     6 -> 39 (+1 deliberately skipped)
  HowToFit     10 -> 15

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@Jammy2211 Jammy2211 added the pending-release PR queued for the next release build label Aug 4, 2026
…ous commit)

The previous commit deleted smoke_tests.txt but, due to a stale pathspec in
the `git add` invocation, staged none of the accompanying changes - so it
removed the allowlist while leaving the old runner that requires it, and CI
failed with "ERROR: no smoke_tests.txt". This commit carries the work that
should have been in it.

run_smoke.py becomes a thin shim over PyAutoHands' autohands/run_python.py,
the same entry point Heart's workspace-validation uses for run_scripts, so
the PR gate and the validation runner share one code path and one exclusion
list (config/build/no_run.yaml). --report-dir is passed deliberately:
run_python.py only propagates failures when a report was built, and without
it the suite always exits 0.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@Jammy2211
Jammy2211 merged commit d0cc2dc into main Aug 4, 2026
6 checks passed
@Jammy2211
Jammy2211 deleted the feature/howto-smoke-all-tutorials branch August 4, 2026 15:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

pending-release PR queued for the next release build

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant