test: smoke should run every tutorial, not a 4-of-26 allowlist

[Jammy2211]

Overview

Shipping #57 revealed that a public teaching notebook was broken in three places and this repo's CI could never have caught any of them. smoke_tests.txt is an allowlist — a script is tested only if someone remembers to add it — so every new tutorial is uncovered from birth.

repo	covered	total
HowToGalaxy	4	26
HowToLens	6	40
HowToFit	10	15

The only backstop is PyAutoHeart's workspace-smoke, which is weekly (Mondays 03:00 UTC) — and the run that caught #57 was a manual workflow_dispatch, not the schedule. A broken public tutorial can sit on main for a week.

The allowlist also concealed a false claim: profile_smoke.yaml caps datasets and its comment asserts "the chapters run correctly at 16x16". That is untrue for chapter 4, and nobody knew because chapter 4 was never in the list.

Plan

• Ship a one-line mesh-shape fix first and on its own — without it Heart's next weekly run stays red (it flips from TypeError to IndexError).
• Delete the allowlist and delegate to PyAutoHands' canonical run_python.py, so PR-smoke and Heart's validation become one code path reading one exclusion list.
• Clean up two stale/mis-stated no_run.yaml exclusions.
• Correct the AGENTS.md claim that contradicts the smoke profile.
• Verify by the executed-script count rising, plus a negative control — a green tick already passes today while testing 4 of 26 files.

Detailed implementation plan

Work Classification

Workspace (three tutorial repos; no library change).

Affected Repositories

• HowToGalaxy (primary)
• HowToLens
• HowToFit

Branch Survey

Repository	Current Branch	Dirty?
./HowToGalaxy	main	clean
./HowToLens	main	clean
./HowToFit	main	clean

No worktree claims → no conflict.

Suggested branch: feature/howto-smoke-all-tutorials
Worktree root: ~/Code/PyAutoLabs-wt/howto-smoke-all-tutorials/

Measured (2026-08-04, every script under each repo's own smoke profile)

repo	result	wall time
HowToGalaxy	25/26 → 26/26 with the mesh fix	3.9 min
HowToLens	39/40 (the 1 is already excluded)	6.4 min
HowToFit	15/15	0.9 min

Cost is not the blocker — HowToGalaxy spends ~2m18s today on 4 scripts.

Phase 1 — mesh shape (urgent, ships first)

scripts/chapter_4_pixelizations/tutorial_3_inversions.py:82 uses shape=dataset.shape_native, coupling the pixelization mesh to image resolution: 10000 mesh pixels at full res, 256 under the cap. The tutorial's pix_indexes = [[445], …] (line 140) then goes out of range → IndexError.

The indices are not the bug — they match the 625-pixel mesh of HowToLens/.../tutorial_3_inversions.py:102 (shape=(25, 25)) they were copied from, and this file's own line 176 already uses shape=(25, 25). Line 82 is the anomaly.

- mesh = ag.mesh.RectangularAdaptDensity(shape=dataset.shape_native)
+ mesh = ag.mesh.RectangularAdaptDensity(shape=(25, 25))

Verified: passes under the smoke profile, 19× faster there (8.2s vs 156.7s).

Phase 2 — delete the allowlist, delegate to the canonical runner

Do not hand-roll discovery. PyAutoHands/autohands/run_python.py is already the canonical script runner Heart's run_scripts invokes (workspace-validation.yml:313,343), and already does all of it:

• build_util.find_scripts_in_folder() (build_util.py:449) — recursive, orders simulator* first then start_here.py, exactly what dataset-dependent tutorials need
• should_skip() against config/build/no_run.yaml — the denylist
• find_profile() + build_env_for_script() — the same env resolution
• skips __init__/README as infrastructure

Each repo's .github/scripts/run_smoke.py collapses to a thin shim invoking run_python.py <project> scripts/; smoke_tests.txt is deleted. smoke_tests.yml keeps calling run_smoke.py, so PyAutoHeart's reusable-workflow contract is untouched.

Trap. run_python.py exits non-zero only when --report-dir is passed (gated on if report is not None). Without it the runner executes everything and always exits 0 — a vacuously green CI, strictly worse than today. The shim must always pass --report-dir, and this is what the negative control below proves.

Exclusion hygiene

• tutorial_searches — excluded in both HowToGalaxy and HowToLens with no stated reason, and passes in both (10.4s / 10.1s). Remove.
• HowToLens tutorial_5_borders — recorded as "Cant get right masks". Controlled re-test on identical dataset files: fails with the cap, passes without it (IndexError: index 371 out of bounds for axis 0 with size 272). Cap-induced; keep the exclusion but correct the reason and tag NEEDS_FIX.
• HowToFit no_run.yaml exists and parses to an empty list — nothing to do.

Docs contradiction

AGENTS.md:34 says PYAUTO_SMALL_DATASETS is "deliberately not used in HowToGalaxy", while profile_smoke.yaml:16 sets it for every script. The cap stays (human decision), so AGENTS.md is what is wrong — correct it and drop the disproven "16x16" assertion from the profile comment.

Key Files

• */.github/scripts/run_smoke.py — the triplicated runner (byte-identical, md5 f105e0e8)
• */smoke_tests.txt — deleted
• */config/build/no_run.yaml — the surviving single exclusion list
• HowToGalaxy/scripts/chapter_4_pixelizations/tutorial_3_inversions.py:82 — Phase 1
• PyAutoHands/autohands/run_python.py, build_util.py:449 — reused, not modified

Verification

1. Executed count per repo: 4 → 26, 6 → 39, 10 → 15, read off the runner's output and checked against discovered-minus-no_run.
2. Negative control — break a tutorial, confirm CI goes red. Catches the --report-dir trap.
3. Add a throwaway tutorial, confirm it runs with no list edit.
4. Four scripts pass in ~0.0s; verified prose-only (0 non-docstring statements), legitimate not vacuous.

Follow-up, deliberately not here

run_smoke.py is triplicated. Centralising it into PyAutoHands would mean the next change lands once — scoped out by the human.

Original Prompt

Click to expand starting prompt

I think its CI should check all tutorials: That coverage gap is arguably the more interesting finding: a public teaching notebook was broken in three places and its own repo's CI could never have caught any of them.

[Jammy2211]

Shipped and merged

Repo	PR	Merged
PyAutoHands	#225	✅
HowToGalaxy	#59 (mesh fix) · #60	✅
HowToLens	#66	✅
HowToFit	#43	✅

Coverage, read from the CI job logs rather than a green tick — green already passed while testing 4 of 26 files:

repo	before	after
HowToGalaxy	4	26 pass, 0 fail
HowToLens	6	39 pass, 0 fail (3.12 and 3.13)
HowToFit	10	15 pass, 0 fail

The blocker that nearly made this worthless

build_util.execute_script() contained if "inversion" in f: — a match on the script path, not the exception — that rewrote any failure to PASSED. On a file containing only raise SystemExit, main recorded PASSED / has_failures=False; with the escape removed, FAILED / True. Two of the three matching scripts workspace-wide are the HowToGalaxy and HowToLens tutorial_3_inversions.py, so it covered exactly the tutorial from #56/#57. The notebook runner never had the clause — which is why the notebook job reported that failure and a script run never would have. Removed in PyAutoHands#225.

What the expansion caught within minutes

On the 3.13 leg, two HowToLens chapter 4 tutorials failed with ModuleNotFoundError: No module named 'jax' — smoke_install.sh installs the [optional] extras only on 3.12, the 3.13 leg deliberately exercising the lean path. Neither was in the old 6-script allowlist. Fixed with the workspace's established find_spec → sys.exit(0) idiom, which is_clean_skip_exit() already recognises.

Also corrected

• tutorial_searches exclusion removed from two repos — no stated reason, passes in both
• HowToLens tutorial_5_borders recorded as "Cant get right masks"; on identical dataset files it fails with the cap and passes without. Cap-induced, now tagged NEEDS_FIX
• AGENTS.md claimed PYAUTO_SMALL_DATASETS is "deliberately not used" while the profile sets it for every script

Left for a human decision

execute_notebooks_in_folder has if "InversionException" in traceback → PASS — same species as the escape removed here, but keyed on exception type rather than filename, so removing it could redden Heart on genuinely data-dependent failures. Deliberately untouched.

Worth a follow-up: run_smoke.py is now identical across the three repos and could be centralised into PyAutoHands so the next change lands once.

Completion record: PyAutoMind/complete/2026/08/howto-smoke-all-tutorials.md.