fix: simulator notebooks cannot `import util` — move helpers into `af.ex.util`

[Jammy2211]

Overview

Four workspace-smoke notebook failures share one cause, ModuleNotFoundError: No module named 'util' (PyAutoHeart workspace-smoke run 30790463134): simulators.ipynb and simulators_sample.ipynb under notebooks/simulators/ in both autofit_workspace and HowToFit. This is not a cwd bug and not a regression of the notebook kernel-cwd fix — import util fails under both the old kernel cwd (notebooks/simulators/) and the new pinned one (workspace root), reproduced directly. The .py siblings pass only because running a script puts its own directory on sys.path[0].

The fix removes the local-module trap entirely: move the four simulate helpers into PyAutoFit's existing af.ex.util, delete both duplicated 346-line workspace util.py copies, and call af.ex.util.* from the four scripts. That works identically for scripts, notebooks and Colab.

Plan

1. Move the four simulate helpers from the two duplicated workspace util.py copies into PyAutoFit's existing af.ex.util, and export them from af.ex.
2. Delete both 346-line workspace util.py files — which also stops generate.py emitting the unusable util.ipynb.
3. Rewrite the four simulator scripts to call af.ex.util.* instead of import util.
4. Fix a second, currently-masked failure in HowToFit/scripts/simulators/simulators.py (__file__ is undefined in a notebook kernel).
5. Regenerate notebooks and verify all four execute end-to-end.
6. Ship library first, then the two workspaces behind the pending-release merge gate.

Detailed implementation plan

Work Classification

Both — library (PyAutoFit) first, then workspaces (autofit_workspace, HowToFit).

Affected Repositories

• PyAutoFit (primary)
• autofit_workspace
• HowToFit

Branch Survey

Repository	Current Branch	Dirty?
./PyAutoFit	main	clean
./autofit_workspace	main	clean
./HowToFit	main	clean

worktree_check_conflict reported CONFLICT on PyAutoFit from point-source-defaults-campaign and nautilus-1core-serial-pool. Both are stale: PR #1441 merged 2026-08-01 12:47 and PR #1443 merged 19:12; PyAutoFit's checkout is on main and clean, and that task's PyAutoLabs-wt/.../PyAutoFit is a symlink to the canonical checkout, not a live worktree. Treated as clear.

Suggested branch: feature/simulator-util-to-af-ex
Worktree root: ~/Code/PyAutoLabs-wt/simulator-util-to-af-ex/

Diagnosis (settled, reproduced)

The real asymmetry is script dir vs cwd: python scripts/simulators/simulators.py puts scripts/simulators/ on sys.path[0] and resolves scripts/simulators/util.py, which is why the .py siblings pass. A notebook kernel has no script dir, sys.path[0] is the cwd, and util.py exists only under scripts/ — so no cwd in the tree resolves it.

notebooks/simulators/util.py used to exist as a hand-maintained duplicate (git show 73aab12:notebooks/simulators/util.py, 326 lines) but cannot survive: PyAutoHands/autohands/generate.py:131 does shutil.rmtree(WORKSPACE_PATH / "notebooks") on every build and then copies only .ipynb, .rst and .md — no .py is ever copied into notebooks/. Restoring the copy would not fix it anyway now that the kernel cwd is pinned to the workspace root.

Sweep complete. Exactly these four notebooks — a grep across every .ipynb in the workspace for local (non-package) module imports returns only these two import util sites in two repos. Only four source files import it. util.py exists in only these two repos, byte-identical apart from four docstring path strings.

Implementation Steps

PyAutoFit — autofit/example/util.py, autofit/example/__init__.py

1. Append simulate_dataset_1d_via_gaussian_from, simulate_data_1d_with_kernel_via_gaussian_from, simulate_dataset_1d_via_profile_1d_list_from, simulate_data_1d_with_kernel_via_profile_1d_list_from to util.py; add the four names to example/__init__.py. Purely additive — nothing removed.
2. matplotlib must stay an in-function import. It is not in PyAutoFit's requirements.txt and the existing plot_profile_1d already defers it (autofit/example/util.py:34). The workspace copies import it at module scope; hoisting that would make matplotlib a hard import of autofit.
3. from autofit import to_dict cannot be used at module scope here. autofit/__init__.py:125 does from . import example as ex but to_dict is not bound until line 172, so the from-import would hit a partially initialised module. Use from autonerves.dictable import to_dict (the real source line 172 re-exports). af.util.numpy_array_to_json is fine via import autofit as af — attribute lookup happens at call time, the example/analysis.py precedent.
4. Add a unit test that each helper writes data.json / noise_map.json into a tmp_path. example/util.py currently has zero test coverage.

autofit_workspace + HowToFit — scripts/simulators/simulators.py, simulators_sample.py

5. Delete scripts/simulators/util.py in both repos.
6. Replace import util with af.ex.util.* usage. simulators_sample.py is byte-identical across the two repos; simulators.py is not.
7. HowToFit only — the masked second bug. Its simulators.py ends with runpy.run_path(path.join(path.dirname(path.abspath(__file__)), "simulators_sample.py")), which survives into notebooks/simulators/simulators.ipynb (cell 41). __file__ is undefined in a notebook kernel, and no .py exists in the notebooks tree anyway — so that notebook fails a second time the moment the util import is fixed. Replace with the root-relative path.join("scripts", "simulators", "simulators_sample.py"), which resolves identically for the script (run from the repo root by convention), the notebook (kernel cwd pinned to root) and Colab (autonerves/setup_colab.py:250 chdirs to root).

Verification

• PyAutoFit unit tests.
• Regenerate notebooks and execute all four via the same runner smoke uses, asserting they pass rather than inspecting the diff.
• Re-run the sweep on the regenerated output to confirm no import util and no __file__ remain in any notebook.

Key Files

• PyAutoFit/autofit/example/util.py — destination for the four helpers
• PyAutoFit/autofit/example/__init__.py — export the four names
• autofit_workspace/scripts/simulators/util.py — deleted
• HowToFit/scripts/simulators/util.py — deleted
• autofit_workspace/scripts/simulators/{simulators,simulators_sample}.py — call af.ex.util.*
• HowToFit/scripts/simulators/{simulators,simulators_sample}.py — call af.ex.util.*; plus the __file__ fix

Brain sizing OVERRIDE (recorded per policy)

The Feature Agent returned too-large (score 13) → split-into-4-phases. The factor breakdown contains no component that measures the change: 3 repos (+6), library+workspace (+2), prompt length (+2), and keyword hits on trap, cross-repo, smoke/regression taken from the prompt's own diagnosis prose (architectural_risk is literally the string "cross-repo"). The real change is four functions moved, two files deleted, four scripts edited — uniform per file, additive API only, nothing removed. Shipped as one task with one small PR per repo, per the documented override precedent.

Original Prompt

Click to expand starting prompt

Simulator notebooks cannot import util — move the helpers into af.ex.util

Type: bug
Target: autofit
Repos:

• PyAutoFit
• autofit_workspace
• HowToFit
  Difficulty: small
  Autonomy: safe
  Priority: normal
  Status: formalised

Four workspace-smoke notebook failures share one cause,
ModuleNotFoundError: No module named 'util':
autofit_workspace/notebooks/simulators/simulators.ipynb,
autofit_workspace/notebooks/simulators/simulators_sample.ipynb,
HowToFit/notebooks/simulators/simulators.ipynb,
HowToFit/notebooks/simulators/simulators_sample.ipynb.
Evidence: PyAutoHeart workspace-smoke run 30790463134.

Diagnosis (done — this is NOT a cwd bug and NOT a regression of the
notebook kernel-cwd fix). import util fails under both the old kernel cwd
(notebooks/simulators/) and the new pinned one (workspace root); reproduced
directly with python -c "import util" from each. The real asymmetry is
script dir vs. cwd: python scripts/simulators/simulators.py puts
scripts/simulators/ on sys.path[0] and so resolves
scripts/simulators/util.py — which is why the .py siblings pass. A notebook
kernel has no script dir, sys.path[0] is the cwd, and util.py exists only
under scripts/, so no cwd in the tree resolves it.

notebooks/simulators/util.py used to exist as a hand-maintained duplicate
(git show 73aab12:notebooks/simulators/util.py, 326 lines). It cannot survive:
PyAutoHands/autohands/generate.py:131 does
shutil.rmtree(WORKSPACE_PATH / "notebooks") on every build and then copies only
.ipynb, .rst and .md — no .py is ever copied into notebooks/.
Restoring the copy would not fix it anyway now that the kernel cwd is pinned to
the workspace root.

Sweep (complete). Exactly these four notebooks — a grep across every
.ipynb in the workspace for local (non-package) module imports returns only
these two import util sites in two repos. Only four source files import it
(scripts/simulators/simulators{,_sample}.py × 2 repos). util.py exists in
only these two repos and the two copies are byte-identical apart from four
docstring path strings (autofit_workspace/dataset vs HowToFit/dataset).
generate.py also converts util.py into a util.ipynb of bare function defs
that nothing can import.

Approved fix (human-chosen). Move the four simulate functions
(simulate_dataset_1d_via_gaussian_from,
simulate_data_1d_with_kernel_via_gaussian_from,
simulate_dataset_1d_via_profile_1d_list_from,
simulate_data_1d_with_kernel_via_profile_1d_list_from) into PyAutoFit's
existing af.ex.util (autofit/example/util.py, already home to
plot_profile_1d), delete both 346-line workspace util.py copies, and call
af.ex.util.* from the four scripts. This removes the local-module trap
entirely — it works for scripts, notebooks and Colab alike
(autonerves/setup_colab.py:250 chdirs to the workspace root, matching the
pinned kernel cwd) — kills the cross-repo duplicate, and removes the useless
generated util.ipynb.

Masked second failure (found while planning). HowToFit/scripts/simulators/ simulators.py ends with a runpy.run_path(path.join(path.dirname( path.abspath(__file__)), "simulators_sample.py")) chain-run that survives into
HowToFit/notebooks/simulators/simulators.ipynb (cell 41). __file__ is
undefined in a notebook kernel, and no .py exists in the notebooks tree
anyway — so that notebook fails a second time the instant the util import is
fixed. Replace with the root-relative path.join("scripts", "simulators", "simulators_sample.py"), which resolves identically for the script (run from
the repo root by convention), the notebook (kernel cwd pinned to root) and
Colab. Note simulators_sample.py is byte-identical across the two repos but
simulators.py is not — this trailing block is HowToFit-only.

Two library-side constraints verified (a naive copy gets both wrong).

1. matplotlib must stay an in-function import. It is not in PyAutoFit's
   requirements.txt and the existing plot_profile_1d already defers it
   (autofit/example/util.py:34); the workspace copies import it at module
   scope, and hoisting that would make matplotlib a hard import of autofit.
2. from autofit import to_dict cannot be used at module scope here.
   autofit/__init__.py:125 does from . import example as ex but to_dict
   is not bound until line 172, so the from-import would hit a partially
   initialised module. Use from autonerves.dictable import to_dict (the real
   source line 172 re-exports). af.util.numpy_array_to_json is fine via
   import autofit as af — attribute lookup happens at call time, the
   example/analysis.py precedent.

autofit/example/util.py currently has no test coverage; add a unit test that
each helper writes data.json / noise_map.json into a tmp_path.

Brain sizing OVERRIDDEN (recorded per policy). The Feature Agent returned
too-large (score 13) → split-into-4-phases. The factor breakdown contains no
component that measures the change: 3 repos (+6), library+workspace (+2),
prompt length (+2), and keyword hits on trap, cross-repo,
smoke/regression taken from this prompt's own diagnosis prose
(architectural_risk is literally the string "cross-repo"). The real change is
four functions moved, two files deleted, four scripts edited — uniform per file,
additive API only, nothing removed. Shipped as one task with one small PR per
repo.

Library first (PyAutoFit), then the two workspaces behind the pending-release
merge gate.

[Jammy2211]

Library merged; workspace PRs parked behind the release gate.

• fix: move 1D simulator helpers into af.ex.util (#1444) #1445 — MERGED 2026-08-03T17:36Z (squashed). All CI green: docs, unittest 3.12, unittest 3.13.
• fix: simulators call af.ex.util instead of a local util module (PENDING RELEASE) autofit_workspace#130 — open, labelled pending-release, fully green (smoke 3.12 + 3.13, navigator paths, catalogue staleness).
• fix: simulators call af.ex.util, and the chain-run resolves in a notebook (PENDING RELEASE) HowToFit#42 — open, labelled pending-release, fully green (smoke 3.12 + 3.13, navigator paths, catalogue staleness, tutorials-complete).

The two workspace PRs need no further work. They call af.ex.util, which exists only on PyAutoFit main — not in any release yet (latest is v2026.7.23.1). Merging them before a release ships would break the .py simulator scripts, which currently work, for anyone on the released library. That is the whole reason for the gate here: the notebooks are already broken, but the scripts are not, and merging early would trade one broken path for two.

Their CI is nonetheless meaningful: workspace smoke resolves each PyAuto library from the same-named branch, so both smokes ran against the new af.ex.util and passed on 3.12 and 3.13.

Repo claims released. All three claims are discharged in PyAutoMind/active.md so nothing else queues behind this. The branches are pushed and green; nothing local is required to merge them. Re-attach with worktree_add_repo simulator-util-to-af-ex <repo> if either PR ever needs an update.

On the release itself — Heart is not release-ready as of 2026-08-03T17:19Z, for reasons unrelated to this task: INSTALL VERIFY FAILED (testpypi; D) and RELEASE VALIDATION integrate:fail. Both are separately tracked. Worth noting this task moves that needle rather than waiting on it: it removes 2 of the 19 workspace-validation failures currently feeding the RED (autofit notebooks/simulators/simulators.ipynb and simulators_sample.ipynb).

[Jammy2211]

Closed out — 2026-08-04

All three PRs merged on 2026-08-03 — PyAutoFit#1445 (17:36:47Z),
autofit_workspace#130 (21:40:46Z), HowToFit#42 (21:40:50Z) — and this issue
auto-closed at merge. The PyAutoMind task entry was never advanced, so closing it
out now. Completion record:
PyAutoMind/complete/2026/08/simulator-util-to-af-ex.md.

Not closed with it: the release drive this task opened is still unfinished
and has been split into its own live task (release-drive-2026-08-03) rather
than buried in the completion record. Stage 2/3 are clear to re-run — both
blockers are gone (autolens_workspace#461 merged 2026-08-03T21:41:40Z;
PyAutoHeart#134 squash-merged 2026-08-04 as 46a331a) — and the PUBLISH step
remains a separate human decision. Note the rehearsal's commit_shas.json is now
stale: main has moved in PyAutoFit (8f706c51) and autofit_workspace_test
(f4c45c1) since, so a re-rehearsal must re-derive them.

That release is still owed: workspace main calls af.ex.util helpers that are on
PyAutoFit main but not yet on PyPI, which the control test proved breaks a user
on a released install (AttributeError on the first simulator call against
2026.7.29.2, and HowToFit ships no datasets at all).