Skip to content

Commit 62a5f3a

Browse files
Jammy2211Jammy2211claude
authored
feat: bootstrap HowToLens from autolens_workspace tutorials (#1)
Extract the howtolens tutorial lecture series into its own repo. Mirrors the autolens_workspace/autogalaxy_workspace/autofit_workspace layout: scripts/chapter_1_introduction … chapter_4_pixelizations scripts/simulator/ (runtime dataset generation) notebooks/ (auto-generated from scripts) config/ (PyAutoLens config files) smoke_tests.txt (subset of chapter 1 for CI) Datasets are generated at runtime — no .fits committed. Tutorials that need pre-existing data auto-invoke their simulator; see tutorial 7's subprocess pattern. CI: .github/workflows/smoke_tests.yml Python 3.12 + 3.13 matrix, checks out all PyAutoLabs libraries, runs the smoke list via run_smoke.py. Slack notify on failure. .github/workflows/url_check.yml copied verbatim from autolens_workspace. Two chapter 1 tutorials are intentionally excluded from the initial smoke list pending a content-alignment pass (documented in smoke_tests.txt): - tutorial_0_visualization — missing Path import + needs pre-simulated dataset that HowToLens does not ship. - tutorial_7_fitting — second dataset load at line ~615 has no auto-sim fallback. Both issues pre-exist upstream in autolens_workspace. Follow-ups tracked on issue #78: 1. This bootstrap PR. 2. howtolens-workspace-cleanup — remove scripts/howtolens/ and notebooks/howtolens/ from autolens_workspace, update cross-refs. 3. howtolens-docs-update — update PyAutoLens docs toctree, paper.md, overview/*.rst to reference the new repo. 4. PyAutoBuild howtolens project target registration. Refs PyAutoLabs/autolens_workspace#78. Co-authored-by: Jammy2211 <JNightingale2211@gmail.com> Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
1 parent a8e31ec commit 62a5f3a

204 files changed

Lines changed: 37586 additions & 1 deletion

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

.github/scripts/run_smoke.py

Lines changed: 112 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,112 @@
1+
"""
2+
Run the workspace smoke test suite.
3+
4+
Reads `smoke_tests.txt` from the workspace root and `config/build/env_vars.yaml`
5+
for per-script env var overrides, then runs each listed script with the
6+
appropriate environment. Continues through failures and exits non-zero
7+
if any script failed.
8+
9+
Mirrors the logic of the `/smoke-test` skill so CI and local runs stay
10+
in sync.
11+
"""
12+
13+
from __future__ import annotations
14+
15+
import os
16+
import subprocess
17+
import sys
18+
import time
19+
from pathlib import Path
20+
21+
import yaml
22+
23+
24+
WORKSPACE = Path(__file__).resolve().parents[2]
25+
SMOKE_FILE = WORKSPACE / "smoke_tests.txt"
26+
ENV_VARS_FILE = WORKSPACE / "config" / "build" / "env_vars.yaml"
27+
SCRIPTS_DIR = WORKSPACE / "scripts"
28+
29+
30+
def load_smoke_scripts() -> list[str]:
31+
scripts: list[str] = []
32+
for line in SMOKE_FILE.read_text().splitlines():
33+
line = line.strip()
34+
if not line or line.startswith("#"):
35+
continue
36+
scripts.append(line)
37+
return scripts
38+
39+
40+
def load_env_config() -> dict:
41+
if not ENV_VARS_FILE.exists():
42+
return {"defaults": {}, "overrides": []}
43+
return yaml.safe_load(ENV_VARS_FILE.read_text()) or {}
44+
45+
46+
def pattern_matches(pattern: str, script_path: str) -> bool:
47+
if "/" in pattern:
48+
return pattern in script_path
49+
return Path(script_path).stem == pattern
50+
51+
52+
def build_env(script_rel: str, cfg: dict) -> dict:
53+
env = os.environ.copy()
54+
defaults = cfg.get("defaults") or {}
55+
env.update({k: str(v) for k, v in defaults.items()})
56+
for override in cfg.get("overrides") or []:
57+
if pattern_matches(override["pattern"], script_rel):
58+
for key in override.get("unset", []):
59+
env.pop(key, None)
60+
for key, val in (override.get("set") or {}).items():
61+
env[key] = str(val)
62+
return env
63+
64+
65+
def run_one(script_rel: str, cfg: dict) -> tuple[str, int, float, str]:
66+
env = build_env(script_rel, cfg)
67+
script_path = SCRIPTS_DIR / script_rel
68+
t0 = time.time()
69+
result = subprocess.run(
70+
[sys.executable, str(script_path)],
71+
cwd=str(WORKSPACE),
72+
env=env,
73+
capture_output=True,
74+
text=True,
75+
)
76+
elapsed = time.time() - t0
77+
output = result.stdout + result.stderr
78+
return script_rel, result.returncode, elapsed, output
79+
80+
81+
def main() -> int:
82+
if not SMOKE_FILE.exists():
83+
print(f"ERROR: no smoke_tests.txt at {SMOKE_FILE}", file=sys.stderr)
84+
return 1
85+
scripts = load_smoke_scripts()
86+
if not scripts:
87+
print("No smoke test scripts listed.")
88+
return 0
89+
cfg = load_env_config()
90+
91+
print(f"Running {len(scripts)} smoke test script(s) from {SMOKE_FILE.name}\n")
92+
failures: list[tuple[str, int, str]] = []
93+
for script_rel in scripts:
94+
print(f"::group::{script_rel}")
95+
name, rc, elapsed, output = run_one(script_rel, cfg)
96+
print(output, end="")
97+
status = "PASS" if rc == 0 else f"FAIL (exit {rc})"
98+
print(f"\n[{status}] {name}{elapsed:.1f}s")
99+
print("::endgroup::")
100+
if rc != 0:
101+
failures.append((name, rc, output))
102+
103+
total = len(scripts)
104+
passed = total - len(failures)
105+
print(f"\n=== Smoke test summary: {passed}/{total} passed ===")
106+
for name, rc, _ in failures:
107+
print(f" FAIL {name} (exit {rc})")
108+
return 0 if not failures else 1
109+
110+
111+
if __name__ == "__main__":
112+
sys.exit(main())

.github/workflows/smoke_tests.yml

Lines changed: 100 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,100 @@
1+
name: Smoke Tests
2+
3+
on: [push, pull_request]
4+
5+
jobs:
6+
smoke:
7+
runs-on: ubuntu-latest
8+
strategy:
9+
fail-fast: false
10+
matrix:
11+
python-version: ['3.12', '3.13']
12+
steps:
13+
- name: Checkout PyAutoConf
14+
uses: actions/checkout@v4
15+
with:
16+
repository: PyAutoLabs/PyAutoConf
17+
path: PyAutoConf
18+
- name: Checkout PyAutoFit
19+
uses: actions/checkout@v4
20+
with:
21+
repository: PyAutoLabs/PyAutoFit
22+
path: PyAutoFit
23+
- name: Checkout PyAutoArray
24+
uses: actions/checkout@v4
25+
with:
26+
repository: PyAutoLabs/PyAutoArray
27+
path: PyAutoArray
28+
- name: Checkout PyAutoGalaxy
29+
uses: actions/checkout@v4
30+
with:
31+
repository: PyAutoLabs/PyAutoGalaxy
32+
path: PyAutoGalaxy
33+
- name: Checkout PyAutoLens
34+
uses: actions/checkout@v4
35+
with:
36+
repository: PyAutoLabs/PyAutoLens
37+
path: PyAutoLens
38+
- name: Checkout HowToLens
39+
uses: actions/checkout@v4
40+
with:
41+
path: workspace
42+
- name: Set up Python ${{ matrix.python-version }}
43+
uses: actions/setup-python@v5
44+
with:
45+
python-version: ${{ matrix.python-version }}
46+
- name: Extract branch name
47+
id: extract_branch
48+
shell: bash
49+
run: |
50+
cd workspace
51+
echo "branch=${GITHUB_HEAD_REF:-${GITHUB_REF#refs/heads/}}" >> "$GITHUB_OUTPUT"
52+
- name: Match library branches
53+
shell: bash
54+
run: |
55+
BRANCH="${{ steps.extract_branch.outputs.branch }}"
56+
for PKG in PyAutoConf PyAutoFit PyAutoArray PyAutoGalaxy PyAutoLens; do
57+
pushd "$PKG"
58+
if [[ -n "$(git ls-remote --heads origin "$BRANCH")" ]]; then
59+
echo "Branch $BRANCH exists in $PKG — checking out"
60+
git fetch origin "$BRANCH"
61+
git checkout "$BRANCH"
62+
else
63+
echo "Branch $BRANCH not in $PKG — staying on main"
64+
fi
65+
popd
66+
done
67+
- name: Install dependencies
68+
run: |
69+
pip install --upgrade pip setuptools wheel
70+
pip install pyyaml
71+
if [ "${{ matrix.python-version }}" = "3.12" ]; then
72+
pip install ./PyAutoConf ./PyAutoFit ./PyAutoArray ./PyAutoGalaxy ./PyAutoLens
73+
pip install "./PyAutoArray[optional]" "./PyAutoGalaxy[optional]" "./PyAutoLens[optional]"
74+
else
75+
pip install ./PyAutoConf ./PyAutoFit ./PyAutoArray ./PyAutoGalaxy ./PyAutoLens
76+
pip install numba
77+
fi
78+
pip install tensorflow-probability==0.25.0
79+
- name: Prepare cache dirs
80+
run: |
81+
mkdir -p /tmp/numba_cache /tmp/matplotlib
82+
- name: Run smoke tests
83+
env:
84+
JAX_ENABLE_X64: "True"
85+
NUMBA_CACHE_DIR: /tmp/numba_cache
86+
MPLCONFIGDIR: /tmp/matplotlib
87+
run: |
88+
cd workspace
89+
python .github/scripts/run_smoke.py
90+
- name: Slack notify on failure
91+
if: ${{ failure() }}
92+
uses: slackapi/slack-github-action@v1.21.0
93+
env:
94+
SLACK_WEBHOOK_URL: ${{ secrets.SLACK_WEBHOOK_URL }}
95+
with:
96+
channel-id: C03S98FEDK2
97+
payload: |
98+
{
99+
"text": "${{ github.repository }}/${{ github.ref_name }} smoke tests (Python ${{ matrix.python-version }}) ${{ job.status }}\n${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}"
100+
}

.github/workflows/url_check.yml

Lines changed: 23 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,23 @@
1+
name: URL Check
2+
3+
on:
4+
push:
5+
branches: [main]
6+
pull_request:
7+
8+
jobs:
9+
url_check:
10+
runs-on: ubuntu-latest
11+
steps:
12+
- name: Checkout repo
13+
uses: actions/checkout@v4
14+
with:
15+
path: repo
16+
- name: Checkout PyAutoBuild
17+
uses: actions/checkout@v4
18+
with:
19+
repository: PyAutoLabs/PyAutoBuild
20+
ref: main
21+
path: PyAutoBuild
22+
- name: Run url_check.sh
23+
run: bash PyAutoBuild/autobuild/url_check.sh repo

.gitignore

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,12 @@
1+
root.log
2+
.idea/
3+
4+
__pycache__/
5+
*.pyc
6+
**/images/
7+
8+
output/
9+
dataset/
10+
notebooks/plot/
11+
test_report.md
12+
test_results/

CITATIONS.rst

Lines changed: 31 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,31 @@
1+
.. _references:
2+
3+
Citations & References
4+
======================
5+
6+
The bibtex entries for **PyAutoLens** and its affiliated software packages can be found
7+
`here <https://github.com/Jammy2211/PyAutoLens/blob/main/files/citations.bib>`_, with example text for citing **PyAutoLens**
8+
in `.tex format here <https://github.com/Jammy2211/PyAutoLens/blob/main/files/citations.tex>`_ format here and
9+
`.md format here <https://github.com/Jammy2211/PyAutoLens/blob/main/files/citations.md>`_. As shown in the examples, we
10+
would greatly appreciate it if you mention **PyAutoLens** by name and include a link to our GitHub page!
11+
12+
**PyAutoLens** is published in the `Journal of Open Source Software <https://joss.theoj.org/papers/10.21105/joss.02825#>`_ and its
13+
entry in the above .bib file is under the citation key ``pyautolens``. Please also cite the MNRAS AutoLens
14+
papers (https://academic.oup.com/mnras/article/452/3/2940/1749640 and https://academic.oup.com/mnras/article-abstract/478/4/4738/5001434?redirectedFrom=fulltext) which are included
15+
under the citation keys ``Nightingale2015`` and ``Nightingale2018``.
16+
17+
You should also specify the non-linear search(es) you use in your analysis (e.g. Dynesty, Emcee, etc) in
18+
the main body of text, and delete as appropriate any packages your analysis did not use. The citations.bib file includes
19+
the citation key for all of these projects.
20+
21+
If you use decomposed mass models (e.g. stellar mass models like an ``Sersic`` or dark matter models like
22+
an ``NFW``) please cite the following paper https://arxiv.org/abs/2106.11464 under
23+
citation key ``Oguri2021``. Our deflection angle calculations are based on this method.
24+
25+
If you specifically use a decomposed mass model with the ``gNFW`` please cite the following paper https://academic.oup.com/mnras/article/488/1/1387/5526256 under
26+
citation key ``Anowar2019``.
27+
28+
The citations.bib file above also includes my work on `using strong lensing to study galaxy structure
29+
<https://ui.adsabs.harvard.edu/abs/2019MNRAS.489.2049N/abstract>`_. If you're feeling kind, please go ahead and stick
30+
a citation in your introduction using \citep{Nightingale2019} or [@Nightingale2019] ;).
31+

CLAUDE.md

Lines changed: 82 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,82 @@
1+
# HowToLens
2+
3+
This is the **HowToLens** tutorial lecture series for `PyAutoLens`, a Python library for strong gravitational lens modeling. Tutorials teach new users what strong lensing is and how to model it from first principles.
4+
5+
## Repository Structure
6+
7+
- `scripts/` — Runnable Python tutorial scripts
8+
- `chapter_1_introduction/` — Grids, profiles, galaxies, ray-tracing, data, fitting
9+
- `chapter_2_lens_modeling/` — Non-linear searches, Bayesian inference, lens modeling
10+
- `chapter_3_search_chaining/` — Search chaining, prior passing, automated pipelines
11+
- `chapter_4_pixelizations/` — Pixelized source reconstruction, inversions, regularization
12+
- `chapter_optional/` — Alternative non-linear searches and advanced topics
13+
- `simulator/` — Simulator scripts that generate the tutorial datasets at runtime
14+
- `notebooks/` — Jupyter notebook versions of scripts (generated from `scripts/`, do not edit directly)
15+
- `config/``PyAutoLens` configuration YAML files
16+
- `dataset/` — Empty in the repo; tutorial datasets are written here at runtime by the simulator scripts
17+
- `output/` — Model-fit results (generated at runtime, not committed)
18+
19+
## Running Scripts
20+
21+
Scripts are run from the repository root so relative paths to `dataset/` and `output/` resolve correctly:
22+
23+
```bash
24+
python scripts/chapter_1_introduction/tutorial_1_grids_and_galaxies.py
25+
```
26+
27+
Tutorials in chapters 1 and 2 that need a dataset invoke the relevant script in `scripts/simulator/` via `subprocess` if the dataset folder does not already exist — there is no manual simulate-then-run step.
28+
29+
**Integration testing / fast mode**: set `PYAUTO_TEST_MODE=1` to skip non-linear search sampling:
30+
31+
```bash
32+
PYAUTO_TEST_MODE=1 python scripts/chapter_2_lens_modeling/tutorial_1_non_linear_search.py
33+
```
34+
35+
**Fast smoke tests**: combine test mode with the skip flags:
36+
37+
```bash
38+
PYAUTO_TEST_MODE=2 PYAUTO_SKIP_FIT_OUTPUT=1 PYAUTO_SKIP_VISUALIZATION=1 PYAUTO_SKIP_CHECKS=1 PYAUTO_FAST_PLOTS=1 python scripts/chapter_1_introduction/tutorial_7_fitting.py
39+
```
40+
41+
Note: `PYAUTO_SMALL_DATASETS` is deliberately **not** used in HowToLens. Tutorials assume the full-resolution simulated datasets that the simulator scripts produce.
42+
43+
**Codex / sandboxed runs**: set writable cache directories so `numba` and `matplotlib` do not fail on unwritable home paths:
44+
45+
```bash
46+
NUMBA_CACHE_DIR=/tmp/numba_cache MPLCONFIGDIR=/tmp/matplotlib python scripts/chapter_1_introduction/tutorial_1_grids_and_galaxies.py
47+
```
48+
49+
## Core API Patterns
50+
51+
Imports used throughout the tutorials:
52+
53+
```python
54+
import autofit as af
55+
import autolens as al
56+
import autolens.plot as aplt
57+
```
58+
59+
## Notebooks vs Scripts
60+
61+
Notebooks in `notebooks/` are generated from the `.py` files in `scripts/` using `generate.py` from the `PyAutoBuild` repo. **Always edit the `.py` scripts**, never the notebooks directly. The `# %%` marker alternates between code and markdown cells.
62+
63+
### Building Notebooks
64+
65+
Run from the workspace root:
66+
67+
```bash
68+
PYTHONPATH=../PyAutoBuild/autobuild python3 ../PyAutoBuild/autobuild/generate.py howtolens
69+
```
70+
71+
The `howtolens` project target in `PyAutoBuild/autobuild/config.yaml` is what drives this.
72+
73+
## Relationship to autolens_workspace
74+
75+
HowToLens is the teaching companion to `autolens_workspace`. Many tutorials (particularly in chapters 2–4) point users to `autolens_workspace` scripts (e.g. `scripts/imaging/modeling.py`, `scripts/guides/...`) as the next destination after the relevant concept has been introduced. Those cross-references use absolute paths like `autolens_workspace/scripts/...` and refer to the separate `autolens_workspace` repository — not to anything inside HowToLens.
76+
77+
## Related Repos
78+
79+
- **PyAutoLens** source: `../PyAutoLens`
80+
- **PyAutoGalaxy** source: `../PyAutoGalaxy`
81+
- **autolens_workspace**: `../autolens_workspace` — main user-facing workspace
82+
- **PyAutoBuild**: `../PyAutoBuild` — notebook generation and CI/CD tooling

0 commit comments

Comments
 (0)