Harden install + notebook flow for the workshop room (council points 1-5)

.github/workflows/smoke.yml

@@ -5,10 +5,11 @@ name: capstone-smoke
 #   validate — fast (~30s): every committed notebook is well-formed and every
 #              script/glue module byte-compiles. Catches a corrupted notebook
 #              or a syntax error without waiting on the heavy stack.
-#   smoke    — full: install the workshop env, register the `workshop` kernel,
-#              fetch the prerecorded session, and run the capstone notebook
-#              headless. Catches a broken install or upstream API drift in CI
-#              rather than in front of the workshop room.
+#   smoke    — full: install the pinned lock, register the `workshop` kernel,
+#              refresh the per-tool teaching notebooks (catches upstream
+#              notebook-CLI drift), fetch the prerecorded session, and run the
+#              capstone notebook headless. Catches a broken install or upstream
+#              API drift in CI rather than in front of the workshop room.
 #
 # Docs-only commits (**.md, LICENSE) are skipped — they can't break either gate.
 
@@ -59,23 +60,30 @@ jobs:
         with:
           python-version: "3.12"
           cache: pip
-          cache-dependency-path: requirements.txt
+          cache-dependency-path: requirements.lock
 
       - name: Install ffmpeg
         run: sudo apt-get update && sudo apt-get install -y ffmpeg
 
       - name: Install dependencies
+        # Install the pinned lock — the reproducible set the workshop room runs.
+        # pywinpty is marked win32-only so the lock installs cleanly on Linux.
         run: |
           python -m pip install --upgrade pip
-          python -m pip install -r requirements.txt
-          # requirements.lock is a Windows-generated freeze (it pins pywinpty)
-          # and won't install on the Linux runner — regenerate it cross-platform
-          # before gating CI on it.
+          python -m pip install -r requirements.lock
 
       - name: Register the workshop Jupyter kernel
         # Mirrors INSTALL.md step 3 — the capstone notebook is pinned to this kernel.
         run: python -m ipykernel install --user --name workshop --display-name "Workshop"
 
+      # Refresh the teaching notebooks from the installed wheels. They're also
+      # committed (so the room has them on clone), but running the fetch here is
+      # what catches upstream notebook-CLI drift: fetch_notebooks.py exits nonzero
+      # if any tool fails to produce at least one notebook, turning a silent
+      # in-room failure into a red build.
+      - name: Refresh teaching notebooks (minisim / minian / eztrack)
+        run: python scripts/fetch_notebooks.py
+
       # The capstone reads the processed stages + raw timestamps, never the raw
       # videos — so --skip-video pulls ~228 MB instead of the full 8.9 GB deposit.
       # Cache it so the fetch only hits the archive when get_data.py changes.

.gitignore

@@ -19,10 +19,9 @@ dist/
 # Jupyter
 .ipynb_checkpoints/
 
-# Teaching notebooks fetched from the installed tools (version-matched, generated
-# by scripts/fetch_notebooks.py — not vendored). The capstone notebook lives in
-# capstone/ and IS committed.
-tutorials/*/notebooks/
+# Teaching notebooks (tutorials/<tool>/notebooks/) are now VENDORED and committed
+# so they're present on clone — see tutorials/<tool>/notebooks/PROVENANCE.md.
+# scripts/fetch_notebooks.py refreshes them (version-matched) but is optional.
 
 # OS / editor
 .DS_Store

INSTALL.md

@@ -22,7 +22,16 @@ You need three things that pip cannot install for you: **Python**, **git**, and
 
 ### Python 3.11–3.13
 
-CaMAP requires Python in this range; 3.12 is recommended.
+CaMAP requires Python in this range; 3.12 is recommended. **3.14 (the current
+newest) and 3.10-or-older will NOT work** — grab 3.12 specifically, not "the
+latest."
+
+> **Symptom of the wrong version:** if `pip install` ends with
+> `No matching distribution found for camap[notebook]` and a line like
+> `Ignored the following versions that require a different python version`,
+> your Python is outside 3.11–3.13 (almost always 3.14 too new, or 3.10 too
+> old). Check with `python --version` and rebuild the venv with 3.12 (below).
+> `scripts/verify.py` flags this up front.
 
 - **Windows:** `winget install Python.Python.3.12`
   (or download from [python.org](https://www.python.org/downloads/) and **check
@@ -84,11 +93,15 @@ source .venv/bin/activate      # macOS/Linux
 
 # install everything (this is a large scientific stack — expect ~5-10 min)
 python -m pip install --upgrade pip
-python -m pip install -r requirements.txt          # newest from PyPI
-# or, for the reproducible known-good set:
-# python -m pip install -r requirements.lock
+python -m pip install -r requirements.lock         # pinned, reproducible (recommended for the workshop)
+# if the lock ever fails to install on your platform, fall back to newest-from-PyPI
+# and let the organizers know:  python -m pip install -r requirements.txt
 ```
 
+> **Which file?** `requirements.lock` is the exact, known-good set the workshop
+> room runs — use it. `requirements.txt` is unpinned (newest from PyPI) and is
+> mainly for maintainers checking upstream drift.
+
 Your prompt should now show `(.venv)`. Re-activate (the `activate` line above) in
 each new terminal.
 
@@ -107,10 +120,31 @@ each new terminal.
 
 ```bash
 python -m ipykernel install --user --name workshop --display-name "Workshop"
-python scripts/fetch_notebooks.py                  # tool notebooks -> tutorials/<tool>/notebooks/
 python scripts/get_data.py                         # prerecorded session data (all stages)
+python scripts/fetch_notebooks.py                  # OPTIONAL: refresh tool notebooks to your installed versions
 ```
 
+The teaching notebooks for each tool are **already committed** under
+`tutorials/<tool>/notebooks/`, so they're present the moment you clone — you
+don't need to fetch anything for them to exist. `fetch_notebooks.py` is an
+optional refresh that pulls copies matching your installed versions; it's
+non-destructive, so if it can't run it simply leaves the committed copies in
+place. (`get_data.py` defaults to all stages, ~8.9 GB incl. raw video; if you're
+only running the capstone, `--skip-video` pulls ~228 MB instead.)
+
+## Step 4 — Verify your install
+
+Run the self-check — it confirms every step above actually worked and tells you
+exactly what to fix if not:
+
+```bash
+python scripts/verify.py
+```
+
+You want an all-`PASS` run (a `WARN` is usually fine). **Do this before the
+workshop** and send the output to the organizers if anything is red — a broken
+setup is easy to fix days early and painful to fix in the room.
+
 Then launch Jupyter and pick the **Workshop** kernel:
 
 ```bash
@@ -130,11 +164,12 @@ jupyter lab
 
 ## newest vs. pinned
 
-- `requirements.txt` — top-level packages, **unpinned** (pulls newest). Good
-  during development.
-- `requirements.lock` — a **full pinned freeze** of a verified-working set.
-  Switch the workshop to this once development locks in, so the room is
-  reproducible. The CI smoke test flags drift in the meantime.
+- `requirements.lock` — a **full pinned freeze** of the verified-working set, and
+  what the workshop room runs. Cross-platform (pywinpty is marked Windows-only)
+  and exercised end-to-end by the CI smoke test. **Use this.**
+- `requirements.txt` — top-level packages, **unpinned** (pulls newest). For
+  maintainers checking upstream drift, or as a fallback if the lock won't install
+  on your platform.
 
 ## Verify the capstone runs
 
@@ -155,4 +190,7 @@ jupyter nbconvert --to notebook --execute --inplace \
 - **oasis-deconv** is optional and intentionally left out — the workshop does
   deconvolution in calab and bypasses CaMAP's built-in OASIS.
 - Some teaching notebooks need a tool's notebook extras (e.g. `minisim[notebook]`);
-  these are already in `requirements.txt`.
+  these are already in `requirements.lock` / `requirements.txt`.
+- The per-tool teaching notebooks are **committed** under `tutorials/<tool>/notebooks/`
+  (see each folder's `PROVENANCE.md`). `scripts/fetch_notebooks.py` refreshes them
+  to your installed versions but is optional and non-destructive.

ORGANIZER.md

@@ -4,6 +4,22 @@ Pre-workshop prep and open verification items, collected here so the
 participant-facing READMEs stay clean. Each item links back to the module it
 came from.
 
+## Participant install — pre-flight gate
+
+Make a passing self-check a **prerequisite**, so broken setups surface days early
+over email — not in a time-boxed room where one stuck laptop blocks everyone.
+
+- [ ] Send the install link ([INSTALL.md](INSTALL.md)) with a deadline (e.g. 2
+      days out): clone → `pip install -r requirements.lock` → register the
+      `workshop` kernel → `python scripts/get_data.py` → **`python scripts/verify.py`**,
+      and reply with the output (or an all-`PASS` screenshot).
+- [ ] Triage anyone who's red over email or a pre-workshop office hour.
+- [ ] **Dry-run the full install on a clean machine of each OS family** you expect
+      (Windows, macOS Apple-Silicon, macOS Intel, Linux). The lock is verified on
+      Linux by CI but is **untested on macOS/ARM** — this dry run, not CI, is the
+      real cross-OS guarantee. If a pinned package has no wheel on some platform,
+      patch `requirements.lock` (env marker) before sending instructions.
+
 ## Data & archive — [`data/README.md`](data/README.md)
 
 - [x] Publish the `prerecorded` deposit and set its DOI in `scripts/get_data.py`

README.md

@@ -79,20 +79,23 @@ bare machine? [INSTALL.md](INSTALL.md) walks through installing all three per OS
 git clone https://github.com/miniscope/workshop-processing-and-analysis.git
 cd workshop-processing-and-analysis
 python -m venv .venv && source .venv/bin/activate   # Windows: .venv\Scripts\Activate.ps1
-python -m pip install -r requirements.txt           # see INSTALL.md for the pinned lock
-# pull each tool's teaching notebooks into tutorials/<tool>/notebooks/
-python scripts/fetch_notebooks.py
-# fetch the workshop data (default: the prerecorded backup session, all stages)
-python scripts/get_data.py
+python -m pip install -r requirements.lock          # pinned, reproducible (see INSTALL.md)
+python -m ipykernel install --user --name workshop --display-name "Workshop"
+python scripts/get_data.py                          # prerecorded backup session (all stages)
+python scripts/verify.py                            # confirm the install before the workshop
 ```
 
+Run `python scripts/verify.py` and aim for an all-`PASS` result **before the
+workshop** — see [INSTALL.md](INSTALL.md) for the full step-by-step.
+
 The workshop's **own** recording (the `live` session) is published mid-workshop;
 grab it on day 2 with the one-line command in
 [Getting the live recording](data/README.md#getting-the-live-recording-day-2).
 
-Each tool ships its teaching notebooks inside its package; `fetch_notebooks.py`
-copies them out (version-matched to what's installed) so participants find them
-right next to each module's README. They're regenerated, not committed.
+Each tool's teaching notebooks are **committed** under `tutorials/<tool>/notebooks/`,
+so they're there on clone. `python scripts/fetch_notebooks.py` is an optional,
+non-destructive refresh that re-pulls them version-matched to your installed
+tools (see each folder's `PROVENANCE.md`).
 
 Then open the module you're on under [`tutorials/`](tutorials/) or [`capstone/`](capstone/).
 
@@ -110,15 +113,16 @@ workshop-processing-and-analysis/
 ├── README.md                 # this file
 ├── INSTALL.md                # venv setup + prerequisites (ffmpeg)
 ├── ORGANIZER.md              # organizer-only prep checklist (not for participants)
-├── requirements.txt          # top-level deps (newest from PyPI)
-├── requirements.lock         # full pinned freeze (verified-working set)
+├── requirements.lock         # full pinned freeze (verified-working set) — use this
+├── requirements.txt          # top-level deps (newest from PyPI) — maintainers/fallback
 ├── data/
 │   ├── README.md             # sessions, archive bundles, local-first resolution
 │   └── sessions/             # data/sessions/<name>/{raw,minian_out,deconv_out,eztrack_out}
 ├── scripts/
 │   ├── get_data.py           # DOI fetch from Dataverse/Zenodo (per session, local-first)
-│   └── fetch_notebooks.py    # copy each tool's bundled notebooks into tutorials/
-├── tutorials/                # one folder per upstream tool (links + run notes)
+│   ├── fetch_notebooks.py    # optional refresh of the committed per-tool notebooks
+│   └── verify.py             # pre-flight self-check (run before the workshop)
+├── tutorials/                # one folder per upstream tool (committed teaching notebooks)
 │   ├── overview/             # processing_overview.ipynb (the pipeline end-to-end)
 │   ├── minisim/
 │   ├── minian/

requirements.lock

@@ -144,7 +144,7 @@ pyparsing==3.3.2
 python-dateutil==2.9.0.post0
 python-json-logger==4.1.0
 pyviz_comms==3.0.6
-pywinpty==3.0.5
+pywinpty==3.0.5 ; sys_platform == "win32"
 PyYAML==6.0.3
 pyzmq==27.1.0
 qdldl==0.1.9.post1

scripts/fetch_notebooks.py

@@ -1,17 +1,22 @@
-"""Materialize each tool's teaching notebooks into its tutorials/<tool>/notebooks/.
+"""Refresh each tool's teaching notebooks in tutorials/<tool>/notebooks/.
 
-The upstream packages (minisim, Minian, eztrack) ship their teaching notebooks
-*inside the installed wheel* and expose a CLI to copy them out, version-matched
-to whatever is installed. Rather than vendoring (and going stale), we fetch them
-on demand into the repo so workshop participants have them right next to each
-module's README.
+A **known-good copy of every teaching notebook is committed to the repo**, so
+they are present the moment you clone — nothing has to succeed for the workshop
+materials to exist. This script is an *optional refresh*: the upstream packages
+(minisim, Minian, eztrack) ship their teaching notebooks inside the installed
+wheel and expose a CLI to copy them out, version-matched to whatever you have
+installed. Run it to pull the copies that match your installed versions.
 
 Run from the repo root with the workshop venv active:
 
     python scripts/fetch_notebooks.py
 
-Re-running refreshes the copies (each destination is cleared first). The fetched
-folders are gitignored — they are generated, not committed.
+It is **non-destructive**: each tool is fetched into a temporary directory and
+only swapped into place if the copy succeeds and yields at least one notebook.
+If a tool's fetch fails (CLI missing, upstream drift, no network), the committed
+copy already in tutorials/<tool>/notebooks/ is left untouched — a failed or
+repeated run can never leave you with fewer notebooks than you started with.
+A nonzero exit means at least one tool could not be refreshed.
 
 Notes:
 - calab (CaTune / CaDecon) ships no notebooks upstream. The workshop's
@@ -25,6 +30,7 @@
 import shutil
 import subprocess
 import sys
+import tempfile
 from pathlib import Path
 
 REPO_ROOT = Path(__file__).resolve().parent.parent
@@ -39,33 +45,57 @@
 
 
 def fetch(label: str, cmd_prefix: list[str], dest: Path) -> bool:
-    """Copy one tool's notebooks into *dest*. Returns True on success."""
+    """Refresh one tool's notebooks into *dest*. Returns True on success.
+
+    Fetches into a temp dir and swaps it into place only if the copy produced
+    at least one notebook, so a failure leaves the committed copy untouched.
+    """
     exe = shutil.which(cmd_prefix[0])
     if exe is None:
-        print(f"  SKIP {label}: '{cmd_prefix[0]}' not found — is the venv active and {label} installed?")
-        return False
-
-    # Idempotent: clear any previous copy (minian/eztrack copy has no --force).
-    if dest.exists():
-        shutil.rmtree(dest)
-    dest.mkdir(parents=True, exist_ok=True)
-
-    cmd = [exe, *cmd_prefix[1:], str(dest)]
-    result = subprocess.run(cmd, cwd=REPO_ROOT)
-    if result.returncode != 0:
-        print(f"  FAIL {label}: '{' '.join(cmd_prefix)} ...' exited {result.returncode}")
+        print(f"  SKIP {label}: '{cmd_prefix[0]}' not found — is the venv active and {label} installed? "
+              f"(keeping the committed copy in {dest.relative_to(REPO_ROOT)})")
         return False
 
-    n = len(list(dest.rglob("*.ipynb")))
-    print(f"  OK   {label}: {n} notebook(s) -> {dest.relative_to(REPO_ROOT)}")
-    return True
+    dest.parent.mkdir(parents=True, exist_ok=True)
+    # Stage in a sibling temp dir (same filesystem, so the swap is a cheap move).
+    tmp = Path(tempfile.mkdtemp(prefix=f".{dest.name}.", dir=dest.parent))
+    try:
+        cmd = [exe, *cmd_prefix[1:], str(tmp)]
+        result = subprocess.run(cmd, cwd=REPO_ROOT)
+        if result.returncode != 0:
+            print(f"  FAIL {label}: '{' '.join(cmd_prefix)} ...' exited {result.returncode} "
+                  f"(kept the committed copy in {dest.relative_to(REPO_ROOT)})")
+            return False
+
+        n = len(list(tmp.rglob("*.ipynb")))
+        if n == 0:
+            print(f"  FAIL {label}: fetch produced no .ipynb files "
+                  f"(kept the committed copy in {dest.relative_to(REPO_ROOT)})")
+            return False
+
+        # Swap in the fresh copy: remove the old, move the staged one into place.
+        if dest.exists():
+            shutil.rmtree(dest)
+        shutil.move(str(tmp), str(dest))
+        tmp = None  # consumed by the move; nothing left to clean up
+        print(f"  OK   {label}: {n} notebook(s) -> {dest.relative_to(REPO_ROOT)}")
+        return True
+    finally:
+        if tmp is not None and tmp.exists():
+            shutil.rmtree(tmp, ignore_errors=True)
 
 
 def main() -> int:
-    print("Fetching teaching notebooks into tutorials/<tool>/notebooks/ ...")
+    print("Refreshing teaching notebooks in tutorials/<tool>/notebooks/ ...")
+    print("(committed copies are used as-is if a tool can't be refreshed)\n")
     ok = sum(fetch(*t) for t in TOOLS)
-    print(f"\n{ok}/{len(TOOLS)} tools fetched. Open them with: jupyter lab")
-    print("calab (CaTune/CaDecon): use tutorials/deconvolution/deconvolution.ipynb "
+    print(f"\n{ok}/{len(TOOLS)} tools refreshed. Open them with: jupyter lab")
+    if ok != len(TOOLS):
+        print("\nNot all tools refreshed (see SKIP/FAIL above). The committed copies are\n"
+              "still in place, so the notebooks are present — but they may not match your\n"
+              "installed versions. Activate the workshop venv and re-run if you need a\n"
+              "version-matched refresh.")
+    print("\ncalab (CaTune/CaDecon): use tutorials/deconvolution/deconvolution.ipynb "
           "(authored in-repo) or `calab tune` / `calab cadecon`.")
     return 0 if ok == len(TOOLS) else 1
 

scripts/validate_notebooks.py

@@ -3,10 +3,8 @@
 
 Validates that every committed .ipynb is well-formed nbformat — a cheap gate
 that catches a corrupted or hand-edited notebook in seconds, without spinning
-up the full scientific stack the capstone smoke test needs.
-
-Skips fetched tutorial notebooks (tutorials/*/notebooks/), which are generated
-by scripts/fetch_notebooks.py and not committed.
+up the full scientific stack the capstone smoke test needs. This includes the
+vendored teaching notebooks under tutorials/<tool>/notebooks/.
 """
 
 from __future__ import annotations
@@ -26,9 +24,8 @@ def iter_notebooks() -> list[Path]:
         rel_parts = set(path.relative_to(REPO_ROOT).parts)
         if rel_parts & SKIP_DIRS:
             continue
-        # tutorials/<tool>/notebooks/ is fetched, not committed
-        if "notebooks" in path.parts and "tutorials" in path.parts:
-            continue
+        # The teaching notebooks under tutorials/<tool>/notebooks/ are now
+        # vendored and committed, so they're validated too.
         notebooks.append(path)
     return sorted(notebooks)