diff --git a/.github/workflows/smoke.yml b/.github/workflows/smoke.yml index 40c16bf..f264eb8 100644 --- a/.github/workflows/smoke.yml +++ b/.github/workflows/smoke.yml @@ -5,10 +5,11 @@ name: capstone-smoke # validate — fast (~30s): every committed notebook is well-formed and every # script/glue module byte-compiles. Catches a corrupted notebook # or a syntax error without waiting on the heavy stack. -# smoke — full: install the workshop env, register the `workshop` kernel, -# fetch the prerecorded session, and run the capstone notebook -# headless. Catches a broken install or upstream API drift in CI -# rather than in front of the workshop room. +# smoke — full: install the pinned lock, register the `workshop` kernel, +# refresh the per-tool teaching notebooks (catches upstream +# notebook-CLI drift), fetch the prerecorded session, and run the +# capstone notebook headless. Catches a broken install or upstream +# API drift in CI rather than in front of the workshop room. # # Docs-only commits (**.md, LICENSE) are skipped — they can't break either gate. @@ -59,23 +60,30 @@ jobs: with: python-version: "3.12" cache: pip - cache-dependency-path: requirements.txt + cache-dependency-path: requirements.lock - name: Install ffmpeg run: sudo apt-get update && sudo apt-get install -y ffmpeg - name: Install dependencies + # Install the pinned lock — the reproducible set the workshop room runs. + # pywinpty is marked win32-only so the lock installs cleanly on Linux. run: | python -m pip install --upgrade pip - python -m pip install -r requirements.txt - # requirements.lock is a Windows-generated freeze (it pins pywinpty) - # and won't install on the Linux runner — regenerate it cross-platform - # before gating CI on it. + python -m pip install -r requirements.lock - name: Register the workshop Jupyter kernel # Mirrors INSTALL.md step 3 — the capstone notebook is pinned to this kernel. run: python -m ipykernel install --user --name workshop --display-name "Workshop" + # Refresh the teaching notebooks from the installed wheels. They're also + # committed (so the room has them on clone), but running the fetch here is + # what catches upstream notebook-CLI drift: fetch_notebooks.py exits nonzero + # if any tool fails to produce at least one notebook, turning a silent + # in-room failure into a red build. + - name: Refresh teaching notebooks (minisim / minian / eztrack) + run: python scripts/fetch_notebooks.py + # The capstone reads the processed stages + raw timestamps, never the raw # videos — so --skip-video pulls ~228 MB instead of the full 8.9 GB deposit. # Cache it so the fetch only hits the archive when get_data.py changes. diff --git a/.gitignore b/.gitignore index 03e49c4..2fbf986 100644 --- a/.gitignore +++ b/.gitignore @@ -19,10 +19,9 @@ dist/ # Jupyter .ipynb_checkpoints/ -# Teaching notebooks fetched from the installed tools (version-matched, generated -# by scripts/fetch_notebooks.py — not vendored). The capstone notebook lives in -# capstone/ and IS committed. -tutorials/*/notebooks/ +# Teaching notebooks (tutorials//notebooks/) are now VENDORED and committed +# so they're present on clone — see tutorials//notebooks/PROVENANCE.md. +# scripts/fetch_notebooks.py refreshes them (version-matched) but is optional. # OS / editor .DS_Store diff --git a/INSTALL.md b/INSTALL.md index b529537..f4379e0 100644 --- a/INSTALL.md +++ b/INSTALL.md @@ -22,7 +22,16 @@ You need three things that pip cannot install for you: **Python**, **git**, and ### Python 3.11–3.13 -CaMAP requires Python in this range; 3.12 is recommended. +CaMAP requires Python in this range; 3.12 is recommended. **3.14 (the current +newest) and 3.10-or-older will NOT work** — grab 3.12 specifically, not "the +latest." + +> **Symptom of the wrong version:** if `pip install` ends with +> `No matching distribution found for camap[notebook]` and a line like +> `Ignored the following versions that require a different python version`, +> your Python is outside 3.11–3.13 (almost always 3.14 too new, or 3.10 too +> old). Check with `python --version` and rebuild the venv with 3.12 (below). +> `scripts/verify.py` flags this up front. - **Windows:** `winget install Python.Python.3.12` (or download from [python.org](https://www.python.org/downloads/) and **check @@ -84,11 +93,15 @@ source .venv/bin/activate # macOS/Linux # install everything (this is a large scientific stack — expect ~5-10 min) python -m pip install --upgrade pip -python -m pip install -r requirements.txt # newest from PyPI -# or, for the reproducible known-good set: -# python -m pip install -r requirements.lock +python -m pip install -r requirements.lock # pinned, reproducible (recommended for the workshop) +# if the lock ever fails to install on your platform, fall back to newest-from-PyPI +# and let the organizers know: python -m pip install -r requirements.txt ``` +> **Which file?** `requirements.lock` is the exact, known-good set the workshop +> room runs — use it. `requirements.txt` is unpinned (newest from PyPI) and is +> mainly for maintainers checking upstream drift. + Your prompt should now show `(.venv)`. Re-activate (the `activate` line above) in each new terminal. @@ -107,10 +120,31 @@ each new terminal. ```bash python -m ipykernel install --user --name workshop --display-name "Workshop" -python scripts/fetch_notebooks.py # tool notebooks -> tutorials//notebooks/ python scripts/get_data.py # prerecorded session data (all stages) +python scripts/fetch_notebooks.py # OPTIONAL: refresh tool notebooks to your installed versions ``` +The teaching notebooks for each tool are **already committed** under +`tutorials//notebooks/`, so they're present the moment you clone — you +don't need to fetch anything for them to exist. `fetch_notebooks.py` is an +optional refresh that pulls copies matching your installed versions; it's +non-destructive, so if it can't run it simply leaves the committed copies in +place. (`get_data.py` defaults to all stages, ~8.9 GB incl. raw video; if you're +only running the capstone, `--skip-video` pulls ~228 MB instead.) + +## Step 4 — Verify your install + +Run the self-check — it confirms every step above actually worked and tells you +exactly what to fix if not: + +```bash +python scripts/verify.py +``` + +You want an all-`PASS` run (a `WARN` is usually fine). **Do this before the +workshop** and send the output to the organizers if anything is red — a broken +setup is easy to fix days early and painful to fix in the room. + Then launch Jupyter and pick the **Workshop** kernel: ```bash @@ -130,11 +164,12 @@ jupyter lab ## newest vs. pinned -- `requirements.txt` — top-level packages, **unpinned** (pulls newest). Good - during development. -- `requirements.lock` — a **full pinned freeze** of a verified-working set. - Switch the workshop to this once development locks in, so the room is - reproducible. The CI smoke test flags drift in the meantime. +- `requirements.lock` — a **full pinned freeze** of the verified-working set, and + what the workshop room runs. Cross-platform (pywinpty is marked Windows-only) + and exercised end-to-end by the CI smoke test. **Use this.** +- `requirements.txt` — top-level packages, **unpinned** (pulls newest). For + maintainers checking upstream drift, or as a fallback if the lock won't install + on your platform. ## Verify the capstone runs @@ -155,4 +190,7 @@ jupyter nbconvert --to notebook --execute --inplace \ - **oasis-deconv** is optional and intentionally left out — the workshop does deconvolution in calab and bypasses CaMAP's built-in OASIS. - Some teaching notebooks need a tool's notebook extras (e.g. `minisim[notebook]`); - these are already in `requirements.txt`. + these are already in `requirements.lock` / `requirements.txt`. +- The per-tool teaching notebooks are **committed** under `tutorials//notebooks/` + (see each folder's `PROVENANCE.md`). `scripts/fetch_notebooks.py` refreshes them + to your installed versions but is optional and non-destructive. diff --git a/ORGANIZER.md b/ORGANIZER.md index 874162c..683d176 100644 --- a/ORGANIZER.md +++ b/ORGANIZER.md @@ -4,6 +4,22 @@ Pre-workshop prep and open verification items, collected here so the participant-facing READMEs stay clean. Each item links back to the module it came from. +## Participant install — pre-flight gate + +Make a passing self-check a **prerequisite**, so broken setups surface days early +over email — not in a time-boxed room where one stuck laptop blocks everyone. + +- [ ] Send the install link ([INSTALL.md](INSTALL.md)) with a deadline (e.g. 2 + days out): clone → `pip install -r requirements.lock` → register the + `workshop` kernel → `python scripts/get_data.py` → **`python scripts/verify.py`**, + and reply with the output (or an all-`PASS` screenshot). +- [ ] Triage anyone who's red over email or a pre-workshop office hour. +- [ ] **Dry-run the full install on a clean machine of each OS family** you expect + (Windows, macOS Apple-Silicon, macOS Intel, Linux). The lock is verified on + Linux by CI but is **untested on macOS/ARM** — this dry run, not CI, is the + real cross-OS guarantee. If a pinned package has no wheel on some platform, + patch `requirements.lock` (env marker) before sending instructions. + ## Data & archive — [`data/README.md`](data/README.md) - [x] Publish the `prerecorded` deposit and set its DOI in `scripts/get_data.py` diff --git a/README.md b/README.md index f6c11bd..8e698c8 100644 --- a/README.md +++ b/README.md @@ -79,20 +79,23 @@ bare machine? [INSTALL.md](INSTALL.md) walks through installing all three per OS git clone https://github.com/miniscope/workshop-processing-and-analysis.git cd workshop-processing-and-analysis python -m venv .venv && source .venv/bin/activate # Windows: .venv\Scripts\Activate.ps1 -python -m pip install -r requirements.txt # see INSTALL.md for the pinned lock -# pull each tool's teaching notebooks into tutorials//notebooks/ -python scripts/fetch_notebooks.py -# fetch the workshop data (default: the prerecorded backup session, all stages) -python scripts/get_data.py +python -m pip install -r requirements.lock # pinned, reproducible (see INSTALL.md) +python -m ipykernel install --user --name workshop --display-name "Workshop" +python scripts/get_data.py # prerecorded backup session (all stages) +python scripts/verify.py # confirm the install before the workshop ``` +Run `python scripts/verify.py` and aim for an all-`PASS` result **before the +workshop** — see [INSTALL.md](INSTALL.md) for the full step-by-step. + The workshop's **own** recording (the `live` session) is published mid-workshop; grab it on day 2 with the one-line command in [Getting the live recording](data/README.md#getting-the-live-recording-day-2). -Each tool ships its teaching notebooks inside its package; `fetch_notebooks.py` -copies them out (version-matched to what's installed) so participants find them -right next to each module's README. They're regenerated, not committed. +Each tool's teaching notebooks are **committed** under `tutorials//notebooks/`, +so they're there on clone. `python scripts/fetch_notebooks.py` is an optional, +non-destructive refresh that re-pulls them version-matched to your installed +tools (see each folder's `PROVENANCE.md`). Then open the module you're on under [`tutorials/`](tutorials/) or [`capstone/`](capstone/). @@ -110,15 +113,16 @@ workshop-processing-and-analysis/ ├── README.md # this file ├── INSTALL.md # venv setup + prerequisites (ffmpeg) ├── ORGANIZER.md # organizer-only prep checklist (not for participants) -├── requirements.txt # top-level deps (newest from PyPI) -├── requirements.lock # full pinned freeze (verified-working set) +├── requirements.lock # full pinned freeze (verified-working set) — use this +├── requirements.txt # top-level deps (newest from PyPI) — maintainers/fallback ├── data/ │ ├── README.md # sessions, archive bundles, local-first resolution │ └── sessions/ # data/sessions//{raw,minian_out,deconv_out,eztrack_out} ├── scripts/ │ ├── get_data.py # DOI fetch from Dataverse/Zenodo (per session, local-first) -│ └── fetch_notebooks.py # copy each tool's bundled notebooks into tutorials/ -├── tutorials/ # one folder per upstream tool (links + run notes) +│ ├── fetch_notebooks.py # optional refresh of the committed per-tool notebooks +│ └── verify.py # pre-flight self-check (run before the workshop) +├── tutorials/ # one folder per upstream tool (committed teaching notebooks) │ ├── overview/ # processing_overview.ipynb (the pipeline end-to-end) │ ├── minisim/ │ ├── minian/ diff --git a/requirements.lock b/requirements.lock index 5e983e3..cf0e058 100644 --- a/requirements.lock +++ b/requirements.lock @@ -144,7 +144,7 @@ pyparsing==3.3.2 python-dateutil==2.9.0.post0 python-json-logger==4.1.0 pyviz_comms==3.0.6 -pywinpty==3.0.5 +pywinpty==3.0.5 ; sys_platform == "win32" PyYAML==6.0.3 pyzmq==27.1.0 qdldl==0.1.9.post1 diff --git a/scripts/fetch_notebooks.py b/scripts/fetch_notebooks.py index 51c48c2..c19bbbc 100644 --- a/scripts/fetch_notebooks.py +++ b/scripts/fetch_notebooks.py @@ -1,17 +1,22 @@ -"""Materialize each tool's teaching notebooks into its tutorials//notebooks/. +"""Refresh each tool's teaching notebooks in tutorials//notebooks/. -The upstream packages (minisim, Minian, eztrack) ship their teaching notebooks -*inside the installed wheel* and expose a CLI to copy them out, version-matched -to whatever is installed. Rather than vendoring (and going stale), we fetch them -on demand into the repo so workshop participants have them right next to each -module's README. +A **known-good copy of every teaching notebook is committed to the repo**, so +they are present the moment you clone — nothing has to succeed for the workshop +materials to exist. This script is an *optional refresh*: the upstream packages +(minisim, Minian, eztrack) ship their teaching notebooks inside the installed +wheel and expose a CLI to copy them out, version-matched to whatever you have +installed. Run it to pull the copies that match your installed versions. Run from the repo root with the workshop venv active: python scripts/fetch_notebooks.py -Re-running refreshes the copies (each destination is cleared first). The fetched -folders are gitignored — they are generated, not committed. +It is **non-destructive**: each tool is fetched into a temporary directory and +only swapped into place if the copy succeeds and yields at least one notebook. +If a tool's fetch fails (CLI missing, upstream drift, no network), the committed +copy already in tutorials//notebooks/ is left untouched — a failed or +repeated run can never leave you with fewer notebooks than you started with. +A nonzero exit means at least one tool could not be refreshed. Notes: - calab (CaTune / CaDecon) ships no notebooks upstream. The workshop's @@ -25,6 +30,7 @@ import shutil import subprocess import sys +import tempfile from pathlib import Path REPO_ROOT = Path(__file__).resolve().parent.parent @@ -39,33 +45,57 @@ def fetch(label: str, cmd_prefix: list[str], dest: Path) -> bool: - """Copy one tool's notebooks into *dest*. Returns True on success.""" + """Refresh one tool's notebooks into *dest*. Returns True on success. + + Fetches into a temp dir and swaps it into place only if the copy produced + at least one notebook, so a failure leaves the committed copy untouched. + """ exe = shutil.which(cmd_prefix[0]) if exe is None: - print(f" SKIP {label}: '{cmd_prefix[0]}' not found — is the venv active and {label} installed?") - return False - - # Idempotent: clear any previous copy (minian/eztrack copy has no --force). - if dest.exists(): - shutil.rmtree(dest) - dest.mkdir(parents=True, exist_ok=True) - - cmd = [exe, *cmd_prefix[1:], str(dest)] - result = subprocess.run(cmd, cwd=REPO_ROOT) - if result.returncode != 0: - print(f" FAIL {label}: '{' '.join(cmd_prefix)} ...' exited {result.returncode}") + print(f" SKIP {label}: '{cmd_prefix[0]}' not found — is the venv active and {label} installed? " + f"(keeping the committed copy in {dest.relative_to(REPO_ROOT)})") return False - n = len(list(dest.rglob("*.ipynb"))) - print(f" OK {label}: {n} notebook(s) -> {dest.relative_to(REPO_ROOT)}") - return True + dest.parent.mkdir(parents=True, exist_ok=True) + # Stage in a sibling temp dir (same filesystem, so the swap is a cheap move). + tmp = Path(tempfile.mkdtemp(prefix=f".{dest.name}.", dir=dest.parent)) + try: + cmd = [exe, *cmd_prefix[1:], str(tmp)] + result = subprocess.run(cmd, cwd=REPO_ROOT) + if result.returncode != 0: + print(f" FAIL {label}: '{' '.join(cmd_prefix)} ...' exited {result.returncode} " + f"(kept the committed copy in {dest.relative_to(REPO_ROOT)})") + return False + + n = len(list(tmp.rglob("*.ipynb"))) + if n == 0: + print(f" FAIL {label}: fetch produced no .ipynb files " + f"(kept the committed copy in {dest.relative_to(REPO_ROOT)})") + return False + + # Swap in the fresh copy: remove the old, move the staged one into place. + if dest.exists(): + shutil.rmtree(dest) + shutil.move(str(tmp), str(dest)) + tmp = None # consumed by the move; nothing left to clean up + print(f" OK {label}: {n} notebook(s) -> {dest.relative_to(REPO_ROOT)}") + return True + finally: + if tmp is not None and tmp.exists(): + shutil.rmtree(tmp, ignore_errors=True) def main() -> int: - print("Fetching teaching notebooks into tutorials//notebooks/ ...") + print("Refreshing teaching notebooks in tutorials//notebooks/ ...") + print("(committed copies are used as-is if a tool can't be refreshed)\n") ok = sum(fetch(*t) for t in TOOLS) - print(f"\n{ok}/{len(TOOLS)} tools fetched. Open them with: jupyter lab") - print("calab (CaTune/CaDecon): use tutorials/deconvolution/deconvolution.ipynb " + print(f"\n{ok}/{len(TOOLS)} tools refreshed. Open them with: jupyter lab") + if ok != len(TOOLS): + print("\nNot all tools refreshed (see SKIP/FAIL above). The committed copies are\n" + "still in place, so the notebooks are present — but they may not match your\n" + "installed versions. Activate the workshop venv and re-run if you need a\n" + "version-matched refresh.") + print("\ncalab (CaTune/CaDecon): use tutorials/deconvolution/deconvolution.ipynb " "(authored in-repo) or `calab tune` / `calab cadecon`.") return 0 if ok == len(TOOLS) else 1 diff --git a/scripts/validate_notebooks.py b/scripts/validate_notebooks.py index 6986fdc..94affdf 100644 --- a/scripts/validate_notebooks.py +++ b/scripts/validate_notebooks.py @@ -3,10 +3,8 @@ Validates that every committed .ipynb is well-formed nbformat — a cheap gate that catches a corrupted or hand-edited notebook in seconds, without spinning -up the full scientific stack the capstone smoke test needs. - -Skips fetched tutorial notebooks (tutorials/*/notebooks/), which are generated -by scripts/fetch_notebooks.py and not committed. +up the full scientific stack the capstone smoke test needs. This includes the +vendored teaching notebooks under tutorials//notebooks/. """ from __future__ import annotations @@ -26,9 +24,8 @@ def iter_notebooks() -> list[Path]: rel_parts = set(path.relative_to(REPO_ROOT).parts) if rel_parts & SKIP_DIRS: continue - # tutorials//notebooks/ is fetched, not committed - if "notebooks" in path.parts and "tutorials" in path.parts: - continue + # The teaching notebooks under tutorials//notebooks/ are now + # vendored and committed, so they're validated too. notebooks.append(path) return sorted(notebooks) diff --git a/scripts/verify.py b/scripts/verify.py new file mode 100644 index 0000000..86039ba --- /dev/null +++ b/scripts/verify.py @@ -0,0 +1,157 @@ +"""Pre-flight self-check for the workshop install. + +Run this AFTER following INSTALL.md, with the workshop venv active: + + python scripts/verify.py + +It does not install or download anything - it just checks that each install +step actually worked and prints one PASS / FAIL / WARN per check, with the exact +command to fix anything that's red. Send the output (or a screenshot of an +all-PASS run) to the organizers before the workshop so a broken setup is found +days early, not in the room. + +Exit code is 0 only if every required check passed. +""" + +from __future__ import annotations + +import os +import shutil +import sys +from pathlib import Path + +REPO_ROOT = Path(__file__).resolve().parent.parent + +# Tool distributions to confirm, with the version pinned in requirements.lock +# (a mismatch is a WARN, not a failure - newest-from-PyPI installs are allowed). +TOOLS = { + "minisim": "1.0.3", + "minian": "2.0.2", + "calab": "0.2.3", + "camap": "0.1.6", + "eztrack": None, # git fork - no PyPI version to compare against +} + +# Teaching-notebook dirs that must hold at least one .ipynb. +NOTEBOOK_DIRS = ["minisim", "minian", "eztrack"] + +# Files the capstone needs from the prerecorded session. +SESSION = REPO_ROOT / "data" / "sessions" / "prerecorded" +DATA_INPUTS = [ + SESSION / "minian_out", + SESSION / "deconv_out", + SESSION / "eztrack_out", + SESSION / "raw" / "neural_timestamp.csv", + SESSION / "raw" / "behavior_timestamp.csv", +] + +_results: list[tuple[str, str, str]] = [] # (status, label, hint) + + +def record(status: str, label: str, hint: str = "") -> None: + _results.append((status, label, hint)) + line = f" [{status:4}] {label}" + print(line if not hint else f"{line}\n -> {hint}") + + +def _nonempty_dir(p: Path) -> bool: + return p.is_dir() and any(p.iterdir()) + + +def check_python() -> None: + v = sys.version_info + ok = (3, 11) <= (v.major, v.minor) <= (3, 13) + label = f"Python {v.major}.{v.minor}.{v.micro} (need 3.11-3.13)" + record("PASS" if ok else "FAIL", label, + "" if ok else "Install Python 3.11-3.13 (3.12 recommended) - see INSTALL.md Step 0.") + + +def check_venv() -> None: + in_venv = sys.prefix != getattr(sys, "base_prefix", sys.prefix) or "VIRTUAL_ENV" in os.environ + record("PASS" if in_venv else "WARN", + f"virtual env active ({Path(sys.prefix).name})", + "" if in_venv else "Activate the venv first (INSTALL.md Step 2), or you may be checking the wrong Python.") + + +def check_ffmpeg() -> None: + exe = shutil.which("ffmpeg") + record("PASS" if exe else "FAIL", "ffmpeg on PATH", + "" if exe else "Install ffmpeg (INSTALL.md Step 0) and open a new terminal so PATH refreshes.") + + +def check_tools() -> None: + from importlib.metadata import PackageNotFoundError, version + for dist, pinned in TOOLS.items(): + try: + got = version(dist) + except PackageNotFoundError: + record("FAIL", f"package '{dist}' installed", + f"pip install -r requirements.lock (then re-run). '{dist}' is missing.") + continue + if pinned and got != pinned: + record("WARN", f"package '{dist}' {got} (lock pins {pinned})", + "Not fatal, but the room is pinned - `pip install -r requirements.lock` for the known-good set.") + else: + record("PASS", f"package '{dist}' {got}") + + +def check_kernel() -> None: + try: + from jupyter_client.kernelspec import KernelSpecManager + specs = KernelSpecManager().find_kernel_specs() + except Exception as exc: # jupyter not installed, or spec dir unreadable + record("FAIL", "Jupyter 'workshop' kernel registered", + f"Could not query kernels ({type(exc).__name__}). " + f"Install deps, then: python -m ipykernel install --user --name workshop --display-name \"Workshop\"") + return + ok = "workshop" in specs + record("PASS" if ok else "FAIL", "Jupyter 'workshop' kernel registered", + "" if ok else 'python -m ipykernel install --user --name workshop --display-name "Workshop"') + + +def check_notebooks() -> None: + for tool in NOTEBOOK_DIRS: + d = REPO_ROOT / "tutorials" / tool / "notebooks" + n = len(list(d.rglob("*.ipynb"))) if d.exists() else 0 + record("PASS" if n else "FAIL", f"teaching notebooks: {tool} ({n} found)", + "" if n else "python scripts/fetch_notebooks.py (committed copies should already be present - check your clone).") + + +def check_data() -> None: + missing = [p for p in DATA_INPUTS + if not (_nonempty_dir(p) if p.suffix == "" else p.exists())] + if missing: + names = ", ".join(p.relative_to(SESSION).as_posix() for p in missing) + record("WARN", "capstone data present", + f"Missing: {names}. Run: python scripts/get_data.py --session prerecorded --skip-video") + else: + record("PASS", "capstone data present (prerecorded)") + + +def main() -> int: + print("Workshop install self-check\n" + "=" * 27) + print(f"repo: {REPO_ROOT}\npython: {sys.executable}\n") + for check in (check_python, check_venv, check_ffmpeg, check_tools, + check_kernel, check_notebooks, check_data): + try: + check() + except Exception as exc: # never let the checker itself crash the gate + record("FAIL", f"{check.__name__} crashed", f"{type(exc).__name__}: {exc}") + + fails = [r for r in _results if r[0] == "FAIL"] + warns = [r for r in _results if r[0] == "WARN"] + passes = [r for r in _results if r[0] == "PASS"] + print("\n" + "-" * 27) + print(f"{len(passes)} passed, {len(warns)} warning(s), {len(fails)} failed.") + if fails: + print("\nNOT READY - fix the FAIL items above and re-run. Send this output to the organizers if stuck.") + return 1 + if warns: + print("\nReady, with warnings (see WARN above) - usually fine, but worth a look.") + else: + print("\nAll checks passed - you're ready for the workshop.") + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/tutorials/eztrack/README.md b/tutorials/eztrack/README.md index 1a3e974..5673ec9 100644 --- a/tutorials/eztrack/README.md +++ b/tutorials/eztrack/README.md @@ -10,8 +10,9 @@ Installed from the fork via `pip install git+https://github.com/daharoni/ezTrack ## Get the notebooks -`scripts/fetch_notebooks.py` copies eztrack's bundled notebooks into -`tutorials/eztrack/notebooks/`. To (re)fetch just eztrack: +eztrack's teaching notebooks are already committed under +`tutorials/eztrack/notebooks/`. `scripts/fetch_notebooks.py` optionally refreshes +them to your installed eztrack. To (re)fetch just eztrack: ```bash eztrack notebooks list # see what's available diff --git a/tutorials/eztrack/notebooks/LocationTracking_BatchProcess.ipynb b/tutorials/eztrack/notebooks/LocationTracking_BatchProcess.ipynb new file mode 100644 index 0000000..215f5a1 --- /dev/null +++ b/tutorials/eztrack/notebooks/LocationTracking_BatchProcess.ipynb @@ -0,0 +1,231 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "id": "0", + "metadata": {}, + "source": [ + "# ezTrack — Location Tracking (batch)\n", + "\n", + "Apply one set of selections and tracking parameters to every video in a folder. Draw the crop / mask / ROIs / scale once on the first video (this assumes every video shares the same framing); a fresh reference frame is built per video." + ] + }, + { + "cell_type": "markdown", + "id": "1", + "metadata": {}, + "source": [ + "## 1. Load packages" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "2", + "metadata": {}, + "outputs": [], + "source": [ + "import holoviews as hv\n", + "import eztrack as ez\n", + "\n", + "hv.extension(\"bokeh\")" + ] + }, + { + "cell_type": "markdown", + "id": "3", + "metadata": {}, + "source": [ + "## 2. Point at the folder\n", + "`ftype` is the video file extension to process. `region_names` names the ROIs to draw." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "4", + "metadata": {}, + "outputs": [], + "source": [ + "session = ez.Session(\n", + " dpath=\"../../PracticeVideos/\",\n", + " ftype=\"mp4\",\n", + " start=0,\n", + " end=None,\n", + " # Speed knobs (reduction factors, 1 = off); outputs stay in original space.\n", + " spatial_downsample=1, # 2 = track at half resolution\n", + " temporal_downsample=1, # 2 = track every other frame (one row per tracked frame)\n", + " region_names=[\"left\", \"right\"],\n", + ")\n", + "\n", + "ez.discover_files(session)\n", + "session.file_names" + ] + }, + { + "cell_type": "markdown", + "id": "5", + "metadata": {}, + "source": [ + "## 3. Draw selections on the first video\n", + "The crop/mask/ROIs/scale you draw here are applied to every file. We load the first file and build its reference so the ROI tool has an image to draw on." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "6", + "metadata": {}, + "outputs": [], + "source": [ + "session.file = session.file_names[0]\n", + "\n", + "hv.output(size=50)\n", + "ez.crop_tool(session)" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "7", + "metadata": {}, + "outputs": [], + "source": [ + "hv.output(size=100)\n", + "ez.mask_tool(session)" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "8", + "metadata": {}, + "outputs": [], + "source": [ + "ez.reference_frame(session, num_frames=50)\n", + "\n", + "hv.output(size=100)\n", + "ez.roi_tool(session)" + ] + }, + { + "cell_type": "markdown", + "id": "9", + "metadata": {}, + "source": [ + "## 4. (Optional) Scale" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "10", + "metadata": {}, + "outputs": [], + "source": [ + "hv.output(size=100)\n", + "ez.distance_tool(session)" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "11", + "metadata": {}, + "outputs": [], + "source": [ + "ez.set_scale(session, real_distance=100, unit=\"cm\")" + ] + }, + { + "cell_type": "markdown", + "id": "12", + "metadata": {}, + "source": [ + "## 5. (Optional) Save / reuse the selections\n", + "`session.selections.save(\"selections.json\")` — or load a previously saved set with `session.selections = ez.Selections.load(\"selections.json\")`." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "13", + "metadata": {}, + "outputs": [], + "source": [ + "session.selections.save(\"selections.json\")" + ] + }, + { + "cell_type": "markdown", + "id": "14", + "metadata": {}, + "source": [ + "## 6. Set tracking parameters\n", + "Same options as the single-video notebook (step 9 there explains each): `threshold_pct` (or `threshold_abs` for a fixed 0–255 cutoff instead of the percentile, with `threshold_on` choosing whether that cutoff is measured against the baseline difference or the raw pixel value), `method`, `window`, and `denoise`/`denoise_kernel` (drop specks / a thin wire). These apply to every video in the batch. (Position outlier removal is applied in the run step below, since the batch has no interactive per-file review.)" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "15", + "metadata": {}, + "outputs": [], + "source": [ + "params = ez.TrackParams(\n", + " threshold_pct=99.5, # used only when threshold_abs is None\n", + " threshold_abs=None, # fixed cutoff (0-255); overrides the percentile when set\n", + " threshold_on=\"difference\", # what threshold_abs measures: \"difference\" (vs baseline) | \"raw\" (pixel value)\n", + " method=\"abs\", # \"abs\" | \"light\" | \"dark\" (raw mode needs \"dark\" or \"light\")\n", + " window=ez.Window(size=100, weight=0.9), # or window=None to disable\n", + " denoise=False, # True = remove small specks / thin wire from the mask\n", + " denoise_kernel=5, # opening kernel in px (only used when denoise=True)\n", + ")" + ] + }, + { + "cell_type": "markdown", + "id": "16", + "metadata": {}, + "source": [ + "## 7. Run the batch\n", + "Writes `