Centralized environment & config reporting for lecture series

[mmcky]

Summary

Umbrella issue for a small initiative to give us a clear, low-noise view of how each lecture series is built and configured — and to surface environment changes in the weekly report without drowning it in noise.

Origin: @jstac asked whether the automated weekly activity report should include environment details, and whether that would be too noisy. Conclusion: full environment/config data should be pull (a status dashboard + per-series pages), and the weekly report should surface only diffs (what changed week-over-week).

Architecture

publish-gh-pages (QuantEcon/actions) ──manifest + repository_dispatch──┐
daily backstop scrape ─────────────────────────────────────────────────┤
                                                                        ▼
                                                      QuantEcon/status-lectures
                                                 (in-repo JSON store + Pages dashboard)
                                                                │
                          ┌─────────────────────────────────────┴──────────────┐
                          ▼                                                      ▼
                   QuantEcon.manual                              action-activity-report
            (cross-cutting table + per-series block)          (week-over-week env diffs)

Decisions

• New repo QuantEcon/status-lectures is the data store + GitHub Pages dashboard (created, public).
• Collection: publish-push primary + daily backstop — publish-time manifests capture the effective (resolved) environment that actually built each site; the daily scrape covers not-yet-migrated repos and flags declared-vs-effective drift.
• Data lives in-repo (GitHub-native), consistent with our no-webapp-hosting ops.
• Naming: adopt a status- prefix convention for status/dashboard repos (data lives inside them).

Fields tracked per series

python version · anaconda/conda pin · jupyter-book version · theme version · sphinx extensions (+versions) · execute mode + timeout · only_build_toc_files · launch buttons · intersphinx targets · container vs conda + container tag/digest · runner (ubuntu vs AWS GPU) · GPU? · cache strategy · build duration.

Workstreams

1. Producer — emit effective-env manifests at publish + daily backstop scrape → Environment & config reporting: publish-time manifest + central data store actions#30
2. Store + dashboard — JSON store, dispatch intake, Pages dashboard → Scaffold: JSON store layout + Pages dashboard + dispatch intake status-lectures#1
3. Manual views — cross-cutting features table + per-series env block → QuantEcon/QuantEcon.manual#94
4. Weekly diffs — environment-change diffs in the activity report → QuantEcon/action-activity-report#25
5. Naming convention — add status- prefix to the repo conventions → QuantEcon/QuantEcon.manual#95

[mmcky]

Suggested development order

The dependency structure here is friendlier than it first looks: the daily backstop scrape means the store doesn't have to wait for the producer. Scraping _config.yml + environment.yml gives us declared config for every series with zero changes to QuantEcon/actions, so the dashboard can be populated and useful on day one — publish-push then upgrades series to effective data one at a time (which is exactly what makes it a migration tracker).

Proposed order:

1. Schema contract (joint: Environment & config reporting: publish-time manifest + central data store actions#30 + Scaffold: JSON store layout + Pages dashboard + dispatch intake status-lectures#1) — agree the v1 record shape and schema_version semantics first; it's the interface every other workstream consumes. Two decisions to settle here: the repository_dispatch payload cap (~64 KB — send curated fields, not a full conda list) and which org credential lecture repos use to dispatch into status-lectures.
2. Store + backstop scrape + dashboard (Scaffold: JSON store layout + Pages dashboard + dispatch intake status-lectures#1) — build the JSON store and the scheduled scrape, render the cross-cutting table from scraped (declared) data. No producer dependency; immediately useful.
3. Producer publish-push (Environment & config reporting: publish-time manifest + central data store actions#30) — emit the effective-environment manifest + dispatch intake, then seed one series end-to-end (suggest lecture-python-intro) and verify the backstop steps aside for it. Roll out to remaining series as they migrate.
4. Manual views (QuantEcon/QuantEcon.manual#94) — once the raw-JSON URLs are stable; reads data/index.json + data/series/<repo>.json.
5. Weekly diffs (QuantEcon/action-activity-report#25) — last, deliberately: diffs come from git history of the store, so this gets more meaningful after a few weeks of accumulated snapshots.

Anytime: the status- naming convention (QuantEcon/QuantEcon.manual#95) is independent and small — may as well land it first.

Development note: I'm treating status-lectures as the primary working repo for this initiative — the sibling repos get cloned into a gitignored repos/ folder there so the schema contract and producer/consumer code can be developed against each other in one workspace.