diff --git a/active.md b/active.md index e4a5fa2b..3b4dcb22 100644 --- a/active.md +++ b/active.md @@ -1,5 +1,21 @@ # Active Tasks +## covariance-interpolator-rng-seed +- issue: https://github.com/PyAutoLabs/PyAutoFit/issues/1450 +- status: library-dev — fix COMMITTED AND PUSHED to `claude/covariance-interpolator-rng-seed-8zexp3` (PyAutoFit 68a8b391). NO PR opened (not requested). Cloud session, no worktree and no `gh` CLI; issue filed via the GitHub MCP surface. +- why it matters: this is the Class B release blocker — the sole reason the 2026.8.2.1 LIVE release failed (PyAutoHands run 30736527569, `release_test_pypi (3.12, PyAutoFit, main)` step 9: 1641 passed, 1 failed, `assert 30.121646313498022 == 25.0 +/- 5`). +- PROMPT PREMISE WAS WRONG, and this is the finding that matters: the unseeded `np.random.random()` in the test body is NOT the dominant cause. Seeding only it leaves std 1.95 (of 2.05) and the test still fails. The dominant source is dynesty's `rstate`, which `NestedSampler` defaults to `Generator(PCG64(None))` — OS entropy, reachable from neither `np.random.seed` nor a local Generator. Third source: the stdlib `random` module, used by `initializer.py:301` for the search's initial unit values. All three must be seeded. +- measured pre-fix failure rate (the prompt's step 2, done before touching anything): 2000 runs under CI conditions → 33/2000 = 1.65% miss `abs=5.0`, 95% Wilson CI 1.18–2.31%. Recovered value mean 25.24, std 2.05, range 17.82–32.04. Percent-level, so per the prompt's own criterion seeding alone would have been papering over something. +- BUT the tolerance is NOT mis-calibrated: with a full untruncated search the same test recovers 25.0496 +/- 0.0039 (n=20) — 500x tighter. The estimator is what is broken, not `abs=5.0`. The autouse `limit_maxcall` fixture (added with PyAutoFit#1386 for speed) caps every search in the module at ONE likelihood call, so the "recovered" value is an unconverged draw. Stage-wise check with seeds fixed: interpolator inputs (x, y, inverse covariance) are bit-identical; ALL variance is inside `search.fit`. +- SIBLING IS WORSE (prompt asked to check): `test_single_variable` contains NO `np.random` call at all, yet gave 500 distinct values over 500 runs and missed its `abs=2.0` tolerance 18/500 = 3.6%. It rides on dynesty's rstate alone — seeding `np.random` would never have touched it. Fixed by the same fixture. +- RULED OUT explicitly: the `scipy.linalg.LinAlgError` guards on `test_interpolate` / `test_relationships` / `test_interpolate_other_field` / `test_linear_analysis_for_value` (`e29c69ef2`) are NOT this nondeterminism in disguise. The `interpolator` conftest fixture uses no RNG; its covariance matrix is bit-identical across builds and RANK-DEFICIENT (rank 6 of 9, each 3x3 block rank 2 of 3, cond 1.3e17). Those guards cover inverting a singular matrix — deterministic input, platform-LAPACK dependent. Separate issue, do not fold in. +- verification: recovered values bit-identical across 8 pytest invocations and in both selection contexts (whole module vs `-k`) — `test_single_variable` 24.577316739353 (dev 0.42 vs tol 2.0), `test_variable_and_constant` 22.718814166116 (dev 2.28 vs tol 5.0). Full suite 1642 passed, 6 skipped. One pre-existing environment failure, `test_nautilus.py::test__single_core_builds_no_pool` (`ModuleNotFoundError: nautilus`, optional dep absent in the sandbox) — reproduced identically with the change stashed, NOT ours. +- KNOWN LIMITATION, do not oversell the fix: under `maxcall=1` both assertions are now frozen-RNG REGRESSION checks, not accuracy checks — a seeded draw 2.28 from the true 25.05 still "passes" at `abs=5.0`. Restoring meaning needs budget (measured `maxcall=200` → std 1.06 @ 0.76 s/draw; `maxcall=1000` → std 0.46 @ 2.67 s/draw; full → std 0.004 @ ~25 s/draw) or a seedable search. +- follow-up filed: `draft/feature/autofit/search_seed_reproducibility.md` — PyAutoFit has NO way to make a search reproducible. `AbstractDynesty.search_kwargs` is a closed dict so `rstate` cannot be passed through even though dynesty accepts it, which is exactly why this fix must monkeypatch a third-party module from a test. Deliberately NOT folded in, to keep the release unblock test-only. +- prompt: active/covariance_interpolator_test_unseeded_rng.md +- worktree: (none — cloud session, worked in the canonical /home/user/PyAutoFit checkout on the mandated branch) +- repos-single-claim: PyAutoFit is the only affected repo, named on this one line deliberately and NOT as a 2-space ` - PyAutoFit` bullet, because worktree_check_conflict reads any such bullet as a live claim. + ## mge-sigma-min-workspace-sweep - issue: https://github.com/PyAutoLabs/autolens_workspace/issues/466 - status: BOTH PHASES MERGED 2026-08-04. Phase 1 autolens_workspace#467 -> 92019316 (issue #466 auto-closed). Phase 2 autogalaxy_workspace#203 -> 8a7df7a6, HowToLens#67 -> 4ff3135c, HowToGalaxy#61 -> 51eed3d6, autogalaxy_assistant#10 -> f6966a64. Upstream PyAutoGalaxy#549 -> 13d3023c. All worktrees removed, all branches deleted local+origin, all five canonical checkouts back on main. Code work COMPLETE; two debts remain (below). diff --git a/draft/bug/autofit/covariance_interpolator_test_unseeded_rng.md b/active/covariance_interpolator_test_unseeded_rng.md similarity index 97% rename from draft/bug/autofit/covariance_interpolator_test_unseeded_rng.md rename to active/covariance_interpolator_test_unseeded_rng.md index c31a5cd1..faeea971 100644 --- a/draft/bug/autofit/covariance_interpolator_test_unseeded_rng.md +++ b/active/covariance_interpolator_test_unseeded_rng.md @@ -7,8 +7,8 @@ Repos: Difficulty: small Autonomy: safe Priority: high -Status: formalised -Issue: (none yet) +Status: issued +Issue: https://github.com/PyAutoLabs/PyAutoFit/issues/1450 ## The defect diff --git a/draft/feature/autofit/search_seed_reproducibility.md b/draft/feature/autofit/search_seed_reproducibility.md new file mode 100644 index 00000000..c0127771 --- /dev/null +++ b/draft/feature/autofit/search_seed_reproducibility.md @@ -0,0 +1,83 @@ +# Give PyAutoFit searches a `seed` — today no search can be made reproducible + +Type: feature +Target: autofit +Repos: +- PyAutoFit +Difficulty: medium +Autonomy: supervised +Priority: medium +Status: formalised +Issue: (none yet) + +## The gap + +There is no supported way to make a PyAutoFit search reproducible. A search's +randomness comes from three places, and the caller can reach none of them: + +1. **The sampler's own generator.** `DynestyStatic.search_internal_from` builds + `dynesty.NestedSampler(...)` and splats `**self.search_kwargs` + (`autofit/non_linear/search/nest/dynesty/search/abstract.py:157`). That + property is a **closed dict** of eleven fixed keys — `bound`, `sample`, + `walks`, … — so an extra kwarg cannot be threaded through it. `dynesty` + itself *does* accept `rstate`, and defaults it to + `np.random.Generator(PCG64(None))` — OS entropy + (`dynesty/dynesty.py:607-608`, `814`). So the one knob that would fix this + exists in the dependency and is unreachable from PyAutoFit. +2. **The initializer.** `autofit/non_linear/initializer.py:301` draws initial + unit values with the stdlib `random.uniform`, i.e. off the process-global + `random` module rather than a search-owned generator. +3. **`numpy.random`**, wherever callers or model code touch the legacy global. + +Seeding any one of these leaves the other two free. + +## Why it matters + +This is the *root* cause behind +`complete/…/covariance_interpolator_test_unseeded_rng.md` +(PyAutoFit#1450) — the flake that killed the **2026.8.2.1 live release**. That +fix had to seed all three sources from a test fixture, and to reach (1) at all +it **monkeypatches `dynesty.dynesty.get_random_generator`** — reaching into a +third-party module's namespace from a test, which is exactly the kind of thing +that breaks silently on a dependency upgrade. + +Two consequences beyond that one test: + +- **Any** test that runs a real search is unseedable by construction, so the + same flake class can reappear anywhere in the suite. `test_single_variable` + in that same file was already flaking at 3.6% on source (1) alone, with no + random call of its own. +- **Users cannot reproduce a fit.** Re-running the same model on the same data + gives a different chain. For a scientific inference library that is a real + gap, not only a test-hygiene one. + +## Proposed work + +1. Add a `seed` parameter to the search classes (`AbstractSearch`, honoured by + the dynesty searches first), defaulting to `None` = today's behaviour, so + this is additive and no existing fit changes. +2. When `seed` is set, derive and pass `rstate` into the dynesty sampler — + either by adding `rstate` to `search_kwargs` or by giving subclasses a hook + to extend it. Check `DynamicNestedSampler` takes the same route. +3. Give the initializer a generator seeded from the same `seed`, instead of the + global `random` module. +4. **Check the identifier and serialization surface before committing to the + shape.** PyAutoFit hashes search configuration into run identifiers + (`__identifier_fields__`) and writes search config to JSON/the database. A + `np.random.Generator` is not JSON-serializable, so store the integer seed, + not a generator object, and decide deliberately whether `seed` belongs in + `__identifier_fields__` — including it changes identifiers for existing + runs, which is the trap PyAutoGalaxy#549 was careful about. +5. Once it exists, simplify the `seed_search_randomness` fixture in + `test_autofit/interpolator/test_covariance.py` to use it and drop the + `dynesty.dynesty` monkeypatch. +6. Consider extending to the other samplers (emcee/zeus already take a seed + concept; nautilus and the MLE searches need checking) — but land dynesty + first rather than blocking on full coverage. + +## Exit criteria + +A search constructed with a fixed `seed` produces a bit-identical result across +repeated runs and across processes; no existing search's identifier or output +changes when `seed` is left unset; the interpolator test fixture no longer +monkeypatches a third-party module.