prompt: route covariance-interpolator-rng-seed (#1450) → start_library

active.md

@@ -1,5 +1,21 @@
 # Active Tasks
 
+## covariance-interpolator-rng-seed
+- issue: https://github.com/PyAutoLabs/PyAutoFit/issues/1450
+- status: library-dev — fix COMMITTED AND PUSHED to `claude/covariance-interpolator-rng-seed-8zexp3` (PyAutoFit 68a8b391). NO PR opened (not requested). Cloud session, no worktree and no `gh` CLI; issue filed via the GitHub MCP surface.
+- why it matters: this is the Class B release blocker — the sole reason the 2026.8.2.1 LIVE release failed (PyAutoHands run 30736527569, `release_test_pypi (3.12, PyAutoFit, main)` step 9: 1641 passed, 1 failed, `assert 30.121646313498022 == 25.0 +/- 5`).
+- PROMPT PREMISE WAS WRONG, and this is the finding that matters: the unseeded `np.random.random()` in the test body is NOT the dominant cause. Seeding only it leaves std 1.95 (of 2.05) and the test still fails. The dominant source is dynesty's `rstate`, which `NestedSampler` defaults to `Generator(PCG64(None))` — OS entropy, reachable from neither `np.random.seed` nor a local Generator. Third source: the stdlib `random` module, used by `initializer.py:301` for the search's initial unit values. All three must be seeded.
+- measured pre-fix failure rate (the prompt's step 2, done before touching anything): 2000 runs under CI conditions → 33/2000 = 1.65% miss `abs=5.0`, 95% Wilson CI 1.18–2.31%. Recovered value mean 25.24, std 2.05, range 17.82–32.04. Percent-level, so per the prompt's own criterion seeding alone would have been papering over something.
+- BUT the tolerance is NOT mis-calibrated: with a full untruncated search the same test recovers 25.0496 +/- 0.0039 (n=20) — 500x tighter. The estimator is what is broken, not `abs=5.0`. The autouse `limit_maxcall` fixture (added with PyAutoFit#1386 for speed) caps every search in the module at ONE likelihood call, so the "recovered" value is an unconverged draw. Stage-wise check with seeds fixed: interpolator inputs (x, y, inverse covariance) are bit-identical; ALL variance is inside `search.fit`.
+- SIBLING IS WORSE (prompt asked to check): `test_single_variable` contains NO `np.random` call at all, yet gave 500 distinct values over 500 runs and missed its `abs=2.0` tolerance 18/500 = 3.6%. It rides on dynesty's rstate alone — seeding `np.random` would never have touched it. Fixed by the same fixture.
+- RULED OUT explicitly: the `scipy.linalg.LinAlgError` guards on `test_interpolate` / `test_relationships` / `test_interpolate_other_field` / `test_linear_analysis_for_value` (`e29c69ef2`) are NOT this nondeterminism in disguise. The `interpolator` conftest fixture uses no RNG; its covariance matrix is bit-identical across builds and RANK-DEFICIENT (rank 6 of 9, each 3x3 block rank 2 of 3, cond 1.3e17). Those guards cover inverting a singular matrix — deterministic input, platform-LAPACK dependent. Separate issue, do not fold in.
+- verification: recovered values bit-identical across 8 pytest invocations and in both selection contexts (whole module vs `-k`) — `test_single_variable` 24.577316739353 (dev 0.42 vs tol 2.0), `test_variable_and_constant` 22.718814166116 (dev 2.28 vs tol 5.0). Full suite 1642 passed, 6 skipped. One pre-existing environment failure, `test_nautilus.py::test__single_core_builds_no_pool` (`ModuleNotFoundError: nautilus`, optional dep absent in the sandbox) — reproduced identically with the change stashed, NOT ours.
+- KNOWN LIMITATION, do not oversell the fix: under `maxcall=1` both assertions are now frozen-RNG REGRESSION checks, not accuracy checks — a seeded draw 2.28 from the true 25.05 still "passes" at `abs=5.0`. Restoring meaning needs budget (measured `maxcall=200` → std 1.06 @ 0.76 s/draw; `maxcall=1000` → std 0.46 @ 2.67 s/draw; full → std 0.004 @ ~25 s/draw) or a seedable search.
+- follow-up filed: `draft/feature/autofit/search_seed_reproducibility.md` — PyAutoFit has NO way to make a search reproducible. `AbstractDynesty.search_kwargs` is a closed dict so `rstate` cannot be passed through even though dynesty accepts it, which is exactly why this fix must monkeypatch a third-party module from a test. Deliberately NOT folded in, to keep the release unblock test-only.
+- prompt: active/covariance_interpolator_test_unseeded_rng.md
+- worktree: (none — cloud session, worked in the canonical /home/user/PyAutoFit checkout on the mandated branch)
+- repos-single-claim: PyAutoFit is the only affected repo, named on this one line deliberately and NOT as a 2-space `  - PyAutoFit` bullet, because worktree_check_conflict reads any such bullet as a live claim.
+
 ## mge-sigma-min-workspace-sweep
 - issue: https://github.com/PyAutoLabs/autolens_workspace/issues/466
 - status: BOTH PHASES MERGED 2026-08-04. Phase 1 autolens_workspace#467 -> 92019316 (issue #466 auto-closed). Phase 2 autogalaxy_workspace#203 -> 8a7df7a6, HowToLens#67 -> 4ff3135c, HowToGalaxy#61 -> 51eed3d6, autogalaxy_assistant#10 -> f6966a64. Upstream PyAutoGalaxy#549 -> 13d3023c. All worktrees removed, all branches deleted local+origin, all five canonical checkouts back on main. Code work COMPLETE; two debts remain (below).

draft/bug/autofit/covariance_interpolator_test_unseeded_rng.md → active/covariance_interpolator_test_unseeded_rng.md

@@ -7,8 +7,8 @@ Repos:
 Difficulty: small
 Autonomy: safe
 Priority: high
-Status: formalised
-Issue: (none yet)
+Status: issued
+Issue: https://github.com/PyAutoLabs/PyAutoFit/issues/1450
 
 ## The defect
 

draft/feature/autofit/search_seed_reproducibility.md

@@ -0,0 +1,83 @@
+# Give PyAutoFit searches a `seed` — today no search can be made reproducible
+
+Type: feature
+Target: autofit
+Repos:
+- PyAutoFit
+Difficulty: medium
+Autonomy: supervised
+Priority: medium
+Status: formalised
+Issue: (none yet)
+
+## The gap
+
+There is no supported way to make a PyAutoFit search reproducible. A search's
+randomness comes from three places, and the caller can reach none of them:
+
+1. **The sampler's own generator.** `DynestyStatic.search_internal_from` builds
+   `dynesty.NestedSampler(...)` and splats `**self.search_kwargs`
+   (`autofit/non_linear/search/nest/dynesty/search/abstract.py:157`). That
+   property is a **closed dict** of eleven fixed keys — `bound`, `sample`,
+   `walks`, … — so an extra kwarg cannot be threaded through it. `dynesty`
+   itself *does* accept `rstate`, and defaults it to
+   `np.random.Generator(PCG64(None))` — OS entropy
+   (`dynesty/dynesty.py:607-608`, `814`). So the one knob that would fix this
+   exists in the dependency and is unreachable from PyAutoFit.
+2. **The initializer.** `autofit/non_linear/initializer.py:301` draws initial
+   unit values with the stdlib `random.uniform`, i.e. off the process-global
+   `random` module rather than a search-owned generator.
+3. **`numpy.random`**, wherever callers or model code touch the legacy global.
+
+Seeding any one of these leaves the other two free.
+
+## Why it matters
+
+This is the *root* cause behind
+`complete/…/covariance_interpolator_test_unseeded_rng.md`
+(PyAutoFit#1450) — the flake that killed the **2026.8.2.1 live release**. That
+fix had to seed all three sources from a test fixture, and to reach (1) at all
+it **monkeypatches `dynesty.dynesty.get_random_generator`** — reaching into a
+third-party module's namespace from a test, which is exactly the kind of thing
+that breaks silently on a dependency upgrade.
+
+Two consequences beyond that one test:
+
+- **Any** test that runs a real search is unseedable by construction, so the
+  same flake class can reappear anywhere in the suite. `test_single_variable`
+  in that same file was already flaking at 3.6% on source (1) alone, with no
+  random call of its own.
+- **Users cannot reproduce a fit.** Re-running the same model on the same data
+  gives a different chain. For a scientific inference library that is a real
+  gap, not only a test-hygiene one.
+
+## Proposed work
+
+1. Add a `seed` parameter to the search classes (`AbstractSearch`, honoured by
+   the dynesty searches first), defaulting to `None` = today's behaviour, so
+   this is additive and no existing fit changes.
+2. When `seed` is set, derive and pass `rstate` into the dynesty sampler —
+   either by adding `rstate` to `search_kwargs` or by giving subclasses a hook
+   to extend it. Check `DynamicNestedSampler` takes the same route.
+3. Give the initializer a generator seeded from the same `seed`, instead of the
+   global `random` module.
+4. **Check the identifier and serialization surface before committing to the
+   shape.** PyAutoFit hashes search configuration into run identifiers
+   (`__identifier_fields__`) and writes search config to JSON/the database. A
+   `np.random.Generator` is not JSON-serializable, so store the integer seed,
+   not a generator object, and decide deliberately whether `seed` belongs in
+   `__identifier_fields__` — including it changes identifiers for existing
+   runs, which is the trap PyAutoGalaxy#549 was careful about.
+5. Once it exists, simplify the `seed_search_randomness` fixture in
+   `test_autofit/interpolator/test_covariance.py` to use it and drop the
+   `dynesty.dynesty` monkeypatch.
+6. Consider extending to the other samplers (emcee/zeus already take a seed
+   concept; nautilus and the MLE searches need checking) — but land dynesty
+   first rather than blocking on full coverage.
+
+## Exit criteria
+
+A search constructed with a fixed `seed` produces a bit-identical result across
+repeated runs and across processes; no existing search's identifier or output
+changes when `seed` is left unset; the interpolator test fixture no longer
+monkeypatches a third-party module.