docs(search): state the opt-in rule for _test_mode_samples_info

[Jammy2211]

Overview

AbstractSearch._test_mode_samples_info() is an opt-in hook, but its docstring
reads as a per-sampler obligation — so the fact that only one of nine searches
overrides it looks like an oversight. It isn't. This issue writes the real rule
into the docstring so the next person hitting a bypass KeyError knows which of
the two fixes applies.

Split out of autofit_workspace_test#83, where the same confusion cost an
investigation.

Plan

• Rewrite the _test_mode_samples_info docstring in
  autofit/non_linear/search/abstract_search.py: state that the hook is opt-in,
  carry the print-vs-assert test, and cite both live precedents.
• Add no override and no test — see the rejected alternatives below.
• Docstring-only: no behaviour change.

Detailed implementation plan

The mechanism

Under PYAUTO_TEST_MODE=2 / =3, AbstractSearch._fit_bypass_test_mode
(abstract_search.py:864) skips the sampler and hand-builds a result: fake
samples at the prior median, plus a samples_info of only
{total_iterations: 1, time: 0.0, log_evidence: <ll>}, merged at line 947 with
self._test_mode_samples_info(). The base hook (line 979) returns {}.

Nine search modules write samples_info in their real path — nautilus, dynesty,
emcee, zeus, bfgs, drawer, blackjax nuts, multi_start_gradient, abstract. Exactly
one overrides the hook: BlackJAXNUTS
(.../mcmc/blackjax/nuts/search.py:386).

Why the asymmetry is correct

Reachability. No library path reads these diagnostic keys under bypass. The
properties that read them — SamplesMCMC.total_steps (samples/mcmc.py:200),
SamplesNest.number_live_points / total_samples / log_evidence
(samples/nest.py:77-92) — live on Samples subclasses the bypass never
constructs; _fit_bypass_test_mode always builds a SamplesPDF. The only
possible consumers are workspace scripts reading samples_info[...] directly.

Consumer sweep across all eleven workspace/tutorial repos finds exactly two
such sites:

Site	Keys	Runs bypassed?
autofit_workspace/scripts/searches/mcmc.py:335	ess_min, num_samples, mean_acceptance, n_divergent, n_logl_evals	yes — no __Env__ declaration
autofit_workspace_test/.../multi_start_gradient_auto_convergence.py:130	total_steps	no — declares ENV: real_search jax

The first is precisely why the NUTS override exists: 5d175ebcc (#1260, May
2026) added both the hook and that override to stop this tutorial crashing under
smoke. The second was fixed on the workspace side (wst#83 / PR#84, merged
f4c45c1).

Nautilus, Dynesty, Emcee and Zeus have no bypassed consumer either — which is why
none of them override the hook. autofit_workspace/scripts/searches/mle.py uses
MultiStartAdam but never touches samples_info, so MultiStartGradient has no
bypassed consumer at all.

The rule both decisions actually followed

Prints → placeholders. Asserts → real search. A tutorial that only
displays diagnostics may legitimately run bypassed, so its search needs the
override with honest empties. A script that asserts on diagnostics must not
run bypassed at all — it declares ENV: real_search, and no override is wanted.

Rejected alternatives

• Add an AbstractMultiStartGradient override (for surface consistency with
  NUTS): rejected. It serves no existing consumer, and a placeholder
  total_steps would let a future assert total_steps < n_steps silently pass
  on a stub 0 — the exact failure mode avoided in wst#83.
• A unit test pinning the bypass keys: rejected. A test can pin
  BlackJAXNUTS's keys but cannot express "and the others deliberately have
  none" without freezing the sampler roster.

Affected Repositories

• PyAutoFit (primary)

Branch Survey

Repository	Current Branch	Dirty?
./PyAutoFit	main	clean

Two active.md entries claimed PyAutoFit and both were stale: the
point-source-defaults-campaign line read "PR #1441 OPEN" (merged
2026-08-01T12:47:32Z) and the nautilus-1core-serial-pool line read "PR OPEN …
restore main after merge" (#1443 merged 2026-08-01T19:12:47Z, merge commit
5bf32dab, contained in origin/main). Both released; neither is a live
conflict.

Suggested branch: feature/test-mode-samples-info-hook-contract

Worktree root: ~/Code/PyAutoLabs-wt/test-mode-samples-info-hook-contract/

Work Classification: Library

Implementation Steps

1. autofit/non_linear/search/abstract_search.py — rewrite the
   _test_mode_samples_info docstring (line 979) per the rule above, naming both
   precedents and the "do NOT add placeholders for an asserting reader" hazard.
2. Run the search unit tests; confirm docstring-only diff.

Key Files

• autofit/non_linear/search/abstract_search.py:979 — the hook to document.
• autofit/non_linear/search/mcmc/blackjax/nuts/search.py:386 — the one override
  (unchanged; cited as precedent).
• autofit/non_linear/search/mle/multi_start_gradient/search.py — the deliberate
  non-override (unchanged; cited as counter-precedent).

Acceptance

The base hook's docstring states the opt-in rule and the print-vs-assert test
explicitly. Docstring-only — no behaviour change, no new override, no new test.

Original Prompt

Click to expand starting prompt

Filed 2026-08-04 as PyAutoMind/draft/bug/autofit/multi_start_test_mode_samples_info_gap.md,
split out of autofit_workspace_test#83:

AbstractMultiStartGradient has no _test_mode_samples_info override (NUTS does).
Decide whether the hook contract is "every sampler with diagnostic samples_info
keys must override it" (→ add the MultiStartGradient override with NaN/0/None
placeholders, per the NUTS precedent) or "the hook is opt-in for keys downstream
code genuinely reads under bypass" (→ leave it, and say so in the hook docstring
so the asymmetry stops looking like an oversight). Weigh against the workspace
doctrine that a bypassed run should fail loudly rather than silently produce
placeholder diagnostics.

[Jammy2211]

Correction to the issue body above: the class is NonLinearSearch (autofit/non_linear/search/abstract_search.py:136), not AbstractSearch. The line references (_fit_bypass_test_mode at :864, the hook at :979, the merge at :947) are correct; only the class name was wrong. The PR uses the right name.

[Jammy2211]

Shipped — 2026-08-04

PRs

• docs(search): state the opt-in rule for _test_mode_samples_info #1449 (MERGED as 8f706c51)

Summary

NonLinearSearch._test_mode_samples_info()'s docstring now states that the hook
is opt-in, carries the print-vs-assert test, and cites both live precedents.
Docstring-only — no behaviour change, no new override, no new test.

Session Notes

The asymmetry was correct; the docstring was wrong. Nine searches write
samples_info; only BlackJAXNUTS overrides the hook. That is right because no
library path reads these keys under bypass — the properties that read them
(SamplesMCMC.total_steps, SamplesNest.total_samples, …) live on Samples
subclasses _fit_bypass_test_mode never constructs, since it always builds a
SamplesPDF. The only consumers are workspace scripts reading samples_info[...]
directly, and across all eleven workspace/tutorial repos there are exactly two.

Why NUTS has one and MultiStartGradient doesn't.
autofit_workspace/scripts/searches/mcmc.py prints NUTS diagnostics, has no
__Env__ declaration, and is line 2 of that workspace's smoke_tests.txt — so it
runs bypassed on every PR, which is what #1260 was fixing. The multi-start
script asserts on total_steps, so it was given ENV: real_search jax
instead (autofit_workspace_test#83 / PR#84).

Alternatives rejected. Adding an AbstractMultiStartGradient override for
surface consistency would serve no existing consumer and would let a future
assert total_steps < n_steps silently pass on a placeholder 0 — strictly worse
than the KeyError it replaces. A unit test can pin BlackJAXNUTS's keys but
cannot express "and the others deliberately have none" without freezing the
sampler roster.

Validation. pytest test_autofit/non_linear/search 172 passed / 1 skipped;
PR CI green on all three checks including docs / docs-build (the one that would
catch malformed RST in a docstring); diff confirmed to contain no non-docstring
line.