Skip to content

Seed the NumPy RNG used for initial weight noise in reweight()#93

Merged
MaxGhenis merged 1 commit into
mainfrom
fix/np-seed
Apr 17, 2026
Merged

Seed the NumPy RNG used for initial weight noise in reweight()#93
MaxGhenis merged 1 commit into
mainfrom
fix/np-seed

Conversation

@MaxGhenis
Copy link
Copy Markdown
Contributor

Summary

Finding #3 (HIGH). Calibration.__init__ called torch.manual_seed(self.seed) but never np.random.seed(self.seed), yet reweight() drew its initial weight noise from the unseeded global np.random stream. Two runs with the same seed therefore produced different initial log-weights and diverged thereafter, breaking the documented reproducibility guarantee and destabilising CI, hyperparameter tuning, and holdout robustness.

reweight() now accepts an explicit seed parameter. When provided it seeds torch (including CUDA) and draws initial noise from a local numpy.random.default_rng(seed), which keeps the caller's global numpy RNG state untouched. Calibration.calibrate() threads its self.seed through.

Test plan

  • Add tests/test_seed_determinism.py: identical seeds → identical weights, different seeds → different weights, calibration does not mutate the caller's global numpy RNG stream.
  • All existing tests pass (uv run pytest tests -x -q -> 18 passed).

🤖 Generated with Claude Code

@vercel
Copy link
Copy Markdown
Contributor

vercel Bot commented Apr 17, 2026

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated (UTC)
microcalibrate Ready Ready Preview, Comment Apr 17, 2026 4:23pm

Request Review

Copy link
Copy Markdown
Contributor Author

@MaxGhenis MaxGhenis left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM (cannot self-approve; posting as comment).

  • Local np.random.default_rng(seed) — does not mutate the caller's global numpy state. The dedicated test test_calibration_does_not_mutate_global_numpy_state verifies this.
  • Explicit seed parameter on reweight(), threaded from Calibration.calibrate() via self.seed. Clean API.
  • torch.manual_seed(seed) + torch.cuda.manual_seed_all(seed) guarded by if seed is not None — preserves the previous non-deterministic behaviour when no seed is given.
  • Three tests (same-seed determinism, different-seed divergence, global-state preservation) cover the reproducibility contract end-to-end with noise_level > 0 so the numpy RNG is actually exercised.

Note: this branch doesn't include the np.maximum(..., 1e-12) guard from #91; once #91 merges, #93 will need a trivial rebase. That's the case for all 7 branches as they're independent off origin/main.

Calibration.__init__ called torch.manual_seed(self.seed) but not
np.random.seed(self.seed), yet reweight() drew its initial weight
noise from the unseeded global np.random stream. Two runs with the
same seed therefore produced different initial log-weights and
diverged thereafter, breaking the documented reproducibility
guarantee and destabilising CI, hyperparameter tuning, and holdout
robustness.

reweight() now accepts an explicit `seed` parameter. When provided it
seeds torch (including CUDA) and draws initial noise from a local
`numpy.random.default_rng(seed)`, which keeps the caller's global
numpy RNG state untouched. Calibration.calibrate() threads its
`self.seed` through.

Adds tests/test_seed_determinism.py covering (a) identical seeds ->
identical weights, (b) different seeds -> different weights, and (c)
calibration does not mutate the caller's global numpy RNG stream.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@MaxGhenis MaxGhenis merged commit 1e6e642 into main Apr 17, 2026
6 checks passed
@MaxGhenis MaxGhenis deleted the fix/np-seed branch April 17, 2026 16:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant