Skip to content

Guard loss() and pct_close() denominators and catch Inf, not just NaN#96

Merged
MaxGhenis merged 1 commit into
mainfrom
fix/metrics-denominator
Apr 17, 2026
Merged

Guard loss() and pct_close() denominators and catch Inf, not just NaN#96
MaxGhenis merged 1 commit into
mainfrom
fix/metrics-denominator

Conversation

@MaxGhenis
Copy link
Copy Markdown
Contributor

Summary

Finding #7 (MED). Both functions divide by target + 1, which crosses zero at target == -1 — producing Inf and silently corrupting the loss. The NaN guard in loss() was specifically torch.isnan, so Inf passed through undetected and calibration diverged without any diagnostic. Any other upstream non-finite value (e.g. an Inf estimate from a broken estimate function) slipped past the same check.

Factored out _safe_denominator(targets) that clamps target + 1 to eps = 1e-12 so the value is unchanged for any well-behaved target but never zero. Switched the guard to torch.isfinite(...).all() so Inf is rejected alongside NaN, with an error message pointing at likely causes.

Test plan

  • Add tests/test_metrics_regression.py: target == -1 no longer produces Inf, explicit Inf estimates now raise, value preservation for normal targets, and pct_close gets the same guard.
  • All existing tests pass (uv run pytest tests -x -q -> 19 passed).

🤖 Generated with Claude Code

Two related problems in src/microcalibrate/utils/metrics.py:

1. Both functions divide by `target + 1`, which crosses zero at
   `target == -1` -- producing Inf and silently corrupting the loss.
   The NaN guard in loss() was specifically `torch.isnan`, so Inf
   passed through undetected and calibration diverged without any
   diagnostic.

2. Any other upstream non-finite value (e.g. an Inf estimate from a
   broken estimate function or dropout misconfiguration) slipped past
   the same isnan check.

Factor out a `_safe_denominator` helper that clamps `target + 1` to
`eps` (1e-12) so the value is unchanged for any well-behaved target
but never zero. Switch the guard to `torch.isfinite(...).all()` so
Inf is rejected alongside NaN, with an error message pointing at
likely causes.

Adds tests/test_metrics_regression.py covering target == -1 (no Inf,
result is finite), explicit Inf estimate (now raises), value
preservation for normal targets, and pct_close with the same guard.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@vercel
Copy link
Copy Markdown
Contributor

vercel Bot commented Apr 17, 2026

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated (UTC)
microcalibrate Ready Ready Preview, Comment Apr 17, 2026 0:40am

Request Review

Copy link
Copy Markdown
Contributor Author

@MaxGhenis MaxGhenis left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM (cannot self-approve; posting as comment).

  • _safe_denominator(targets) clamps targets + 1 to eps = 1e-12, preventing divide-by-zero when target == -1. For any well-behaved target (target + 1 >= eps) the numeric result is unchanged — verified by test_loss_preserves_values_for_well_behaved_targets.
  • Guard broadened from torch.isnan(rel_error).any() to not torch.isfinite(rel_error).all(), so Inf is now caught (was silently passing through before). Error message is informative.
  • pct_close uses the same _safe_denominator, so the metric stays finite on pathological targets.

Scope note (not blocking): finding #7 also flagged the "magnitude asymmetry" of the +1 smoothing for sub-unit targets. This PR deliberately preserves the original +1 semantics and only guards the denominator floor — a narrow, conservative fix. Revisiting the principled scale (e.g. max(|target|, eps)) is a bigger behaviour change best handled separately. Leaving as-is is fine.

@MaxGhenis MaxGhenis merged commit a0e3e61 into main Apr 17, 2026
6 checks passed
@MaxGhenis MaxGhenis deleted the fix/metrics-denominator branch April 17, 2026 16:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant