953 stochastic noise by maxwhitemet · Pull Request #2275 · metoppv/improver

maxwhitemet · 2026-01-08T12:09:22Z

Addresses #953.

This PR adds the plugin, CLI, and tests for stochastic noise generation.

The acceptance tests show the impact, with data available here.

Testing:

Ran tests and they passed OK
Added new tests for the new feature(s)

cpelley · 2026-01-12T08:44:28Z

@maxwhitemet, could you merge/rebase master into your branch. The failing "CI Tests / Test-Coverage (pull_request)" should hopefully be addressed by the now merged #2273

There does appear to be a new issue potentially with the improver documentation building now though... (different issue)

cpelley · 2026-01-13T11:41:26Z

Ignore the test coverage failure. Fixed in #2282

gavinevans

Thanks @maxwhitemet 👍

I've added some comments below.

envs/environment_b.yml

improver/cli/stochastic_noise.py

envs/conda-forge.yml

improver/precipitation/stochastic_noise.py

improver_tests/precipitation/stochastic_noise/test_StochasticNoise.py

improver/precipitation/stochastic_noise.py

maxwhitemet

Thank you for the review @gavinevans. I have now made the requested changes.

envs/conda-forge.yml

envs/environment_b.yml

improver/cli/stochastic_noise.py

improver/precipitation/stochastic_noise.py

improver_tests/precipitation/stochastic_noise/test_StochasticNoise.py

gavinevans

I've added some minor comments.

improver/precipitation/stochastic_noise.py

improver_tests/precipitation/stochastic_noise/test_StochasticNoise.py

maxwhitemet

Thanks @gavinevans. I have implemented your feedback

improver/precipitation/stochastic_noise.py

improver_tests/precipitation/stochastic_noise/test_StochasticNoise.py

bayliffe

Thanks Max, a few comments. The thrust of some of my comments it to make this less precipitation specific, but I've not highlighted everywhere this needs to happen, so think about variable names (e.g. dry_mask) and comments to make it less precip specific, using precip as an example if that's instructive.

I'm also interested in you making changes to better handle the expected shape of cubes and the coordinate order, rather than assuming realization is first and simply indexing it with that assumption.

bayliffe · 2026-02-16T09:58:44Z

improver/cli/stochastic_noise.py

+            pysteps.noise.fftgenerators.initialize_nonparam_2d_ssft_filter.
+            Provide as Python dict string,
+            e.g., "{'win_size': (100, 100), 'overlap': 0.3}".
+            Recommended keys: win_size, overlap, war_thr.


If we don't want to re-define these here, can we include a link to the pysteps documentation that can be clicked in the read-the-docs pages to make these quick to get to.

https://pysteps.readthedocs.io/en/stable/generated/pysteps.noise.fftgenerators.initialize_nonparam_2d_ssft_filter.html#pysteps.noise.fftgenerators.initialize_nonparam_2d_ssft_filter

bayliffe · 2026-02-17T10:03:37Z

improver/cli/stochastic_noise.py

+            Recommended keys: overlap, seed.
+        db_threshold:
+            Threshold value below which data will be set to a constant in dB scale
+            to avoid issues with log(0).


Suggested change

to avoid issues with log(0).

to avoid issues with log(0). Value provided in units of `db_threshold_units`.

bayliffe · 2026-02-17T10:06:50Z

improver/cli/stochastic_noise.py

+        scale_dry_noise:
+            If True, noise in dry regions (where template.data <= 0) will be scaled
+            such that the maximum noise value in those regions is zero and all other
+            noise values are negative.
+            This prevents the addition of positive noise to dry regions, which could
+            artificially increase precipitation values where the input cube
+            indicates no precipitation should occur.
+            Default is False.


You've written a relatively generic CLI here and then this one option is incredibly specific to precipitation. Could this be rewritten to be scale_zero_noise? There is the question of what happens to diagnostic for which the value could be negative as we've assumed a zero-bounded quantity by the looks of this. I'll get to that later I guess. It probably has implications for whether my suggested variable name is sensible.

bayliffe · 2026-02-17T10:22:59Z

improver/precipitation/stochastic_noise.py

+    correlated rather than independent. This is particularly useful for Ensemble Copula
+    Coupling-Quantile (ECC-Q) realization generation, where post-processing may indicate
+    non-zero precipitation should occur at locations where all raw ensemble members had
+    zero. In ECC reordering, these locations create ties (all raw members have identical
+    zero values) that cannot be meaningfully reordered. The spatially-structured noise
+    breaks these ties by adding small contiguous precipitation patches in dry regions,


We could make this generic by using the precipitation as an example rather than making it the sole use case as it appears as currently written, or just write in more general terms.

bayliffe · 2026-02-17T10:23:25Z

improver/precipitation/stochastic_noise.py

+                Number of worker threads for parallel FFT computation.
+                If not specified, uses the smaller of the plugin's default (number of
+                available CPUs) or the number of realizations in the input cube.
+            scale_dry_noise:


See comment in CLI.

bayliffe · 2026-02-17T10:37:33Z

improver/precipitation/stochastic_noise.py

+            return self.db_threshold
+
+    def _to_dB(self, cube: Cube) -> Cube:
+        """Convert cube data to dB scale with thresholding.


This description could be a little more expansive.

bayliffe · 2026-02-17T10:43:27Z

improver/precipitation/stochastic_noise.py

+        template_dB = self._to_dB(template.copy())
+
+        # Build delayed processing tasks for each realization
+        n_realiz = template.coord("realization").points.size


Go on, we can stretch to n_realizations

bayliffe · 2026-02-17T10:45:28Z

improver/precipitation/stochastic_noise.py

+        n_realiz = template.coord("realization").points.size
+        tasks = []
+        for k in range(n_realiz):
+            realiz_data = template_dB.data[k].astype(np.float32)


There is an implicit assumption that the first index of the cube here corresponds to realization but I don't think that's been written anywhere, checked, or enforeced.

bayliffe · 2026-02-17T11:07:23Z

improver/precipitation/stochastic_noise.py

+        # noise addition only to dry regions)
+        if not np.any(dry_mask):
+            output_cube = input_cube.copy()
+            return output_cube


Why are we copying this to return it? You could quite happily just return input_cube.

bayliffe · 2026-02-17T11:09:10Z

improver_tests/precipitation/stochastic_noise/test_StochasticNoise.py

+    assert result.data.dtype == np.float32
+
+    # All values in simple_cube are non-zero, so output should equal input
+    np.testing.assert_array_equal(result.data, simple_cube.data)


This can go away with the process method.

bayliffe · 2026-02-17T11:42:18Z

I forgot to say that this code is currently within the precipitation directory. If you do make it less precip-specific then it should probably move.

maxwhitemet mentioned this pull request Jan 8, 2026

Add acceptance test data for stochastic noise metoppv/improver_test_data#119

Open

maxwhitemet added 6 commits January 12, 2026 09:25

Initial creation of plugin

43f4b5f

Remove redundant unit conversion

6af5792

Add stochastic noise finalisations, including tests

aba7a64

Fix reStructuredText issue with Sphinx

3130aec

Add pysteps to Sphinx's mock imports

403a074

Add pytest to environments

897e0cb

maxwhitemet force-pushed the 953_stochastic_noise branch from 84d3c1b to 897e0cb Compare January 12, 2026 09:56

gavinevans requested changes Jan 16, 2026

View reviewed changes

maxwhitemet added 2 commits January 21, 2026 15:06

Undo environemnt changes

08001d9

Implement review feedback

a2487f9

maxwhitemet commented Jan 21, 2026

View reviewed changes

gavinevans requested changes Jan 27, 2026

View reviewed changes

maxwhitemet added 2 commits January 28, 2026 11:12

Add pytest importskip for pysteps

e312ee8

Implement review feedback

c4738bc

maxwhitemet commented Jan 28, 2026

View reviewed changes

Resolve ModuleNotFoundError

8ef5ec0

gavinevans approved these changes Jan 28, 2026

View reviewed changes

bayliffe requested changes Feb 17, 2026

View reviewed changes

	to avoid issues with log(0).
	to avoid issues with log(0). Value provided in units of `db_threshold_units`.

Comments

Conversation

maxwhitemet commented Jan 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

cpelley commented Jan 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

cpelley commented Jan 13, 2026

Uh oh!

gavinevans left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

maxwhitemet left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

gavinevans left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

maxwhitemet left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

bayliffe left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

bayliffe commented Feb 17, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

maxwhitemet commented Jan 8, 2026 •

edited

Loading

cpelley commented Jan 12, 2026 •

edited

Loading