Update to work with Pydantic upgrade of mt-metadata #389
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
local tests show: 30 failed, 46 passed, 41 warnings, 1 error in 360.48s (0:06:00)
Introduces a minimal conftest.py with fixtures for creating and cleaning up synthetic MTH5 files, and configures pytest to filter noisy warnings. Adds a pytest-based test that writes a TF object to a zrr file, reads it back, and asserts array equality, ensuring xdist safety and proper cleanup.
Added new pytest-based synthetic tests for Aurora and MTH5 processing, including feature weighting, Fourier coefficients, decimation, STFT agreement, and frequency band definition. Enhanced conftest.py with fixtures for synthetic test paths, file cleanup, and monkeypatching to sanitize provenance comments, improving test isolation and reliability.
Updated tests to use the same num_samples_window value when manually specifying band_edges, ensuring alignment with FFT harmonics and consistent transfer function results. Also removed unnecessary skip/xfail markers from pytest-based tests.
Introduces worker-safe pytest fixtures for synthetic MTH5 test files, replacing direct calls to file creation functions in tests. Updates processing helpers and all synthetic tests to accept or use these fixtures, improving test isolation, parallelism, and reliability. Also adds support for passing custom MTH5 file paths to processing helpers.
Removed unittest-based synthetic test modules and replaced them with pytest equivalents for metadata and multi-run tests. This improves test maintainability and integration with modern Python testing workflows.
Removed the unittest-based transfer function kernel test and added comprehensive pytest suites for ApodizationWindow and WindowingScheme classes. The new tests cover window generation, properties, taper families, sliding window operations, FFT, edge cases, and integration workflows, improving coverage and compatibility with pytest-xdist.
Introduces comprehensive tests for nan_to_mean, handle_nan, and time_axis_match functions in aurora.time_series.xarray_helpers. Covers edge cases, multiple channels, time axis mismatches, and data integrity, optimized for pytest-xdist parallel execution.
Deleted test_apodization_window.py, test_windowing_scheme.py, and test_xarray_helpers.py from the tests/time_series directory. These files contained unit tests for apodization windows, windowing schemes, and xarray helpers, respectively.
Introduces a comprehensive pytest test suite for the aurora.transfer_function.cross_power module. Tests cover channel name generation, transfer function computation, mathematical properties, edge cases, data integrity, numerical stability, return value characteristics, and consistency across calls. Optimized for parallel execution with pytest-xdist.
Added a comprehensive pytest suite for aurora.transfer_function.regression.helper_functions covering rme_beta, simple_solve_tf, and direct_solve_tf, including edge cases, mathematical properties, and data integrity. Removed the unittest-based cross_power test file to focus on regression testing for helper functions.
Introduces a comprehensive pytest suite for the RegressionEstimator base class, covering initialization, OLS estimation, QR decomposition, underdetermined systems, input type handling, xarray conversion, data validation, numerical stability, edge cases, data integrity, deterministic behavior, mathematical properties, and return value checks. These tests ensure correctness, robustness, and compatibility with various data types and scenarios.
Deleted test_base.py and test_helper_functions.py from tests/transfer_function/regression. These files contained unit tests for regression estimators and helper functions, possibly as part of a test suite cleanup or migration.
Introduces a new, fully refactored Parkfield test suite in tests/parkfield/test_parkfield_pytest.py, organized into multiple test classes with 25+ focused tests covering calibration, single-station and remote-reference processing, data integrity, and numerical validation. Adds extensive reusable fixtures to tests/conftest.py for efficient resource management and pytest-xdist compatibility. Includes a detailed REFACTORING_SUMMARY.md documenting the migration from three monolithic test files to a single, maintainable, and parallelizable suite with improved coverage and maintainability.
Updated attribute names from station/ch1/ch2 to station_1/channel_1/channel_2 in feature_weights.py and related test code for consistency. Improved logging for feature type and validation. Adjusted test deserialization logic to handle nested dicts and removed xfail marker from feature weighting test.
Updated docs/examples/dataset_definition.ipynb to use Windows-style paths, added 'channel_nomenclature.keyword', replaced nulls with empty strings, and changed 'units' from 'counts' to 'digital counts'. Also updated import paths, output examples, and warning messages for better Windows compatibility and current metadata conventions. Dropped Python 3.9 from test matrix in .github/workflows/tests.yaml.
Ensure CONFIG_PATH directory exists before saving JSON configs in make_processing_configs.py. Update test_decimation_methods_agree and test_stft_methods_agree to accept synthetic_test_paths argument for improved test setup.
Comment out the execution of Jupyter Notebooks in the CI workflow.
Added a step to execute Jupyter Notebooks and catch failures.
Comment out the test execution step and update notebook paths.
Activate virtual environment before executing notebooks.
Updated the GitHub Actions workflow for testing, including changes to installation steps, cache handling, and notebook execution.
Activate virtual environment before executing Jupyter Notebooks.
Set minimum required version of mth5 to 0.6.0 in pyproject.toml to ensure compatibility with recent features and fixes.
kkappler
approved these changes
Jan 24, 2026
Collaborator
There was a problem hiding this comment.
Before merging into main,
- return versions to 3.10-3.12
- run slow tests at least once
| "decimation_level": 0, | ||
| "frequency_max": 0.23828125, | ||
| "frequency_min": 0.19140625, | ||
| "frequency_max": 0.119140625, |
Collaborator
There was a problem hiding this comment.
The frequencies changed, but the indices did not -- something seems amiss
Collaborator
There was a problem hiding this comment.
actaully, this is because the default window size changed to 256 from 128, so this is actually expected.
| "minimum_cycles": 10, | ||
| "max_iterations": 10, | ||
| "max_redescending_iterations": 2, | ||
| "minimum_cycles": 1, |
Collaborator
There was a problem hiding this comment.
This seems too low -- if there is only 1 cycle the TF estimate will not be trustowrthy.
| "harmonic_indices": null, | ||
| "method": "fft", | ||
| "min_num_stft_windows": 2, | ||
| "min_num_stft_windows": 0, |
| test_data_set.network = "BK" | ||
| test_data_set.station = "PKD,SAO" | ||
| test_data_set.starttime = UTCDateTime("2004-09-28T00:00:00.000000Z") | ||
| test_data_set.endtime = UTCDateTime("2004-09-28T01:59:59.975000Z") |
Collaborator
There was a problem hiding this comment.
If the tests are passing, i guess this is OK ... but this could be double checked.
Collaborator
There was a problem hiding this comment.
It's OK to remove these .md files
Collaborator
There was a problem hiding this comment.
.prof (test profiling) can also be removed.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
mt-metadatahas been updated to use Pydantic under the hood. This PR will update Aurora to operate using the updated versions of mt-metadata and mth5.