Port test suite to offline fixture-based architecture by nick-gorman · Pull Request #72 · UNSW-CEEM/NEMOSIS

nick-gorman · 2026-04-23T14:30:02Z

Summary

Replaces the legacy network-hitting tests with a hermetic offline suite running against a local HTTP server that serves pre-filtered AEMO data committed under tests/fixtures/data/. 711 tests pass in ~80s, fully offline.

See testing_and_maintenance.md for suite layout, fixture system, boundary-test matrix, known quirks, and how to add tests for new tables.

test_performance_stats.py (covers custom_tables) stays ignored in CI pending offline port.

Test plan

CI runs uv run pytest tests/ --ignore=tests/test_performance_stats.py and passes

🤖 Generated with Claude Code

Replaces legacy network-hitting tests with a hermetic offline suite running against a local HTTP server serving pre-filtered AEMO data committed under tests/fixtures/data/. 711 tests pass in ~80s. See testing_and_maintenance.md for layout, fixtures, and how to add tests. test_performance_stats.py (custom_tables) stays ignored pending port. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

New workflow files added in a PR can sit in approval limbo. Fold the offline test job into the existing cicd.yml (already registered with Actions) so PR runs trigger automatically. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

Bootstraps CI while the `pull_request` trigger isn't yet on master. Remove this branch from the push trigger before merging. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

The cherry-picked commit from mdavis-xyz/NEMOSIS#61 points NEMOSIS at the .xlsx URL; the local fixture tree mirrors URL paths, so the file needs the matching extension for the dummy server to serve it. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

`static_table_url` had `/-/media/Files/Electricity/NEM/...` for the Generators XL and `/-/media/files/electricity/nem/...` for the VARIABLES_FCAS_4_SECOND CSV — same AEMO media dir, different casings. AEMO case-folds (verified: both casings return HTTP 200 with the xlsx content-type against the live CDN), but the test fixture tree can only exist at one casing on case-insensitive filesystems, so Linux CI was 404ing on the mixed-case URL. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

numpy 2.0.2 (previously locked) has no binary wheel for Python 3.13 on macOS-latest; uv falls back to building from source and fails on string_fastsearch.h. `uv lock --upgrade-package numpy` resolves to 2.2.6 (Py 3.10-3.12) and 2.4.4 (Py 3.13), both with wheels. CI-only fix — pip users don't consult uv.lock. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

nick-gorman · 2026-04-24T00:33:36Z

@mdavis-xyz I'm thinking to merge this PR onto main and then rebase your PR off main to include the refactored test suite.

I've taken a pretty different approch to speeding up the tests, such that they can now run in cicd. I'll leave this here for a few days or a week in case you want review, make comments, or suggest a different approach.

mdavis-xyz

Looks good. This is a huge improvement. I've made some suggestions.

mdavis-xyz · 2026-04-27T07:06:49Z

+      fail-fast: false
+      matrix:
+        os: [ubuntu-latest, windows-latest, macos-latest]
+        python-version: ["3.10", "3.11", "3.12", "3.13"]


I'd suggest adding Python 3.14 as well. That's been released on python.org

mdavis-xyz · 2026-04-27T07:09:46Z

+
+EXPECTED_BIDTYPES = {
+    "ENERGY", "RAISEREG", "LOWERREG",
+    "RAISE5MIN", "RAISE60SEC", "LOWER5MIN", "LOWER60SEC",


Should we add 1SEC too? I'm surprised this test passes.

Actually, it's missing the 6SEC timeframes as well. What's going on here? Is that because the particular generators chosen don't bid in those markets? If so, please add a comment next to this variable.

mdavis-xyz · 2026-04-27T07:13:47Z

+def test_trading_day_buffer_back_at_start_of_month(nemosis_fixture, era_start):
+    """A calendar-midnight query on day 1 must reach into the prev-month
+    archive for the prior trading day's 00:05 → 04:00 rows and stitch them
+    to day 1's 04:05 → onward rows. The [day1 00:00, day1 05:15] window is


I don't understand the explanation here.

If you're testing the misalignment between trading day and month boundaries, I would expect that we'd need to fetch 04:05-00:00 from the last day of the previous month. Why would we need 00:05-04:00 from the previous month?

mdavis-xyz · 2026-04-27T07:18:58Z

+    )
+
+    assert not data.empty
+    assert set(data["REGIONID"]) == {"SA1", "NSW1"}


Reading the code for this test, it's not clear what this is testing.

Why would other regions be missing? The test name says that regions are filtered, but I don't see any row filter in the dynamic_data_compiler args.

Ah, I see. Now I've spotted the explanation of how the data is filtered before becoming a test fixture.

mdavis-xyz · 2026-04-27T07:36:16Z

 allow-direct-references = true

 [tool.hatch.build.targets.wheel]
 packages = ["src/nemosis"]


We're adding a lot of large .zip files for the tests. Will they be packaged and included in the final pip installable file? (Thus making it larger)

I think this line means that they won't. If so, I'd suggest adding a one line comment here saying so. (I've never heard of hatch before, so I don't think it's obvious.)

mdavis-xyz · 2026-04-27T07:37:04Z

@@ -0,0 +1,224 @@
+# Testing and maintenance


Maybe update CONTRIBUTING.md to link to this document.

mdavis-xyz · 2026-04-27T07:37:39Z

+`SimpleHTTPServer` rooted at the fixture tree, which mirrors AEMO's URL paths verbatim — at test
+time NEMOSIS's hostname is swapped for the local server and everything else is real.
+
+The whole suite is fast (under a minute) and network-free.


Under a minute? Wooh hoo! Great work

nick-gorman and others added 7 commits April 24, 2026 00:28

Move test job into cicd.yml

ce78e47

New workflow files added in a PR can sit in approval limbo. Fold the offline test job into the existing cicd.yml (already registered with Actions) so PR runs trigger automatically. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

TEMP: trigger CI on offline-test-suite branch pushes

acf5f80

Bootstraps CI while the `pull_request` trigger isn't yet on master. Remove this branch from the push trigger before merging. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

Change registration list from xls to xlsx, fixes #60

ebf27ea

mdavis-xyz reviewed Apr 27, 2026

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Port test suite to offline fixture-based architecture#72

Port test suite to offline fixture-based architecture#72
nick-gorman wants to merge 7 commits intomasterfrom
offline-test-suite

nick-gorman commented Apr 23, 2026

Uh oh!

nick-gorman commented Apr 24, 2026

Uh oh!

mdavis-xyz left a comment

Uh oh!

mdavis-xyz Apr 27, 2026

Uh oh!

mdavis-xyz Apr 27, 2026

Uh oh!

mdavis-xyz Apr 27, 2026

Uh oh!

mdavis-xyz Apr 27, 2026

Uh oh!

mdavis-xyz Apr 27, 2026

Uh oh!

mdavis-xyz Apr 27, 2026

Uh oh!

mdavis-xyz Apr 27, 2026

Uh oh!

mdavis-xyz Apr 27, 2026

Uh oh!

mdavis-xyz Apr 27, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

nick-gorman commented Apr 23, 2026

Summary

Test plan

Uh oh!

nick-gorman commented Apr 24, 2026

Uh oh!

mdavis-xyz left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants