Implement functions and classes in ngen to produce Catchment output netcdfs [NGWPC-10597] by sivasankkar · Pull Request #177 · NGWPC/ngen

sivasankkar · 2026-04-29T19:54:55Z

Enable functionality in ngen to write all catchment outputs to a netcdf file for non-MPI ngen runs. This PR is created following NGWPC-9011 PR (#106). However, that didn't make it into development because it remained in draft. We squashed all earlier commits and rebased with ngen development for this PR. This PR might show more changes than what is needed for NetCDF operations. The changes listed below are specific to NetCDF related changes. This works only with NHF geopackage schema.

Additions

NetCDFManager.hpp - header file for handling all netcdf operations.
NetCDFManager.cpp - class file for handling all netcdf operations.
NetCDFFile.hpp - header file for methods/functions to handle all netcdf file level operations coming from the manager.
NetCDFFile.cpp - header file for methods/functions to handle all netcdf file level operations coming from the manager.
NetCDFVar.hpp - header file for methods/functions to handle all netcdf variable level operations coming from NetCDFManager and NetCFFile.
NetCDFVar.cpp - class file for methods/functions to handle all netcdf variable level operations coming from NetCDFManager and NetCFFile.
geopackages - Added two geopackage samples that are in NHF schema for testing.
PayloadConfig.hpp - Added a global Payload struct that can be used to update log messages for status communications

Removals

NetCDFCreator.cpp - This is the old NetCDF Wrapper class that used the C++ library.
NetCDFCreator.hpp - This is the old NetCDF Wrapper header

Changes

CmakeLists.txt in src/core, src/forcing, src/test, src/utilities/mdframe and the root: Updates to include the new NetCDF Wrapper classes and the custom NetCDF library.
Bmi_Formulation.hpp: Added get_output_variable_units and set_output_variable_units function to obtain the output variable units to write to netcdf attributes.
Bmi_Multi_Formulation.cpp: Set the output variable units read from realization config.
Bmi_Module_Formulation.cpp: Set the output variable units read from realization config. Also added payload messages in the Update function.
NgenSimulation.hpp: Included NetCDFManager header and pointer for that class. Included functions for payload messages
NgenSimulation.cpp: Added a function call to write the output variable values to netcdf after every time step. Added functions for payload messages.
Formulation_Manager.hpp: Added a function to retrieve all catchment formulations for iterating through for netcdf file creation.
Layer.hpp and cpp: Added a function to gather catchment outputs to a vector for each timestep.
NGen.cpp: Added a function call to create the NetCDF writer/creator instance.
NetCDFCreatorTest.cpp - class containing two unit tests - one for the catchment and another for the output values updated to use the new NetCDF Wrapper classes and library.
handler_netcdf.cpp - Updated code and function calls to use the new NetCDF Wrapper classes for reading/writing.
mdframe_netcdf_Test.cpp - Updated code and function calls to use the new NetCDF Wrapper classes for reading/writing.
NetCDFPerFeatureDataProvider.cpp - Updated code and function calls to use the new NetCDF Wrapper classes for reading forcing data.
Geopackage_Test.cpp - Updated unit tests to use the new NHF schema.
BMI_C_Adapter.cpp, BMI_Cpp_Adapter.cpp, BMI_Py_Adapter.cpp, BMI_Fortran_Adapter.hpp - Added payload messages for initiating and initiated statuses

Testing

Tested with data for basin 01123000. The NetCDF output file was created successfully for both MPI and non-MPI runs.
Unit tests

Screenshots

Notes

Todos

Checklist

Testing checklist (automated report can be put here)

Target Environment support

Linux

idtodd · 2026-06-16T15:05:41Z

            std::string output = std::to_string(output_time_index)+","+current_timestamp+","+
                r_c->get_output_line_for_timestep(output_time_index)+"\n";
-            r_c->write_output(output);
+
+            for (const auto& fmt : output_formats)
+            {
+                if (fmt == "csv"){
+                    r_c->write_output(output);
+                }
+                if(fmt == "netcdf"){
+                    //capture all the output values for this timestep to write to netcdf
+                    #if NGEN_WITH_NETCDF
+                        catchment_output_values[id] = r_c->get_output_line_for_timestep(output_time_index);
+                    #endif //NGEN_WITH_NETCDF
+                }
+            }


Work can be reduced here. Call just r_c->get_output_line_for_timestep on line 63. Line 69 can add the other information, and line 74 doesn't have to recreate the string.

Fixed as suggested.

idtodd · 2026-06-16T19:02:16Z

+#include <NGenConfig.h>
+
+NetCDFManager::NetCDFManager(std::shared_ptr<realization::Formulation_Manager> manager, 
+        const std::string& output_name, Simulation_Time const& sim_time, int mpi_rank, int mpi_num_procs)


Suggested change

const std::string& output_name, Simulation_Time const& sim_time, int mpi_rank, int mpi_num_procs)

const std::string& output_name, Simulation_Time const& sim_time, int mpi_rank, int mpi_num_procs) : rank_(mpi_rank), num_procs_(mpi_num_procs)

The workflow should ensure that rank is 0 and num procs is 1 when NGEN is compiled without MPI, so it's safe to just always initialize them at the start instead of in the constructor.

Fixed. Removed initialization of rank_ and num_procs_ in the header. Assigning values as suggested.

idtodd · 2026-06-16T19:25:55Z

+    std::vector<int> catchments_in_proc; 
+    catchments_in_proc.reserve(num_catchments);
+    for (auto const& formulation_info : manager_->get_all_formulations())
+    {
+        int catchment_id;
+        std::string catchm = formulation_info.first;
+        if (catchm.rfind("cat-", 0) == 0){
+            catchment_id = std::stoi(catchm.substr(4));
+        }
+        else{
+            catchment_id = std::stoi(formulation_info.first);
+        }
+        catchments_in_proc.push_back(catchment_id);
+    }
+    gather_all_catchments(catchments_in_proc);


Catchment IDs now need to be int_64_t for NHF 1.2. All conversions of catchment IDs to numbers need to us atoll, stoll, etc. I'm just marking this first place I noticed using int for catchment IDs, but please review the rest of where this type of conversion happens and update accordingly.

One note is that the IDs should never be less than or equal to 0.

Changed all int to int64_t for catchments. All NetCDF operations related to catchments now use long long. Tested the changes and everything works.

jswade-rtx · 2026-06-18T20:59:16Z


 # Uncomment when building locally
-# FROM ngen-bmi-forcing AS base
+# FROM forcing:local AS base


This is likely an unintended change.

This has been refactored out in favor of a command line build argument in the latest version of the Dockerfile.

Likely need to rebase this branch to development

I added it for testing purposes. It has been replaced and updated after rebase.

commit 971c830 Author: siva.selvanathan Added checks for MPI and output variables in realization to prevent NetCDF creation. Added appropriate log messages. commit b82797b Author: siva.selvanathan Date: Mon Jan 26 21:46:41 2026 -0500 Added a couple of macro If blocks to prevent attempts to write to a single NetCDF in parallel fashion. commit cf5e694 Author: siva.selvanathan <sselvanathan_IE@Dewberry.com> Date: Mon Jan 26 14:52:17 2026 -0500 Included num_proc if clause to prevent MPI Barrier being called for a single process. Also, removed an unused function from the test class. commit f4b8819 Author: siva.selvanathan <sselvanathan_IE@Dewberry.com> Date: Mon Jan 26 11:29:36 2026 -0500 Unit tests for netcdf creators. This build does not support MPI functionality. commit fff50aa Author: siva.selvanathan <sselvanathan_IE@Dewberry.com> Date: Tue Jan 13 21:18:32 2026 -0500 Added dependent libraries to test_ngen_simulation for a successful build. commit 915c0c3 Author: siva.selvanathan <sselvanathan_IE@Dewberry.com> Date: Tue Jan 13 17:24:56 2026 -0500 Changes made to enable MPI for netcdf writing. commit fc5eec5 Author: siva.selvanathan <sselvanathan_IE@Dewberry.com> Date: Mon Jan 12 09:19:57 2026 -0500 Made changes to enable MPI for catchment NetCDF creation. Build is still failing. commit 43fb963 Author: siva.selvanathan <sselvanathan_IE@Dewberry.com> Date: Sun Jan 11 21:11:14 2026 -0500 Added pybind include directory to check pipeline build commit 53f7022 Author: siva.selvanathan <sselvanathan_IE@Dewberry.com> Date: Sun Jan 11 17:14:34 2026 -0500 NetCDF for catchment outputs funcitonality implemented. commit 74a624a Author: siva.selvanathan <sselvanathan_IE@Dewberry.com> Date: Wed Dec 24 09:17:12 2025 -0500 Included proper headers and target directories to CMakeFile to make the program compilable. commit 3dcac1a Author: siva.selvanathan <sselvanathan_IE@Dewberry.com> Date: Tue Dec 23 14:26:04 2025 -0500 Removed all class variables and constructor implementation for NetCDFCreator. Currently, the code is skeletal with the focus on getting a successful compile. commit 4358f6a Merge: 6fcb180 b44c038 Author: siva.selvanathan <sselvanathan_IE@Dewberry.com> Date: Thu Dec 18 13:44:24 2025 -0500 Merge branch 'ssn_9011_netcdf_for_catchments' of https://github.com/NGWPC/ngen into ssn_9011_netcdf_for_catchments commit 6fcb180 Author: siva.selvanathan <sselvanathan_IE@Dewberry.com> Date: Thu Dec 18 12:26:03 2025 -0500 first commit with new netcdfcreator class commit b44c038 Author: siva.selvanathan <sselvanathan_IE@Dewberry.com> Date: Thu Dec 18 12:26:03 2025 -0500 first commit with new netcdfcreator class

…s updates with logger.

… changes to NgenSimulation and NetCDF Data provider. The old C++ class NetCDFCreator hasn't been removed yet.

…table fails due to shared library issue.

…ary.

…r defined output formats.

…hould help in docker build as well.

commit 971c830 Author: siva.selvanathan Added checks for MPI and output variables in realization to prevent NetCDF creation. Added appropriate log messages. commit b82797b Author: siva.selvanathan Date: Mon Jan 26 21:46:41 2026 -0500 Added a couple of macro If blocks to prevent attempts to write to a single NetCDF in parallel fashion. commit cf5e694 Author: siva.selvanathan <sselvanathan_IE@Dewberry.com> Date: Mon Jan 26 14:52:17 2026 -0500 Included num_proc if clause to prevent MPI Barrier being called for a single process. Also, removed an unused function from the test class. commit f4b8819 Author: siva.selvanathan <sselvanathan_IE@Dewberry.com> Date: Mon Jan 26 11:29:36 2026 -0500 Unit tests for netcdf creators. This build does not support MPI functionality. commit fff50aa Author: siva.selvanathan <sselvanathan_IE@Dewberry.com> Date: Tue Jan 13 21:18:32 2026 -0500 Added dependent libraries to test_ngen_simulation for a successful build. commit 915c0c3 Author: siva.selvanathan <sselvanathan_IE@Dewberry.com> Date: Tue Jan 13 17:24:56 2026 -0500 Changes made to enable MPI for netcdf writing. commit fc5eec5 Author: siva.selvanathan <sselvanathan_IE@Dewberry.com> Date: Mon Jan 12 09:19:57 2026 -0500 Made changes to enable MPI for catchment NetCDF creation. Build is still failing. commit 43fb963 Author: siva.selvanathan <sselvanathan_IE@Dewberry.com> Date: Sun Jan 11 21:11:14 2026 -0500 Added pybind include directory to check pipeline build commit 53f7022 Author: siva.selvanathan <sselvanathan_IE@Dewberry.com> Date: Sun Jan 11 17:14:34 2026 -0500 NetCDF for catchment outputs funcitonality implemented. commit 74a624a Author: siva.selvanathan <sselvanathan_IE@Dewberry.com> Date: Wed Dec 24 09:17:12 2025 -0500 Included proper headers and target directories to CMakeFile to make the program compilable. commit 3dcac1a Author: siva.selvanathan <sselvanathan_IE@Dewberry.com> Date: Tue Dec 23 14:26:04 2025 -0500 Removed all class variables and constructor implementation for NetCDFCreator. Currently, the code is skeletal with the focus on getting a successful compile. commit 4358f6a Merge: 6fcb180 b44c038 Author: siva.selvanathan <sselvanathan_IE@Dewberry.com> Date: Thu Dec 18 13:44:24 2025 -0500 Merge branch 'ssn_9011_netcdf_for_catchments' of https://github.com/NGWPC/ngen into ssn_9011_netcdf_for_catchments commit 6fcb180 Author: siva.selvanathan <sselvanathan_IE@Dewberry.com> Date: Thu Dec 18 12:26:03 2025 -0500 first commit with new netcdfcreator class commit b44c038 Author: siva.selvanathan <sselvanathan_IE@Dewberry.com> Date: Thu Dec 18 12:26:03 2025 -0500 first commit with new netcdfcreator class

…s updates with logger.

… changes to NgenSimulation and NetCDF Data provider. The old C++ class NetCDFCreator hasn't been removed yet.

…table fails due to shared library issue.

…ary.

…r defined output formats.

…hould help in docker build as well.

…o use enums. Checked in with latest updates.

…h EWTS and other submodules).

…e output format requested.

…mmunicator is not null.

jswade-rtx

Changes look good. I performed many tests for calibration/validation/forecast/regionalization/default run types to produce CSV, NetCDF/CSV, and NetCDF outputs using both single and multiple processors.

jswade-rtx · 2026-06-24T14:51:09Z

Related PRS:
nwm-rte: NGWPC/nwm-rte#4
msw-mgr: NGWPC/nwm-msw-mgr#85
fcst-mgr: NGWPC/nwm-fcst-mgr#38
cal-mgr: NGWPC/nwm-cal-mgr#65

sivasankkar requested review from cmaynard-ngwpc and idtodd April 29, 2026 19:57

sivasankkar force-pushed the ngwpc_10597_netcdf_creation branch from cae217c to 61be6c6 Compare April 29, 2026 20:49

sivasankkar marked this pull request as draft April 29, 2026 21:13

sivasankkar force-pushed the ngwpc_10597_netcdf_creation branch from 1f1199b to f84be25 Compare May 13, 2026 17:22

sivasankkar force-pushed the ngwpc_10597_netcdf_creation branch from 7d99ade to b42b585 Compare June 12, 2026 22:18

sivasankkar marked this pull request as ready for review June 12, 2026 23:56

sivasankkar force-pushed the ngwpc_10597_netcdf_creation branch from 944da40 to 61e9454 Compare June 16, 2026 12:19

idtodd reviewed Jun 16, 2026

View reviewed changes

jswade-rtx reviewed Jun 18, 2026

View reviewed changes

sivasankkar added 19 commits June 19, 2026 15:14

Made updates to CmakeLists and netcdf creator class to accomodate ewt…

1cc6d66

…s updates with logger.

Changed logger path to ewts logger in the unit test cpp file.

05d2aa3

Commit with C++ wrappers for using NetCDF C library. It also includes…

24a75a8

… changes to NgenSimulation and NetCDF Data provider. The old C++ class NetCDFCreator hasn't been removed yet.

Added more updates due to C library.

6419331

Adding a bak file generated

249e93b

Commit with successful build for ngen with MPI on and off. Test execu…

4991735

…table fails due to shared library issue.

Implementation of catchment output NetCDF creation with NetCDF C libr…

a29256b

…ary.

Final changes that includes creating netcdf and also accounts for use…

b8029cf

…r defined output formats.

Changes to use HDF5 and NetCDF that is installed in usr/local. This s…

b3978f4

…hould help in docker build as well.

Check in commits trying to fix serialization

37a4fea

Changes to use of EWTS logger or standard logger

b633c4e

Added updates to write Payload messages for status communications.

372326c

Added missing double quotes for payload message.

1365ab9

Added curly braces around the payload message.

8de2535

update submodules to latest origin/development commits

0bfe6d5

Committing with a lot of log messages for testing a bug.

4daa334

Made updates to CmakeLists and netcdf creator class to accomodate ewt…

119ed68

…s updates with logger.

sivasankkar added 12 commits June 19, 2026 15:51

Commit with C++ wrappers for using NetCDF C library. It also includes…

f851666

… changes to NgenSimulation and NetCDF Data provider. The old C++ class NetCDFCreator hasn't been removed yet.

Commit with successful build for ngen with MPI on and off. Test execu…

2d57e50

…table fails due to shared library issue.

Implementation of catchment output NetCDF creation with NetCDF C libr…

c9ba14f

…ary.

Final changes that includes creating netcdf and also accounts for use…

c41dba5

…r defined output formats.

Changes to use HDF5 and NetCDF that is installed in usr/local. This s…

687ca4a

…hould help in docker build as well.

Check in commits trying to fix serialization

e0197a5

Changes to use of EWTS logger or standard logger

e212dd1

Added updates to write Payload messages for status communications.

fc826d6

update submodules to latest origin/development commits

76dbd98

Committing with a lot of log messages for testing a bug.

263f023

Removed log messages and redefinitions caused by rebase).

139d48f

Added new LogLevel for status , refactored the PayloadConfig struct t…

45df04b

…o use enums. Checked in with latest updates.

sivasankkar force-pushed the ngwpc_10597_netcdf_creation branch from d2e042d to 45df04b Compare June 19, 2026 20:14

sivasankkar added 4 commits June 22, 2026 10:11

More commits for netcdf generation after rebasing to development (wit…

75e816d

…h EWTS and other submodules).

Call nc_close after simulations, delete MPI_Comm_Free in NetCDFManager.

f198e55

Placed netcdf closing within NGEN_WITH_NETCDF and also checked for th…

41cc549

…e output format requested.

Added a check for NetCDFManager to execute MPI_Barrier only if the co…

de6adc7

…mmunicator is not null.

jswade-rtx self-requested a review June 24, 2026 14:48

jswade-rtx approved these changes Jun 24, 2026

View reviewed changes

Committing troute has to match development

78bb88c

sivasankkar merged commit 87b3bca into development Jun 25, 2026
8 checks passed

	const std::string& output_name, Simulation_Time const& sim_time, int mpi_rank, int mpi_num_procs)
	const std::string& output_name, Simulation_Time const& sim_time, int mpi_rank, int mpi_num_procs) : rank_(mpi_rank), num_procs_(mpi_num_procs)

Uh oh!

Conversation

sivasankkar commented Apr 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Additions

Removals

Changes

Testing

Screenshots

Notes

Todos

Checklist

Testing checklist (automated report can be put here)

Target Environment support

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

cmaynard-ngwpc Jun 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jswade-rtx left a comment

Choose a reason for hiding this comment

Uh oh!

jswade-rtx commented Jun 24, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

sivasankkar commented Apr 29, 2026 •

edited

Loading

cmaynard-ngwpc Jun 18, 2026 •

edited

Loading