Network Storage (NetworkIO) Implementation for SST Firefly NIC by RishankPratikHPLabs · Pull Request #2643 · sstsimulator/sst-elements

RishankPratikHPLabs · 2026-03-15T12:58:09Z

Network Storage (NetworkIO) Implementation for SST Firefly NIC

What this does

Adds network-attached SSD storage simulation to the Firefly NIC. Compute nodes can now issue async read/write operations to storage nodes over the simulated network, with a SimpleSSD model handling the storage-side latency and bandwidth.

We needed this to study I/O performance in distributed storage setups — how network latency, SSD throughput, node placement, and access patterns interact — without needing real hardware. This plugs into the existing Ember/Firefly/Hermes stack, so existing MPI simulations are unaffected.

How it works

The diagram above shows a read operation end-to-end. The client application calls `nwio_read()`, which goes through RDMA over the Slingshot network model to reach the SSD server. On the server side, the request gets scheduled into one of N queue lanes (round-robin), processed by the SSD with a bandwidth+latency delay, and the response is sent back over the network to the client's RDMA memory.

Here's what each layer does:

Hermes API (networkIOapi.h) — defines two calls: networkIORead(dest, offset, length, callback) and networkIOWrite(offset, src, length, callback). Both are async — you pass a callback that fires when the op completes.

Hades (hadesNetworkIO) — takes the global byte offset and figures out which storage node to hit. Uses a simple modulo scheme: nodeIndex = (offset / storageNodeCapacity) % numStorageNodes. Then hands it down to the NIC.

Firefly NIC (nicNetworkIO, nicNetworkIOSendEntry, nicNetworkIOStream) — this is where most of the work happens:

On the compute side: the NIC creates a send entry (read or write), generates a response key (respKey) with a callback attached, and queues a small request packet: [MsgHdr::NetworkIO][Read|Write][offset][addr][length][respKey]. The respKey gets stored so the NIC can match the ACK later.
On the storage side: a NetworkIOStream receives the packet, extracts the op type and respKey, and passes it to SimpleSSD (or falls back to DMA if SimpleSSD isn't loaded). When the SSD finishes, the stream creates a tiny ACK packet containing just the respKey and sends it back.
Back on the compute side: another NetworkIOStream receives the ACK, pulls out the respKey, looks up the stored callback via getRespKeyValue(), and invokes it. This completes the async operation.

NetworkIO traffic stays separated from MPI — it uses MsgHdr::NetworkIO (vs MsgHdr::Msg), its own NetworkIOStream (vs MsgStream), and vNic 0.

Ember (TestNetworkIO motif) — the workload layer. Issues configurable read/write calls with messageSize, iterations, op type, and fileSize params. Randomizes offsets within the file range.

SimpleSSD — sits on each storage node as a subcomponent. Has a multi-lane queue structure (nSSDsPerNode × queuesCountPerSSD lanes). Requests go round-robin across lanes. Each request gets a delay of overheadLatency + (bytes / bandwidth), scheduled via a self-link timer. When the delay fires, the completion callback runs and the ACK gets sent.

SimpleSSD model

firefly.SimpleSSD is a subcomponent with these params:

Parameter	Default	What it does
`nSSDsPerNode`	1	SSDs per storage node
`queuesCountPerSSD`	4	Queue lanes per SSD
`readBandwidthPerSSD_GBps`	6.25	Read bandwidth (GB/s)
`writeBandwidthPerSSD_GBps`	6.25	Write bandwidth (GB/s)
`readOverheadLatency_ns`	500	Fixed read overhead (ns)
`writeOverheadLatency_ns`	500	Fixed write overhead (ns)

Delay = overhead + (bytes / bandwidth). Requests go round-robin across all SSD×queue lanes.

Node setup

Some nodes are compute, some are storage. Configured in loadNetworkIO:

Compute nodes run workloads (TestNetworkIO) and issue I/O
Storage nodes run Null (idle) and serve requests through SimpleSSD
The split uses SSD_START_NODE and SSD_NODES vars in the load file

Files

New files (20):

hermes/networkIOapi.h — NetworkIO API
firefly/hadesNetworkIO.{cc,h} — offset-to-node mapping
firefly/nicNetworkIO.{cc,h} — NIC command handling
firefly/nicNetworkIOSendEntry.h — packet construction
firefly/nicNetworkIOStream.{cc,h} — incoming packet processing
firefly/storageModel/simpleSSD.{cc,h} — SSD simulation
ember/libs/emberNetworkIOLib.h — library wrapper
ember/libs/networkIOEvents/emberNetworkIO{Event,ReadEvent,WriteEvent}.h — events
ember/networkIO/emberNetworkIOGen.{cc,h} — generator base
ember/networkIO/motifs/emberTestNetworkIO.{cc,h} — test motif
ember/test/loadNetworkIO — test config
ember/test/networkIOParams.py — platform params

Modified files (16):

firefly/nic.{cc,h} — NetworkIO handler + SimpleSSD setup
firefly/nicEvents.h — new event types
firefly/nicRecvCtx.cc, nicRecvMachine.h — stream dispatch
firefly/nicSendEntry.h, nicSendMachine.{cc,h} — send support
firefly/nicVirtNic.h, virtNic.{cc,h} — delegation methods
firefly/Makefile.am, hermes/Makefile.am, ember/Makefile.am — build updates
ember/.gitignore — ignore patterns
CONTRIBUTORS.TXT — added HPE entry

Testing

Tested with three configs, all pass:

Test	Topology	Compute	SSD	messageSize	Sim Time
1	torus 2x2 (4 nodes)	2	2	1000	2.697 us
2	torus 4x2 (8 nodes)	6	2	1000	3.661 us
3	torus 4x2 (8 nodes)	2	6	1000	1.731 us

Built with SST Core + Elements from devel, GCC 14.

How to run

Build:

git clone https://github.com/RishankPratikHPLabs/nwStorage_sst-elements.git
cd nwStorage_sst-elements && git checkout nwStorage-feature
./autogen.sh
CC=gcc-14 CXX=g++-14 ./configure --prefix=$SST_INSTALL_DIR --with-sst-core=$SST_CORE_HOME
make -j$(nproc) && make install

Verify SimpleSSD registered:

sst-info firefly.SimpleSSD

Run the default test (2 compute + 2 SSD):

cd src/sst/elements/ember/test
sst emberLoad.py -- --topo=torus --shape=2x2 --numNodes=4 --numCores=1 --platform=networkIO --loadFile=loadNetworkIO

To change the node layout, edit loadNetworkIO:

[VAR] TOTAL_NODES=8
[VAR] COMPUTE_NODES=6
[VAR] SSD_START_NODE=6
[VAR] SSD_NODES=2

Then run with a matching topology:

sst emberLoad.py -- --topo=torus --shape=4x2 --numNodes=8 --numCores=1 --platform=networkIO --loadFile=loadNetworkIO

Motif params in loadNetworkIO: messageSize=<bytes>, iterations=<N>, op=read|write, fileSize=<bytes>.

Co-authored-by: Rishank Pratik rishank.pratik@hpe.com
Co-authored-by: Pawan Kumar pawan.kumar4@hpe.com
Co-authored-by: Sumant Kalra sumant.kalra@hpe.com
Co-authored-by: Shridhar Joshi shridhar@hpe.com

SSD module integration with Firefly NIC for realistic storage I/O modeling. End-to-end I/O path simulation: compute nodes -> network -> SSD storage nodes. Distributed address mapping with round-robin striping across storage devices. Direct integration with Ember motif framework for HPC workload simulation. New components: - NetworkIO API layer in Hermes interface - HadesNetworkIO with distributed address mapping - NIC-level packet handling for NetworkIO (read, write, ACK) - SimpleSSD storage device model (configurable bandwidth, latency, multi-queue bus) - Ember NetworkIO motif and test configurations Co-authored-by: Rishank Pratik <rishank.pratik@hpe.com> Co-authored-by: Pawan Kumar <pawan.kumar4@hpe.com> Co-authored-by: Sumant Kalra <sumant.kalra@hpe.com> Co-authored-by: Shridhar Joshi <shridhar@hpe.com>

sst-autotester · 2026-03-15T13:19:38Z

Status Flag 'Pre-Test Inspection' - - This Pull Request Requires Inspection... The code must be inspected by a member of the Team before Testing/Merging
NO INSPECTION HAS BEEN PERFORMED ON THIS PULL REQUEST! - This PR must be inspected by setting label 'AT: PRE-TEST INSPECTED'.

Add SPDX-FileCopyrightText and SPDX-License-Identifier (BSD-3-Clause) headers to all new and modified files per HPE OSRB attribution requirements. Update CONTRIBUTORS.TXT with HPE entry. Fix corrupted copyright line in nicEvents.h.

sst-autotester · 2026-03-16T06:20:11Z

Status Flag 'Pre-Test Inspection' - - This Pull Request Requires Inspection... The code must be inspected by a member of the Team before Testing/Merging
NO INSPECTION HAS BEEN PERFORMED ON THIS PULL REQUEST! - This PR must be inspected by setting label 'AT: PRE-TEST INSPECTED'.

feldergast

Code generally looks good. There are a few comments about some code clean-up. There will also need to be at least one test, preferably more to test the new code. The tests should use the new merlin.base derived python modules from the merlin library and should not use emberload.py as this uses the old deprecated merlin python libraries.

feldergast · 2026-04-20T17:05:18Z

+    m_networkIOLib = static_cast<EmberNetworkIOLib*>(getLib("networkIO"));
+    assert(m_shmemLib);
+    assert(m_networkIOLib);
+}


New line at end of line

feldergast · 2026-04-20T17:05:46Z

+
+[JOB_ID] 2
+[NID_LIST] generateNidList=generateNidListRange({SSD_START_NODE},{SSD_NODES})
+[MOTIF] Null


New line at end of line

feldergast · 2026-04-20T17:13:12Z

+        params.find<uint32_t>("verboseMask",-1),
+        Output::STDOUT );
+
+    auto parse = [](const std::string& s) 


Is this used somewhere that I'm missing?

feldergast · 2026-04-20T17:14:34Z

+        m_storageNodesList.push_back(m_ssd_start_node + i);
+    }
+
+    m_storageNodeCapacity = params.find<UnitAlgebra>("storageNodeCapacity", "1GiB").getRoundedValue();


May want to consider a check on the units to makes sure they are passing in Bytes for this.

feldergast · 2026-04-20T17:15:45Z

+
+    int nodeIndex = (offset/m_storageNodeCapacity)%m_storageNodesList.size();
+    return m_storageNodesList.at(nodeIndex);
+}


feldergast · 2026-04-20T17:22:52Z

+    m_nic.qSendEntry(entry);
+
+    delete event;
+}


feldergast · 2026-04-20T17:51:18Z

        m_toCoreLink->send( delay , event );
    }

+    void sendNetworkIO( SimTime_t delay, SST::Event * event ) {


This function could likely just use the sendShmem() function above, but give it a more generic name. Could likely even just be called send() since it has different parameters than the original send() function above.

feldergast · 2026-04-20T17:54:15Z

+{
+    m_dbg.debug(CALL_INFO,2,0,"src=%#lx len=%zu\n", src, len);
+    sendCmd(0, new NicNetworkIOWriteCmdEvent(  targetNid, src, len, callback ) );
+}


feldergast · 2026-04-20T17:56:49Z

+
+    virtual ~Interface() = default;
+
+    // Network IO READ - reads from network storage to local buffer


Comments about function would be better formatted as doxygen comments as we are trying to move everything to use doxygen.

RishankPratikHPLabs force-pushed the nwStorage-feature branch from 630bdc2 to f85e717 Compare March 16, 2026 06:16

feldergast requested changes Apr 20, 2026

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Network Storage (NetworkIO) Implementation for SST Firefly NIC#2643

Network Storage (NetworkIO) Implementation for SST Firefly NIC#2643
RishankPratikHPLabs wants to merge 2 commits into
sstsimulator:develfrom
RishankPratikHPLabs:nwStorage-feature

RishankPratikHPLabs commented Mar 15, 2026

Uh oh!

sst-autotester commented Mar 15, 2026

Uh oh!

sst-autotester commented Mar 16, 2026

Uh oh!

feldergast left a comment

Uh oh!

feldergast Apr 20, 2026

Uh oh!

feldergast Apr 20, 2026

Uh oh!

feldergast Apr 20, 2026

Uh oh!

feldergast Apr 20, 2026

Uh oh!

feldergast Apr 20, 2026

Uh oh!

feldergast Apr 20, 2026

Uh oh!

feldergast Apr 20, 2026

Uh oh!

feldergast Apr 20, 2026

Uh oh!

feldergast Apr 20, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants


		virtual ~Interface() = default;

		// Network IO READ - reads from network storage to local buffer

Conversation

RishankPratikHPLabs commented Mar 15, 2026