SCIM

[bgmeulem]

Refactor the network parameter builder

Problem

The network parameters as built by the top-level pipeline I.create_network_parameters contained many hardcoded paths to activity data.
This activity data was undocumented.
This method is poorly generalizable to toher researchers or model systems

Solution

Network parameters can now be built using the NetworkParamBuilder, adding separately:

• Network embedding data
• synapse dynamics
• activity data (ongoing or evoked, or whatever you want)
• any additional info

I updated the tutorials and top-level pipeline I.create_network_parameters to match the new syntax. As long as you use positional arguments (not keyword arguments), the top-level pipelines and results are identical.

Testing and API stability

I tested backwards compatibility by cross-referencing with marias network parameter files from the decoding project for morphologies 64, 71, 88, 89, 91, for all columns, for all stimuli, and for positoin 1 in each column. All network parameters are exactly the same, except for the excitatory synaptic weights, which I didn't scale (see test code below)

To do so, I pre-processed the activity data so we don't need to infer the C2-relative stimulus / recording site.
Instead, I did the following:

• pre-generated all activity data for different whisker stimuli
• kept the original data we actually use in a subfolder "empirical_recordings"
• Added a notebook on how we augmented this data, and a readme on what the data format is and what everything means

Passing a string to the "whisker" argument is now interpreted as a globstring. The top-level pipeline expects you to then additionally pass a directory it should search in. If not, it falls back to a user-configured default data directory. This is currently set to our example data folder.

When it comes to generating activity data from a template or from scratch, all input data is now explicitly documented, and configured in config/user, instead of hardcoded in the run scripts. The tutorial explores this data more extensively and clearly now.

from typing import Mapping

def dicts_equal(d1, d2, ignore_keys=None, _path="root"):
    if ignore_keys is None: ignore_keys = set()
    elif isinstance(ignore_keys, str): ignore_keys = {ignore_keys}
    else: ignore_keys = set(ignore_keys)

    if not isinstance(d1, Mapping): return d1 == d2:

    filtered1 = {k: v for k, v in d1.items() if k not in ignore_keys}
    filtered2 = {k: v for k, v in d2.items() if k not in ignore_keys}

    if filtered1.keys() != filtered2.keys(): return False
    for k in filtered1:
        path = _path + "." + k
        is_equal = dicts_equal(filtered1[k], filtered2[k], ignore_keys, _path=path)
        return is_equal: 


dbdir = '/gpfs/soma_fs/scratch/royo/results/20240826_simulations_for_RM_generation_multiple_columns_4/64/db'
for grid in ["A1", "A2", "A3", "A4", "B1", "B2", 'B3', 'B4', 'C1', 'C2', 'C3', 'C4', 'D1', 'D2', 'D3', 'D4', 'Alpha', 'Beta', 'Gamma', 'Delta']:
    for stim in ["B1", "A1", "A2", "A3", "A4", "B1", "B2", 'B3', 'B4', 'C1', 'C2', 'C3', 'C4', 'D1', 'D2', 'D3', 'D4', 'Alpha', 'Beta', 'Gamma', 'Delta']:
        for posdir in [e for e in Path(f'{dbdir}/network_embeddings/{grid}_grid').glob("pos_1_synapses_*")]:
            cell_number_file_name = posdir / 'NumberOfConnectedCells.csv'
            syn = [e for e in posdir.glob('pos_1_synapses_*.syn')][0]
            con = [e for e in posdir.glob('pos_1_synapses_*.con')][0]
            target_paramfile_maria= posdir / 'model_1_{}_stimuli_network.param'.format(stim)
            if not I.os.path.exists(target_paramfile_maria): continue
            netp_maria = I.scp.build_parameters(target_paramfile_maria)
            netp_file = 'test_reproducibility_network.param'
            I.create_evoked_network_parameter(  # <--- new network param builder
                None, #ongoing_template_param_name, 
                str(cell_number_file_name), 
                str(syn), 
                str(con), 
                stim, 
                netp_file
            )
            new_netp = I.scp.build_parameters(netp_file)
            assert dicts_equal(new_netp.network, netp_maria.network, ignore_keys=["weight"])  # <-- must be the same

[codecov]

Codecov Report

❌ Patch coverage is 68.18182% with 7 lines in your changes missing coverage. Please review.
✅ Project coverage is 47.75%. Comparing base (e9b0a2b) to head (2781093).
⚠️ Report is 55 commits behind head on develop.

Files with missing lines	Patch %	Lines
config/isf_logging.py	25.00%	6 Missing ⚠️
...ta_base/db_initializers/somatic_summation_model.py	0.00%	1 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff             @@
##           develop     #465      +/-   ##
===========================================
+ Coverage    47.18%   47.75%   +0.56%     
===========================================
  Files          243      244       +1     
  Lines        22476    22207     -269     
===========================================
- Hits         10605    10604       -1     
+ Misses       11871    11603     -268     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.