MS Processing

This repo consolidates the code to gather, process, and split mass spectrometry (MS) data for downstream model training and benchmarking.

---

Repository Structure

MS_processing/
├── dataprocessing/          # Main data collection and processing pipeline
├── data_splitting/          # Train/val/test split generation
├── data_splits/             # Output split JSON files (per dataset)
├── analysis/                # Exploratory notebooks for dataset statistics
├── dataprocessing_BM/       # Formatting scripts for baseline model benchmarks
├── CFs/                     # ClassyFire-annotated fragment results (per dataset/split)
├── data/                    # Data directory (raw, formatted, cleaned, final, unlabelled)
├── data_BM/                 # Benchmark-ready formatted data
├── models/                  # Downloaded model weights (deepms, dreaMS)
└── LICENSE

---

Data Sources

Labelled data

Open-source libraries (downloaded automatically via dataprocessing/01_download_data.py):

• GNPS — multiple sub-libraries (MGF format)
• MassBank — NIST-formatted MSP
• MoNA — LC-MS/MS positive mode spectra
• Extra libraries — NIH NP, MCE scaffold/bio libraries (Zenodo)

Commercial datasets (must be obtained separately):

• NIST2020 — obtained from collaborators; processed by 06_process_NIST2020.py
• NIST2023 — recently purchased; processed by 07_process_NIST2023.py

Benchmark datasets (downloaded automatically):

• MassSpecGym — processed by 08_process_MassSpecGym.py
• CANOPUS — processed by 09_process_CANOPUS.py

Unlabelled data

• GeMS (GeMS_A, GeMS_B) — downloaded from HuggingFace

---

Processing Pipeline (dataprocessing/)

Run scripts in numbered order:

Script	Description
01_download_data.py	Downloads raw MS data (GNPS, MassBank, MoNA, extras), unlabelled GeMS data, and model weights (deepms, dreaMS)
02_format_data.py	Parses MGF/MSP files into a unified schema using matchms
03_clean_data.py	Applies matchms filtering pipelines: removes non-MS2 spectra, normalises adducts and ionisation modes, filters for valid SMILES/InChIKey
04_merge_data.py	Merges all cleaned datasets; assigns global indices and adduct/instrument mappings
05_download_pubchem.py	Downloads PubChem SDF compounds; caches fingerprints (Morgan, MACCS) per InChIKey
06_process_NIST2020.py	Processes commercial NIST2020 dataset
07_process_NIST2023.py	Processes commercial NIST2023 dataset
08_process_MassSpecGym.py	Processes MassSpecGym benchmark dataset
09_process_CANOPUS.py	Processes CANOPUS training dataset
10_add_mol_info.py	Adds ClassyFire taxonomic classifications and molecular fingerprints per unique InChIKey
11_get_buddy_frag_CF.py	Uses MSBuddy to assign subformulas to fragment peaks
add_01_get_frags.py	Computes fragment ion annotations via bond-breakage rules (lone-pair and C–C bonds)
add_02_get_adduct_instrument_list.py	Generates adduct and instrument-type lookup tables

Configuration (config.py)

Defines all download URLs, output folder paths, and normalisation mappings:

• Adduct normalisation (e.g. M+H → [M+H]+)
• Instrument type normalisation (e.g. LC-ESI-Orbitrap → LC-ESI-ITFT)
• Standard output folders: data/raw, data/formatted, data/cleaned, data/final, data/unlabelled

Utilities (dataprocessing/utils/)

File	Contents
utils.py	I/O helpers, spectrum loading, fingerprint computation
chem_utils.py	RDKit-based chemistry utilities
pyclassyfire.py	ClassyFire API client
formula.py	Molecular formula parsing and arithmetic
check_utils.py	Sanity-check helpers

---

Data Splitting (data_splitting/)

Script	Description
01_compute_MCES_dist.py	Computes pairwise Minimum Common Edge Subgraph (MCES) distances between all unique SMILES in a dataset; writes to HDF5
02_split_data.py	Generates train/val/test splits using scaffold-based or InChIKey-based strategies; optionally sieves for CF-annotated spectra; computes cosine-score-based retrieval ranks
03_subset_data.py	Creates subsets of splits filtered by collision energy bin, instrument type, and adduct
03b_add_mist_splits.py	Converts split JSON files into TSV format required by MIST
03c_add_mist_splits_sampling.py	Same as above, for sampling-based splits

Utilities (data_splitting/utils/)

File	Contents
utils.py	Split I/O, data loading from folder
chem_utils.py	Scaffold generation, InChIKey-based grouping

---

Split Files (data_splits/)

Organised by dataset (canopus, massspecgym, nist2020, nist2023). Each dataset folder contains:

• splits/ — JSON files mapping each spectrum ID to train, val, or test
• splits_sampling/ — Sampling-based splits (massspecgym, nist2023 only)
• noisy_lookup.pkl — Lookup table for noisy spectra
• incon_smiles_inchikey.pkl — Records of inconsistent SMILES/InChIKey mappings

Split strategies

Split name	Description
inchikey_vanilla	InChIKey-based split; no filtering beyond basic cleaning
inchikey_cleaned	InChIKey-based split; requires CF fragment annotations
scaffold_vanilla	Murcko scaffold-based split
scaffold_cleaned	Scaffold split; requires CF fragment annotations
CF_inchikey_* / CF_scaffold_*	Splits restricted to spectra with chemical formula fragment labels
random	Random split
LS	Library search split
*_sieved	Variant filtered to spectra with ≥5 peaks
*_no_ambiguous	Variant excluding ambiguous scaffold assignments
*_subsampled / *_downsampled	Reduced-size variants

---

Analysis (analysis/)

Notebook	Description
data_statistics.ipynb	Dataset size, adduct/instrument distributions, collision energy statistics
get_dataset_leakage.ipynb	Analyses compound overlap between datasets to identify potential data leakage

Utilities (analysis/utils/)

Shared chemistry and I/O helpers for notebook use.

---

Baseline Model Formatting (dataprocessing_BM/)

Contains cloned model repos and custom formatting scripts:

• mist/ — MIST model repository
• ms-pred/ — MS-Pred model repository
• scripts/ — Custom preprocessing scripts:
  • 00_format_spec.py — Converts internal pickle format to model-specific input formats
  • iceberg/01_preprocess.sh — Preprocessing pipeline for ICEBERG
  • mist/ — Scripts for MIST: subformula annotation (01_run_subform.sh), MAGMA fragmentation (02_run_magma.sh), retrieval HDF building (03_retrieval_hdf.py), MGF prediction/augmentation (05_predict_mgf.sh, 06_buid_aug_mgf.sh)
  • mol_libraries/ — Molecular library preparation (biomolecules, HMDB, PubChem subsets)

---

ClassyFire Results (CFs/)

ClassyFire taxonomic classification results for fragment ions, organised by dataset and split type:

CFs/
├── canopus/
│   ├── inchikey_vanilla_split/
│   ├── scaffold_vanilla_split/
│   ├── random_split/
│   └── LS_split/
├── massspecgym/     (+ sieved variants)
└── nist2023/        (+ sieved variants)

---

Models (models/)

Pre-trained model weights downloaded by 01_download_data.py:

Model	Files
MS2DeepScore (deepms)	ms2deepscore_model.pt, settings.json
dreaMS	embedding_model.ckpt, SSL_model.ckpt