issue of CaseGroup2 when extracting data described in a catalog csv file.

[weiming9115]

@jkrasting @bitterbark I used the MDTF catalog builder to generate a csv file (see attached), and tried following the MDTF.ipynb template to read the data with CaseGroup2 from the runtime_config.yml.

# Provide an MDTF case settings file
groups = [
    CaseGroup2(
        "/pscratch/sd/w/wmtsai/MDTF_POD_diags/MDTF-diagnostics/templates/runtime_config.mcstest.yml"
        #date_range=("2005-01-01", "2005-12-31"),
        #mapping=mapping,
    )
]

An error message indicated something was wrong with the csv file

"ComputeError: Path "/pscratch/sd/w/wmtsai//MDTF_POD_diags/catalogs/MPI-ESM1-2-HR_historical_r1i1p1f1_full.csv" contained empty path segment

This error occurred with the following context stack:
	[1] 'slice'
	[2] 'sink'"

After checking the csv file, we found that the init_year column is empty. The path to input data exists and is correct "/pscratch/sd/w/wmtsai/MDTF_POD_diags/inputdata/model/MPI-ESM1-2-HR_historical_r1i1p1f1_gn_200501010130-200912312230/3hr/MPI-ESM1-2-HR_historical_r1i1p1f1_gn_200501010130-200912312230.pr.3hr.nc".

I manually filled it out with the year "2005", but the same error message still appeared. The same, unmodified yml and csv files worked fine under the MDFT framework command-line mode. I've attached the associated files below.

---

MPI-ESM1-2-HR_historical_r1i1p1f1_full.csv

settings.jsonc

runtime_config.mcstest.yml

[aradhakrishnanGFDL]

Hi @weiming9115 time_range is used by esnb, and MDTF. I wonder if you can try with just the YYYY-MM-DD format in the csv (like how you tried and commented off data_range.) and see if the error changes

[weiming9115]

Hi @aradhakrishnanGFDL. I tried, but the same error appeared. I just realized the error may result from the absolute path where "//" occurs when combining the root path with the catalog csv file. That kind of explains the "empty path segment" found in the error message.

I don't see the path of csv file described in my runtime_config.yml, so it must be hard-wired somewhere.

[aradhakrishnanGFDL]

Interesting, where is the "//" ? The path column in csv seems ok?
The json file is mentioned as DATA_CATALOG in your yml
DATA_CATALOG: "/pscratch/sd/w/wmtsai/MDTF_POD_diags/catalogs/MPI-ESM1-2-HR_historical_r1i1p1f1_full.json"
The csv file must be referenced from the json file. Do you have that specified? That's typically how tools like intake-esm get the csv path, so I am hoping esnb does that too.

[weiming9115]

@aradhakrishnanGFDL Yes, I have that specified. There was a typo in the json file I missed, and it's fixed now. Thanks!.

The CaseGroup2 can now be executed, while the returned dataset is empty somehow (inferred from "is_resolved" and "is_loaded"?). If it was run correctly, should I see the required variables that are specified in the "settings.jsonc" in group[0]?

Here shows inputdata described in the csv file, and the sliced time range is fully covered.

[aradhakrishnanGFDL]

Good progress. Please document the typo here so we can track the error reporting improvements for esnb separately.

resolved has to be True, yes.

e.g. [CaseGroup <ESM4 Historical + Future>  n_sources=2  resolved=True  loaded=False
   * CaseExperiment ESM4_historical_D1: catalog=True date_range=['1993-01-01', '2022-12-31']
   * CaseExperiment ESM4_ssp585_D1: catalog=True date_range=['1993-01-01', '2022-12-31']]

[bitterbark]

Wei-ming, I'm looking at your casegroup output and indeed the is_loaded = False shows a problem. (My test notebook and data shows True). However, I'm not sure what could be going wrong. Can you upload the runtime_config_mcstest.yml file?

[bitterbark]

I'm noticing a difference between the source field between my output and yours. Mine shows all the settings, not just the name of the yml file.

So I'm wondering if you need to load the yml file before giving it to the casegroup2 function. In my notebook I read in the source file and then DATA_CATALOG is just a field in that. Feel free to grab the code from my notebook to try; you'll see I am currently running it in interactive mode where I set all of that interactively.

[weiming9115]

@bitterbark, Thanks! I can now read the data by adapting some code from your notebook. I was following the template (mdtf.ipynb) and the below are some modifications I had and differences I spotted:

1. diag = NotebookDiagnostic(settings_dict): Instead of reading a path as described in the template (mdtf.ipynb), a dictionary is now used here (settings_dict = json_utils.read_json(settings_file_path)).

2. Different format / variable names of the CSV file: the information in the CSV file you provided looks slightly different from what I have created by catalog_builder.py. For example, there is no "cell method" and "cell measure" in my CSV file. After manually modifying my file to the same format as yours, the cell "diag.resolve(groups)" can be run. This mismatched format probably explains why I had an error message about "cell_method". My CSV file is attached below.

3. Despite all the cells running successfully, I still have "is_resolved" and "is_loaded" showing "False". However, this does not affect the following operations, e.g., diag.resolve(groups) and diag.open().

---

The original CSV file generated by catalog_builder.py
esm_catalog_CESM2_historical_r11i1p1f1.csv

esm_catalog_CESM2_historical_r11i1p1f1.json

runtime_config_esnb.yml