Describe the bug
There is a race condition in split-netcdf:
https://github.com/NOAA-GFDL/fre-cli/blob/main/fre/pp/split_netcdf_script.py#L67-L69
if not os.path.isdir(output_subdir):
os.mkdir(output_subdir)
When there are multiple splitters running at once, two of them get past the if at the same time, the first one does the mkdir, and the second fails with "FileExistsError"
To Reproduce
@nikizadehgfdl experienced this while testing. Because many split-netcdf tasks start immediate after the stage-history and regrid-xy tasks, the split-netcdf tasks have a tendency to fail due to this, even though the mkdir is if-guarded.
Expected behavior
If the directory exists, it's fine. use a mkdir -p instead.
Additional context
Add any other context about the problem here.
Describe the bug
There is a race condition in split-netcdf:
https://github.com/NOAA-GFDL/fre-cli/blob/main/fre/pp/split_netcdf_script.py#L67-L69
When there are multiple splitters running at once, two of them get past the if at the same time, the first one does the mkdir, and the second fails with "FileExistsError"
To Reproduce
@nikizadehgfdl experienced this while testing. Because many split-netcdf tasks start immediate after the stage-history and regrid-xy tasks, the split-netcdf tasks have a tendency to fail due to this, even though the mkdir is if-guarded.
Expected behavior
If the directory exists, it's fine. use a
mkdir -pinstead.Additional context
Add any other context about the problem here.