Workbench - daskify tripyview - new version #204
Draft
patrickscholz wants to merge 156 commits into
Draft
Conversation
…or dask client initialisation, adding spinupcycles index to input paths ...
…r computation in time spice
…ed horizontal coarse graining alogorythm, chunk horizontal plooting for pcolor plot,
…here are still to many arrows in quiver plot at large grids
…rove transport and heatflux computation through transects now project transport on normal vector of main transect
…hich is more universal across the package
…ure environment if inline, notebook or widget can be chossen by input
… spinupcycle number to the inputpath ...
…ssue when reference plot, add loading of bolus velocity when doing streaml
…lose memory leakages and minimize dask task graph
… dask optimisation
…y with the land sea mask patch
Contributor
|
This appears to be a major upgrade. As we are looking towards making tripyview an automatic post-processing for any fesom run done with esm_tools (also where fesom is a component model in coupled setups). Would it be feasible to pick this up again? |
Collaborator
Author
|
@JanStreffing: In principle yes, this branch is supposed to be the next release version for tripyview. Im still tinkering here and there a bit around as my own alpha tester whenever i encounter some issues while using it. This version supposedly should work a bit better for large meshes because now it is fully dask parallelized before it was a mixture of dask and joblib. I also quite substantially overhauled the reading and processing of the the mesh object, as well as the 1d,2d,3d vector rotation and elem2node interpolation with numba, which also had quite a performance gain. |
… absolute path of the fesom_logo.png
…ector routine in polar projection, now we do an own and point method to do this vector rotation into the projection by our self
…more features into the plot
Use default Kernel "python3" instead. This should fix issues like `Unexpected Error: No such kernel named py39_new` when running tripyrun. Error was raised as papermill by default tries executing the notebooks using the kernel specified in the notebook metadata. This fix tells papermill to use the Kernel called "python3" and ignore notebook metadata. Details: Once any conda environment is activated it seems that the kernel "python3" is associated with the activated conda environment independent of the name of the conda environment. Only if no environment is activated "python3" is the name of the kernel associated with the default e.g. Levante python env. This can be verified by running `jupyter kernelspec list` Only tested on Levante with python 3.12
tripyrun/papermill: Ignore TemplateNotebook Kernel
papermill does not seem to like "=" signs in comments It would come up with the error: ``` Unable to parse line 6 'parallel_nthread = 2 # number of threads per worker, i.e. number worker = parallel_nprc/parallel_nthread'. ```
Update all Notebooks to Suppress Tripyrun Parsing Error
Newer environments (e.g. python 3.12) would raise error about lon coordinate of climatology being read only when trying to roll across +-180 degrees, (i.e. lon.flags.writeable was False)
This is likely because of changes in dependencies and whether views or copies are returned.
Now always force copy.
Original Error Message:
```
ValueError Traceback (most recent call last)
Cell In[6], line 11
7 if (vname in ['temp', 'sst', 'salt', 'sss', 'pdens'] or 'sigma' in vname) and (depth != 'bottom'):
8 ts = clock.time()
9 if vname in ['temp', 'sst'] and which_clim.lower()=='woa18': clim_vname = 't00an1'
10 elif vname in ['salt', 'sss'] and which_clim.lower()=='woa18': clim_vname = 's00an1'
---> 11 clim = tpv.load_climatology(mesh, clim_path, clim_vname, depth=depth)
12 print(' --> elasped time to load clim: {:3.2f} min.'.format( (clock.time()-ts)/60 ))
13 print(' --> clim uses {:3.2f} Gb:'.format(clim.nbytes/(1024**3)))
14 print('')
File ~/tripyview/tripyview/sub_climatology.py:182, in load_climatology(mesh, datapath, vname, mon, depth, depidx, do_zarithm, do_hinterp, do_zinterp, descript, do_ptemp, pref, do_compute, do_load, do_persist, do_zweight, do_hweight, chunks, do_parallel, **kwargs)
180 idx = np.where(lon>mesh.focus+180.0)[0]
181 idx_roll = idx[0]
--> 182 lon[idx] = lon[idx]-360.0
183 elif any(lon<mesh.focus-180.0):
184 idx = np.where(lon<mesh.focus+180.0)[0]
ValueError: assignment destination is read-only
```
`AS` got changed into `YS` in recent pandas/xarray versions
In newer environments (e.g. python 3.12) this warning would be issued by all workers: ``` FutureWarning: Usage of 'use_cftime' as a kwarg is deprecated. Please pass a 'CFDatetimeCoder' instance initialized with 'use_cftime' to the 'decode_times' kwarg instead. Example usage: time_coder = xr.coders.CFDatetimeCoder(use_cftime=True) ds = xr.open_dataset(decode_times=time_coder) ```
Workbench
A part of the code used to attempt to flip the transect direction if it is running in that direction. It however assumed numpy arrays. Thus it would crash on lists.
Transects running NE-->SW now do not crash anymore
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.