multiple nodes for batch run

[QianqianHan96]

Hi Francesco,

Sarah suggested me to use multiple nodes if time is a concern. For now 64 CPU in one node is enough, but maybe it is good to know for the future for running bigger area.
Is multiple nodes already in current script, jobs=6 or 4 (I set into 4)? If so, why you use 96 CPU from 6 nodes, why not 96 CPU from one node? Is this plan for bigger computation in advance?

Another question is: if I use 20 mins from a node, will SURF charge me 1 hour or 20 mins, because the price I saw are all in 1 hour.
@fnattino @SarahAlidoost

[fnattino]

Hi @QianqianHan96, yes, the code snippet above can use multiple nodes.

If so, why you use 96 CPU from 6 nodes, why not 96 CPU from one node? Is this plan for bigger computation in advance?

The reason why I have run calculations using that setup is that I could use a small job for the Jupyter server (16 cores on a thin node, i.e. the smallest allocation you can get on Snellius) and then use the SLURMCluster to provide 96 CPUs only when I need them (they potentially come from other nodes).

Instead, when using the LocalCluster, you need to start the job which runs the Jupyter server on an allocation that already includes the 96 CPUs, and you keep them busy for all the time you are "experimenting".

Apart from the fact that you are then charged for more CPU time, booking 96 CPUs simultaneously for a longer time period often involves more waiting time (if you instead ask for 6 times 16 CPU jobs with shorter wall time your jobs will likely start earlier).

Another question is: if I use 20 mins from a node, will SURF charge me 1 hour or 20 mins, because the price I saw are all in 1 hour.

I don't think billing is rounded to the the hour. You should be charged only for the 20 minutes (times the number of cores you are occupying, times the weight factor of the node - fat nodes and GPU nodes are more expensive).

Pinging @SarahAlidoost for a clarification: when you suggested @QianqianHan96 to used multiple nodes, did you mean to increase the number of nodes used in one calculation or did you mean to run multiple calculations at the same time (keeping one node per calculation)? I thought Sarah meant the latter: for instance, instead of running one year after the other on one node, you could run e.g. 20 years simultaneously on 20 different nodes.

[QianqianHan96]

Hi Francesco,

Thanks a lot for your explanation.

Maybe you are right that Sarah meant the latter one. When I talked with Sarah today, I was just thinking how to use multiple nodes on dask. Then I remember I saw 6 jobs run simultaneously on 6 nodes when I run your script, so I was curious is the current script for running one year (above snippet) already involved multiple nodes. So the 6 times 16 CPU jobs means using 6 nodes, right?
[One small question: by LocalCluster, you mean the one I use sbatch script to submit jobs, by SlurmCluster, you mean ask more CPUs in jupyter notebook? I thought this two ways are asking for the same cluster.]

Following Sarah's advice, do you know how to run 20 years simultaneously on 20 different nodes? My idea was to copy same jupyter notebook into 20, maybe too dumb way.

If I set walltime as 1 hour, and I finished my task earlier (e.g. 20 mins). Then I client.shutdown() in Jupyter notebook or scancel job in terminal, will they stop charging me or still charge me until the end of the walltime?

[fnattino]

Then I remember I saw 6 jobs run simultaneously on 6 nodes when I run your script, so I was curious is the current script for running one year (above snippet) already involved multiple nodes. So the 6 times 16 CPU jobs means using 6 nodes, right?

It means that you are running 6 jobs on 16 CPUs each. The jobs can run on the same node or on a different one, depending on what is available at that moment.

One small question: by LocalCluster, you mean the one I use sbatch script to submit jobs, by SlurmCluster, you mean ask more CPUs in jupyter notebook? I thought this two ways are asking for the same cluster.

The LocalCluster is what you have used for example in this notebook. Note the following code (in cell [3]):

client = Client(n_workers=16, threads_per_worker=2)

is exactly the same this as the following (just shorter):

cluster = LocalCluster(n_workers=16, threads_per_worker=2)
client = Client(cluster)

So, when you create a LocalCluster, you create a Dask cluster running on the same node and in the same job allocation in which you are running Jupyter. Of course, if you want to use 32 CPUs as in the example above (16 workers with 2 threads each) your initial job needs to include at least 32 CPUs.

A SLURMCluster, allows you to create a Dask cluster by submitting new jobs. So the initial job in which you run Jupyter can be arbitrarily small (potentially, even 1 CPU only, but this is not allowed on Snellius, for which the minimum allocation is 16 CPUs). For instance the following snippet submit 4 jobs that start 8 workers with 2 threads each (thus, using 16 CPUs each):

cluster = SLURMCluster(processes=8, cores=16)
cluster.scale(4)
client = Client(cluster)

Following Sarah's advice, do you know how to run 20 years simultaneously on 20 different nodes? My idea was to copy same jupyter notebook into 20, maybe too dumb way.

I think the best way to do that would be to convert the notebook into a Python script that takes some input (e.g. the year of interest), then you can submit 20 jobs with 20 different inputs. Let me or @SarahAlidoost know if you need help with this!

If I set walltime as 1 hour, and I finished my task earlier (e.g. 20 mins). Then I client.shutdown() in Jupyter notebook or scancel job in terminal, will they stop charging me or still charge me until the end of the walltime?

You are only charged for the time that you actually use, so 20 min in your case (you can make sure you don't have any more running jobs with squeue -u $USER).

[QianqianHan96]

Hi Francesco,

Thanks a lot for your detailed explanation. I understand now.

I think the best way to do that would be to convert the notebook into a Python script that takes some input (e.g. the year of interest), then you can submit 20 jobs with 20 different inputs. Let me or @SarahAlidoost know if you need help with this!

Could you help me convert the notebook into a Python script when you have time? Maybe just use year 2014 and 2015 as example, because input data of other years are not ready yet. This notebook is the latest one (https://github.com/EcoExtreML/Emulator/blob/main/2daskParallel/0117_1year_125degrees.ipynb)?

[SarahAlidoost]

Hi Francesco,

Thanks a lot for your detailed explanation. I understand now.

I think the best way to do that would be to convert the notebook into a Python script that takes some input (e.g. the year of interest), then you can submit 20 jobs with 20 different inputs. Let me or @SarahAlidoost know if you need help with this!

Could you help me convert the notebook into a Python script when you have time? Maybe just use year 2014 and 2015 as example, because input data of other years are not ready yet. This notebook is the latest one (https://github.com/EcoExtreML/Emulator/blob/main/2daskParallel/0117_1year_125degrees.ipynb)?

You can convert a notebook to python using the tool nbconvert, see installation and how to use it. Here is an example:

jupyter nbconvert --to python your_notebook.ipynb

Let me know if you need help with this.

[QianqianHan96]

Thanks, Sarah. If just convert jupyter notebook to python script and submit 20 jobs with 20 sbatch script, I know how to do it. I thought submitting 20 jobs in one sbatch script.

[QianqianHan96]

@fnattino Francesco, one thing is if I convert notebook to python script, can I still use SLURMcluster like what I am doing now?

[fnattino]

Hi @QianqianHan96, yes, you can use the SLURMCluster as in the notebooks, but it might be useful to add a call to wait for the full cluster to connect before actually go on and run the calculations.

Something like the following should work:

NWORKERS = 4 

cluster = SLURMCluster(...)
cluster.scale(NWORKERS)
client = Client(cluster)
client.wait_for_workers(NWORKERS) # this blocks the execution until all required workers connect to the cluster