what is the role of workers with locally_exlusive_torch or locally_exclusive_cl?

[ryan-budde]

I'm getting back in to motion correction, and I am testing out some new options.

I ran a test on 10 minutes of NPX 1.0 data. I used the locally_exclusive peak detection and completed the run in 212 seconds (timed using python's perf_counter) with 12 workers using a 28-logical processors ~3.7 GHz CPU (2024 i7). I then tested locally_exclusive_torch and completed the run in 216 seconds using the same settings on a 4000 Ada GPU. My task manager shows essentially 100% utilization of either CPU or GPU during these runs.

I noticed on the progress bar that the torch version also said it was using 12 workers.

What is the meaning of workers when we run something on the GPU? My understanding is that this is not typical language for the parallel processing on the GPU, and that typically we just let cuda run its magic "beneath the hood"

Would you recommend changing the number of workers when running these steps on GPU? Or does that not have a meaningful impact? If a process is parallelizable I would expect more of a performance improvement moving from CPU to GPU. Do you have any ideas if the performance I observed is expected?

[alejoe91]

Hi @ryan-budde

This is a very important concept. The current parallelization level only happens at the process/thread level, so if you use n_jobs=12, it will spawn/fork 12 sub-processes or threads (you can control this).
However, each job has the freedom to use the GPU, but we have no mechanism in place to control for GPU usage. For example, if you have a 8GB GPU and each of your 12 jobs will request 1GB CPU RAM, things will crash.

My gut feeling is that if your process is highly GPU parallelized (e.g., peak detection with torch), you would gain in performance using fewer jobs and larger chunks. By default we use 1s chunks, so you can try 5s chunks and try to get an optimal number of jobs/chunk duration that maxes out your GPU RAM.

Maybe @samuelgarcia @cwindolf @yger could comment too!

[samuelgarcia]

Note locally_exclusive_torch can also use the cpu device on from torch so there is no garanty that you are using the gpu.

locally_exclusive_cl is a very old implementaion I did using opencl, it used to be fast but you need use only on process so n_jobs=1.

Theorically mixing multi process and GPU code is very hard because we need to open one cuda command queue per process, if I am correct, and the multi process machinery of spikeinterface do not handle it at all.
For you GPU test did you set n_jobs=1 ? I am suprosed that it is working with multi process.