Library seems to use up all available GPU memory.

[noahmartinwilliams]

I'm not sure if this is a problem with accelerate-llvm-ptx or cuda, but whenever I run "stack test" for this on the commit with checksum 49432b9265cc72606daf256b9ca641724e1edae9 it runs out of GPU memory and crashes. I'm gonna see if I can come up with a simpler test, but this is a problem I've had for years now with every attempt I've made to develop a neural net library using accelerate. I have an NVIDIA GeForce RTX 3050 Laptop GPU with 4 GiB of memory on Arch Linux (up to date as of April 5th, 2026).

Also when I run the nvtop command to see how much of the gpu is being used it's often at around 5% GPU usage.

[tomsmeding]

There have been various issues with Accelerate's GPU backend over the years, but this is not one of them as far as I know. If you could do a little reduction that would be amazing.

[noahmartinwilliams]

I tried asking ChatGPT for help with making a minimal example, and had this conversation with it: https://chatgpt.com/share/69e04775-feb8-83ea-88d3-f7d6576f88cc

Long story short: ChatGPT says that accelerate doesn't work well when arrays are frequently being reshaped (which is a thing that sometimes happens in my code).

[tomsmeding]

I only skimmed the tail end of that conversation, and as expected there are some misunderstandings of how Accelerate works.

• Accelerate kernels do not specialise on the array size — this is always a runtime parameter of the compiler kernel. Thus bucketing is pointless.
• Using run1/runN has very little to do eith GPU memory allocation, but it does prevent (when used correctly) repeated re-compilation of your Accelerate program, which is a good thing.
• What is most relevant is that the combination of 1. the fact that the current Accelerate backends do not in-place update arrays, and 2. array allocation on the GPU is managed, indirectly, by the Haskell GC. Together this means that the output of every compiled kernel is potentially a new array in GPU memory, and freeing of these arrays happens only after a while; thus if you run kernels often enough, fast enough, you can fill up GPU memory.
• However: the Accelerate GPU backend does have a facility for removing buffers from the GPU if necessary to make space for new allocations, by copying them to CPU memory. I'm not sure what happens when CPU memory fills up — likely the Haskell GC kicks in and cleans up things, which also results in GPU buffers being freed.

The eviction logic in the last bullet point lives in https://github.com/AccelerateHS/accelerate/blob/master/src/Data/Array/Accelerate/Array/Remote/LRU.hs, which logs interesting information under the dump-gc flag. So for debugging this, two good sources of information are:

1. The Show output of the program you're actually running. This tells us how many kernels you really have after fusion. I doubt that reshapes themselves are a problem, except if they are arranged/shared in such an unfortunate manner to create array copies even after fusion, and we can see if this is happening by looking at the fused program in show.
2. Run your program with +ACC -ddump-gc as additional arguments on the command line (this works like +RTS flags for the GHC RTS, see accelerate.cabal under the debug flag). To be able to do this you have to set the debug flag on the accelerate package, which you can do with cabal using constraints: accelerate +debug in cabal.project(.local), for example.

Point 2 you should be able to read in whether my diagnosis is correct; for point 1, if you're not sure what you're looking at, feel free to post here.

Final note: the Show Acc instance prints hashes of the compiled kernels if +ACC -dverbose is passed, which may be helpful to correlate it to other debug output.