nvidia persistence mode is deprecated warning

[dwilson1251]

Warning message following run of script in log

NVRM: Persistence mode is deprecated and will be removed in a future release. Please use nvidia-persistenced instead

Looks like this is a utility that is compiled with the linux driver, however not sure if it's available for the one used w/unraid. Looking through the syslog at boot it doens't seem to load it at system init, but not an expert here. I followed the following references below and modified the script to use nvidia-persistenced --persistence-mode instead of nvidia-smi --persistence-mode=1.

That removed the error and returned the following in the syslog nvidia-persistenced: Started (22826). I'm not 100% clear on how this impacts the intent of this script, stability issues, etc, but will run it to see if any.

Reference Links:
https://download.nvidia.com/XFree86/Linux-x86_64/396.51/README/nvidia-persistenced.html
https://docs.nvidia.com/deploy/driver-persistence/index.html
http://manpages.ubuntu.com/manpages/trusty/man1/alt-nvidia-331-persistenced.1.html
https://github.com/NVIDIA/nvidia-persistenced

[dwilson1251]

@SpaceinvaderOne - Quick update on this issue, it appears the using nvidia-persistenced --persistence-mode is a valid option, and negates the need for a periodic script. I'm not absolutely positive, but after about 24 hours of intermittent use the power draw seems to go back to the base state as expected.

[Solverz-0]

@SpaceinvaderOne - Quick update on this issue, it appears the using nvidia-persistenced --persistence-mode is a valid option, and negates the need for a periodic script. I'm not absolutely positive, but after about 24 hours of intermittent use the power draw seems to go back to the base state as expected.

I found the same result with nvidia-smi --persistence-mode=1 if I just ran it manually, the state goes to P0 when running a process and back to P8 when no processes are running.

Of course due to the way unraid works, this would need to be run after every reboot or fresh system start.

[ich777]

@dwilson1251 & @Solverz-0 nowadays it is enough to run nvidia-persistenced once at boot, no need to run it in intervals (except you want to get the power savings that @SpaceinvaderOne implemented), once at boot is enough. How you set that up it's up to you, it's enough to put the command in the go file or you simply run a User Script that is executed ad first boot.

But keep in mind that if you have two Nvidia GPUs in your system an you use one for Docker and one for a VM (which is not stubbed - not bound to VFIO) and you start the VM with nvidia-persistenced running in the background you will most likely crash your server.
However you could of course send a kill signal to nvidia-persistenced before the VM is started and run nvidia-persistenced again after the VM is started, but that is a bit more involved.

[SpaceinvaderOne]

Hi @ich777 @dwilson1251 @Solverz-0 Yeah, I've steered clear of changing the script to include nvidia-persistenced mainly because, just as you mentioned, @ich777 , having it run while passing a GPU through to a VM makes the server crashing unless we stub it. But if we stub the card for passthrough, we're left without any power management when it's not being used by a VM because it won't have a driver, except for when it's actively passed through and relying on the guest's drivers.

So i thought it better to not use nvidia-persistenced in the script. Nvidia's been flagging the deprecation warning for ages now, but fingers crossed, it stays operational. I'd like to think Nvidia might keep supporting it, to maintain backward compatibility for people and businesses running on older setups, cause if they pull out the feature it might break systems and workflows around it, so hopefully it will stay around for now.

But yeah, @ich777, you're right, if you dont need to pass through a GPU that's not stubbed, puting nvidia-persistenced into the go file or a script to run at first boot is probably better.

[sage2050]

I just came across this script today and it seems like persistence-mode=1 has finally been depricated:

nvidia-smi --persistence-mode=1
Unable to set persistence mode for GPU 00000000:01:00.0: Unknown Error
Terminating early due to previous errors.

[ich777]

You could also run a Qemu hook so that nvidia-persistenced is killed before the VM is started, with that you basically ensure you don‘t crash the server.

But keep in mind that such a script is completely custom and needs possibly more attention.

[sage2050]

if you're an idiot like me and forget to turn off persistenced before trying to pass a gpu to a vm, you can still ssh in and run 'killall nvidia-persistenced' and it fixes the lockup

[sage2050]

libvirt hook to stop and start persistence mode with a specified VM:

create script_name.sh and put it in /etc/libvirt/hooks/qemu.d/

#!/bin/bash

if [ "$1" == [ "your VM name" ] && [ "$2" == "prepare" ]
then
killall nvidia-persistenced
elif [ "$1" == [ "your VM name" ] && [ "$2" == "release" ]
then
nvidia-persistenced

echo auto | tee /sys/bus/pci/devices/????:??:??.?/power/control

fi

exit 0