Invoke-IcingaCheckCPU Timeouts after Update

[MoniDevAlex]

Hello,
since the last Update to the newest Icinga Agent, Icinga Powershell Framework and Powershell PLugins we are experiencing problems with the Invoke-IcingaCheckCPU check.
On some of our servers the service will run into a timeout without any specific reason why.
The plugin output in Icinga is just

<Timeout exceeded.>

Command Executed is the following

'C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe' '-C' 'try { Use-Icinga -Minimal; } catch { Write-Output '\''The Icinga PowerShell Framework is either not installed on the system or not configured properly. Please check https://icinga.com/docs/windows for further details'\''; Write-Output '\''Error:'\'' $($_.Exception.Message)Components:`r`n$( Get-Module -ListAvailable '\''icinga-powershell-*'\'' )`r`n'\''Module-Path:'\''`r`n$($Env:PSModulePath); exit 3; }; Exit-IcingaExecutePlugin -Command '\''Invoke-IcingaCheckCPU'\'' ' '-DisableProcessList' '-Warning' '80%' '-Critical' '90%' '-Core' '_Total' '-Verbosity' '2' '-ThresholdInterval' '15m'

No other Service has problems so far, only the CPU one and it worked fine beforehand.
For the 15 Minute Time Index we have this implemented on our hosts:

Register-IcingaServiceCheck -CheckCommand 'Invoke-IcingaCheckCpu' -Interval 300 -TimeIndexes 5m, 15m

In the Event Logs we found these entries on the hosts with the problem:

The Problem itself will manifest mostly at random after some time, directly after installing the Icinga Agent or after a reboot of the server and then will disappear either after some minutes, hours or days.
If you run the command in a local Powershell Session on the host it will just close the terminal after some time.
Forcing Icinga to check again in the WebUI sometimes returns a valid result but the service will then again go Unknown and back into a timeout.
Hosts are mostly Windows Server 2019 or 2022 all on the same patch version.
We can't really find a common thing under those hosts fora root cause of the problem because some hosts have no problems so far and others have the timeout only sometimes or all the time.

Installed Version for Icinga Agent are as following:
Icinga Agent - 2.15.1
Powershell Framwork - 1.14.1
Powershell Plugins - 1.14.0
Icinga Service - 1.3.0

[LordHepipud]

Thank you for the issue.

Can you please try to run the plugin in a PowerShell like this and share the result:

icinga { Invoke-IcingaCheckCPU -DisableProcessList -Warning '80%' -Critical '90%' -Core '_Total' -Verbosity 3; };

and

icinga { Invoke-IcingaCheckCPU -DisableProcessList -Warning '80%' -Critical '90%' -Core '_Total' -Verbosity 3 -ThresholdInterval '15m'; };

Can you please also share the output of

Show-Icinga;

and

Test-IcingaForWindows;

Thank you in advance!

[MoniDevAlex]

Hello and thank you for the quick answere.
Here is the Output from the commands:

Windows PowerShell
Copyright (C) Microsoft Corporation. All rights reserved.

PS C:\Windows\system32> icinga { Invoke-IcingaCheckCPU -DisableProcessList -Warning '80%' -Critical '90%' -Core '_Total' -Verbosity 3; };
[OK] CPU Load (All must be [OK])
\_ [OK] Overall Load: 27.930874%
\_ [OK] Socket #0 (All must be [OK])
   \_ [OK] Core Total: 21.16496%
\_ [OK] Socket #1 (All must be [OK])
   \_ [OK] Core Total: 9.220256%
\_ [OK] Socket #2 (All must be [OK])
   \_ [OK] Core Total: 45.83968%
\_ [OK] Socket #3 (All must be [OK])
   \_ [OK] Core Total: 35.4986%
| totalload::ifw_cpu::load=27.930874%;80;90;0;100 0_total::ifw_cpu::load=21.16496%;80;90;0;100 1_total::ifw_cpu::load=9.220256%;80;90;0;100 2_total::ifw_cpu::load=45.83968%;80;90;0;100 3_total::ifw_cpu::load=35.4986%;80;90;0;100
0
PS C:\Program Files\WindowsPowerShell\Modules\icinga-powershell-framework> icinga { Invoke-IcingaCheckCPU -DisableProcessList -Warning '80%' -Critical '90%' -Core '_Total' -Verbosity 3 -ThresholdInterval '15m'; };
[OK] CPU Load (All must be [OK])
\_ [OK] Overall Load: 46.736956%
\_ [OK] Socket #0 (All must be [OK])
   \_ [OK] Core Total: 46.81020%
\_ [OK] Socket #1 (All must be [OK])
   \_ [OK] Core Total: 47.52755%
\_ [OK] Socket #2 (All must be [OK])
   \_ [OK] Core Total: 47.995467%
\_ [OK] Socket #3 (All must be [OK])
   \_ [OK] Core Total: 44.614607%
| totalload::ifw_cpu::load=32.472108%;80;90;0;100 0_total::ifw_cpu::load=26.8289%;80;90;0;100 1_total::ifw_cpu::load=22.10818%;80;90;0;100 2_total::ifw_cpu::load=63.96246%;80;90;0;100 3_total::ifw_cpu::load=16.98889%;80;90;0;100
0
PS C:\Program Files\WindowsPowerShell\Modules\icinga-powershell-framework> Show-Icinga;
Icinga for Windows environment:
-----------


Environment configuration:

PowerShell Root                 => C:\Program Files\WindowsPowerShell\Modules\
Icinga for Windows Service Path => C:\Program Files\icinga-framework-service\
Icinga for Windows Service User => NT Authority\SYSTEM
Icinga for Windows Service Pid  => 2148
Icinga for Windows JEA Pid      =>
Icinga Agent Path               => C:\Program Files\ICINGA2\
Icinga Agent User               => NT Authority\SYSTEM
Defined Default User            => NT Authority\SYSTEM
Icinga Managed User             => False
PowerShell Version              => 5.1.17763.8510
Operating System                => Microsoft Windows Server 2019 Datacenter
Operating System Version        => 10.0.17763
JEA Context                     =>
JEA Session File                =>
Api Check Forwarder             => True
Debug Mode                      => False

Icinga for Windows Certificate:

Issuer  => CN=Icinga CA
Subject => CN=HOST

List of configured background daemons on this system:

Start-IcingaServiceCheckDaemon
-----------
-StartupDelay => 15

Start-IcingaWindowsRESTApi
-----------
-Port => 5667

List of configured background service checks on this system:
=> https://icinga.com/docs/icinga-for-windows/latest/doc/110-Installation/06-Collect-Metrics-over-Time/

Invoke-IcingaCheckCPU
-----------
Arguments    =>
CheckCommand => Invoke-IcingaCheckCPU
Id           => 5275219864641021224811420224776891459631192206
Interval     => 300
TimeIndexes  => 5m, 15m

Invoke-IcingaCheckCpu
-----------
Arguments    =>
CheckCommand => Invoke-IcingaCheckCpu
Id           => 169352231691581241292321561901571481822401822020625532194
Interval     => 300
TimeIndexes  => 5m, 15m

List of configured repositories on this system. The list order matches the apply order:

Icinga Stable
-----------
CloneSource  =>
Enabled      => True
LocalPath    =>
Order        => 0
RemotePath   => C:\Path\to\Repo
UseSCP       => False

Installed components on this system:

Component    Version   Available
---          ---       ---
agent        2.15.2    2.15.2
framework    1.14.1    1.14.1
plugins      1.14.0    1.14.0
service      1.3.0     1.3.0

Available versions flagged with "*" mean that this component is locked to this version
PS C:\Program Files\WindowsPowerShell\Modules\icinga-powershell-framework> Test-IcingaForWindows;
[Notice]: Collecting Icinga for Windows environment information
[Passed]: The Icinga Agent service and the Icinga Agent are installed on the system
[Passed]: The Icinga for Windows service is installed on the system
[Passed]: The Icinga for Windows service binary does exist: "C:\Program Files\icinga-framework-service\icinga-service.exe"
[Passed]: Your service installation is not affected by IWKB000009
[Passed]: Your service installation is properly referring to "icinga-powershell-framework.psd1" for module imports.
[Passed]: The Icinga Agent service user "NT Authority\SYSTEM" is matching the Icinga for Windows service user "NT Authority\SYSTEM"
[Passed]: The specified user "NT Authority\SYSTEM" is allowed to run as service
[Passed]: Directory "C:\ProgramData\icinga2\etc" is fully accessible by "NT Authority\SYSTEM"
[Passed]: Directory "C:\ProgramData\icinga2\var" is fully accessible by "NT Authority\SYSTEM"
[Passed]: Directory "C:\Program Files\WindowsPowerShell\Modules\icinga-powershell-framework\cache" is fully accessible by "NT Authority\SYSTEM"
[Passed]: Directory "C:\Program Files\WindowsPowerShell\Modules\icinga-powershell-framework\config" is fully accessible by "NT Authority\SYSTEM"
[Passed]: Directory "C:\Program Files\WindowsPowerShell\Modules\icinga-powershell-framework\certificate" is fully accessible by "NT Authority\SYSTEM"
[Passed]: The Icinga Agent state file is healthy
[Passed]: Icinga Agent configuration is valid
[Passed]: Icinga Agent debug log is disabled
[Passed]: The Icinga for Windows REST-Api is configured to start with the daemon
[Passed]: The Icinga for Windows REST-Api is configured to allow API checks
[Passed]: The Icinga for Windows certificate is installed on the system and possibly signed by a valid CA
[Warning]: Icinga for Windows is configured without a JEA-Profile. It is highly recommended to use JEA for advanced security and easier permission handling
[Passed]: The Icinga for Windows service is running
[Passed]: The Icinga for Windows REST-Api responded successfully on this machine
PS C:\Program Files\WindowsPowerShell\Modules\icinga-powershell-framework>

After that I've checked the Service in Icinga again and it recovered for one execution, after that it went Unknown again with the Timout message.