ARM runner has been stuck for multiple days

[Biswa96]

This CI job is running for days https://github.com/msys2-arm/msys2-autobuild/actions/runs/6508662089

@jeremyd2019

[jeremyd2019]

It managed to hang up right as I was leaving on a long weekend trip, and I didn't notice until I got back. I wanted to get a fresh runner going anyway, for the latest Windows Updates, but was waiting until I got back to try to avoid any issues while I was gone 😁. New runner is going now

[Biswa96]

@jeremyd2019 Would you like to check if this CI job is stuck again https://github.com/msys2-arm/msys2-autobuild/actions/runs/6570301888 ?

[lazka]

@jeremyd2019 https://github.com/msys2-arm/msys2-autobuild/actions/runs/6714121489/job/18246869719

[lazka]

@jeremyd2019 likely stuck https://github.com/msys2-arm/msys2-autobuild/actions/runs/6980352526/job/18995436393

[jeremyd2019]

I've been ruminating on the idea of some sort of 'watchdog' to detect and kill stuck pacman processes automatically, but I haven't settled on the best language/technology to do so. It seems like python would be most convenient since autobuild is already python, I could put a background thread like I did to try polling the token, but I'm not familiar with process querying/killing modules.

What I've got so far is a cygwin commands to get the cygwin pid of the process I want to kill (what I really want is the child pacman process, this gets the newest pacman process older than 1800 seconds)

pgrep -xn -O 1800 pacman

coupled with the script I already had (because when stuck in this state cygwin kill is not sufficient)
https://github.com/jeremyd2019/winautoconfig/blob/master/msys2-runner-setup/setupscripts/wkill.sh

[Biswa96]

It would be a bit clear if the reason of such CI failure is explained.

[jeremyd2019]

lost power, so any lack of runner in the near future will be due to that

power is back

[lazka]

@jeremyd2019 https://github.com/msys2-arm/msys2-autobuild/actions/runs/7355161710

[lazka]

@jeremyd2019 https://github.com/msys2-arm/msys2-autobuild/actions/runs/7862352714

[jeremyd2019]

unstuck it. the powershell variant in git-for-windows/git-for-windows-automation#61 (comment) was intriguing, it seems like it could be close to being turned into a 'watchdog', would just need to also query CreationDate field to see any pacman processes that have been running a long time (like a half hour? or hour?), and then arrange for it to run continuously (scheduled task?). Of course, I'd much rather get whatever bug is causing this fixed...

[lazka]

https://github.com/msys2-arm/msys2-autobuild/actions/runs/8938789079/job/24553638109

[jeremyd2019]

There's a stuck job now, but it doesn't seem to be the runner this time. Probably something on Github's end.

[Biswa96]

stuck again https://github.com/msys2-arm/msys2-autobuild/actions/runs/9739288128