Bound freeipmi subprocess runtime and fix config-pipe goroutine leak - #359
Open
tgriffitts-ns wants to merge 1 commit into
Open
Bound freeipmi subprocess runtime and fix config-pipe goroutine leak#359tgriffitts-ns wants to merge 1 commit into
tgriffitts-ns wants to merge 1 commit into
Conversation
Two related fixes for a failure mode where a long-lived ipmi_exporter process permanently wedges: every scrape reports ipmi_up=0 with a near-zero ipmi_scrape_duration_seconds, and the process never recovers without a restart. We have observed this fleet-wide on exporter processes with 180+ day uptimes. freeipmi.Execute runs the freeipmi CLI tools via exec.Command(...).CombinedOutput(), which has no timeout. When a BMC or the KCS interface stops responding (contention, firmware hang), the tool blocks and the scrape goroutine blocks with it indefinitely. freeipmiConfigPipe opens the FIFO it just created with O_WRONLY. Opening a FIFO write-only blocks until a reader appears. If the freeipmi subprocess is killed or never opens the config file, that open blocks forever and the writer goroutine leaks -- one per scrape -- accumulating goroutines and file descriptors until the process is exhausted. Fixes: - Add an ExecuteTimeout (wired from a new --freeipmi.timeout flag, default 20s) and run the subprocess with exec.CommandContext so a hung tool is killed (SIGKILL) instead of blocking the scrape forever. A DeadlineExceeded is reported as a clear "timed out" error. 0 disables the timeout, preserving the previous behaviour. - Open the config pipe O_RDWR instead of O_WRONLY so the open returns immediately (the goroutine is its own reader) and the write always completes; also return after an open error instead of falling through to Write on an invalid handle. Signed-off-by: Troy Griffitts <tgriffitts@netskope.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this fixes
We run
ipmi_exporterfleet-wide and hit a failure mode where a long-livedexporter process permanently wedges: every scrape reports
ipmi_up 0with anear-zero
ipmi_scrape_duration_seconds, and the process never recovers withouta restart. We have observed this on exporter processes with 180+ day uptimes.
Root cause is two unbounded blocking operations in the FreeIPMI path:
1.
freeipmi.Executehas no subprocess timeoutExecuteruns the FreeIPMI CLI tools viaexec.Command(...).CombinedOutput(),which has no timeout. When a BMC or the KCS interface stops responding (bus
contention, firmware hang), the tool blocks and the scrape goroutine blocks with
it indefinitely.
Fix: add an
ExecuteTimeout(wired from a new--freeipmi.timeoutflag,default
20s) and run the subprocess withexec.CommandContext, so a hung toolis killed (SIGKILL) rather than blocking the scrape forever. A deadline hit is
reported as a clear
timed out running <cmd> after <dur> (killed)error.Setting
0disables the timeout and preserves the previous behaviour.2.
freeipmiConfigPipeleaks a goroutine per scrapeThe config FIFO is opened
O_WRONLY. Opening a FIFO write-only blocks until areader appears. If the FreeIPMI subprocess is killed (e.g. by the new timeout)
or never opens the config file, that open blocks forever and the writer
goroutine leaks — one per scrape — accumulating goroutines and file descriptors
until the process is exhausted. This is what turns a transient BMC hang into a
permanent wedge.
Fix: open the pipe
O_RDWRinstead ofO_WRONLY. On Linux,O_RDWRon aFIFO returns immediately (the goroutine is its own reader), so the write always
completes and the goroutine always exits. Also
returnafter an open errorinstead of falling through to
Writeon an invalid handle.Behaviour change
--freeipmi.timeout(default20s). Existing deployments get a20s per-subprocess bound by default; set
--freeipmi.timeout=0to restore theprevious unbounded behaviour.
Testing
go build ./...,go vet ./...clean.after a BMC hang; with the fix the hung subprocess is killed at the timeout,
the goroutine no longer leaks, and subsequent scrapes recover.