Skip to content

Bound freeipmi subprocess runtime and fix config-pipe goroutine leak - #359

Open
tgriffitts-ns wants to merge 1 commit into
prometheus-community:masterfrom
tgriffitts-ns:freeipmi-subprocess-timeout
Open

Bound freeipmi subprocess runtime and fix config-pipe goroutine leak#359
tgriffitts-ns wants to merge 1 commit into
prometheus-community:masterfrom
tgriffitts-ns:freeipmi-subprocess-timeout

Conversation

@tgriffitts-ns

Copy link
Copy Markdown

What this fixes

We run ipmi_exporter fleet-wide and hit a failure mode where a long-lived
exporter process permanently wedges: every scrape reports ipmi_up 0 with a
near-zero ipmi_scrape_duration_seconds, and the process never recovers without
a restart. We have observed this on exporter processes with 180+ day uptimes.

Root cause is two unbounded blocking operations in the FreeIPMI path:

1. freeipmi.Execute has no subprocess timeout

Execute runs the FreeIPMI CLI tools via exec.Command(...).CombinedOutput(),
which has no timeout. When a BMC or the KCS interface stops responding (bus
contention, firmware hang), the tool blocks and the scrape goroutine blocks with
it indefinitely.

Fix: add an ExecuteTimeout (wired from a new --freeipmi.timeout flag,
default 20s) and run the subprocess with exec.CommandContext, so a hung tool
is killed (SIGKILL) rather than blocking the scrape forever. A deadline hit is
reported as a clear timed out running <cmd> after <dur> (killed) error.
Setting 0 disables the timeout and preserves the previous behaviour.

2. freeipmiConfigPipe leaks a goroutine per scrape

The config FIFO is opened O_WRONLY. Opening a FIFO write-only blocks until a
reader appears. If the FreeIPMI subprocess is killed (e.g. by the new timeout)
or never opens the config file, that open blocks forever and the writer
goroutine leaks — one per scrape — accumulating goroutines and file descriptors
until the process is exhausted. This is what turns a transient BMC hang into a
permanent wedge.

Fix: open the pipe O_RDWR instead of O_WRONLY. On Linux, O_RDWR on a
FIFO returns immediately (the goroutine is its own reader), so the write always
completes and the goroutine always exits. Also return after an open error
instead of falling through to Write on an invalid handle.

Behaviour change

  • New flag --freeipmi.timeout (default 20s). Existing deployments get a
    20s per-subprocess bound by default; set --freeipmi.timeout=0 to restore the
    previous unbounded behaviour.
  • No metric or output format changes.

Testing

  • go build ./..., go vet ./... clean.
  • Verified against a wedged host: prior to the fix the exporter stops responding
    after a BMC hang; with the fix the hung subprocess is killed at the timeout,
    the goroutine no longer leaks, and subsequent scrapes recover.

Two related fixes for a failure mode where a long-lived ipmi_exporter
process permanently wedges: every scrape reports ipmi_up=0 with a
near-zero ipmi_scrape_duration_seconds, and the process never recovers
without a restart. We have observed this fleet-wide on exporter
processes with 180+ day uptimes.

freeipmi.Execute runs the freeipmi CLI tools via
exec.Command(...).CombinedOutput(), which has no timeout. When a BMC or
the KCS interface stops responding (contention, firmware hang), the tool
blocks and the scrape goroutine blocks with it indefinitely.

freeipmiConfigPipe opens the FIFO it just created with O_WRONLY. Opening
a FIFO write-only blocks until a reader appears. If the freeipmi
subprocess is killed or never opens the config file, that open blocks
forever and the writer goroutine leaks -- one per scrape -- accumulating
goroutines and file descriptors until the process is exhausted.

Fixes:

- Add an ExecuteTimeout (wired from a new --freeipmi.timeout flag,
  default 20s) and run the subprocess with exec.CommandContext so a hung
  tool is killed (SIGKILL) instead of blocking the scrape forever. A
  DeadlineExceeded is reported as a clear "timed out" error. 0 disables
  the timeout, preserving the previous behaviour.

- Open the config pipe O_RDWR instead of O_WRONLY so the open returns
  immediately (the goroutine is its own reader) and the write always
  completes; also return after an open error instead of falling through
  to Write on an invalid handle.

Signed-off-by: Troy Griffitts <tgriffitts@netskope.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant