Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions changelog.d/host_metrics_oom_kill.feature.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
The `host_metrics` source now exposes an `oom_kill` counter metric on Linux,
reporting the number of Out-Of-Memory kill events recorded by the kernel.

authors: simonhammes
61 changes: 61 additions & 0 deletions src/sources/host_metrics/memory.rs
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,9 @@ use vector_lib::event::MetricTags;
use super::HostMetrics;
use crate::internal_events::HostMetricsScrapeDetailError;

#[cfg(target_os = "linux")]
const OOM_KILL: &str = "oom_kill";

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I was a bit unsure regarding the naming of this metric.

Should this be called memory_oom_kill_events? Or oom_kill_events?


impl HostMetrics {
pub async fn memory_metrics(&self, output: &mut super::MetricsBuffer) {
output.name = "memory";
Expand Down Expand Up @@ -122,4 +125,62 @@ impl HostMetrics {
}
}
}

#[cfg(target_os = "linux")]
pub async fn vmstat_metrics(&self, output: &mut super::MetricsBuffer) {
output.name = "memory";

// Spawn blocking task to avoid blocking the async runtime with synchronous I/O
let result = tokio::task::spawn_blocking(procfs::vmstat)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Honor PROCFS_ROOT for vmstat

When host_metrics is run with the documented PROCFS_ROOT override (for example a container mounting the host proc tree under /host/proc or tests using a fixture), this calls procfs::vmstat, which reads the process' default /proc/vmstat; init_roots only configures heim, so the new memory metric ignores the configured proc root while the existing memory metrics honor it. This can report OOM kills from the wrong proc tree or fail even though the configured proc root is valid, so read vmstat through the configured procfs root instead of the crate default.

Useful? React with 👍 / 👎.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As far as I'm aware, /proc/vmstat is not cgroup-aware and therefore returns the values from the host.

In addition, the procfs crate does not support customization of the procfs root iirc.

.await
.unwrap_or_else(|join_error| {
Err(procfs::ProcError::Other(format!(
"Failed to join blocking task: {}",
join_error
)))
});

match result {
Ok(stats) => {
if let Some(&oom_kill) = stats.get("oom_kill") {
output.counter(OOM_KILL, oom_kill as f64, MetricTags::default());

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Use a total-suffixed counter name

This emits a public counter named oom_kill, but Vector's metric naming spec requires counters to end in _total, and the other host memory counters follow that pattern (memory_swapped_in_bytes_total, memory_swapped_out_bytes_total). Shipping the new metric without the suffix makes the API non-compliant and any later rename would be breaking for users, so use a total-suffixed counter name and update the docs/tests together.

Useful? React with 👍 / 👎.

}
}
Err(error) => {
emit!(HostMetricsScrapeDetailError {
message: "Failed to load vmstat info.",
error,
});
}
}
}
}

#[cfg(all(test, target_os = "linux"))]
mod tests {
use crate::event::metric::MetricValue;
use crate::sources::host_metrics::{HostMetrics, HostMetricsConfig, MetricsBuffer};

use super::OOM_KILL;

#[tokio::test]
async fn generates_vmstat_oom_kill_metric() {
let mut buffer = MetricsBuffer::new(None);
HostMetrics::new(HostMetricsConfig::default())
.vmstat_metrics(&mut buffer)
.await;
let metrics = buffer.metrics;

assert_eq!(metrics.len(), 1);

let metric = &metrics[0];
assert_eq!(metric.name(), OOM_KILL);
assert!(
matches!(metric.value(), MetricValue::Counter { .. }),
"oom_kill metric should be a counter"
);

let tags = metric.tags().expect("metric must have tags");
assert_eq!(tags.get("collector"), Some("memory"));
}
}
2 changes: 2 additions & 0 deletions src/sources/host_metrics/mod.rs
Original file line number Diff line number Diff line change
Expand Up @@ -423,6 +423,8 @@ impl HostMetrics {
if self.config.has_collector(Collector::Memory) {
self.memory_metrics(&mut buffer).await;
self.swap_metrics(&mut buffer).await;
#[cfg(target_os = "linux")]
self.vmstat_metrics(&mut buffer).await;
}
if self.config.has_collector(Collector::Network) {
self.network_metrics(&mut buffer).await;
Expand Down
9 changes: 5 additions & 4 deletions website/cue/reference/components/sources/host_metrics.cue
Original file line number Diff line number Diff line change
Expand Up @@ -166,6 +166,7 @@ components: sources: host_metrics: {
memory_total_bytes: _host & _memory_gauge & {description: "The total number of bytes of main memory."}
memory_used_bytes: _host & _memory_linux & {description: "The number of bytes of main memory used by programs or caches."}
memory_wired_bytes: _host & _memory_macos & {description: "The number of wired bytes of main memory."}
oom_kill: _host & _memory_counter & _linux & {description: "The number of Out-Of-Memory (OOM) kill events recorded."}

// Host network
network_receive_bytes_total: _host & _network_counter & {description: "The number of bytes received on this interface."}
Expand All @@ -177,15 +178,15 @@ components: sources: host_metrics: {
network_transmit_packets_total: _host & _network_nomac & {description: "The number of packets transmitted on this interface."}

// Host tcp
tcp_connections_total: _host & _tcp_linux & _tcp_gauge & {description: "The number of TCP connections."}
tcp_tx_queued_bytes_total: _host & _tcp_linux & {
tcp_connections_total: _host & _linux & _tcp_gauge & {description: "The number of TCP connections."}
tcp_tx_queued_bytes_total: _host & _linux & {
description: "The number of bytes in the send queue across all connections."
type: "gauge"
tags: _host_metrics_tags & {
collector: examples: ["tcp"]
}
}
tcp_rx_queued_bytes_total: _host & _tcp_linux & {
tcp_rx_queued_bytes_total: _host & _linux & {
description: "The number of bytes in the receive queue across all connections."
type: "gauge"
tags: _host_metrics_tags & {
Expand Down Expand Up @@ -300,7 +301,7 @@ components: sources: host_metrics: {
}
}

_tcp_linux: {relevant_when: "OS is Linux"}
_linux: {relevant_when: "OS is Linux"}
_tcp_gauge: {
type: "gauge"
tags: _host_metrics_tags & {
Expand Down
Loading