feat(gpu): add collectXidKernelLogs to collect Xid errors from the kernel ring buffer

[matthewnewman43]

What does this PR do?

Adds features.gpu.collectXidKernelLogs (default false). When enabled alongside features.logCollection.enabled, the node Agent tails the kernel ring buffer via a journald integration config and tags the messages source:kernel / service:kernel.

Motivation

NVIDIA reports GPU faults as Xid errors, and the driver emits them to the kernel ring buffer — not to any container's stdout. The out-of-the-box Xid log monitor and the GPU monitoring views both query source:kernel.

Today there is no supported way to get those messages through the operator: the GPU feature configures metrics and system-probe only, and nothing in the operator collects kernel logs. Users have to hand-roll an extraConfd journald config and hand-mount the host journal, and if they get the tagging wrong the Xid monitor silently matches nothing.

So GPU monitoring can be fully enabled and Xid alerting still be dead, with no signal that anything is missing. This closes that gap with a supported toggle.

What it does

When features.gpu.collectXidKernelLogs: true and features.logCollection.enabled: true:

• creates a ConfigMap with a journald config (config_id: kernel, include_matches: [_TRANSPORT=kernel], source: kernel, service: kernel), mounted at conf.d/journald.d
• mounts the host journal (/var/log/journal) and /etc/machine-id read-only into the core Agent

Design notes

Opt-in, not implied by gpu.enabled. Kernel log collection is billable log ingestion, and volume rises exactly when a node starts throwing Xids — the worst moment for a bill surprise. logcollection is currently the only feature that sets DD_LOGS_ENABLED, and I did not want a metrics toggle to start log ingestion as a side effect. So this is explicitly opt-in and defaults off.

No-op rather than error when log collection is off. The journald config is inert without the logs Agent, so enabling this alone does nothing. Open to a status condition or a validation error instead if maintainers prefer a louder failure — flagging it as a deliberate choice, not an oversight.

Collects the whole kernel transport, not only Xid lines. The name is about intent. Filtering to Xid alone would be cheaper, but Xid triage leans on surrounding kernel context (bus-off, ECC, thermal, PCIe AER), and the GPU views query kernel logs beyond Xid lines. Filtering would break those and make the data much less useful for root-cause work.

config_id: kernel is a compatibility contract, not an arbitrary name. The Agent's journald launcher keys tailers on journald:<config_id> and skips a config whose identifier is already tailed (launcher.go, Identifier() in pkg/logs/tailers/journald/tailer.go). Consequences worth knowing:

• Another journald config using a different config_id that also matches _TRANSPORT=kernel results in the same messages being tailed twice.
• A catch-all journald config with empty include_units already sweeps in kernel messages, so pairing this with one needs exclude_matches: [_TRANSPORT=kernel] on the catch-all to avoid duplication.
• The dedup is journald-only. A node already tailing /var/log/kern.log as a file gets duplicates with no warning — different launcher.

There is a companion Helm-chart change that adds the same config_id: kernel config with the matching catch-all exclusion: DataDog/k8s-datadog-agent-ops#9347 (internal). If both land, the chart's entry should be dropped in favor of this one rather than relying on launcher dedup to arbitrate — which config wins there is arrival-order dependent.

/var/log/journal is mounted without DirectoryOrCreate. On a node with a volatile-only journal, creating an empty directory would make the tailer silently collect nothing rather than fail visibly.

Possible impact

None when the flag is unset — the field defaults to false, and every code path is behind it. Consistent with the sibling GPU booleans (privilegedMode, patchCgroupPermissions), it is not explicitly defaulted, so no existing defaulting fixtures change.

Requires Agent 7.39.0+ for include_matches / exclude_matches, and a systemd journal on the node. Nodes that log to /var/log/kern.log instead are not covered by this PR.

Testing

• New unit tests cover the log-collection gate (5 cases), the mounts when enabled, absence of mounts when disabled, and the config_id / tagging contract.
• go build ./..., go test ./internal/controller/datadogagent/feature/gpu/... pass.
• make generate && make manifests re-run for deepcopy and CRDs.

Draft: I would like maintainer input on the opt-in-vs-default question and on the no-op-vs-error behavior before this is polished.

[datadog-official]

🎯 Code Coverage (details)
• Patch Coverage: 94.44%
• Overall Coverage: 50.18% (+0.09%)

_{This comment will be updated automatically if new data arrives.
🔗 Commit SHA: 39cf0b5 | Docs | Datadog PR Page | Give us feedback!}