Enable config sync by default when CNM direct send is enabled - #3368
Draft
rahulkaukuntla wants to merge 2 commits into
Draft
Enable config sync by default when CNM direct send is enabled#3368rahulkaukuntla wants to merge 2 commits into
rahulkaukuntla wants to merge 2 commits into
Conversation
Direct send is on by default for NPM and USM on agents >= 7.81.0, which makes system-probe send network payloads itself. system-probe wires the no-op secrets component, so an ENC[...] api_key is never resolved there and the direct sender fails to start. Config sync is what supplies the resolved value from the core agent, and both of its settings default to off. Set DD_AGENT_IPC_PORT and DD_AGENT_IPC_CONFIG_REFRESH_INTERVAL on the core agent, process-agent and system-probe containers whenever direct send is enabled, reusing the values the otelcollector and hostprofiler features already set so the shared core agent container cannot end up with conflicting ports. User-provided values still win.
|
🎯 Code Coverage (details) 🔗 Commit SHA: 786530c | Docs | View more details | Give us feedback! |
The helper's statements were only exercised indirectly from the npm and usm packages, so per-package patch coverage counted all of them as uncovered.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What does this PR do?
When CNM direct send is enabled for NPM or USM, also set
DD_AGENT_IPC_PORT=5009andDD_AGENT_IPC_CONFIG_REFRESH_INTERVAL=60on the core agent, process-agent and system-probe containers, so that config sync is enabled by default alongside direct send.Both env vars are applied with a merge func that keeps any value the user already provided, so an explicit
override.nodeAgent.enventry still wins.Motivation
Direct send makes system-probe submit network payloads itself instead of routing them through process-agent. system-probe wires the no-op secrets component, so it cannot resolve an
ENC[...]secret handle — ifapi_keycomes from a secret backend, system-probe receives the handle verbatim and the direct sender fails to start, taking the wholenetwork_tracermodule (NPM and USM) down with it. Config sync is the mechanism that hands system-probe the api_key already resolved by the core agent.Config sync is off by default and needs two independent settings:
agent_ipc.config_refresh_intervalmust be positive, and there must be a transport (agent_ipc.portpositive, oragent_ipc.use_socket). Bothconfig_refresh_intervalandportdefault to0, so neither is satisfied out of the box.This matters here because the Operator enables direct send by default for NPM and USM on agents >= 7.81.0 (
agent.datadoghq.com/cnm-direct-send-enabledis an opt-out, not an opt-in). Any customer on a secret-backedapi_keywith NPM or USM therefore loses network monitoring unless they know to turn config sync on themselves. Enabling it here by default removes that footgun.Related agent-side change: DataDog/datadog-agent#55014, which turns the resulting failure into an explicit error naming the config sync settings, and scopes it to the direct send path so a secret-backed
api_keyno longer breaks unrelated system-probe modules.Additional Notes
The port and interval values match what the
otelcollectorandhostprofilerfeatures already set, for exactly the same secrets reason. That consistency is deliberate: all of these land on the shared core agent container, so a mismatch would make the effective IPC port depend on feature ordering. The shared helper lives infeature/utilsrather than being duplicated a third and fourth time.Clusters that use socket-based IPC instead (
agent_ipc.use_socket: true) are unaffected by the port var — both the config sync client (comp/core/configsync/impl/module.go) and the IPC listener (comp/api/api/apiimpl/listener/common.go) checkuse_socketfirst and never readagent_ipc.portwhen it is set, so no extra TCP listener is opened. The refresh interval is still required in that setup, since it is gated independently of the transport.One precedence caveat for consumers that configure the agent through a
datadog.yamlvalues file: env vars outrank file config, soDD_AGENT_IPC_CONFIG_REFRESH_INTERVAL=60will override anagent_ipc.config_refresh_intervalset that way. Tuning the interval now requiresoverride.nodeAgent.env. This is pre-existing behaviour for the otelcollector and hostprofiler features.An equivalent change is needed in
DataDog/helm-charts, which also enables direct send by default.Minimum Agent Versions
No new minimum. This only takes effect where direct send is already enabled, which the Operator gates on Agent >= 7.81.0.
Describe your test plan
Unit tests in
feature/npmandfeature/usmcover both directions: with direct send enabled (the default on a supported agent version) the two env vars are expected on the system-probe container and asserted present on the core agent container; with direct send disabled by annotation, or on an agent below 7.81.0, they are asserted absent.go test ./internal/... ./api/...andmake lintboth pass locally.For manual verification, deploy a DatadogAgent with NPM enabled and
global.credentials.apiSecretpointing at a secret-backend-resolvedapi_key, then confirm system-probe logsconfigsync enabled (agent_ipc 'localhost:5009' ...)and that thenetwork_tracermodule starts rather than failing with an unresolved secret handle.Checklist
bug,enhancement,refactoring,documentation,tooling, and/ordependenciesqa/skip-qalabel