Skip to content

Fix CWAgent container insights - #2249

Open
Paamicky wants to merge 4 commits into
mainfrom
mcommey/OtelContainerInsight
Open

Fix CWAgent container insights#2249
Paamicky wants to merge 4 commits into
mainfrom
mcommey/OtelContainerInsight

Conversation

@Paamicky

Copy link
Copy Markdown
Contributor

Description of the issue

While bringing OTEL Container Insights (OCI) into the CloudWatch Agent, several pieces of functionality present in the Helm implementation were dropped or omitted, leaving the agent-generated config diverged from the chart. Most notably: OCI logs were never delivered to CloudWatch Logs because the log-group/log-stream metadata was missing; cluster-scoped metrics (apiserver, kube-state-metrics) carried the scraper node's identity. Attributes limits of <150 guaranteed to the Zeus endpoint was broken as well as Batching.

Description of changes

Realigns the agent's OTEL Container Insights pipelines with the Helm implementation and fixes the dropped/omitted functionality:

  • OCI logs delivery: route CI application/host logs through the shared logs pipeline, which supplies the log-group/log-stream metadata that was missing (logs now reach CloudWatch Logs), including host-log gating (host logs omit
    service.name/deployment.environment.name).
  • schema_url reordering: move the schema_url clear into the shared metrics + logs pipelines (after the re-stamping
    resourcedetection/opentelemetry) and remove the redundant per-CI-pipeline clears.
  • Cluster-metric identity suppression: for cluster-scoped metrics (apiserver, kube-state-metrics), suppress the scraper node's host.*, cloud.availability_zone, and ec2.tag.* so series stay stable across scraper reschedules.
  • Attribute limit: add awsattributelimit (150-cap) on the shared metrics pipeline to stay under the CloudWatch OTLP backend limit.
  • Batching: set the shared metrics batch to 1000/10s which affects all OTLP metrics
  • resourcedetection: unify the CI resourcedetection definition (detectors: [eks, env, ec2], timeout: 2s) across pipelines.

All changes are gated to EKS OTEL Container Insights; ECS, host metrics, and non-CI pipelines are unaffected. Goldens regenerated.

License

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

Tests

  • Unit: go build ./... and go test ./translator/... pass (goldens updated); gofmt clean.
  • End-to-end on EKS (cluster mcommey-otelci-final, agent built from this branch, deployed via operator):
  • OCI logs now reach CloudWatch with the correct and verified attributes
  • Cluster metrics (apiserver/KSM via Query Studio): no host., no cloud.availability_zone, no ec2.tag.; node metrics retain node identity.
  • Metrics export cleanly

Requirements

  1. Run make fmt and make fmt-sh
  2. Run make lint

@Paamicky
Paamicky requested a review from movence August 13, 2026 18:04
@Paamicky Paamicky self-assigned this Aug 13, 2026
@Paamicky Paamicky added the ready for testing Indicates this PR is ready for integration tests to run label Aug 13, 2026
@Paamicky
Paamicky requested a review from sky333999 August 17, 2026 18:04
@Paamicky
Paamicky force-pushed the mcommey/OtelContainerInsight branch from aa5093b to 4be3af9 Compare August 17, 2026 19:08
@Paamicky
Paamicky marked this pull request as ready for review August 17, 2026 19:08
@Paamicky
Paamicky requested a review from a team as a code owner August 17, 2026 19:08
@Paamicky Paamicky changed the title Fix CWAgent container infights Fix CWAgent container insights Aug 17, 2026
@Paamicky
Paamicky force-pushed the mcommey/OtelContainerInsight branch from bae8c04 to 452caf0 Compare August 17, 2026 20:05
@Paamicky
Paamicky force-pushed the mcommey/OtelContainerInsight branch from 452caf0 to a0cb8bf Compare August 17, 2026 20:18
@github-actions

Copy link
Copy Markdown
Contributor

Binary Size Report

linux/amd64

Binary PR vs main (9884add) vs v1.300072.0
amazon-cloudwatch-agent 193.3 MB ${\color{red}▲}$ +4.1 KB (+0.0%) ${\color{red}▲}$ +4.1 KB (+0.0%)
amazon-cloudwatch-agent-config-wizard 2.0 MB +0 B +0 B
config-downloader 2.0 MB +0 B +0 B
config-translator 2.0 MB +0 B +0 B
start-amazon-cloudwatch-agent 2.5 MB +0 B +0 B
workload-discovery 3.0 MB +0 B +0 B
Total 204.9 MB ${\color{red}▲}$ +4.1 KB (+0.0%) ${\color{red}▲}$ +4.1 KB (+0.0%)
linux/amd64 amazon-cloudwatch-agent (last 9 main commits + this PR)

 193 ┤                                        
     ┤                                    ███ 
     ┤                                    ███ 
     ┤                ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ███ 
     ┤                ███ ███ ███ ███ ███ ███ 
     ┤                ███ ███ ███ ███ ███ ███ 
     ┤            ███ ███ ███ ███ ███ ███ ███ 
     ┤            ███ ███ ███ ███ ███ ███ ███ 
 193 ┤▄▄▄ ▄▄▄ ▄▄▄ ███ ███ ███ ███ ███ ███ ███ 
 MB  └────────────────────────────────────────
      ab55b8d                     v1.30007PR0
linux/arm64
Binary PR vs main (9884add) vs v1.300072.0
amazon-cloudwatch-agent 177.1 MB +0 B +0 B
amazon-cloudwatch-agent-config-wizard 2.0 MB +0 B +0 B
config-downloader 2.0 MB +0 B +0 B
config-translator 2.0 MB +0 B +0 B
start-amazon-cloudwatch-agent 2.4 MB +0 B +0 B
workload-discovery 2.9 MB +0 B +0 B
Total 188.4 MB +0 B +0 B
windows/amd64
Binary PR vs main (9884add) vs v1.300072.0
amazon-cloudwatch-agent.exe 196.1 MB ${\color{red}▲}$ +8.7 KB (+0.0%) ${\color{red}▲}$ +8.7 KB (+0.0%)
amazon-cloudwatch-agent-config-wizard.exe 2.1 MB +0 B +0 B
config-downloader.exe 2.1 MB +0 B +0 B
config-translator.exe 2.1 MB +0 B +0 B
start-amazon-cloudwatch-agent.exe 2.3 MB +0 B +0 B
workload-discovery.exe 2.9 MB +0 B +0 B
Total 207.7 MB ${\color{red}▲}$ +8.7 KB (+0.0%) ${\color{red}▲}$ +8.7 KB (+0.0%)
Investigating size changes

Use go-size-analyzer to compare binaries:

GOEXPERIMENT=jsonv2 go install github.com/Zxilly/go-size-analyzer/cmd/gsa@latest
gsa diff --old <baseline-binary> --new <new-binary>

@Paamicky
Paamicky force-pushed the mcommey/OtelContainerInsight branch from 839d539 to 0fe7082 Compare August 18, 2026 15:38
@Paamicky
Paamicky requested a review from jefchien August 18, 2026 16:05
@Paamicky
Paamicky force-pushed the mcommey/OtelContainerInsight branch from 0fe7082 to 553469b Compare August 18, 2026 19:14
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ready for testing Indicates this PR is ready for integration tests to run

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant