Skip to content

fix: Restore colored logs, strip ANSI escape codes in Vector agent#1237

Open
sbernauer wants to merge 4 commits into
mainfrom
fix/restore-log-colors
Open

fix: Restore colored logs, strip ANSI escape codes in Vector agent#1237
sbernauer wants to merge 4 commits into
mainfrom
fix/restore-log-colors

Conversation

@sbernauer

@sbernauer sbernauer commented Jun 30, 2026

Copy link
Copy Markdown
Member

Description

Different take on #1183

I (once again) run into this when a customer screenshared an operator log and we just saw a big wall of (white) text. Previously I was able to tell immediately if everything is green or not.

To improve this I need to understand the motivation behind the change in #1183.

I assume it is the following, but please correct me if I'm wrong:
containerdebug runs in the product container and produces ANSI codes. When ingesting that into the vector agent they are kept and end up in OpenSearch/Graylog.

I propose to leave the logs colored (the operators as well as the various tools such as containerdebug or cert-tools). Vector has a built-in function, using that we can strip the ANSI escape codes in the vector sidecar, so they don't end up in OpenSearch/Graylog/whatever.

Testing

I tested this using the logging demo and inspecting the prepare container of trino, as it's cert-tools uses ANSI stuff.
This PR worked perfectly fine, colored output in k9s and no weird characters in OpenSearch.
This solution also has the benefit that it works for all tools and not only for the stackable-specifc ones using stackable-telemetry.

Definition of Done Checklist

  • Not all of these items are applicable to all PRs, the author should update this template to only leave the boxes in that are relevant
  • Please make sure all these things are done and tick the boxes

Author

  • Changes are OpenShift compatible
  • CRD changes approved
  • CRD documentation for all fields, following the style guide.
  • Integration tests passed (for non trivial changes) - testing described above
  • Changes need to be "offline" compatible

Reviewer

  • Code contains useful comments
  • Code contains useful logging statements
  • (Integration-)Test cases added
  • Documentation added or updated. Follows the style guide.
  • Changelog updated
  • Cargo.toml only contains references to git tags (not specific commits or branches)

Acceptance

  • Feature Tracker has been updated
  • Proper release label has been added

@sbernauer sbernauer self-assigned this Jun 30, 2026
@sbernauer sbernauer moved this to Development: Waiting for Review in Stackable Engineering Jun 30, 2026
@lfrancke

Copy link
Copy Markdown
Member

This needs to work outside of Vector as well for those not using Vector and I don't think this necessarily does.
This is also unplanned work.

The reason is that we receive support tickets with stuff like this attached:

[2m2026-04-27T07:29:02.953575Z�[0m �[32m INFO�[0m �[1mcontainerdebug init�[0m�[2m:�[0m �[2mcontainerdebug�[0m�[2m:�[0m Starting containerdebug �[3mbuilt_info.pkg_version�[0m�[2m=�[0m"0.2.0" �[3mbuilt_info.git_version�[0m�[2m=�[0m"0.2.0" �[3mbuilt_info.target�[0m�[2m=�[0m"x86_64-unknown-linux-gnu" �[3mbuilt_info.built_time_utc�[0m�[2m=�[0m"Wed, 23 Jul 2025 17:59:35 +0000" �[3mbuilt_info.rustc_version�[0m�[2m=�[0m"rustc 1.85.0 (4d91de4e4 2025-02-17)"
�[2m2026-04-27T07:29:02.979837Z�[0m �[32m INFO�[0m �[1mcontainerdebug run�[0m�[2m:�[0m�[1mSystemInformation::collect�[0m�[2m:�[0m�[1mResources::collect�[0m�[2m:�[0m �[2mcontainerdebug::system_information::resources�[0m�[2m:�[0m cpus �[3mcpus.physical�[0m�[2m=�[0m16 �[3mcpus.cores.physical�[0m�[2m=�[0m8
�[2m2026-04-27T07:29:02.979891Z�[0m �[32m INFO�[0m �[1mcontainerdebug run�[0m�[2m:�[0m�[1mSystemInformation::collect�[0m�[2m:�[0m�[1mResources::collect�[0m�[2m:�[0m �[2mcontainerdebug::system_information::resources�[0m�[2m:�[0m open files limit �[3mopen_files.limit�[0m�[2m=�[0m1048576
�[2m2026-04-27T07:29:02.979902Z�[0m �[32m INFO�[0m �[1mcontainerdebug run�[0m�[2m:�[0m�[1mSystemInformation::collect�[0m�[2m:�[0m�[1mResources::collect�[0m�[2m:�[0m �[2mcontainerdebug::system_information::resources�[0m�[2m:�[0m memory �[3mmemory.total�[0m�[2m=�[0m133889966080 �[3mmemory.free�[0m�[2m=�[0m29899767808 �[3mmemory.available�[0m�[2m=�[0m66511568896 �[3mmemory.used�[0m�[2m=�[0m67378397184

I assume this is kubectl logs or something similar.

NickLarsenNZ
NickLarsenNZ previously approved these changes Jun 30, 2026

@NickLarsenNZ NickLarsenNZ left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm happy with this if @lfrancke is.

@lfrancke

Copy link
Copy Markdown
Member

I'm not, see: #1237 (comment)

@NickLarsenNZ

Copy link
Copy Markdown
Member

This needs to work outside of Vector as well for those not using Vector and I don't think this necessarily does. This is also unplanned work.

The reason is that we receive support tickets with stuff like this attached:

[2m2026-04-27T07:29:02.953575Z�[0m �[32m INFO�[0m �[1mcontainerdebug init�[0m�[2m:�[0m �[2mcontainerdebug�[0m�[2m:�[0m Starting containerdebug �[3mbuilt_info.pkg_version�[0m�[2m=�[0m"0.2.0" �[3mbuilt_info.git_version�[0m�[2m=�[0m"0.2.0" �[3mbuilt_info.target�[0m�[2m=�[0m"x86_64-unknown-linux-gnu" �[3mbuilt_info.built_time_utc�[0m�[2m=�[0m"Wed, 23 Jul 2025 17:59:35 +0000" �[3mbuilt_info.rustc_version�[0m�[2m=�[0m"rustc 1.85.0 (4d91de4e4 2025-02-17)"
�[2m2026-04-27T07:29:02.979837Z�[0m �[32m INFO�[0m �[1mcontainerdebug run�[0m�[2m:�[0m�[1mSystemInformation::collect�[0m�[2m:�[0m�[1mResources::collect�[0m�[2m:�[0m �[2mcontainerdebug::system_information::resources�[0m�[2m:�[0m cpus �[3mcpus.physical�[0m�[2m=�[0m16 �[3mcpus.cores.physical�[0m�[2m=�[0m8
�[2m2026-04-27T07:29:02.979891Z�[0m �[32m INFO�[0m �[1mcontainerdebug run�[0m�[2m:�[0m�[1mSystemInformation::collect�[0m�[2m:�[0m�[1mResources::collect�[0m�[2m:�[0m �[2mcontainerdebug::system_information::resources�[0m�[2m:�[0m open files limit �[3mopen_files.limit�[0m�[2m=�[0m1048576
�[2m2026-04-27T07:29:02.979902Z�[0m �[32m INFO�[0m �[1mcontainerdebug run�[0m�[2m:�[0m�[1mSystemInformation::collect�[0m�[2m:�[0m�[1mResources::collect�[0m�[2m:�[0m �[2mcontainerdebug::system_information::resources�[0m�[2m:�[0m memory �[3mmemory.total�[0m�[2m=�[0m133889966080 �[3mmemory.free�[0m�[2m=�[0m29899767808 �[3mmemory.available�[0m�[2m=�[0m66511568896 �[3mmemory.used�[0m�[2m=�[0m67378397184

I think I understand what you mean... We offer vector side car (@sbernauer fixed it for that case). But if they are using a privileged container to scrape pod logs from the host (otel-collector, alloy, fluent-bit, etc...) then it doesn't solve it for them.

I assume this is kubectl logs or something similar.

I assume it would display fine, unless they pipe it through something that can't handle it.

@NickLarsenNZ NickLarsenNZ dismissed their stale review June 30, 2026 12:35

This is contentious

@Techassi

Copy link
Copy Markdown
Member

I would say this needs a lot more investigation:

  • How does kubectl logs behave?
  • How does logging to a file behave? We should not emit escape codes.
  • How does the strange piping to Vector behave?
  • Etc...

@lfrancke

Copy link
Copy Markdown
Member

Yes. And that's time I don't want us to spend right now, sorry.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: Development: Waiting for Review

Development

Successfully merging this pull request may close these issues.

4 participants