Excessive logging pattern when relating to otel-collector causing very high CPU utilization.

[PawelMack]

Steps to reproduce

1. deploy psql
2. deploy otel-collector
3. relate psql with otel-collector (that is related to COS)

Expected behavior

Otel collector receivers are not configured with /**. However, that still doesn't resolve the issue completely as per psql log rotation #736

    include:
    - /snap/opentelemetry-collector/25/shared-logs/patroni/*.log

    include:
    - /snap/opentelemetry-collector/25/shared-logs/pgbackrest/*.log

    include:
    - /snap/opentelemetry-collector/25/shared-logs/pgbouncer/*.log

    include:
    - /snap/opentelemetry-collector/25/shared-logs/postgresql/*.log

Actual behavior

As of right now the receivers have /**

    include:
    - /snap/opentelemetry-collector/25/shared-logs/patroni/**

    include:
    - /snap/opentelemetry-collector/25/shared-logs/pgbackrest/**

    include:
    - /snap/opentelemetry-collector/25/shared-logs/pgbouncer/**

    include:
    - /snap/opentelemetry-collector/25/shared-logs/postgresql/**

Causing quite some pressure on CPU

:/var/lib/juju/storage/rootfs/9/2/var/log/postgresql$ ll *.log | wc -l 
10080

sudo ls -l /proc/$(pgrep -x otelcol)/fd | wc -l
1143

Versions

Operating system: Ubuntu 22.04.5 LTS

Juju CLI: 3.6.23-genericlinux-amd64

Juju agent: 3.6.21

Charm revision: 468 /14.12

LXD:

Log output

Additional context

I find the following issues to be relevant.

grafana-agent-#153
#736
Also path_exclude config option doesn't apply here as it only works on /var/log/

[syncronize-issues-to-jira]

Thank you for reporting your feedback to us!

The internal ticket has been created: https://warthogs.atlassian.net/browse/DPE-10357.

This message was autogenerated

[ggouzi]

Workaround: Add another config file (otel charm will override the existing one) to override the include/exclude to only take into account *.log files and replace start_at: beginning to start_at: end.

grep -E '^\s+filelog/charmed-postgresql.*:$' /etc/otelcol/config.d/otel-collector*.yaml | \
  sed 's/^\s*//' | \
  sed 's/:$//' | \
  while read receiver; do
    dir=$(echo "$receiver" | grep -oP '/snap/.*?(?=/otel-collector)')
    echo "  ${receiver}:"
    echo "    include:"
    echo "    - ${dir}/*.log"
    echo "    exclude:"
    echo "    - ${dir}/**"
    echo "    start_at: end"
  done | \
  { echo "receivers:"; cat; } | \
  sudo tee /etc/otelcol/config.d/aaa-override.yaml
sudo systemctl restart snap.opentelemetry-collector.opentelemetry-collector.service

• Before:

• After:

[odufourc]

This workaround is not really viable for long term because Opentelemetry-collector charm might delete and remove any supplementary file in /etc/otelcol/config.d in some cases.

A fix on the postgresql charm side is required.

[taurus-forever]

@odufourc can you please point to the best fix on postgresql charm (from your point of view)?

[sed-i]

Seems to be due to log rotation, and having configured otel to scrape postgresql/**.

I believe the proposal in the messages above is to change to *.log (I imagine to avoid scraping rotated files such as *.log.1 etc?)

The trouble is that the ** glob is hard-coded in otelcol itself.
We (o11y) could add a new config option or reuse path_exclude.
I created canonical/opentelemetry-collector-operator#324 to capture this.

Related: canonical/opentelemetry-collector-operator#129

[sed-i]

We have a stop-gap fix in

• fix: apply path_exclude config to snap log jobs opentelemetry-collector-operator#327
• fix: apply path_exclude config to snap log jobs (backport #327) opentelemetry-collector-operator#334

which applies otelcol's path_excldue config option to snap plugs as well as /var/log. This means that with a carefully crafted glob we could exclude the *.log.* rotated files.

Note that otelcol config option would apply to all subordinate units, so the path_exclude glob should be carefully constructed.
We have a complementary issue opened to give charm devs the ability to define custom globs.