Add loki logpush via promtail

[konradkonrad]

This aims to recreate the functionality from
shutter-network/shutter-keyper-deployment#10 usable with the DAppnodePackage.

Because we can not install the docker loki plugin, and everything is running in one container through supervisord, we need to resort to promtail and some extra log redirections to keep the docker logging working.

Fixes #10

[konradkonrad]

This is unfortunately still not ready -- env expansion in promtail configuration is still not working (on dappnode). This leads to authorization errors:

root@dappnode:~# docker logs DAppNodePackage-shutter.shutter-api-gnosis.dnp.dappnode.eth |& head -n 150
2025-04-10 13:21:14,109 INFO Set uid to user 0 succeeded
2025-04-10 13:21:14,117 INFO RPC interface 'supervisor' initialized
2025-04-10 13:21:14,117 CRIT Server 'unix_http_server' running without any HTTP authentication checking
2025-04-10 13:21:14,117 INFO supervisord started with pid 1
2025-04-10 13:21:15,121 INFO spawned: 'configure' with pid 7
2025-04-10 13:21:15,124 INFO spawned: 'promtail' with pid 8
[INFO | configure] Configuration file already exists. Removing it...
[INFO | configure] Generating configuration files...
---
# Loki Config
# (version=3.4.3, branch=release-3.4.x, revision=e038d0fc)
global:
  file_watch_config:
    min_poll_frequency: 250ms
    max_poll_frequency: 250ms
server:
  http_listen_network: tcp
  http_listen_address: ""
 […]
  profiling_enabled: false
  enable_runtime_reload: false
client:
  url: https://logs.metrics.shutter.network/insert/loki/api/v1/push
  batchwait: 1s
  batchsize: 1048576
  basic_auth:
    username: $(PUSHGATEWAY_USERNAME)
    password: <secret>

[konradkonrad]

This is ready now!

[ulope]

Apart from the question about rotation, LGTM.

[ulope]

Env var interpolation should allow to make autostart and autorestart conditional on the SHUTTER_PUSH_METRICS_ENABLED variable

[ulope]

Does the "rotation" with only one file work correctly with promtail without causing log lines to be missed?

Reading the promtail logs on logrotation suggests it might not.

[konradkonrad]

I saw that documentation, but I could not fully figure out, whether it applies in this case. My assessment was that using a single file would actually work as intended, but now that I re-read your link I am not sure anymore.

Do you have another suggestion how to achieve what we need?:

• redirect output to file AND stdout (to keep docker logging working)
• keep the logfile from growing indefinitely

[konradkonrad]

I've added a comment but decided to leave it like this.