Tkit Logging

[maybeec]

Discuss harvesting of Tkit Logging

[hohwille]

In devonfw we use SLF4J. As implementation we suggest logback. Additional for cloud projects we use logstash-logback-encoder and directly log JSON to standard out that then the platform (docker/kubernetes/... can directly feed to the log monitoring like ELK/Graylog/Splunk).

The only speciallities on top of that we have in devonfw are:

• protection against Log Forging attacks OOTB
• MDC for correlationID and specific performance logging. However, we planned to go for openTelemetry and this would also be my suggestion here.

I assume this all fits easily with Quarkus and TKit.

[SchneiderMalte]

All functions covered:
Logging via JBoss Log Manager. Adapters to other logging libraries like Slf4j. https://quarkus.io/guides/logging
Centralized log management through quarkus-logging-gelf extension. https://quarkus.io/guides/centralized-log-management
Endpoint protection with default security mechanisms (authorization). https://github.com/quarkiverse/quarkus-logging-manager . Enough for protection against log forging as Slf4j doesn't come with standard?
OpenTelemetry extension for tracing made recently available. https://quarkus.io/blog/quarkus-1-13-0-final-released/

[maybeec]

@yntelectual what does tkit provide here in addition to slf4j and opentelemetry?

[matus-m]

tkit-quarkus-log provides 3 extensions for a) automatic logging of all business method invocations(start exec + params, end exec + result + time, possiblity to specify custom structure...) b) logging of all HTTP communication(JaxRS filters logging outgoing and incoming request, URL, status Code) and c) optional, slightly extended, structured JSON logger(there is already a built-in json logger in quarkus, tkit one adds some advanced features like nested json support, stacktrace splitting, env keys printing... but for regular cases the built-in json formatter is perfectly fine).
There are some additional features that allow control over how are parameters formatted during logs, exclusion of beans/methods/params, MDC scope management,...
There is no extension on top of Opentelemetry/tracing atm.
In K8S envs, the services usually dont actively push their logs anywhere, but rather leave that to the platform EFK/ELK stack, which processes docker daemon logs and pushed them to central log service.

[maybeec]

We will focus on OpenTelemetry and SLF4J + support for JSON log output, which is already supported by Quarkus.

[maybeec]

https://quarkus.io/guides/opentracing

[SchneiderMalte]

Some design decision thoughts:

• auto-generation of logging on HTTP communication and business method invocations in tkit -> seems like an important feature quarkus logging doesn't provide by itself
• openTelemetry: only supported as experimental extension, so no guideline yet. Only tracing component supported (so no advantages for metrics or logging).

EDIT: reference projects version wont allow openTelemetry extension as it is only included in quarkus since 1.13.0. Changing project version conflicts with tkit jpa dependency yet supporting only 1.12.0.
So far, automatic tracing to both jaeger and zipkin exporter possible in quarkus (although extension is named jaeger-exporter).
As mentioned earlier, logging and metric compatibility in quarkus with openTelemetry still in development -> write own extension or keep that telemetry data separated and not integrated in otel-collector.

[hohwille]

So if I got everything correct, we already have an easy first agreement:
In our code we will use Slf4j for logging.

Can we already make use of the latest versions with fluent APIs?

Will we also agree on static logger pattern?
https://github.com/devonfw/devon4j/blob/master/documentation/guide-logging.asciidoc#logger-access
(independent from the question if you want to code this manually or use lombok)

Do we have a common agreement on how to use the log-levels:
https://github.com/devonfw/devon4j/blob/master/documentation/guide-logging.asciidoc#how-to-log

We can forget about the devon4j log files and log patterns. IMHO we also can agree on JSON logging to standard out and letting the platform care about log-shipping into ELK/GrayLog/Splunk or whatever log monitoring you may be using.

How do we protect quarkus apps against log-forging:
https://github.com/devonfw/devon4j/blob/master/documentation/guide-logging.asciidoc#security

So is quarkus forcing to use JBoss Log Manager as the implementation of Slf4j. In the end, I do not care much about the implementation but does it support the Slf4j features such as markers, etc.? Can I emit custom fields to the log that will become specific JSON properties? I am asking as I have a longer project backtrack with JBoss Logging products that have turned out to be really crappy.

[matus-m]

Slf4j + static logger pattern are all good and should be default(preferably the lombok version).

The JSON logging should be default, but in dev (and probably test) mode it should be disabled, as it greatly reduces log clarity for humans(Quarkus dev mode property %dev.quarkus.log.console.json=false

How do we protect quarkus apps against log-forging:

This should be a no-issue with JSON logs, as the actual log message is just a field in a json document, new lines etc are preserved (so you get a multiline string inside the json) and there is no way how you could break out of the message field from the json log entry(same behaviour is with stacktraces btw: they are stored in the message field, with newlines preserved, so if you log viewer supports it, you will see them as multiline text).

As for custom fields: MDC is of course supported:

MDC.put("myKey1", "value1");
//log msg with line break
log.info("My Message\n with new line");

Will produce :

{"timestamp":"2021-06-11T09:44:42.773+02:00","sequence":3609,"loggerClassName":"org.slf4j.impl.Slf4jLogger","loggerName":"org.acme.MyClass","level":"INFO","message":"My Message\n with new line","threadName":"Quarkus Main Thread","threadId":85,"mdc":{"myKey1":"value1"},"ndc":"","hostName":"mbp.local","processName":"code-with-quarkus-dev.jar","processId":53271}

If more control over the fields is required, we can use the tkit-quarkus-log-json appendder(allows fields rename, nested json objects, MDC expansion, env var logging, stack trace splitting etc...) but for most cases the default JSON appender is fine for prod. (there are also other extension out there on github that do similar things)
Example JSON log entry produced by tkit-quarkus-log-json showcasing the MDC expansion to top lvl fields (e.g. X-B3-TraceId)

As for markers, we have not yet tested it.

It is also possible to add any other logging impl (via extension) if none of the existing handlers meet the requirements.