Spring Boot app crashes on start-up when logz.io appender is active

[bascan]

One of our Spring Boot applications crashes when the logz.io appender is activated through the logback-spring.xml. Unfortunately, we weren't able to debug our issue. It only occurs when we deploy the app to a pod on GKE and not on our local Docker container.

Here's the stacktrace that was logged by our pod:

12:23:49.856 [main] ERROR o.s.boot.SpringApplication - Application run failed
java.lang.IllegalStateException: Logback configuration error detected: 
ERROR in ch.qos.logback.core.joran.spi.Interpreter@70:20 - RuntimeException in Action for tag [appender] java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask@51650883[Not completed, task = java.util.concurrent.Executors$RunnableAdapter@1a15b789[Wrapped task = io.logz.sender.LogzioSender$$Lambda$214/1201004240@57f791c6]] rejected from java.util.concurrent.ScheduledThreadPoolExecutor@6c4f9535[Shutting down, pool size = 1, active threads = 1, queued tasks = 0, completed tasks = 9]
	at org.springframework.boot.logging.logback.LogbackLoggingSystem.loadConfiguration(LogbackLoggingSystem.java:169)
	at org.springframework.boot.logging.AbstractLoggingSystem.initializeWithConventions(AbstractLoggingSystem.java:82)
	at org.springframework.boot.logging.AbstractLoggingSystem.initialize(AbstractLoggingSystem.java:60)
	at org.springframework.boot.logging.logback.LogbackLoggingSystem.initialize(LogbackLoggingSystem.java:117)
	at org.springframework.boot.context.logging.LoggingApplicationListener.initializeSystem(LoggingApplicationListener.java:264)
	at org.springframework.boot.context.logging.LoggingApplicationListener.initialize(LoggingApplicationListener.java:237)
	at org.springframework.boot.context.logging.LoggingApplicationListener.onApplicationEnvironmentPreparedEvent(LoggingApplicationListener.java:200)
	at org.springframework.boot.context.logging.LoggingApplicationListener.onApplicationEvent(LoggingApplicationListener.java:173)
	at org.springframework.context.event.SimpleApplicationEventMulticaster.doInvokeListener(SimpleApplicationEventMulticaster.java:172)
	at org.springframework.context.event.SimpleApplicationEventMulticaster.invokeListener(SimpleApplicationEventMulticaster.java:165)
	at org.springframework.context.event.SimpleApplicationEventMulticaster.multicastEvent(SimpleApplicationEventMulticaster.java:139)
	at org.springframework.context.event.SimpleApplicationEventMulticaster.multicastEvent(SimpleApplicationEventMulticaster.java:127)
	at org.springframework.boot.context.event.EventPublishingRunListener.environmentPrepared(EventPublishingRunListener.java:74)
	at org.springframework.boot.SpringApplicationRunListeners.environmentPrepared(SpringApplicationRunListeners.java:54)
	at org.springframework.boot.SpringApplication.prepareEnvironment(SpringApplication.java:338)
	at org.springframework.boot.SpringApplication.run(SpringApplication.java:297)
	at de.meindach.B2bApplicationKt.main(B2bApplication.kt:22)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:564)
	at org.springframework.boot.loader.MainMethodRunner.run(MainMethodRunner.java:48)
	at org.springframework.boot.loader.Launcher.launch(Launcher.java:87)
	at org.springframework.boot.loader.Launcher.launch(Launcher.java:50)
	at org.springframework.boot.loader.JarLauncher.main(JarLauncher.java:51)

Appender configuration:

        <appender name="LOGZIO" class="io.logz.logback.LogzioLogbackAppender">
            <token>****</token>
            <logzioUrl>https://listener.logz.io:8071</logzioUrl>
            <logzioType>java</logzioType>
            <addHostname>true</addHostname>
            <encoder class="net.logstash.logback.encoder.LoggingEventCompositeJsonEncoder">
                <providers>
                    <mdc/>
                    <timestamp/>
                    <pattern>
                        <pattern>
                            {
                            "env": "****",
                            "appname": "****"
                            }
                        </pattern>
                    </pattern>
                    <logstashMarkers/>
                    <logLevel>
                        <fieldName>loglevel</fieldName>
                    </logLevel>
                    <loggerName>
                        <fieldName>logger</fieldName>
                    </loggerName>
                    <message/>
                    <stackTrace>
                        <throwableConverter class="net.logstash.logback.stacktrace.ShortenedThrowableConverter">
                            <maxDepthPerThrowable>30</maxDepthPerThrowable>
                            <maxLength>2048</maxLength>
                            <shortenedClassNameLength>20</shortenedClassNameLength>
                            <exclude>^sun\.reflect\..*\.invoke</exclude>
                            <exclude>^net\.sf\.cglib\.proxy\.MethodProxy\.invoke</exclude>
                            <rootCauseFirst>true</rootCauseFirst>
                        </throwableConverter>
                    </stackTrace>
                </providers>
            </encoder>
        </appender>

Relevant Maven dependencies:

        <dependency>
            <groupId>io.logz.logback</groupId>
            <artifactId>logzio-logback-appender</artifactId>
            <version>1.0.22</version>
        </dependency>
        <dependency>
            <groupId>net.logstash.logback</groupId>
            <artifactId>logstash-logback-encoder</artifactId>
            <version>5.3</version>
        </dependency>

After removing all <appender-ref ref="LOGZIO"/> occurrences from logback-spring.xml the app successfully starts.

How can I find out what goes wrong in the io.logz.sender.LogzioSender class?

[idohalevi]

@bascan
Looks like the application failed to run and that's why you received a runtime exception.
The failed application closes the executer and prevents the sender from drain logs.

Can you check this option?

[bascan]

@idohalevi We checked that option. The application runs fine when I remove all <appender-ref ref="LOGZIO"/> occurrences. The issue seems to be caused by the appender. Is there anything else we can try?

[idohalevi]

@bascan
Sorry for the late response. Any updates on the issue?

[martijnblankestijn]

We also see this behaviour in 1-10% of the cases with our pod.

The application is running on
Kubernetes: v1.12.9+IKS
Docker image: openjdk:11.0.2-jre
Spring Boot: 2.1.6.RELEASE
Logback: 1.2.3