Skip to content

SQS messaging shutdown error #3285

@HaohongLin1

Description

@HaohongLin1

import io.quarkus.runtime.ShutdownDelayInitiated;
import io.smallrye.reactive.messaging.ChannelRegistry;
import io.smallrye.reactive.messaging.PausableChannel;
import jakarta.annotation.Priority;
import jakarta.enterprise.context.ApplicationScoped;
import jakarta.enterprise.context.BeforeDestroyed;
import jakarta.enterprise.event.Observes;
import jakarta.inject.Inject;
import lombok.extern.slf4j.Slf4j;

import java.util.Set;

@ApplicationScoped
@Slf4j
public class MessagingGracefulShutdown {

  @Inject
  ChannelRegistry registry;

  void pauseBeforeDestroy(@Observes @Priority(1) @BeforeDestroyed(ApplicationScoped.class) Object ev) {
    log.info("Application shutdown starting. Pausing incoming channels...");
    Set<String> incoming = registry.getIncomingNames();
    for (String channelName : incoming) {
      PausableChannel pausable = registry.getPausable(channelName);
      if (pausable != null) {
        pausable.pause();
        log.info("Paused channel: {}", channelName);
      } else {
        log.debug("Channel {} is not pausable; skipping", channelName);
      }
    }
  }

  @ShutdownDelayInitiated
  void shutdownDelayInitiated() {
    log.info("shutdownDelayInitiated Application shutdown starting. Pausing incoming channels...");
    Set<String> incoming = registry.getIncomingNames();
    for (String channelName : incoming) {
      PausableChannel pausable = registry.getPausable(channelName);
      if (pausable != null) {
        pausable.pause();
        log.info("shutdownDelayInitiated Paused channel: {}", channelName);
      } else {
        log.debug("shutdownDelayInitiated Channel {} is not pausable; skipping", channelName);
      }
    }
  }
}

Quarkus graceful shutdown is enabled

#For graceful shutdown, only works at image creation and cannot be overridden.
quarkus.shutdown.delay-enabled=true
quarkus.shutdown.delay=5
quarkus.shutdown.timeout=20

# And the pool
smallrye.messaging.worker.resend-message.max-concurrency=1024
smallrye.messaging.worker.resend-message.shutdown-timeout=15000
smallrye.messaging.worker.resend-message.shutdown-check-interval=100

# The maximum number of worker thread for delete-message
smallrye.messaging.worker.delete-message.max-concurrency=1024
smallrye.messaging.worker.delete-message.shutdown-timeout=15000
smallrye.messaging.worker.delete-message.shutdown-check-interval=100

And the logs:

2026-01-15 15:40:17.533 INFO  [cid=] [main] [com.ppb.platform.personalisation.msq.MessagingGracefulShutdown] shutdownDelayInitiated Application shutdown starting. Pausing incoming channels...
2026-01-15 15:40:24.533 INFO  [cid=] [main] [com.ppb.platform.personalisation.msq.MessagingGracefulShutdown] shutdownDelayInitiated Paused channel: inbound
2026-01-15 15:40:27.105 INFO  [cid=] [main] [com.ppb.platform.personalisation.msq.MessagingGracefulShutdown] shutdownDelayInitiated Paused channel: delete
2026-01-15 15:40:52.130 ERROR [cid=] [main] [io.quarkus.runtime.shutdown.ShutdownRecorder] Timed out waiting for graceful shutdown, shutting down anyway. [Error Occurred After Shutdown]
2026-01-15 15:40:52.221 INFO  [cid=] [main] [com.ppb.platform.personalisation.msq.MessagingGracefulShutdown] Application shutdown starting. Pausing incoming channels...
2026-01-15 15:40:52.222 INFO  [cid=] [main] [com.ppb.platform.personalisation.msq.MessagingGracefulShutdown] Paused channel: inbound
2026-01-15 15:40:52.222 INFO  [cid=] [main] [com.ppb.platform.personalisation.msq.MessagingGracefulShutdown] Paused channel: delete
2026-01-15 15:40:52.255 ERROR [cid=] [vert.x-eventloop-thread-0] [io.smallrye.reactive.messaging.aws.sqs] SRMSG19302: Error while receiving the message from channel 'inbound' [Error Occurred After Shutdown]: software.amazon.awssdk.core.exception.SdkClientException: Task java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask@5b0c13b7[Not completed, task = java.util.concurrent.Executors$RunnableAdapter@51da574b[Wrapped task = software.amazon.awssdk.core.internal.http.pipeline.stages.AsyncRetryableStage$RetryingExecutor$$Lambda/0x00000008010c5f98@75423dd3]] rejected from java.util.concurrent.ScheduledThreadPoolExecutor@66b62ebd[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 0]
All I try to avoid is stop Smallrye polling messages when ShutdownDelayInitiated is triggered. The error was triggered during Quarkus shutdown period, the AWS SDK submitted the retry to a already shut-down pool ScheduledThreadPoolExecutor.

How can I fix this issue? I try to avoid losing sqs messages during Quarkus shutdown.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions