Skip to content

fix: suppress dead-letter logging during a terminating shutdown when disabled (#3256)#3262

Open
He-Pin wants to merge 1 commit into
mainfrom
fix/3256-deadletter-shutdown-suppress
Open

fix: suppress dead-letter logging during a terminating shutdown when disabled (#3256)#3262
He-Pin wants to merge 1 commit into
mainfrom
fix/3256-deadletter-shutdown-suppress

Conversation

@He-Pin

@He-Pin He-Pin commented Jun 28, 2026

Copy link
Copy Markdown
Member

Motivation

When pekko.log-dead-letters-during-shutdown = off (which is the default,
reference.conf:63), dead letters were still logged throughout the entire
CoordinatedShutdown sequence. DeadLetterListener had no shutdown awareness
and was only stopped at the very end of shutdown (finalTerminate), so every
dead letter produced while the system was terminating — potentially for the
whole (and sometimes lengthy) shutdown — was logged unconditionally. The
setting was effectively ignored until the very last moment.

Fixes #3256.

Modification

DeadLetterListener now suppresses dead-letter logging while a coordinated
shutdown that actually terminates this ActorSystem is in progress, when
log-dead-letters-during-shutdown is disabled.

Crucially, whether the shutdown terminates the system is taken from the
effective per-reason configuration, using the exact same resolution that
CoordinatedShutdown itself uses
(CoordinatedShutdown.confWithOverrides(conf, shutdownReason()).getBoolean("terminate-actor-system")).
This matters because terminate-actor-system can be overridden per shutdown
reason. A coordinated shutdown that does not terminate the system (globally
via terminate-actor-system = off, or only for a specific reason via
reason-overrides) leaves a shutdown reason set on a still-running system —
that must not suppress logging, otherwise dead-letter logging would be
silently disabled for the remaining lifetime of the running system.

The CoordinatedShutdown extension reference is cached and the effective
decision is memoized (computed at most once — the shutdown reason is set once
and never changes), so the per-dead-letter path stays allocation-free and cheap
(a volatile read of the shutdown reason during normal operation, no extension
lookup).

Result

  • Dead letters are no longer logged during a terminating shutdown when
    log-dead-letters-during-shutdown = off — the setting is now honored for the
    whole shutdown, not just after the listener is finally stopped.
  • Logging continues normally for a non-terminating coordinated shutdown
    (globally or per shutdown reason) and during normal operation.
  • All new members are private; no public API or binary-compatibility
    change
    actor / mimaReportBinaryIssues is clean.

Tests

  • sbt "actor-tests/testOnly org.apache.pekko.event.DeadLetterListenerShutdownSpec"
    → 4 passed, plus DeadLetterSupressionSpec and DeadLetterSuspensionSpec
    (no regression).
  • Directional tests (a paused first CoordinatedShutdown phase keeps the
    listener alive mid-shutdown) that fail before the fix:
    • a terminating shutdown must suppress the dead letter;
    • a non-terminating shutdown (global terminate-actor-system = off) must keep
      logging;
    • a non-terminating shutdown via a per-reason override must keep logging
      (this one fails against an earlier base-setting-only gate);
    • logging must continue even after a non-terminating run completes.
  • sbt "actor/mimaReportBinaryIssues" → no binary issues.

References

Fixes #3256


This is an original contribution to Apache Pekko, made under the Apache License
2.0 (i.e. the changes are now Apache licensed). No third-party or Akka-derived
code is included.

@He-Pin He-Pin force-pushed the fix/3256-deadletter-shutdown-suppress branch from ce0dc5a to 164b485 Compare July 1, 2026 13:31
…disabled

Motivation:
With pekko.log-dead-letters-during-shutdown = off (the default), dead letters
were still logged during the entire CoordinatedShutdown sequence. The
DeadLetterListener had no shutdown awareness and was only stopped at the very
end of shutdown (finalTerminate), so dead letters produced while the system was
terminating - potentially for the whole shutdown duration - were logged
unconditionally.

Modification:
- DeadLetterListener now suppresses logging while a coordinated shutdown that
  actually terminates this ActorSystem is in progress, when
  log-dead-letters-during-shutdown is disabled. Whether the shutdown terminates
  the system is taken from the EFFECTIVE per-reason configuration
  (CoordinatedShutdown.confWithOverrides), the same resolution CoordinatedShutdown
  uses, so a non-terminating run - including one where only a specific shutdown
  reason overrides terminate-actor-system to off - keeps logging enabled instead
  of silently disabling it for the rest of a still-running system's life. The
  CoordinatedShutdown extension reference is cached and the effective decision is
  computed at most once to keep the dead-letter path cheap.

Result:
Dead letters are no longer logged during a terminating shutdown when
log-dead-letters-during-shutdown = off, while logging continues normally for a
non-terminating coordinated shutdown (globally or per reason) and during normal
operation.

Tests:
- actor-tests/testOnly org.apache.pekko.event.DeadLetterListenerShutdownSpec
  (plus DeadLetterSupressionSpec, DeadLetterSuspensionSpec) - all passed.
  Added directional tests that fail before the fix: a terminating shutdown that
  must suppress, and non-terminating shutdowns (global and per-reason override)
  that must keep logging.
- actor/mimaReportBinaryIssues - no binary issues.

References:
Fixes #3256
@He-Pin He-Pin force-pushed the fix/3256-deadletter-shutdown-suppress branch from 164b485 to 1cdc7a9 Compare July 6, 2026 16:49
@He-Pin He-Pin requested a review from pjfanning July 6, 2026 16:54
@He-Pin He-Pin marked this pull request as ready for review July 6, 2026 17:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

log-dead-letters-during-shutdown=off not respected during CoordinatedShutdown

1 participant