[Questions] Shovel fails to (re)declare a quorum queue which fails to start a Raft replica due to {error, already_started} returned somewhere

[PhBouzid]

First of all, I would like to thank you for your work on RabbitMQ. It has been a reliable and fundamental component of our system for many years.

Community Support Policy

• I have read RabbitMQ's Community Support Policy
• I run RabbitMQ 4.x, the only series currently covered by community support
• I promise to provide all relevant information (versions, logs from all nodes, rabbitmq-diagnostics output, detailed reproduction steps)

RabbitMQ version used

• 3.13.6

• 4.1.x (we faced an issue with shovel recreationl)

Erlang version used

26.2.x

Operating system (distribution) used

Kubernetes

How is RabbitMQ deployed?

Community Docker image

rabbitmq-diagnostics status output

See https://www.rabbitmq.com/docs/cli to learn how to use rabbitmq-diagnostics

/ $ rabbitmq-diagnostics cluster_status
Cluster status of node rabbit@rmqlocal-2.rmqlocal.rabbitmq.svc.cluster.local ...
Basics

Cluster name: rabbitmq
Total CPU cores available cluster-wide: 24

Disk Nodes

rabbit@rmqlocal-0.rmqlocal.rabbitmq.svc.cluster.local
rabbit@rmqlocal-1.rmqlocal.rabbitmq.svc.cluster.local
rabbit@rmqlocal-2.rmqlocal.rabbitmq.svc.cluster.local

Running Nodes

rabbit@rmqlocal-0.rmqlocal.rabbitmq.svc.cluster.local
rabbit@rmqlocal-1.rmqlocal.rabbitmq.svc.cluster.local
rabbit@rmqlocal-2.rmqlocal.rabbitmq.svc.cluster.local

Versions

rabbit@rmqlocal-2.rmqlocal.rabbitmq.svc.cluster.local: RabbitMQ 3.13.6 on Erlang 26.2.5.2
rabbit@rmqlocal-0.rmqlocal.rabbitmq.svc.cluster.local: RabbitMQ 3.13.6 on Erlang 26.2.5.2
rabbit@rmqlocal-1.rmqlocal.rabbitmq.svc.cluster.local: RabbitMQ 3.13.6 on Erlang 26.2.5.2

CPU Cores

Node: rabbit@rmqlocal-2.rmqlocal.rabbitmq.svc.cluster.local, available CPU cores: 8
Node: rabbit@rmqlocal-0.rmqlocal.rabbitmq.svc.cluster.local, available CPU cores: 8
Node: rabbit@rmqlocal-1.rmqlocal.rabbitmq.svc.cluster.local, available CPU cores: 8

Maintenance status

Node: rabbit@rmqlocal-2.rmqlocal.rabbitmq.svc.cluster.local, status: not under maintenance
Node: rabbit@rmqlocal-0.rmqlocal.rabbitmq.svc.cluster.local, status: not under maintenance
Node: rabbit@rmqlocal-1.rmqlocal.rabbitmq.svc.cluster.local, status: not under maintenance

Alarms

(none)

Network Partitions

(none)

Listeners

Node: rabbit@rmqlocal-2.rmqlocal.rabbitmq.svc.cluster.local, interface: [::], port: 15692, protocol: http/prometheus, purpose: Prometheus exporter API over HTTP
Node: rabbit@rmqlocal-2.rmqlocal.rabbitmq.svc.cluster.local, interface: [::], port: 25672, protocol: clustering, purpose: inter-node and CLI tool communication
Node: rabbit@rmqlocal-2.rmqlocal.rabbitmq.svc.cluster.local, interface: [::], port: 5672, protocol: amqp, purpose: AMQP 0-9-1 and AMQP 1.0
Node: rabbit@rmqlocal-2.rmqlocal.rabbitmq.svc.cluster.local, interface: [::], port: 15672, protocol: http, purpose: HTTP API
Node: rabbit@rmqlocal-0.rmqlocal.rabbitmq.svc.cluster.local, interface: [::], port: 15672, protocol: http, purpose: HTTP API
Node: rabbit@rmqlocal-0.rmqlocal.rabbitmq.svc.cluster.local, interface: [::], port: 15692, protocol: http/prometheus, purpose: Prometheus exporter API over HTTP
Node: rabbit@rmqlocal-0.rmqlocal.rabbitmq.svc.cluster.local, interface: [::], port: 25672, protocol: clustering, purpose: inter-node and CLI tool communication
Node: rabbit@rmqlocal-0.rmqlocal.rabbitmq.svc.cluster.local, interface: [::], port: 5672, protocol: amqp, purpose: AMQP 0-9-1 and AMQP 1.0
Node: rabbit@rmqlocal-1.rmqlocal.rabbitmq.svc.cluster.local, interface: [::], port: 15672, protocol: http, purpose: HTTP API
Node: rabbit@rmqlocal-1.rmqlocal.rabbitmq.svc.cluster.local, interface: [::], port: 15692, protocol: http/prometheus, purpose: Prometheus exporter API over HTTP
Node: rabbit@rmqlocal-1.rmqlocal.rabbitmq.svc.cluster.local, interface: [::], port: 25672, protocol: clustering, purpose: inter-node and CLI tool communication
Node: rabbit@rmqlocal-1.rmqlocal.rabbitmq.svc.cluster.local, interface: [::], port: 5672, protocol: amqp, purpose: AMQP 0-9-1 and AMQP 1.0

Feature flags

Flag: classic_mirrored_queue_version, state: enabled
Flag: classic_queue_type_delivery_support, state: enabled
Flag: detailed_queues_endpoint, state: enabled
Flag: direct_exchange_routing_v2, state: enabled
Flag: drop_unroutable_metric, state: enabled
Flag: empty_basic_get_metric, state: enabled
Flag: feature_flags_v2, state: enabled
Flag: implicit_default_bindings, state: enabled
Flag: khepri_db, state: disabled
Flag: listener_records_in_ets, state: enabled
Flag: maintenance_mode_status, state: enabled
Flag: message_containers, state: enabled
Flag: message_containers_deaths_v2, state: enabled
Flag: quorum_queue, state: enabled
Flag: quorum_queue_non_voters, state: enabled
Flag: restart_streams, state: enabled
Flag: stream_filtering, state: enabled
Flag: stream_queue, state: enabled
Flag: stream_sac_coordinator_unblock_group, state: enabled
Flag: stream_single_active_consumer, state: enabled
Flag: stream_update_config_command, state: enabled
Flag: tracking_records_in_ets, state: enabled
Flag: user_limits, state: enabled
Flag: virtual_host_metadata, state: enabled

Logs from node 1 (with sensitive values edited out)

See https://www.rabbitmq.com/docs/logging to learn how to collect logs

# PASTE LOG HERE, BETWEEN BACKTICKS

Logs from node 2 (if applicable, with sensitive values edited out)

See https://www.rabbitmq.com/docs/logging to learn how to collect logs

# PASTE LOG HERE, BETWEEN BACKTICKS

Logs from node 3 (if applicable, with sensitive values edited out)

See https://www.rabbitmq.com/docs/logging to learn how to collect logs

2025-09-02 06:20:59.290729+00:00 [info] <0.4448006.0> connection <0.4448006.0> (169.254.169.5:47904 -> 10.128.2.159:5672 - Shovel env-1.quote_ibi_tracking.indexation-v1-env-1-exported): user 'env-1.259b3d9bfb' authenticated and granted access to vhost 'env-1.public'�[0m
�[38;5;160m2025-09-02 06:20:59.296506+00:00 [error] <0.4448032.0>     supervisor: {<0.4448032.0>,ra_server_sup}�[0m
�[38;5;160m2025-09-02 06:20:59.296506+00:00 [error] <0.4448032.0>     errorContext: start_error�[0m
�[38;5;160m2025-09-02 06:20:59.296506+00:00 [error] <0.4448032.0>     reason: {already_started,<0.4141943.0>}�[0m
�[38;5;160m2025-09-02 06:20:59.296506+00:00 [error] <0.4448032.0>     offender: [{pid,undefined},�[0m
�[38;5;160m2025-09-02 06:20:59.296506+00:00 [error] <0.4448032.0>                {id,�[0m
�[38;5;160m2025-09-02 06:20:59.296506+00:00 [error] <0.4448032.0>                 'env-1.public-exported_env-1.quote-tracking.quote.state.change-v2'},�[0m
�[38;5;160m2025-09-02 06:20:59.296506+00:00 [error] <0.4448032.0>                {mfargs,�[0m
�[38;5;160m2025-09-02 06:20:59.296506+00:00 [error] <0.4448032.0>                 {ra_server_proc,start_link,�[0m
�[38;5;160m2025-09-02 06:20:59.296506+00:00 [error] <0.4448032.0>                  [#{id =>�[0m
�[38;5;160m2025-09-02 06:20:59.296506+00:00 [error] <0.4448032.0>                      {'env-1.public-exported_env-1.quote-tracking.quote.state.change-v2',�[0m
�[38;5;160m2025-09-02 06:20:59.296506+00:00 [error] <0.4448032.0>                       'rabbit@rmqlocal-2.rmqlocal.rabbitmq.svc.cluster.local'},�[0m
�[38;5;160m2025-09-02 06:20:59.296506+00:00 [error] <0.4448032.0>                     machine =>�[0m
�[38;5;160m2025-09-02 06:20:59.296506+00:00 [error] <0.4448032.0>                      {module,rabbit_fifo,�[0m
�[38;5;160m2025-09-02 06:20:59.296506+00:00 [error] <0.4448032.0>                       #{name =>�[0m
�[38;5;160m2025-09-02 06:20:59.296506+00:00 [error] <0.4448032.0>                          'env-1.public-exported_env-1.quote-tracking.quote.state.change-v2',�[0m
�[38;5;160m2025-09-02 06:20:59.296506+00:00 [error] <0.4448032.0>                         max_length => undefined,max_bytes => undefined,�[0m
�[38;5;160m2025-09-02 06:20:59.296506+00:00 [error] <0.4448032.0>                         created => 1756794059295,�[0m
�[38;5;160m2025-09-02 06:20:59.296506+00:00 [error] <0.4448032.0>                         queue_resource =>�[0m
�[38;5;160m2025-09-02 06:20:59.296506+00:00 [error] <0.4448032.0>                          {resource,<<"env-1.public-exported">>,�[0m
�[38;5;160m2025-09-02 06:20:59.296506+00:00 [error] <0.4448032.0>                           queue,�[0m
�[38;5;160m2025-09-02 06:20:59.296506+00:00 [error] <0.4448032.0>                           <<"env-1.quote-tracking.quote.state.change-v2">>},�[0m
�[38;5;160m2025-09-02 06:20:59.296506+00:00 [error] <0.4448032.0>                         dead_letter_handler => undefined,�[0m
�[38;5;160m2025-09-02 06:20:59.296506+00:00 [error] <0.4448032.0>                         become_leader_handler =>�[0m
�[38;5;160m2025-09-02 06:20:59.296506+00:00 [error] <0.4448032.0>                          {rabbit_quorum_queue,become_leader,�[0m
�[38;5;160m2025-09-02 06:20:59.296506+00:00 [error] <0.4448032.0>                           [{resource,<<"env-1.public-exported">>,�[0m
�[38;5;160m2025-09-02 06:20:59.296506+00:00 [error] <0.4448032.0>                             queue,�[0m
�[38;5;160m2025-09-02 06:20:59.296506+00:00 [error] <0.4448032.0>                             <<"env-1.quote-tracking.quote.state.change-v2">>}]},�[0m
�[38;5;160m2025-09-02 06:20:59.296506+00:00 [error] <0.4448032.0>                         overflow_strategy => drop_head,�[0m
�[38;5;160m2025-09-02 06:20:59.296506+00:00 [error] <0.4448032.0>                         delivery_limit => undefined,expires => undefined,�[0m
�[38;5;160m2025-09-02 06:20:59.296506+00:00 [error] <0.4448032.0>                         msg_ttl => undefined,�[0m
�[38;5;160m2025-09-02 06:20:59.296506+00:00 [error] <0.4448032.0>                         single_active_consumer_on => false,�[0m
�[38;5;160m2025-09-02 06:20:59.296506+00:00 [error] <0.4448032.0>                         max_in_memory_length => undefined,�[0m
�[38;5;160m2025-09-02 06:20:59.296506+00:00 [error] <0.4448032.0>                         max_in_memory_bytes => undefined}},�[0m
�[38;5;160m2025-09-02 06:20:59.296506+00:00 [error] <0.4448032.0>                     parent => <0.4448032.0>,membership => voter,�[0m
�[38;5;160m2025-09-02 06:20:59.296506+00:00 [error] <0.4448032.0>                     friendly_name =>�[0m
�[38;5;160m2025-09-02 06:20:59.296506+00:00 [error] <0.4448032.0>                      "queue 'env-1.quote-tracking.quote.state.change-v2' in vhost 'env-1.public-exported'",�[0m
�[38;5;160m2025-09-02 06:20:59.296506+00:00 [error] <0.4448032.0>                     cluster_name =>�[0m
�[38;5;160m2025-09-02 06:20:59.296506+00:00 [error] <0.4448032.0>                      'env-1.public-exported_env-1.quote-tracking.quote.state.change-v2',�[0m
�[38;5;160m2025-09-02 06:20:59.296506+00:00 [error] <0.4448032.0>                     reply_to =>�[0m
�[38;5;160m2025-09-02 06:20:59.296506+00:00 [error] <0.4448032.0>                      {#Ref<0.2647127664.4065591297.144166>,<0.4448027.0>},�[0m
�[38;5;160m2025-09-02 06:20:59.296506+00:00 [error] <0.4448032.0>                     uid => <<"MAASENQDU53CNRU6V8">>,�[0m
�[38;5;160m2025-09-02 06:20:59.296506+00:00 [error] <0.4448032.0>                     initial_members =>�[0m
�[38;5;160m2025-09-02 06:20:59.296506+00:00 [error] <0.4448032.0>                      [{'env-1.public-exported_env-1.quote-tracking.quote.state.change-v2',�[0m
�[38;5;160m2025-09-02 06:20:59.296506+00:00 [error] <0.4448032.0>                        'rabbit@rmqlocal-2.rmqlocal.rabbitmq.svc.cluster.local'},�[0m
�[38;5;160m2025-09-02 06:20:59.296506+00:00 [error] <0.4448032.0>                       {'env-1.public-exported_env-1.quote-tracking.quote.state.change-v2',�[0m
�[38;5;160m2025-09-02 06:20:59.296506+00:00 [error] <0.4448032.0>                        'rabbit@rmqlocal-1.rmqlocal.rabbitmq.svc.cluster.local'},�[0m
�[38;5;160m2025-09-02 06:20:59.296506+00:00 [error] <0.4448032.0>                       {'env-1.public-exported_env-1.quote-tracking.quote.state.change-v2',�[0m
�[38;5;160m2025-09-02 06:20:59.296506+00:00 [error] <0.4448032.0>                        'rabbit@rmqlocal-0.rmqlocal.rabbitmq.svc.cluster.local'}],�[0m
�[38;5;160m2025-09-02 06:20:59.296506+00:00 [error] <0.4448032.0>                     log_init_args =>�[0m
�[38;5;160m2025-09-02 06:20:59.296506+00:00 [error] <0.4448032.0>                      #{snapshot_interval => 8192,�[0m
�[38;5;160m2025-09-02 06:20:59.296506+00:00 [error] <0.4448032.0>                        uid => <<"MAASENQDU53CNRU6V8">>},�[0m
�[38;5;160m2025-09-02 06:20:59.296506+00:00 [error] <0.4448032.0>                     metrics_key =>�[0m
�[38;5;160m2025-09-02 06:20:59.296506+00:00 [error] <0.4448032.0>                      {resource,<<"env-1.public-exported">>,queue,�[0m
�[38;5;160m2025-09-02 06:20:59.296506+00:00 [error] <0.4448032.0>                       <<"env-1.quote-tracking.quote.state.change-v2">>},�[0m
�[38;5;160m2025-09-02 06:20:59.296506+00:00 [error] <0.4448032.0>                     ra_event_formatter =>�[0m
�[38;5;160m2025-09-02 06:20:59.296506+00:00 [error] <0.4448032.0>                      {rabbit_quorum_queue,format_ra_event,�[0m
�[38;5;160m2025-09-02 06:20:59.296506+00:00 [error] <0.4448032.0>                       [{resource,<<"env-1.public-exported">>,queue,�[0m
�[38;5;160m2025-09-02 06:20:59.296506+00:00 [error] <0.4448032.0>                         <<"env-1.quote-tracking.quote.state.change-v2">>}]},�[0m
�[38;5;160m2025-09-02 06:20:59.296506+00:00 [error] <0.4448032.0>                     tick_timeout => 5000,�[0m
�[38;5;160m2025-09-02 06:20:59.296506+00:00 [error] <0.4448032.0>                     system_config =>�[0m
�[38;5;160m2025-09-02 06:20:59.296506+00:00 [error] <0.4448032.0>                      #{message_queue_data => off_heap,name => quorum_queues,�[0m
�[38;5;160m2025-09-02 06:20:59.296506+00:00 [error] <0.4448032.0>                        names =>�[0m
�[38;5;160m2025-09-02 06:20:59.296506+00:00 [error] <0.4448032.0>                         #{directory => ra_directory,�[0m
�[38;5;160m2025-09-02 06:20:59.296506+00:00 [error] <0.4448032.0>                           closed_mem_tbls => ra_log_closed_mem_tables,�[0m
�[38;5;160m2025-09-02 06:20:59.296506+00:00 [error] <0.4448032.0>                           directory_rev => ra_directory_reverse,�[0m
�[38;5;160m2025-09-02 06:20:59.296506+00:00 [error] <0.4448032.0>                           log_ets => ra_log_ets,log_meta => ra_log_meta,�[0m
�[38;5;160m2025-09-02 06:20:59.296506+00:00 [error] <0.4448032.0>                           log_sup => ra_log_sup,�[0m
�[38;5;160m2025-09-02 06:20:59.296506+00:00 [error] <0.4448032.0>                           open_mem_tbls => ra_log_open_mem_tables,�[0m
�[38;5;160m2025-09-02 06:20:59.296506+00:00 [error] <0.4448032.0>                           segment_writer => ra_log_segment_writer,�[0m
�[38;5;160m2025-09-02 06:20:59.296506+00:00 [error] <0.4448032.0>                           server_sup => ra_server_sup_sup,wal => ra_log_wal,�[0m
�[38;5;160m2025-09-02 06:20:59.296506+00:00 [error] <0.4448032.0>                           wal_sup => ra_log_wal_sup},�[0m
�[38;5;160m2025-09-02 06:20:59.296506+00:00 [error] <0.4448032.0>                        data_dir =>�[0m
�[38;5;160m2025-09-02 06:20:59.296506+00:00 [error] <0.4448032.0>                         "/var/lib/rabbitmq/mnesia/rabbit@rmqlocal-2.rmqlocal.rabbitmq.svc.cluster.local/quorum/rabbit@rmqlocal-2.rmqlocal.rabbitmq.svc.cluster.local",�[0m
�[38;5;160m2025-09-02 06:20:59.296506+00:00 [error] <0.4448032.0>                        wal_max_size_bytes => 536870912,�[0m
�[38;5;160m2025-09-02 06:20:59.296506+00:00 [error] <0.4448032.0>                        wal_max_batch_size => 4096,�[0m
�[38;5;160m2025-09-02 06:20:59.296506+00:00 [error] <0.4448032.0>                        compress_mem_tables => false,�[0m
�[38;5;160m2025-09-02 06:20:59.296506+00:00 [error] <0.4448032.0>                        default_max_append_entries_rpc_batch_size => 16,�[0m
�[38;5;160m2025-09-02 06:20:59.296506+00:00 [error] <0.4448032.0>                        segment_compute_checksums => true,�[0m
�[38;5;160m2025-09-02 06:20:59.296506+00:00 [error] <0.4448032.0>                        wal_compute_checksums => true,�[0m
�[38;5;160m2025-09-02 06:20:59.296506+00:00 [error] <0.4448032.0>                        wal_data_dir =>�[0m
�[38;5;160m2025-09-02 06:20:59.296506+00:00 [error] <0.4448032.0>                         "/var/lib/rabbitmq/mnesia/rabbit@rmqlocal-2.rmqlocal.rabbitmq.svc.cluster.local/quorum/rabbit@rmqlocal-2.rmqlocal.rabbitmq.svc.cluster.local",�[0m
�[38;5;160m2025-09-02 06:20:59.296506+00:00 [error] <0.4448032.0>                        segment_max_entries => 4096,�[0m
�[38;5;160m2025-09-02 06:20:59.296506+00:00 [error] <0.4448032.0>                        segment_max_pending => 1024,�[0m
�[38;5;160m2025-09-02 06:20:59.296506+00:00 [error] <0.4448032.0>                        wal_max_entries => undefined,�[0m
�[38;5;160m2025-09-02 06:20:59.296506+00:00 [error] <0.4448032.0>                        wal_write_strategy => default,�[0m
�[38;5;160m2025-09-02 06:20:59.296506+00:00 [error] <0.4448032.0>                        wal_sync_method => datasync,�[0m
�[38;5;160m2025-09-02 06:20:59.296506+00:00 [error] <0.4448032.0>                        wal_garbage_collect => false,wal_pre_allocate => false,�[0m
�[38;5;160m2025-09-02 06:20:59.296506+00:00 [error] <0.4448032.0>                        wal_min_bin_vheap_size => 46422,�[0m
�[38;5;160m2025-09-02 06:20:59.296506+00:00 [error] <0.4448032.0>                        wal_min_heap_size => 233,server_min_heap_size => 233,�[0m
�[38;5;160m2025-09-02 06:20:59.296506+00:00 [error] <0.4448032.0>                        server_min_bin_vheap_size => 46422,�[0m
�[38;5;160m2025-09-02 06:20:59.296506+00:00 [error] <0.4448032.0>                        default_max_pipeline_count => 4096,�[0m
�[38;5;160m2025-09-02 06:20:59.296506+00:00 [error] <0.4448032.0>                        snapshot_chunk_size => 1000000,�[0m
�[38;5;160m2025-09-02 06:20:59.296506+00:00 [error] <0.4448032.0>                        receive_snapshot_timeout => 30000,�[0m
�[38;5;160m2025-09-02 06:20:59.296506+00:00 [error] <0.4448032.0>                        low_priority_commands_flush_size => 16,�[0m
�[38;5;160m2025-09-02 06:20:59.296506+00:00 [error] <0.4448032.0>                        low_priority_commands_in_memory_size => 16}}]}},�[0m
�[38;5;160m2025-09-02 06:20:59.296506+00:00 [error] <0.4448032.0>                {restart_type,transient},�[0m
�[38;5;160m2025-09-02 06:20:59.296506+00:00 [error] <0.4448032.0>                {significant,false},�[0m
�[38;5;160m2025-09-02 06:20:59.296506+00:00 [error] <0.4448032.0>                {shutdown,5000},�[0m
�[38;5;160m2025-09-02 06:20:59.296506+00:00 [error] <0.4448032.0>                {child_type,worker}]�[0m
�[38;5;160m2025-09-02 06:20:59.296506+00:00 [error] <0.4448032.0> �[0m
�[38;5;160m2025-09-02 06:20:59.299088+00:00 [error] <0.4448029.0> ra: failed to start a server #{id => {'env-1.public-exported_env-1.quote-tracking.quote.state.change-v2','rabbit@rmqlocal-1.rmqlocal.rabbitmq.svc.cluster.local'},machine => {module,rabbit_fifo,#{name => 'env-1.public-exported_env-1.quote-tracking.quote.state.change-v2',max_length => undefined,max_bytes => undefined,created => 1756794059295,queue_resource => {resource,<<109,97,97,115,46,101,110,118,45,49,45,98,115,115,46,112,117,98,108,105,99,45,101,120,112,111,114,116,101,100>>,queue,<<101,110,118,45,49,45,98,115,115,46,113,117,111,116,101,45,116,114,97,99,107,105,110,103,46,113,117,111,116,101,46,115,116,97,116,101,46,99,104,97,110,103,101,45,118,50,45,100,108,113>>},dead_letter_handler => undefined,become_leader_handler => {rabbit_quorum_queue,become_leader,[{resource,<<109,97,97,115,46,101,110,118,45,49,45,98,115,115,46,112,117,98,108,105,99,45,101,120,112,111,114,116,101,100>>,queue,<<101,110,118,45,49,45,98,115,115,46,113,117,111,116,101,45,116,114,97,99,107,105,110,103,46,113,117,111,116,101,46,115,116,97,116,101,46,99,104,97,110,103,101,45,118,50,45,100,108,113>>}]},overflow_strategy => drop_head,delivery_limit => undefined,expires => undefined,msg_ttl => undefined,single_active_consumer_on => false,max_in_memory_length => undefined,max_in_memory_bytes => undefined}},membership => voter,friendly_name => 

rabbitmq.conf

See https://www.rabbitmq.com/docs/configure#config-location to learn how to find rabbitmq.conf file location

## clustering
cluster_formation.peer_discovery_backend = rabbit_peer_discovery_k8s
cluster_formation.k8s.host = kubernetes.default.svc.cluster.local
cluster_formation.k8s.address_type = hostname
cluster_partition_handling = autoheal

## queue master locator
queue_master_locator = min-masters

## enable guest user
loopback_users.guest = false

## logging
log.file = false
log.upgrade.file = false
log.upgrade.level = none
log.console = true
management.load_definitions = /etc/rabbitmq/logging_definitions.json


## custom
cluster_name = rabbitmq
vm_memory_high_watermark.relative = 0.80
total_memory_available_override_value = 2000MiB

Steps to deploy RabbitMQ cluster

We deploy RabbitMQ using Helm.

Steps to reproduce the behavior in question

1. Create vhost

2. Create quorum queue

3. Create shovel between queues

4. Use the system normally (publish/consume messages)

Delete:

1. Shovel

2. Quorum queue

3. Vhost
   Recreate all of the above with the same names

Repeat steps 1–6 multiple times

advanced.config

See https://www.rabbitmq.com/docs/configure#config-location to learn how to find advanced.config file location

# PASTE advanced.config HERE, BETWEEN BACKTICKS

Application code

# PASTE CODE HERE, BETWEEN BACKTICKS

Kubernetes deployment file

What problem are you trying to solve?

We are running RabbitMQ 3.13.6 and are observing an issue where quorum queues become impossible to re-declare after multiple recreate cycles involving shovels and queues. At some point, the quorum queue enters an inconsistent state and cannot be created again, even though it was deleted via API.

We would like to ask:

Is this a known issue in RabbitMQ 3.13.6?

Could upgrading to RabbitMQ 4.1.5 potentially fix this behavior?

Or are we performing the recreate process incorrectly?

Process Description

In our system, we dynamically manage infrastructure and regularly recreate both quorum queues and shovels.

Our typical workflow:

1. Create vhost

2. Create quorum queue

3. Create shovel between queues

4. Use the system normally (publish/consume messages)

Delete:

1. Shovel

2. Quorum queue

3. Vhost

Repeat the same process again with the same names

This recreate cycle can be executed many times automatically.

After several iterations, quorum queues stop being recreated correctly and start failing during startup or declaration.

I found that there was the similar issue without answer:
rabbitmq/ra#410
#13827

[michaelklishin]

RabbitMQ 3.13 has been out of community support for well over a year now.

According to the logs, a quorum queue Raft cluster fails to start a replica because a process it attempts to start is already running. There is only one place in our Raft implementation where that specific message is logged, and it does not have an {error, already_started} clause.

It would be trivial to add but 3.13 won't get any more public releases and 4.1 will follow suite in Jan 2026. So the only way out for you for now is to reduce the churn of quorum queues (which is required to reproduce per your own words) and then upgrade to, say, 4.2.2 when it ships this month,
then contribute a small change to https://github.com/rabbitmq/ra/blame/09577ea3790640a661baba13fb2a4d6bc56e780b/src/ra.erl#L435-L443 (this is my current understanding after a few minutes of investigating) that will ship in 4.2.x.

[michaelklishin]

@PhBouzid can you share the rest of your logs? Specifically we need to clarify if this message can be present in the logs:

rabbitmq-server/deps/rabbit/src/rabbit_quorum_queue.erl

Lines 932 to 935 in b909a81

	?LOG_WARNING(
	"Force delete of ~w failed with: ~w"
	"This may require manual data clean up",
	[S, Err]),

That would mean that a part of queue state related to its Raft member failed to be deleted.

Quorum queues were not designed for churn and in every scenario we could think of there's only so much a QQ can do to protect itself (or its future incarnations) from churn. But some team members want to reconstruct the sequence of events.