Outbound delivery sometimes gets stuck

[cuu508]

I run a maddy server which takes messages on submission port, and delivers them to remote mail servers. The sending volume is around 1 message per second. These are transactional emails from a web application, the sending volume fluctuates a bit, but I'm careful not to send big bursts of messages.

I've noticed two cases about a month apart where outbound delivery got stuck for 10 or so minutes, and then got "unstuck" all by itself. Here's what I saw in the logs:

• normally there's a mix of submission: incoming message, submission: accepted, queue: delivered, and other messages.
• symptoms of being stuck: I still see submission: ... messages in logs, I see the number of files in /var/lib/maddy/remote_queue climbing, but no queue: ... or remote: ... log messages
• getting unstuck: I see a wave of messages like this:

remote: QUIT error        {"msg_id":"xxxxxxxxx","reason":"write tcp 111.111.111:1234-\u003e64.233.165.26:25: write: broken pipe","remote_server":"aspmx.l.google.com."}

And then things proceed normally, maddy gradually eats through the backlog of queued messages and delivers them.

I've been staring at logs, at my configuration, and at maddy docs, but cannot figure out if it is an issue with my configuration or with maddy itself. My configuration below with a few changes: I replaced my server's hostname with example.com, and usernames, passwords, tokens with dummy values.

$(public_ip) = 111.111.111.111

hostname mail.example.com

tls {
    loader acme {
        hostname mail.example.com
        email contact@example.com
        agreed
        challenge dns-01
        dns cloudflare {
            api_token "token-goes-here"
        }
    }
}

msgpipeline webhook {
    default_destination {
        check {
            command /usr/bin/curl -fsS -o /dev/null --data-binary @- https://example.com/bounce_handler/
            command /usr/bin/cp /dev/stdin bounces/{msg_id}.txt
        }

        deliver_to dummy
    }
}

smtp tcp://$(public_ip):25 {
    limits {
        # Up to 20 msgs/sec across max. 10 SMTP connections.
        all rate 20 1s
        all concurrency 10
    }

    destination mail.example.com {
        deliver_to &webhook
    }

    default_destination {
        reject 551 5.7.1 "Delivery not authorized, message refused"
    }
}

submission tls://$(public_ip):465 {
    limits {
        # Up to 50 msgs/sec across any amount of SMTP connections.
        all rate 50 1s
    }

    auth pass_table static {
        entry smtpuser "smtppassword"
    }

    modify {
        # Bounces should be delivered to us, not the main MX
        replace_sender regexp "(.+)@example.com" "$1@mail.example.com"
    }

    source mail.example.com {
        default_destination {
            modify {
                dkim {
                    domains example.com
                    selector mail
                    sign_subdomains yes
                }
            }
            deliver_to &remote_queue
        }
    }

    default_source {
        reject 551 5.7.1 "Bad from address, we only accept email from example.com"
    }
}

target.queue remote_queue {
    target &outbound_delivery
    # retry for ~32 hours:
    max_tries 17

    autogenerated_msg_domain mail.example.com
    bounce {
        deliver_to &webhook
    }
}

target.remote outbound_delivery {
    local_ip $(public_ip)
    limits {
        # Up to 20 msgs/sec across max. 10 SMTP connections for each recipient domain.
        destination rate 20 1s
        destination concurrency 10
    }
    mx_auth {
        dane
        mtasts
    }
}

I'm hoping someone recognizes the symptoms and can give me any hints about what might be going wrong. Thank you in advance for any tips!

[foxcpp]

Might be related: #596