How to Flush Postfix Queue One Email at a Time with Rate Limiting


2 views

When dealing with Postfix mail queues, you might encounter situations where remote servers reject connections with errors like "Too many connections, slow down." Even if you've configured Postfix to limit sending rates for specific domains (e.g., one email per second), the queue might still get stuck. The standard postqueue -f command flushes all emails at once, which doesn't help in these scenarios.

The issue arises because postqueue -f attempts to deliver all queued messages simultaneously. This can overwhelm receiving servers with strict connection limits. What we need is a controlled, sequential delivery mechanism.

Here's a bash script that processes the Postfix queue one message at a time with configurable delays:

#!/bin/bash
DELAY_SECONDS=2  # Adjust this value as needed

while true; do
    # Get the queue ID of the first message
    QUEUE_ID=$(postqueue -p | grep -v '^ *(' | awk 'NR==1{print $1}')
    
    # Exit if queue is empty
    if [ -z "$QUEUE_ID" ]; then
        echo "Queue is empty"
        break
    fi
    
    # Attempt delivery of single message
    echo "Processing message $QUEUE_ID"
    postsuper -d "$QUEUE_ID"
    
    # Wait before processing next message
    sleep $DELAY_SECONDS
done

For more advanced queue management, you can use Postfix's qshape tool to identify problematic domains and process them selectively:

#!/bin/bash
DELAY=2  # Seconds between messages
DOMAIN="example.com"  # Target domain

qshape deferred | grep "$DOMAIN" | awk '{print $1}' | while read QUEUE_ID; do
    echo "Processing $QUEUE_ID for $DOMAIN"
    postsuper -d "$QUEUE_ID"
    sleep $DELAY
done

For a more permanent solution, consider adjusting these Postfix main.cf parameters:

default_destination_concurrency_limit = 1
default_destination_rate_delay = 1s
smtp_destination_concurrency_limit = 1

To monitor the queue processing in real-time, you can use:

watch -n 1 'postqueue -p | head -n 20'

For very large queues, consider processing in batches with longer delays between batches:

#!/bin/bash
BATCH_SIZE=10
DELAY_BETWEEN_BATCHES=30

postqueue -p | grep -v '^ *(' | awk '{print $1}' | \
xargs -n $BATCH_SIZE | while read BATCH; do
    echo "Processing batch: $BATCH"
    postsuper -d $BATCH
    sleep $DELAY_BETWEEN_BATCHES
done

For regular maintenance, set up a cron job to process the queue during off-peak hours:

# Run every hour at 5 minutes past the hour
5 * * * * /usr/local/bin/process_postfix_queue.sh

When dealing with remote SMTP servers that enforce connection limits (like "Too many connections, slow down" errors), using postqueue -f becomes counterproductive. The atomic nature of this command floods the receiving server, often resulting in persistent queue backlogs.

For granular control, we'll leverage Postfix's postqueue -p output parsing combined with postsuper -d for surgical queue manipulation. Here's a Bash script that processes one message per execution:


#!/bin/bash
# Single-message queue processor
QUEUE_ID=$(postqueue -p | grep -v '^ *(' | awk 'BEGIN { RS = "" } { print $1 }' | head -1)
if [[ -n "$QUEUE_ID" ]]; then
  postsuper -d "$QUEUE_ID"
  logger -t postfix "Processed queue ID $QUEUE_ID"
fi

To enforce timing between deliveries (e.g., 2-second intervals), combine with cron and sleep:


*/1 * * * * /usr/local/bin/process_postfix_queue && sleep 2

For persistent issues with specific domains, implement sender-dependent transport maps:


# /etc/postfix/transport
problem-domain.com  slow:

# /etc/postfix/master.cf
slow    unix  -       -       n       -       -       smtp
    -o syslog_name=postfix-slow
    -o smtp_destination_concurrency_limit=1
    -o smtp_destination_rate_delay=1s

Before implementing throttling, identify problem patterns with:


postqueue -p | awk '/@problem-domain\.com/ {print $5}' | sort | uniq -c | sort -n