How to Prevent Duplicate Email Delivery in Postfix When Using Aliases with Overlapping Recipients


2 views

When implementing email distribution lists through Postfix aliases, a common pain point emerges in scenarios where recipients are included both in the alias expansion and explicitly in message headers. Consider this canonical example from /etc/aliases:


# Standard alias definition
all@company.com:    dev-team@company.com, qa@company.com, manager@company.com

When someone sends to all@company.com while CC'ing dev-team@company.com, Postfix's parallel delivery mechanism processes these as separate recipients, resulting in duplicate messages for overlapping addresses.

Postfix intentionally doesn't perform recipient deduplication during SMTP transactions for performance reasons. The cleanup daemon processes messages through these sequential steps:

  1. Header/body processing
  2. Alias expansion
  3. Canonical mappings
  4. Routing decision

The critical detail is that alias expansion happens before the final recipient list is compiled for delivery.

Option 1: Recipient Canonical Mapping

Add a recipient_canonical_maps pattern that normalizes addresses to a single form:


# /etc/postfix/recipient_canonical
dev-team@company.com     all@company.com
qa@company.com           all@company.com
manager@company.com      all@company.com

Then in main.cf:


recipient_canonical_maps = hash:/etc/postfix/recipient_canonical

Option 2: Header Check Filtering

Implement a before-queue content filter using a policy server:


smtpd_recipient_restrictions =
    check_policy_service inet:127.0.0.1:10031

Sample Python policy server (save as dedup_server.py):


import sys
import email
from collections import OrderedDict

def process_message():
    recipients = set()
    while True:
        line = sys.stdin.readline()
        if line == "\n":
            break
        key, value = line.split('=', 1)
        if key == 'recipient':
            recipients.add(value.strip())
    
    # Deduplicate logic
    filtered = list(OrderedDict.fromkeys(recipients))
    for r in filtered:
        print(f"action=OK recipient={r}")
    print("action=DUNNO\n")

if __name__ == "__main__":
    process_message()

Option 3: Post-Delivery Deduplication

For Dovecot users, implement a Sieve filter:


require ["duplicate", "variables"];

if duplicate :seconds 300 :header "message-id" {
    discard;
}

Each approach has distinct performance characteristics:

Method Processing Stage Overhead
Canonical Maps SMTP Transaction Low (hash lookup)
Policy Server Pre-queue Medium (network I/O)
Sieve Filter Post-delivery High (message parsing)

For large installations, consider implementing a custom smtpd_proxy_filter:


smtpd_proxy_filter = dedup-filter:10025

With this C++ filter skeleton (compile with -lboost-regex):


#include <boost/regex.hpp>
#include <unordered_set>

void process_email() {
    std::unordered_set<std::string> recipients;
    boost::regex recipient_pattern("^RCPT TO:<(.*?)>");
    
    // Process SMTP commands
    while(get_command()) {
        if(boost::smatch matches; 
           regex_search(buffer, matches, recipient_pattern)) {
            recipients.insert(matches[1]);
        }
    }
    
    // Rewrite transaction
    for(const auto& r : recipients) {
        send_command("RCPT TO:<" + r + ">");
    }
}

Many Postfix administrators encounter duplicate email delivery when using distribution aliases combined with CC recipients. The classic scenario:

# /etc/aliases
all@domain.com: foo@domain.com, bar@domain.com, baz@domain.com

When someone sends to all@domain.com while CC'ing foo@domain.com, Postfix's parallel delivery mechanism creates duplicates.

Postfix processes recipients in parallel without full expansion, which provides performance benefits but causes this duplicate delivery. The official FAQ explains this as intentional design.

Option 1: Recipient Deduplication with Cleanup Service

Create a custom cleanup service that filters duplicates before delivery:

# master.cf addition
cleanup_dedup unix  n       -       n       -       0       cleanup
  -o header_checks=regexp:/etc/postfix/dedup_recipients

Then create the regexp file:

# /etc/postfix/dedup_recipients
/^To:.*(foo@domain.com).*CC:.*\1/i IGNORE
/^CC:.*(foo@domain.com).*To:.*\1/i IGNORE

Option 2: Using Before-Queue Filter

Implement a before-queue content filter with Amavis or custom script:

# main.cf
content_filter = smtp-amavis:[127.0.0.1]:10024
receive_override_options = no_address_mappings

Then add recipient processing in your filter script:

#!/usr/bin/perl
use Email::Simple;
my $email = Email::Simple->new($stdin);
my %seen;
$email->header_set('To', join ', ', grep !$seen{$_}++, split /,/, $email->header('To'));
# Similar for CC/BCC headers

Option 3: Postfix After-Queue Processing

Use Dovecot's deliver with sieve filtering:

# /etc/dovecot/conf.d/90-sieve.conf
plugin {
  sieve = ~/.dovecot.sieve
  sieve_global = /var/lib/dovecot/sieve/default.sieve
}

Create a sieve script:

require ["duplicate", "variables"];
if duplicate :seconds 60 {
  discard;
}

When implementing these solutions, consider:

  • Performance impact on high-volume servers
  • Interaction with existing content filters like Amavis
  • Logging requirements for troubleshooting
  • Edge cases like mailing lists or forwarded messages

For Mac OS X Server specifically, you'll need to modify the Postfix configuration through /Library/Server/Mail/Config/postfix/main.cf rather than the standard locations.

# OS X Server-specific master.cf override
sudo serveradmin settings mail:postfix:master_cf:cleanup_dedup = "unix n - n - 0 cleanup -o header_checks=regexp:/etc/postfix/dedup_recipients"