When implementing email distribution lists through Postfix aliases, a common pain point emerges in scenarios where recipients are included both in the alias expansion and explicitly in message headers. Consider this canonical example from /etc/aliases
:
# Standard alias definition
all@company.com: dev-team@company.com, qa@company.com, manager@company.com
When someone sends to all@company.com
while CC'ing dev-team@company.com
, Postfix's parallel delivery mechanism processes these as separate recipients, resulting in duplicate messages for overlapping addresses.
Postfix intentionally doesn't perform recipient deduplication during SMTP transactions for performance reasons. The cleanup
daemon processes messages through these sequential steps:
- Header/body processing
- Alias expansion
- Canonical mappings
- Routing decision
The critical detail is that alias expansion happens before the final recipient list is compiled for delivery.
Option 1: Recipient Canonical Mapping
Add a recipient_canonical_maps
pattern that normalizes addresses to a single form:
# /etc/postfix/recipient_canonical
dev-team@company.com all@company.com
qa@company.com all@company.com
manager@company.com all@company.com
Then in main.cf
:
recipient_canonical_maps = hash:/etc/postfix/recipient_canonical
Option 2: Header Check Filtering
Implement a before-queue content filter using a policy server:
smtpd_recipient_restrictions =
check_policy_service inet:127.0.0.1:10031
Sample Python policy server (save as dedup_server.py
):
import sys
import email
from collections import OrderedDict
def process_message():
recipients = set()
while True:
line = sys.stdin.readline()
if line == "\n":
break
key, value = line.split('=', 1)
if key == 'recipient':
recipients.add(value.strip())
# Deduplicate logic
filtered = list(OrderedDict.fromkeys(recipients))
for r in filtered:
print(f"action=OK recipient={r}")
print("action=DUNNO\n")
if __name__ == "__main__":
process_message()
Option 3: Post-Delivery Deduplication
For Dovecot users, implement a Sieve filter:
require ["duplicate", "variables"];
if duplicate :seconds 300 :header "message-id" {
discard;
}
Each approach has distinct performance characteristics:
Method | Processing Stage | Overhead |
---|---|---|
Canonical Maps | SMTP Transaction | Low (hash lookup) |
Policy Server | Pre-queue | Medium (network I/O) |
Sieve Filter | Post-delivery | High (message parsing) |
For large installations, consider implementing a custom smtpd_proxy_filter
:
smtpd_proxy_filter = dedup-filter:10025
With this C++ filter skeleton (compile with -lboost-regex
):
#include <boost/regex.hpp>
#include <unordered_set>
void process_email() {
std::unordered_set<std::string> recipients;
boost::regex recipient_pattern("^RCPT TO:<(.*?)>");
// Process SMTP commands
while(get_command()) {
if(boost::smatch matches;
regex_search(buffer, matches, recipient_pattern)) {
recipients.insert(matches[1]);
}
}
// Rewrite transaction
for(const auto& r : recipients) {
send_command("RCPT TO:<" + r + ">");
}
}
Many Postfix administrators encounter duplicate email delivery when using distribution aliases combined with CC recipients. The classic scenario:
# /etc/aliases
all@domain.com: foo@domain.com, bar@domain.com, baz@domain.com
When someone sends to all@domain.com
while CC'ing foo@domain.com
, Postfix's parallel delivery mechanism creates duplicates.
Postfix processes recipients in parallel without full expansion, which provides performance benefits but causes this duplicate delivery. The official FAQ explains this as intentional design.
Option 1: Recipient Deduplication with Cleanup Service
Create a custom cleanup service that filters duplicates before delivery:
# master.cf addition
cleanup_dedup unix n - n - 0 cleanup
-o header_checks=regexp:/etc/postfix/dedup_recipients
Then create the regexp file:
# /etc/postfix/dedup_recipients
/^To:.*(foo@domain.com).*CC:.*\1/i IGNORE
/^CC:.*(foo@domain.com).*To:.*\1/i IGNORE
Option 2: Using Before-Queue Filter
Implement a before-queue content filter with Amavis or custom script:
# main.cf
content_filter = smtp-amavis:[127.0.0.1]:10024
receive_override_options = no_address_mappings
Then add recipient processing in your filter script:
#!/usr/bin/perl
use Email::Simple;
my $email = Email::Simple->new($stdin);
my %seen;
$email->header_set('To', join ', ', grep !$seen{$_}++, split /,/, $email->header('To'));
# Similar for CC/BCC headers
Option 3: Postfix After-Queue Processing
Use Dovecot's deliver with sieve filtering:
# /etc/dovecot/conf.d/90-sieve.conf
plugin {
sieve = ~/.dovecot.sieve
sieve_global = /var/lib/dovecot/sieve/default.sieve
}
Create a sieve script:
require ["duplicate", "variables"];
if duplicate :seconds 60 {
discard;
}
When implementing these solutions, consider:
- Performance impact on high-volume servers
- Interaction with existing content filters like Amavis
- Logging requirements for troubleshooting
- Edge cases like mailing lists or forwarded messages
For Mac OS X Server specifically, you'll need to modify the Postfix configuration through /Library/Server/Mail/Config/postfix/main.cf
rather than the standard locations.
# OS X Server-specific master.cf override
sudo serveradmin settings mail:postfix:master_cf:cleanup_dedup = "unix n - n - 0 cleanup -o header_checks=regexp:/etc/postfix/dedup_recipients"