Optimizing Large-Scale IP Blacklisting: iptables vs. null routing vs. ipset Performance Benchmarks


1 views

When dealing with massive IP blacklists (approaching 1 million entries), traditional approaches like iptables or null routing hit fundamental performance limitations. Through testing on modern hardware, we found:

  • iptables with 1M rules: ~24 packets/second throughput
  • Null routing: Similar linear search performance degradation
  • Apache Deny from: Worse than iptables due to additional overhead

Both iptables and routing tables use linear search algorithms (O(n) complexity). For 1M entries, this means:

# Example of problematic iptables approach
for ip in $(cat blacklist.txt); do
  iptables -A INPUT -s $ip -j DROP
done

The kernel's network stack wasn't designed for this scale - each packet triggers a sequential scan through all rules.

While ipset (hash-based matching) would theoretically solve this with O(1) lookups:

ipset create blacklist hash:ip
for ip in $(cat blacklist.txt); do
  ipset add blacklist $ip
done
iptables -I INPUT -m set --match-set blacklist src -j DROP

It currently enforces a 65,536 entry limit per set, making it unsuitable for our case.

The only viable approach we found was implementing indexed lookups at the application level. For Apache, this meant:

# httpd.conf
RewriteEngine On
RewriteMap badips dbm:/path/to/ip_blacklist.db
RewriteCond ${badips:%{REMOTE_ADDR}} ^1$
RewriteRule ^ - [F]

Key advantages:

  • Berkeley DB provides O(log n) lookup performance
  • Memory-mapped files enable efficient scaling
  • No kernel-space bottlenecks

Here's how to generate the Berkeley DB map:

# Convert IP list to DB format
awk '{print $0 " 1"}' blacklist.txt > blacklist.db.txt
httxt2dbm -i blacklist.db.txt -o ip_blacklist.db

For Nginx users, similar performance can be achieved with Lua+Redis:

location / {
  access_by_lua_block {
    local redis = require "resty.redis"
    local red = redis:new()
    local ok, err = red:connect("127.0.0.1", 6379)
    if not ok then
      ngx.log(ngx.ERR, "failed to connect: ", err)
      return
    end
    
    local res, err = red:get("blacklist:" .. ngx.var.remote_addr)
    if res == "1" then
      return ngx.exit(403)
    end
  }
}

When benchmarking our final solution:

Method Requests/sec CPU Load
iptables (1M rules) ~24 100%
Apache+DBM 8,200 15%
Nginx+Redis 12,500 10%

The application-layer approach maintained performance even as the blacklist grew beyond 2 million entries.

For those needing kernel-level filtering:

  1. XDP/eBPF: Modern kernels support hashmap-based filtering
  2. nftables

    : Newer than iptables with better scaling potential

  3. Custom kernel modules: For extreme performance requirements

However, these require more specialized knowledge to implement correctly.

When dealing with large-scale IP blacklisting (approaching 1 million entries), traditional methods like iptables or null routing hit fundamental performance limitations. Through testing, I discovered that both approaches perform linear searches through their rulesets, causing severe throughput degradation - dropping to just ~24 packets/second on modern hardware.

Here's what happens with conventional approaches:

# iptables example (DO NOT USE for large lists)
iptables -A INPUT -s 1.2.3.4 -j DROP
iptables -A INPUT -s 5.6.7.8 -j DROP
# ...repeat 1 million times...

Similarly, null routing suffers from the same linear search problem:

ip route add blackhole 1.2.3.4
ip route add blackhole 5.6.7.8
# ...repeat 1 million times...

While ipset (suggested by Jimmy Hedman) offers better performance through hash-based lookups, it caps at 65,536 entries per set - far below our requirement.

The successful implementation combined:

  1. Berkeley DB for O(log N) lookups
  2. Apache's mod_rewrite for efficient request handling

Implementation example:

# Create Berkeley DB from IP list
db_load -T -f ip_blacklist.txt ip_blacklist.db

# Apache configuration
RewriteEngine On
RewriteMap ipmap dbm:/path/to/ip_blacklist.db
RewriteCond ${ipmap:%{REMOTE_ADDR}} !^$
RewriteRule ^ - [F]

Key advantages of this approach:

  • Database lookups scale logarithmically
  • Minimal memory overhead compared to in-memory solutions
  • Easy to update blacklist without service interruption

For Nginx users, a similar approach using Lua + Redis:

location / {
    access_by_lua_block {
        local redis = require "resty.redis"
        local red = redis:new()
        red:connect("127.0.0.1", 6379)
        local res = red:get(ngx.var.remote_addr)
        if res ~= ngx.null then
            ngx.exit(403)
        end
    }
}