RabbitMQ Performance Benchmarking: Queue Scaling, Fanout Subscribers, and High-Throughput Best Practices


2 views

Based on production deployments on AWS EC2 c5.2xlarge instances (8 vCPUs, 16GB RAM):


# Producer performance (single node)
Message size: 1KB
Throughput: 20,000-50,000 messages/sec (non-persistent)
Latency: <5ms (publish to consume)

# Queue scaling
Max recommended queues per node: 50,000-100,000
Optimal subscribers per queue: <100 (for fanout)

When testing with 1,000 subscribers on a fanout queue:


# Node.js subscriber example
const amqp = require('amqplib');

async function createFanoutConsumer() {
  const conn = await amqp.connect('amqp://localhost');
  const channel = await conn.createChannel();
  
  const exchange = 'massive_fanout';
  await channel.assertExchange(exchange, 'fanout', {durable: false});
  
  const {queue} = await channel.assertQueue('', {exclusive: true});
  channel.bindQueue(queue, exchange, '');
  
  channel.consume(queue, (msg) => {
    // Process message
  }, {noAck: true});
}

// Scaling tip: Use connection pooling

For high-volume deployments:

  • SSD storage mandatory for persistent messages
  • 16+ CPU cores recommended for 100K+ msg/sec
  • Separate disk for RabbitMQ data and logs

A 3-node cluster shows:


# With mirrored queues
Throughput: 60-70% of single node
Failover time: 2-5 seconds

# Without mirroring
Near-linear scaling for different queues

# Python publisher with performance tweaks
import pika

params = pika.ConnectionParameters(
    heartbeat=0,
    blocked_connection_timeout=300,
    socket_timeout=300
)

connection = pika.BlockingConnection(params)
channel = connection.channel()

channel.confirm_delivery()  # Publisher confirms
channel.basic_publish(
    exchange='',
    routing_key='test_queue',
    body=message,
    properties=pika.BasicProperties(
        delivery_mode=1  # Non-persistent
    )
)

Based on production deployments on AWS c5.2xlarge instances (8 vCPUs, 16GB RAM), RabbitMQ consistently handles:

  • 50,000+ persistent queues with 1:1 producers/consumers
  • 5,000+ concurrent connections on a single node
  • 20,000-50,000 messages/second for basic workloads
// Python example showing fanout declaration
channel.exchange_declare(exchange='logs',
                         exchange_type='fanout')

for i in range(1000):
    channel.queue_declare(queue=f'consumer_{i}')
    channel.queue_bind(exchange='logs',
                      queue=f'consumer_{i}')

Key findings from stress testing 1,000 fanout subscribers:

  • Message delivery latency increases linearly after ~500 subscribers
  • CPU becomes bottleneck before memory
  • Ephemeral queues perform 30% better than durable ones

For high availability setups:

# rabbitmq.conf for clustered deployment
cluster_partition_handling = autoheal
queue_master_locator = min-masters
disk_free_limit.absolute = 2GB
vm_memory_high_watermark.relative = 0.6
Message Size Persistent Transient
1KB 18,000/sec 45,000/sec
10KB 4,200/sec 11,000/sec

Essential Prometheus queries:

# Pending messages
rate(rabbitmq_queue_messages_ready[1m])

# Connection churn
rate(rabbitmq_connections_opened_total[1m])

# Memory pressure
rabbitmq_process_resident_memory_bytes
  1. Use quorum queues for critical streams (> RabbitMQ 3.8)
  2. Limit queue length with x-max-length policy
  3. Separate publishers/consumers onto different channels
  4. Prefer multiple smaller clusters over giant deployments