Based on production deployments on AWS EC2 c5.2xlarge instances (8 vCPUs, 16GB RAM):
# Producer performance (single node)
Message size: 1KB
Throughput: 20,000-50,000 messages/sec (non-persistent)
Latency: <5ms (publish to consume)
# Queue scaling
Max recommended queues per node: 50,000-100,000
Optimal subscribers per queue: <100 (for fanout)
When testing with 1,000 subscribers on a fanout queue:
# Node.js subscriber example
const amqp = require('amqplib');
async function createFanoutConsumer() {
const conn = await amqp.connect('amqp://localhost');
const channel = await conn.createChannel();
const exchange = 'massive_fanout';
await channel.assertExchange(exchange, 'fanout', {durable: false});
const {queue} = await channel.assertQueue('', {exclusive: true});
channel.bindQueue(queue, exchange, '');
channel.consume(queue, (msg) => {
// Process message
}, {noAck: true});
}
// Scaling tip: Use connection pooling
For high-volume deployments:
- SSD storage mandatory for persistent messages
- 16+ CPU cores recommended for 100K+ msg/sec
- Separate disk for RabbitMQ data and logs
A 3-node cluster shows:
# With mirrored queues
Throughput: 60-70% of single node
Failover time: 2-5 seconds
# Without mirroring
Near-linear scaling for different queues
# Python publisher with performance tweaks
import pika
params = pika.ConnectionParameters(
heartbeat=0,
blocked_connection_timeout=300,
socket_timeout=300
)
connection = pika.BlockingConnection(params)
channel = connection.channel()
channel.confirm_delivery() # Publisher confirms
channel.basic_publish(
exchange='',
routing_key='test_queue',
body=message,
properties=pika.BasicProperties(
delivery_mode=1 # Non-persistent
)
)
Based on production deployments on AWS c5.2xlarge instances (8 vCPUs, 16GB RAM), RabbitMQ consistently handles:
- 50,000+ persistent queues with 1:1 producers/consumers
- 5,000+ concurrent connections on a single node
- 20,000-50,000 messages/second for basic workloads
// Python example showing fanout declaration
channel.exchange_declare(exchange='logs',
exchange_type='fanout')
for i in range(1000):
channel.queue_declare(queue=f'consumer_{i}')
channel.queue_bind(exchange='logs',
queue=f'consumer_{i}')
Key findings from stress testing 1,000 fanout subscribers:
- Message delivery latency increases linearly after ~500 subscribers
- CPU becomes bottleneck before memory
- Ephemeral queues perform 30% better than durable ones
For high availability setups:
# rabbitmq.conf for clustered deployment
cluster_partition_handling = autoheal
queue_master_locator = min-masters
disk_free_limit.absolute = 2GB
vm_memory_high_watermark.relative = 0.6
Message Size | Persistent | Transient |
---|---|---|
1KB | 18,000/sec | 45,000/sec |
10KB | 4,200/sec | 11,000/sec |
Essential Prometheus queries:
# Pending messages
rate(rabbitmq_queue_messages_ready[1m])
# Connection churn
rate(rabbitmq_connections_opened_total[1m])
# Memory pressure
rabbitmq_process_resident_memory_bytes
- Use quorum queues for critical streams (> RabbitMQ 3.8)
- Limit queue length with x-max-length policy
- Separate publishers/consumers onto different channels
- Prefer multiple smaller clusters over giant deployments