Best Practices for Safely Restarting MongoDB in Production with Minimal Downtime


2 views

When dealing with MongoDB in production environments, restart operations require careful consideration of:

- Write concern durability
- Journaling configurations 
- Replica set status (if applicable)
- Active connections

The safest approach involves using MongoDB's built-in shutdown command:

use admin
db.shutdownServer()

For systems with authentication enabled:

mongo --username admin --password --authenticationDatabase admin \
--eval "db.shutdownServer()"

For replica set members, step down primaries first:

rs.stepDown(300)  // 300-second wait period

Then restart secondaries one at a time:

// Connect to each secondary
use admin
db.shutdownServer()

If standard shutdown fails (should be last resort):

sudo kill -2 pgrep mongod  // SIGINT for clean shutdown
// Wait 2 minutes before restart
sudo systemctl start mongod

Check logs and connection status:

tail -f /var/log/mongodb/mongod.log
mongo --eval "db.serverStatus().connections"

Example script for controlled restart:

#!/bin/bash

# Check if primary
IS_PRIMARY=$(mongo --quiet --eval 'rs.isMaster().ismaster')

if [ "$IS_PRIMARY" = "true" ]; then
    echo "Stepping down primary"
    mongo --eval "rs.stepDown(300)"
    sleep 60
fi

echo "Shutting down MongoDB"
mongo admin --eval "db.shutdownServer()"

sleep 120  # Wait for complete shutdown

echo "Restarting MongoDB"
sudo systemctl start mongod

Key metrics to watch:

- Opcounter metrics
- Replication lag
- Queue lengths
- Page faults

When dealing with production MongoDB instances, the restart procedure needs special consideration for:

  • Write operation safety (acknowledged writes)
  • Replica set consistency
  • Journal file synchronization
  • Connection draining

For standalone MongoDB instances:

# Connect to the mongo shell
mongo --eval "db.adminCommand({shutdown: 1, force: false})"

# Wait for clean shutdown (check logs)
tail -f /var/log/mongodb/mongod.log

# Verify process termination
ps aux | grep mongod

# Restart the service
sudo systemctl start mongod

For replica set members, follow this sequence:

// Step down secondary first (if needed)
rs.stepDown(300)

// Then restart secondaries one by one
mongo --eval "db.adminCommand({shutdown: 1})"

// Finally restart primary (after step down)

Ensure these settings are properly configured in mongod.conf:

storage:
  journal:
    enabled: true
    commitIntervalMs: 100
    
operationProfiling:
  slowOpThresholdMs: 100

net:
  maxIncomingConnections: 1000

For zero-downtime restarts in Kubernetes environments:

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: mongodb
spec:
  updateStrategy:
    type: RollingUpdate
    rollingUpdate:
      partition: 0

Essential checks to perform:

db.serverStatus().uptime
db.serverStatus().connections
db.serverStatus().storageEngine.supportsCommittedReads