MySQL Docker Container Crashing After Upgrade: Fixing “Unable to lock ./ibdata1” Errors


2 views

When upgrading Docker from 1.4 to 1.5 while maintaining the same host-mounted /var/lib/mysql directory, MySQL containers begin crashing with critical InnoDB errors. The key symptoms appear in logs as:

2015-02-13 09:08:02 510 [ERROR] InnoDB: Unable to lock ./ibdata1, error: 11
2015-02-13 09:08:02 510 [Note] InnoDB: Check for another mysqld process

The error occurs because Docker 1.5 introduced changes in filesystem handling for mounted volumes. The new version maintains stricter file locks, which conflicts with MySQL's expectation of exclusive access to its data files.

Three specific problems combine:

  • File lock contention between host and container
  • Potential permission mismatches in mounted volume
  • Residual lock files from previous container instances

First, ensure no orphaned MySQL processes are running:

# On host machine
sudo lsof | grep /var/lib/mysql
sudo kill -9 [process_ids]

Then clean up any residual files:

sudo rm -f /var/lib/mysql/ibdata1
sudo rm -f /var/lib/mysql/ib_logfile*

Option 1: Rebuild with proper volume permissions

docker run --name mysql-container \
  -v /host/mysql:/var/lib/mysql \
  -e MYSQL_ROOT_PASSWORD=yourpassword \
  --restart unless-stopped \
  mysql:5.6 --innodb_use_native_aio=0

Option 2: Use named volumes instead of host mounts

docker volume create mysql_data
docker run --name mysql-container \
  -v mysql_data:/var/lib/mysql \
  -e MYSQL_ROOT_PASSWORD=yourpassword \
  mysql:5.6

For production environments, add these MySQL configuration parameters:

[mysqld]
innodb_flush_method=O_DIRECT
innodb_use_native_aio=0
innodb_buffer_pool_size=1G
innodb_log_file_size=256M

If issues persist, check these diagnostic commands:

# Verify file permissions
ls -la /var/lib/mysql

# Check container logs
docker logs --tail 50 mysql-container

# Inspect volume mounts
docker inspect mysql-container | grep -A 10 Mounts

After upgrading Docker from 1.4 to 1.5 and rebuilding my MySQL image while keeping the host-mounted /var/lib/mysql volume, I encountered a puzzling issue: the container would consistently exit after exactly 5 minutes. The error logs revealed a critical pattern:

2015-02-13 09:08:02 510 [ERROR] InnoDB: Unable to lock ./ibdata1, error: 11
2015-02-13 09:08:02 510 [Note] Check that you do not already have another mysqld process...

This error typically occurs when:

  1. Another MySQL instance is accessing the same data files
  2. File permission issues exist on the mounted volume
  3. Incomplete shutdown left lock files behind
  4. Docker version changes affected volume mounting behavior

First, check for running MySQL processes:

ps aux | grep mysqld
docker ps -a | grep mysql

Inspect file permissions on your mounted volume:

ls -la /path/to/host/mysql/data
stat /path/to/host/mysql/data/ibdata1

Solution 1: Clean Shutdown and Restart

# Force remove any lingering containers
docker rm -f mysql_container

# Verify no lock files remain
rm -f /path/to/host/mysql/data/ib_logfile*
rm -f /path/to/host/mysql/data/ibdata1.lock

# Start fresh container with proper privileges
docker run -d \
  --name mysql_container \
  -v /path/to/host/mysql/data:/var/lib/mysql \
  -e MYSQL_ROOT_PASSWORD=yourpassword \
  --privileged=true \
  mysql:5.6

Solution 2: File System Permissions Fix

# Set proper ownership (usually UID 999 for MySQL in Docker)
chown -R 999:999 /path/to/host/mysql/data

# Ensure correct permissions
chmod -R 755 /path/to/host/mysql/data

Solution 3: Docker Version-Specific Workaround

# For Docker 1.5+ compatibility issues
docker run -d \
  --name mysql_container \
  -v /path/to/host/mysql/data:/var/lib/mysql:rw \
  --security-opt seccomp:unconfined \
  --cap-add SYS_NICE \
  mysql:5.6

To avoid similar issues in future upgrades:

  1. Always backup your data volume before Docker upgrades
  2. Document your exact volume mounting parameters
  3. Consider using named volumes instead of host paths
  4. Test upgrades in a staging environment first

If problems persist, try these diagnostic commands:

# Check Docker storage driver compatibility
docker info | grep "Storage Driver"

# Inspect container logs in real-time
docker logs -f mysql_container

# Verify volume mounting
docker inspect mysql_container | grep Mounts -A 20

Remember that MySQL in Docker requires careful handling of persistent data. The key is ensuring exclusive access to the data files while maintaining proper file system permissions across host-container boundaries.