MySQL Crash Recovery: Diagnosing and Fixing InnoDB Assertion Failure (frag_n_used > 0) After Abnormal Shutdown


1 views

When MySQL crashes with an InnoDB assertion failure like frag_n_used > 0 in fsp0fsp.c, it typically indicates corruption in the tablespace during crash recovery. The error occurs when InnoDB attempts to reconstruct partially written data pages after an abnormal shutdown.

First, attempt to start MySQL in recovery mode:

# Add to your my.cnf under [mysqld] section
innodb_force_recovery = 1  # Try values 1 through 6 incrementally

If level 1 doesn't work, increase the value up to 6 (most aggressive). Each level adds more recovery capabilities:

  • 1 (SRV_FORCE_IGNORE_CORRUPT) - Skip corrupt pages
  • 4 (SRV_FORCE_NO_IBUF_MERGE) - Disable insert buffer merge
  • 6 (SRV_FORCE_NO_LOG_REDO) - Skip redo log processing

Once you get the server running in recovery mode, immediately dump your critical data:

mysqldump -u root -p --single-transaction --skip-lock-tables \
--databases critical_db > emergency_backup.sql

The stack trace reveals the crash occurred during rollback of transaction ID 0 562485192. Check your InnoDB status:

# Run when MySQL is operational
SHOW ENGINE INNODB STATUS\G

Look for these key sections in the output:

---TRANSACTION 0 562485192, ACTIVE 0 sec, process no 12345, OS thread id 1234567890
15 row operations to undo
MySQL thread id 123, query id 456 localhost root

For severe corruption, consider using Percona's recovery tools if you're using XtraDB:

# Install Percona recovery toolkit
sudo apt-get install percona-xtrabackup

# Create a physical backup of the data files
innobackupex --defaults-file=/etc/mysql/my.cnf --user=root --password=xxx /backup/

Then examine the tablespace with InnoDB's offline recovery mode:

# Add to my.cnf and restart
[mysqld]
innodb_file_per_table=1
innodb_read_only=1
innodb_force_recovery=6

To prevent future occurrences:

  1. Implement proper monitoring for crash-safe replication:
    # Configure your replicas with
    sync_binlog=1
    innodb_flush_log_at_trx_commit=1
    
  2. Set up regular verification of InnoDB tables:
    # Add to cron job
    mysqlcheck -u root -p --all-databases --check --silent
    

For unrecoverable corruption, you may need to rebuild from backups. Here's a safe restore procedure:

# Stop MySQL
sudo service mysql stop

# Move corrupted data away
mv /var/lib/mysql/ibdata1 /var/lib/mysql/ibdata1.corrupt
mv /var/lib/mysql/ib_logfile* /var/lib/mysql/corrupt/

# Restore from backup
mysql -u root -p < full_backup.sql

# Restart MySQL
sudo service mysql start

When your MySQL server fails with an InnoDB assertion error like Failing assertion: frag_n_used > 0 followed by a segmentation fault, it typically indicates severe tablespace corruption. The error occurs during crash recovery when MySQL attempts to roll back uncommitted transactions.

InnoDB: Assertion failure in thread 1873206128 in file ../../../storage/innobase/fsp/fsp0fsp.c line 1593
InnoDB: Failing assertion: frag_n_used > 0

First, attempt to start MySQL with forced recovery mode. Add this to your my.cnf:

[mysqld]
innodb_force_recovery = 1

If level 1 doesn't work, progressively try higher levels up to 6. Each level adds more aggressive recovery:

# Try levels sequentially, stopping when MySQL starts
innodb_force_recovery = 1  # SRV_FORCE_IGNORE_CORRUPT
innodb_force_recovery = 2  # SRV_FORCE_NO_BACKGROUND
innodb_force_recovery = 3  # SRV_FORCE_NO_TRX_UNDO
innodb_force_recovery = 4  # SRV_FORCE_NO_IBUF_MERGE
innodb_force_recovery = 5  # SRV_FORCE_NO_UNDO_LOG_SCAN
innodb_force_recovery = 6  # SRV_FORCE_NO_LOG_REDO

Once started in recovery mode, immediately dump your critical data:

mysqldump --single-transaction --routines --triggers \
    --databases db1 db2 > emergency_backup.sql

For heavily corrupted databases, use dedicated tools:

# Using mysqlfrm to recover table structures
mysqlfrm --server=user:pass@localhost:3306 /var/lib/mysql/dbname/*.frm

# Using undrop-for-innodb for page-level recovery
./stream_parser -f /var/lib/mysql/ibdata1
./c_parser -5f pages-ibdata1/FIL_PAGE_INDEX/0000000000000001.page

After successful recovery, verify your data integrity:

mysqlcheck --all-databases --check-upgrade --auto-repair
mysql_upgrade --force

To prevent future occurrences:

# Add these to my.cnf
innodb_doublewrite = 1
innodb_flush_log_at_trx_commit = 1
sync_binlog = 1

# Regular maintenance script
#!/bin/bash
mysql -e "SET GLOBAL innodb_max_dirty_pages_pct = 0;"
mysql -e "FLUSH TABLES WITH READ LOCK; SET GLOBAL read_only = ON;"
rsync -avz /var/lib/mysql /backup/
mysql -e "SET GLOBAL read_only = OFF; UNLOCK TABLES;"