When MySQL crashes with an InnoDB assertion failure like frag_n_used > 0
in fsp0fsp.c
, it typically indicates corruption in the tablespace during crash recovery. The error occurs when InnoDB attempts to reconstruct partially written data pages after an abnormal shutdown.
First, attempt to start MySQL in recovery mode:
# Add to your my.cnf under [mysqld] section innodb_force_recovery = 1 # Try values 1 through 6 incrementally
If level 1 doesn't work, increase the value up to 6 (most aggressive). Each level adds more recovery capabilities:
- 1 (SRV_FORCE_IGNORE_CORRUPT) - Skip corrupt pages
- 4 (SRV_FORCE_NO_IBUF_MERGE) - Disable insert buffer merge
- 6 (SRV_FORCE_NO_LOG_REDO) - Skip redo log processing
Once you get the server running in recovery mode, immediately dump your critical data:
mysqldump -u root -p --single-transaction --skip-lock-tables \ --databases critical_db > emergency_backup.sql
The stack trace reveals the crash occurred during rollback of transaction ID 0 562485192. Check your InnoDB status:
# Run when MySQL is operational SHOW ENGINE INNODB STATUS\G
Look for these key sections in the output:
---TRANSACTION 0 562485192, ACTIVE 0 sec, process no 12345, OS thread id 1234567890 15 row operations to undo MySQL thread id 123, query id 456 localhost root
For severe corruption, consider using Percona's recovery tools if you're using XtraDB:
# Install Percona recovery toolkit sudo apt-get install percona-xtrabackup # Create a physical backup of the data files innobackupex --defaults-file=/etc/mysql/my.cnf --user=root --password=xxx /backup/
Then examine the tablespace with InnoDB's offline recovery mode:
# Add to my.cnf and restart [mysqld] innodb_file_per_table=1 innodb_read_only=1 innodb_force_recovery=6
To prevent future occurrences:
- Implement proper monitoring for crash-safe replication:
# Configure your replicas with sync_binlog=1 innodb_flush_log_at_trx_commit=1
- Set up regular verification of InnoDB tables:
# Add to cron job mysqlcheck -u root -p --all-databases --check --silent
For unrecoverable corruption, you may need to rebuild from backups. Here's a safe restore procedure:
# Stop MySQL sudo service mysql stop # Move corrupted data away mv /var/lib/mysql/ibdata1 /var/lib/mysql/ibdata1.corrupt mv /var/lib/mysql/ib_logfile* /var/lib/mysql/corrupt/ # Restore from backup mysql -u root -p < full_backup.sql # Restart MySQL sudo service mysql start
When your MySQL server fails with an InnoDB assertion error like Failing assertion: frag_n_used > 0
followed by a segmentation fault, it typically indicates severe tablespace corruption. The error occurs during crash recovery when MySQL attempts to roll back uncommitted transactions.
InnoDB: Assertion failure in thread 1873206128 in file ../../../storage/innobase/fsp/fsp0fsp.c line 1593 InnoDB: Failing assertion: frag_n_used > 0
First, attempt to start MySQL with forced recovery mode. Add this to your my.cnf:
[mysqld] innodb_force_recovery = 1
If level 1 doesn't work, progressively try higher levels up to 6. Each level adds more aggressive recovery:
# Try levels sequentially, stopping when MySQL starts innodb_force_recovery = 1 # SRV_FORCE_IGNORE_CORRUPT innodb_force_recovery = 2 # SRV_FORCE_NO_BACKGROUND innodb_force_recovery = 3 # SRV_FORCE_NO_TRX_UNDO innodb_force_recovery = 4 # SRV_FORCE_NO_IBUF_MERGE innodb_force_recovery = 5 # SRV_FORCE_NO_UNDO_LOG_SCAN innodb_force_recovery = 6 # SRV_FORCE_NO_LOG_REDO
Once started in recovery mode, immediately dump your critical data:
mysqldump --single-transaction --routines --triggers \ --databases db1 db2 > emergency_backup.sql
For heavily corrupted databases, use dedicated tools:
# Using mysqlfrm to recover table structures mysqlfrm --server=user:pass@localhost:3306 /var/lib/mysql/dbname/*.frm # Using undrop-for-innodb for page-level recovery ./stream_parser -f /var/lib/mysql/ibdata1 ./c_parser -5f pages-ibdata1/FIL_PAGE_INDEX/0000000000000001.page
After successful recovery, verify your data integrity:
mysqlcheck --all-databases --check-upgrade --auto-repair mysql_upgrade --force
To prevent future occurrences:
# Add these to my.cnf innodb_doublewrite = 1 innodb_flush_log_at_trx_commit = 1 sync_binlog = 1 # Regular maintenance script #!/bin/bash mysql -e "SET GLOBAL innodb_max_dirty_pages_pct = 0;" mysql -e "FLUSH TABLES WITH READ LOCK; SET GLOBAL read_only = ON;" rsync -avz /var/lib/mysql /backup/ mysql -e "SET GLOBAL read_only = OFF; UNLOCK TABLES;"