Why is the ibdata1 file continuously growing in MySQL?
2014-02-28 00:42
459 查看
We receive this question about the ibdata1 file in MySQL very often in Percona Support.
The panic starts when the monitoring server sends an alert about the storage of the MySQL server – saying that the disk is about to get filled.
After some research you realize that most of the disk space is used by the InnoDB’s shared tablespace ibdata1. You have innodb_file_per_table enabled, so the question is:
data dictionary aka metadata of InnoDB tables
change buffer
doublewrite buffer
undo logs
Some of them can be configured on Percona Server to avoid becoming too large. For example you can set a maximum size for change buffer with innodb_ibuf_max_size (Half the size of the InnoDB buffer pool) or store the doublewrite buffer on a separate file with innodb_doublewrite_file.
In MySQL 5.6 you can also create external UNDO tablespaces so they will be in their own files instead of stored inside ibdata1. Check following documentation link.
That will show us very valuable information. We start checking the TRANSACTIONS section and we find this:
This is the most common reason, a pretty old transaction created 14 days ago. The status is ACTIVE, that means InnoDB has created a snapshot of the data so it needs to maintain old pages in undo to be able to provide a consistent view of the database since that transaction was started. If your database is heavily write loaded that means lots of undo pages are being stored.
If you don’t find any long-running transaction you can also monitor another variable from the INNODB STATUS, the “History list length.” It shows the number of pending purge operations. In this case the problem is usually caused because the purge thread (or master thread in older versions) is not capable to process undo records with the same speed as they come in.
It is pretty easy to use:
It has 19272 UNDO_LOG pages from a total of 20608. That’s the 93% of the tablespace.
The second way to check the content of a tablespace are the InnoDB Ruby Tools made by Jeremy Cole. It is a more advanced tool to examine the internals of InnoDB. For example we can use the space-summary parameter to get a list with every page and its data type. We can use standard Unix tools to get the number of UNDO_LOG pages:
Altough in this particular case innochecksum is faster and easier to use I recommend you to play with Jeremy’s tools to learn more about the data distribution inside InnoDB and its internals.
OK, now we know where the problem is. The next question:
If the problem is caused by the purge thread then the solution is usually to upgrade to a newer version where you can use a dedicated purge thread instead of the master thread. More information on the following documentation link.
When you delete some rows, the pages are marked as deleted to reuse later but the space is never recovered. The only way is to start the database with fresh ibdata1. To do that you would need to take a full logical backup with mysqldump. Then stop MySQL and remove all the databases, ib_logfile* and ibdata* files. When you start MySQL again it will create a new fresh shared tablespace. Then, recover the logical dump.
Monitoring the database to avoid these kind of problems is also very recommended. Our MySQL Monitoring Plugins includes a Nagios script that can alert you if it finds a too old running transaction.
quoted from:
http://www.mysqlperformanceblog.com/2013/08/20/why-is-the-ibdata1-file-continuously-growing-in-mysql/
The panic starts when the monitoring server sends an alert about the storage of the MySQL server – saying that the disk is about to get filled.
After some research you realize that most of the disk space is used by the InnoDB’s shared tablespace ibdata1. You have innodb_file_per_table enabled, so the question is:
What is stored in ibdata1?
When you have innodb_file_per_table enabled, the tables are stored in their own tablespace but the shared tablespace is still used to store other InnoDB’s internal data:data dictionary aka metadata of InnoDB tables
change buffer
doublewrite buffer
undo logs
Some of them can be configured on Percona Server to avoid becoming too large. For example you can set a maximum size for change buffer with innodb_ibuf_max_size (Half the size of the InnoDB buffer pool) or store the doublewrite buffer on a separate file with innodb_doublewrite_file.
In MySQL 5.6 you can also create external UNDO tablespaces so they will be in their own files instead of stored inside ibdata1. Check following documentation link.
What is causing the ibdata1 to grow that fast?
Usually the first command that we need to run when there is a MySQL problem is:SHOW ENGINE INNODB STATUS\G
That will show us very valuable information. We start checking the TRANSACTIONS section and we find this:
---TRANSACTION 36E, ACTIVE 1256288 sec MySQL thread id 42, OS thread handle 0x7f8baaccc700, query id 7900290 localhost root show engine innodb status Trx read view will not see trx with id >= 36F, sees < 36F
This is the most common reason, a pretty old transaction created 14 days ago. The status is ACTIVE, that means InnoDB has created a snapshot of the data so it needs to maintain old pages in undo to be able to provide a consistent view of the database since that transaction was started. If your database is heavily write loaded that means lots of undo pages are being stored.
If you don’t find any long-running transaction you can also monitor another variable from the INNODB STATUS, the “History list length.” It shows the number of pending purge operations. In this case the problem is usually caused because the purge thread (or master thread in older versions) is not capable to process undo records with the same speed as they come in.
How can I check what is being stored in the ibdata1?
Unfortunately MySQL doesn’t provide information of what is being stored on that ibdata1 shared tablespace but there are two tools that will be very helpful. First a modified version of innochecksum made by Mark Callaghan and published in this bug report.It is pretty easy to use:
# ./innochecksum /var/lib/mysql/ibdata1 0 bad checksum 13 FIL_PAGE_INDEX 19272 FIL_PAGE_UNDO_LOG 230 FIL_PAGE_INODE 1 FIL_PAGE_IBUF_FREE_LIST 892 FIL_PAGE_TYPE_ALLOCATED 2 FIL_PAGE_IBUF_BITMAP 195 FIL_PAGE_TYPE_SYS 1 FIL_PAGE_TYPE_TRX_SYS 1 FIL_PAGE_TYPE_FSP_HDR 1 FIL_PAGE_TYPE_XDES 0 FIL_PAGE_TYPE_BLOB 0 FIL_PAGE_TYPE_ZBLOB 0 other 3 max index_id
It has 19272 UNDO_LOG pages from a total of 20608. That’s the 93% of the tablespace.
The second way to check the content of a tablespace are the InnoDB Ruby Tools made by Jeremy Cole. It is a more advanced tool to examine the internals of InnoDB. For example we can use the space-summary parameter to get a list with every page and its data type. We can use standard Unix tools to get the number of UNDO_LOG pages:
# innodb_space -f /var/lib/mysql/ibdata1 space-summary | grep UNDO_LOG | wc -l 19272
Altough in this particular case innochecksum is faster and easier to use I recommend you to play with Jeremy’s tools to learn more about the data distribution inside InnoDB and its internals.
OK, now we know where the problem is. The next question:
How can I solve the problem?
The answer to this question is easy. If you can still commit that query, do it. If not you’ll have to kill the thread to start the rollback process. That will just stop ibdata1 from growing but it is clear that your software has a bug or someone made a mistake. Now that you know how to identify where is the problem you need to find who or what is causing it using your own debugging tools or the general query log.If the problem is caused by the purge thread then the solution is usually to upgrade to a newer version where you can use a dedicated purge thread instead of the master thread. More information on the following documentation link.
Is there any way to recover the used space?
No, it is not possible at least in an easy and fast way. InnoDB tablespaces never shrink… see the following 10-year old bug reportrecently updated by James Day (thanks):When you delete some rows, the pages are marked as deleted to reuse later but the space is never recovered. The only way is to start the database with fresh ibdata1. To do that you would need to take a full logical backup with mysqldump. Then stop MySQL and remove all the databases, ib_logfile* and ibdata* files. When you start MySQL again it will create a new fresh shared tablespace. Then, recover the logical dump.
Summary
When the ibdata1 file is growing too fast within MySQL it is usually caused by a long running transaction that we have forgotten about. Try to solve the problem as fast as possible (commiting or killing a transaction) because you won’t be able to recover the wasted disk space without the painfully slow mysqldump process.Monitoring the database to avoid these kind of problems is also very recommended. Our MySQL Monitoring Plugins includes a Nagios script that can alert you if it finds a too old running transaction.
quoted from:
http://www.mysqlperformanceblog.com/2013/08/20/why-is-the-ibdata1-file-continuously-growing-in-mysql/
相关文章推荐
- Mysql 出现错误The server is not configured as slave; fix in config file or with CHANGE MASTER TO
- Why is the "DEBUG: Error 2836" message generated in the DB2 V8, V9.1, or V9.5 installation log file on Windows system?
- InnoDB: The Auto-extending innodb_system data file './ibdata1' is of a different size 640 pages (rounded down to MB) than specified in the .cnf file: initial 768 pages, max 0 (relevant if non-zero) pa
- mysql5.7.12/13在安装新实例时报错:InnoDB: auto-extending data file ./ibdata1 is of a different size 640 pages (rounded down to MB) than specified in the .cnf file: initial 768 pages, max 0 (relevant if non-zero
- InnoDB: auto-extending data file ./ibdata1 is of a different size 640 pages (rounded down to MB) than specified in the .cnf file: initial 768 pages, max 0 (relevant if non-zero) pages!
- Is there any best way to reduce the size of ibdata in mysql.?
- 解决linux下sudo更改文件权限报错xxxis not in the sudoers file. This incident will be reported.
- [转]Linux之"xxx is not in the sudoers file"解决方法
- 报错:An error occurred at line: 22 in the generated java file The method getJspApplicationContext(ServletContext) is undefined for the type JspFactory
- Centos下无法格式化硬盘/dev/sd* is apparently in use by the system; will not make a filesystem here!
- 解决mysql_connect(): The mysql extension is deprecated and will beremoved in the future: use mysqli or
- linux *** is not in the sudoers file. This incident will be reported.
- Linux配置之解决CentOS中:xx is not in the sudoers file的问题
- 解决cakePHP中的. session_start()[function.session-start];open_basedir restnction in effect.File(c:/windows/temp/)is not within the a
- xxx is not in the sudoers file. This incident will be reported.
- Ubuntu技巧之 is not in the sudoers file解决方法
- xxx is not in the sudoers file.This incident will be reported.的解决方法
- xxx is not in the sudoers file.This incident will be reported.的解决方法
- xxx is not in the sudoers file解决方法
- 解决Deprecated: mysql_connect(): The mysql extension is deprecated and will be removed in the future: