记一次mysql故障恢复

事情要从俩月前的一个坑说起，一台新的测试服务器，新项目一元夺宝用的。

配置aws上的一台云主机，系统盘8G，一块300G的云硬盘。

拿到机器后，另一运维小哥安装php，nginx，mysql等软件。 安装完后，然忘了挂载云硬盘。 随着根分区日志越来越多，mysql数据量增大。。。磁盘已满，mysql自行挂了。。。哭。。 

开发找上门来，mysql插不进去了，我连上服务器，ps -ef。。 没有进程。

尝试重启。。失败。

查看错误日志。。如下：

1 161206 08:07:41 mysqld_safe Starting mysqld daemon with databases from /data/mysql
2 2016-12-06 08:07:41 4473 [Note] InnoDB: The InnoDB memory heap is disabled
3 2016-12-06 08:07:41 4473 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
4 2016-12-06 08:07:41 4473 [Note] InnoDB: Compressed tables use zlib 1.2.3
5 2016-12-06 08:07:41 4473 [Note] InnoDB: Using Linux native AIO
6 2016-12-06 08:07:41 4473 [Note] InnoDB: Using CPU crc32 instructions
7 2016-12-06 08:07:41 4473 [Note] InnoDB: Initializing buffer pool, size = 128.0M
8 2016-12-06 08:07:41 4473 [Note] InnoDB: Completed initialization of buffer pool
9 2016-12-06 08:07:41 4473 [Note] InnoDB: Highest supported file format is Barracuda.
10 2016-12-06 08:07:41 4473 [Note] InnoDB: Log scan progressed past the checkpoint lsn 416493596
11 2016-12-06 08:07:41 4473 [Note] InnoDB: Database was not shutdown normally!
12 2016-12-06 08:07:41 4473 [Note] InnoDB: Starting crash recovery.
13 2016-12-06 08:07:41 4473 [Note] InnoDB: Reading tablespace information from the .ibd files...
14 2016-12-06 08:07:41 4473 [Note] InnoDB: Restoring possible half-written data pages
15 2016-12-06 08:07:41 4473 [Note] InnoDB: from the doublewrite buffer...
16 InnoDB: Doing recovery: scanned up to log sequence number 416570218
17 2016-12-06 08:07:41 4473 [Note] InnoDB: Starting an apply batch of log records to the database...
18 InnoDB: Progress in percent: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99
19 InnoDB: Apply batch completed
20 InnoDB: Last MySQL binlog file position 0 177880661, file name mysql-bin.000020
21 2016-12-06 08:07:42 4473 [Note] InnoDB: 128 rollback segment(s) are active.
22 2016-12-06 08:07:42 4473 [Note] InnoDB: Waiting for purge to start
23 2016-12-06 08:07:42 4473 [Note] InnoDB: 5.6.14 started; log sequence number 416570218
24 /usr/local/app/mysql-5.6.14/bin/mysqld: Error writing file '/data/mysql/mysql-slow.log' (Errcode: 28 - No space left on device)
25 2016-12-06 08:07:42 4473 [ERROR] Could not use /data/mysql/mysql-slow.log for logging (error 28). Turning logging off for the whole duration of the MySQL server process. To turn it on again: fix the cause, shutdown the MySQL server and restart it.
26 08:07:42 UTC - mysqld got signal 11 ;
27 This could be because you hit a bug. It is also possible that this binary
28 or one of the libraries it was linked against is corrupt, improperly built,
29 or misconfigured. This error can also be caused by malfunctioning hardware.
30 We will try our best to scrape up some info that will hopefully help
31 diagnose the problem, but since we have already crashed,
32 something is definitely wrong and this may fail.
33
34 key_buffer_size=33554432
35 read_buffer_size=33554432
36 max_used_connections=0
37 max_threads=1000
38 thread_count=0
39 connection_count=0
40 It is possible that mysqld could use up to
41 key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 65582393 K  bytes of memory
42 Hope that's ok; if not, decrease some variables in the equation.
43
44 Thread pointer: 0x0
45 Attempting backtrace. You can use the following information to find out
46 where mysqld died. If you see no messages after this, something went
47 terribly wrong...
48 stack_bottom = 0 thread_stack 0x40000
49 /usr/local/app/mysql-5.6.14/bin/mysqld(my_print_stacktrace+0x35)[0x8dd875]
50 /usr/local/app/mysql-5.6.14/bin/mysqld(handle_fatal_signal+0x40b)[0x64ddbb]
51 /lib64/libpthread.so.0(+0xf7e0)[0x7f926e2b77e0]
52 The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
53 information that should help you find out what is causing the crash.
54 161206 08:07:42 mysqld_safe mysqld from pid file /var/run/mysqld/mysqld.pid ended

第25行亮了。No space left on device。找到问题了，df一看，根分区使用率100%。。。剩下/app目录300G使用率1%在那孤零零挂着。

解决办法：

移动原有文件到数据分区

# mv /data/mysql  /app/

将

datadir = /data/mysql 改为datadir = /app/mysql

重启正常。