Mysql PXC 问题从节点不能启动问题

昨天公司的测试服务器由于别的空间满了导致mysql不能访问，后来释放了部分空间后，虽然主节点可以正常重启。但是从节点就起不来了。从节点日志中报如下错误：

/var/lib/mysql/mysql-error.log:

xbstream: Can't create/write to file '././backup-my.cnf' (Errcode: 13 - Permission denied)
xbstream: failed to create file.
2015/06/02 15:00:01 socat[17491] E write(1, 0x161d400, 1512): Broken pipe
WSREP_SST: [ERROR] Error while getting data from donor node:  exit codes: 1 1 (20150602 15:00:01.125)
WSREP_SST: [ERROR] Cleanup after exit with status:32 (20150602 15:00:01.128)
2015-06-02 15:00:01 17227 [ERROR] WSREP: Process completed with error: wsrep_sst_xtrabackup-v2 --role 'joiner' --address '192.168.100.135' --auth 'root:root123' --datadir '/var/lib/mysql/' --defaults-file '/etc/my.cnf' --parent '17227'  '' : 32 (Broken pipe)
2015-06-02 15:00:01 17227 [ERROR] WSREP: Failed to read uuid:seqno from joiner script.
2015-06-02 15:00:01 17227 [ERROR] WSREP: SST failed: 32 (Broken pipe)
2015-06-02 15:00:01 17227 [ERROR] Aborting

其中可以看到Can’t create/write to file ‘././backup-my.cnf’，这时可以同时查看主节点日志：

innobackup.backup.log

xtrabackup: Error writing file 'UNOPENED' (Errcode: 32 - Broken pipe)
[02] Compressing and streaming ./mysql/slave_worker_info
4000
.ibd
[02]        ...done

类似一堆这种错误，猜测是xtrabackup备份时有文件不能写，故打算重新做次备份。故首先删除datadir(/var/lib/mysql)并重建一个（注意将其权限改为mysql:mysql），然后为确保没有mysql进程以及能重新其起来，一是查看ps -ef|grep mysql，二是删除/var/lock/subsys/mysql空文件，这时再启动，可以发现主节点不报错了，但是从节点依然报错：

WSREP_SST: [ERROR] Cleanup after exit with status:1 (20150603 16:13:46.587)
2015-06-03 16:13:46 23751 [ERROR] WSREP: Process completed with error: wsrep_sst_xtrabackup-v2 --role 'joiner' --address '192.168.100.135' --auth 'root:root123' --datadir '/var/lib/mysql/' --defaults-file '/etc/my.cnf' --parent '23751'  '' : 1 (Operation not permitted)
2015-06-03 16:13:46 23751 [ERROR] WSREP: Failed to read uuid:seqno from joiner script.
2015-06-03 16:13:46 23751 [ERROR] WSREP: SST failed: 1 (Operation not permitted)
2015-06-03 16:13:46 23751 [ERROR] Aborting

但是可以发现和刚才报错不一样了，之后查了很多资料，虽然有很多类似的现象，但是仍有细微之处不太一样，最后谷歌中有个哥们，他把my.cnf中的wsrep_sst_method值从xtrabackup-v2改成了rsync，然后再新同步，竟然能成功，然后又改回xtrabackup-v2再重启，竟然也能成功，这个还是不能理解，但是确实解决了这个问题。先mark一下。