Greenplum集群状态检查及失败segment修复(上)
2016-08-30 00:00
761 查看
一、GP集群运行状态检查
查gp_segment_configuration，status='d'表示segment失败，status='u'表示segment正常SELECT * FROM gp_segment_configuration WHERE status='d';
gpstate查看各个primary、mirror segment状态信息
集群状态：
9个 primary segment失败
3个 mirror segment失败
此时，GP集群增删改查均正常，猜测是mirror segment起了作用。
[gpadmin@gpm ~]$ gpstate 20160829:17:36:08:035340 gpstate:gpm:gpadmin-[INFO]:-Starting gpstate with args: 20160829:17:36:08:035340 gpstate:gpm:gpadmin-[INFO]:-local Greenplum Version: 'postgres (Greenplum Database) 4.3.8.2 build 1' 20160829:17:36:08:035340 gpstate:gpm:gpadmin-[INFO]:-master Greenplum Version: 'PostgreSQL 8.2.15 (Greenplum Database 4.3.8.2 build 1) on x86_64-unknown-linux-gnu, compiled by GCC gcc (GCC) 4.4.2 compiled on May 9 2016 22:28:40' 20160829:17:36:09:035340 gpstate:gpm:gpadmin-[INFO]:-Obtaining Segment details from master... 20160829:17:36:09:035340 gpstate:gpm:gpadmin-[INFO]:-Gathering data from segments... . 20160829:17:36:10:035340 gpstate:gpm:gpadmin-[INFO]:-Greenplum instance status summary 20160829:17:36:10:035340 gpstate:gpm:gpadmin-[INFO]:----------------------------------------------------- 20160829:17:36:10:035340 gpstate:gpm:gpadmin-[INFO]:- Master instance = Active 20160829:17:36:10:035340 gpstate:gpm:gpadmin-[INFO]:- Master standby = gps1 20160829:17:36:10:035340 gpstate:gpm:gpadmin-[INFO]:- Standby master state = Standby host passive 20160829:17:36:10:035340 gpstate:gpm:gpadmin-[INFO]:- Total segment instance count from metadata = 24 20160829:17:36:10:035340 gpstate:gpm:gpadmin-[INFO]:----------------------------------------------------- 20160829:17:36:10:035340 gpstate:gpm:gpadmin-[INFO]:- Primary Segment Status 20160829:17:36:10:035340 gpstate:gpm:gpadmin-[INFO]:----------------------------------------------------- 20160829:17:36:10:035340 gpstate:gpm:gpadmin-[INFO]:- Total primary segments = 12 20160829:17:36:10:035340 gpstate:gpm:gpadmin-[INFO]:- Total primary segment valid (at master) = 3 20160829:17:36:10:035340 gpstate:gpm:gpadmin-[WARNING]:-Total primary segment failures (at master) = 9 <<<<<<<< 20160829:17:36:10:035340 gpstate:gpm:gpadmin-[WARNING]:-Total number of postmaster.pid files missing = 9 <<<<<<<< 20160829:17:36:10:035340 gpstate:gpm:gpadmin-[INFO]:- Total number of postmaster.pid files found = 3 20160829:17:36:10:035340 gpstate:gpm:gpadmin-[WARNING]:-Total number of postmaster.pid PIDs missing = 9 <<<<<<<< 20160829:17:36:10:035340 gpstate:gpm:gpadmin-[INFO]:- Total number of postmaster.pid PIDs found = 3 20160829:17:36:10:035340 gpstate:gpm:gpadmin-[WARNING]:-Total number of /tmp lock files missing = 9 <<<<<<<< 20160829:17:36:10:035340 gpstate:gpm:gpadmin-[INFO]:- Total number of /tmp lock files found = 3 20160829:17:36:10:035340 gpstate:gpm:gpadmin-[WARNING]:-Total number postmaster processes missing = 9 <<<<<<<< 20160829:17:36:10:035340 gpstate:gpm:gpadmin-[INFO]:- Total number postmaster processes found = 3 20160829:17:36:10:035340 gpstate:gpm:gpadmin-[INFO]:----------------------------------------------------- 20160829:17:36:10:035340 gpstate:gpm:gpadmin-[INFO]:- Mirror Segment Status 20160829:17:36:10:035340 gpstate:gpm:gpadmin-[INFO]:----------------------------------------------------- 20160829:17:36:10:035340 gpstate:gpm:gpadmin-[INFO]:- Total mirror segments = 12 20160829:17:36:10:035340 gpstate:gpm:gpadmin-[INFO]:- Total mirror segment valid (at master) = 9 20160829:17:36:10:035340 gpstate:gpm:gpadmin-[WARNING]:-Total mirror segment failures (at master) = 3 <<<<<<<< 20160829:17:36:10:035340 gpstate:gpm:gpadmin-[WARNING]:-Total number of postmaster.pid files missing = 3 <<<<<<<< 20160829:17:36:10:035340 gpstate:gpm:gpadmin-[INFO]:- Total number of postmaster.pid files found = 9 20160829:17:36:10:035340 gpstate:gpm:gpadmin-[WARNING]:-Total number of postmaster.pid PIDs missing = 3 <<<<<<<< 20160829:17:36:10:035340 gpstate:gpm:gpadmin-[INFO]:- Total number of postmaster.pid PIDs found = 9 20160829:17:36:10:035340 gpstate:gpm:gpadmin-[WARNING]:-Total number of /tmp lock files missing = 3 <<<<<<<< 20160829:17:36:10:035340 gpstate:gpm:gpadmin-[INFO]:- Total number of /tmp lock files found = 9 20160829:17:36:10:035340 gpstate:gpm:gpadmin-[WARNING]:-Total number postmaster processes missing = 3 <<<<<<<< 20160829:17:36:10:035340 gpstate:gpm:gpadmin-[INFO]:- Total number postmaster processes found = 9 20160829:17:36:10:035340 gpstate:gpm:gpadmin-[WARNING]:-Total number mirror segments acting as primary segments = 9 <<<<<<<< 20160829:17:36:10:035340 gpstate:gpm:gpadmin-[INFO]:- Total number mirror segments acting as mirror segments = 3 20160829:17:36:10:035340 gpstate:gpm:gpadmin-[INFO]:-----------------------------------------------------
gpstate -m列出mirror 实例的状态和配置信息。
失败的primary segment被mirror segment顶起来了，可以看到，有9个mirror segment Acting as Primary。
[gpadmin@gpm ~]$ gpstate -m 20160829:17:38:03:035819 gpstate:gpm:gpadmin-[INFO]:-Starting gpstate with args: -m 20160829:17:38:03:035819 gpstate:gpm:gpadmin-[INFO]:-local Greenplum Version: 'postgres (Greenplum Database) 4.3.8.2 build 1' 20160829:17:38:03:035819 gpstate:gpm:gpadmin-[INFO]:-master Greenplum Version: 'PostgreSQL 8.2.15 (Greenplum Database 4.3.8.2 build 1) on x86_64-unknown-linux-gnu, compiled by GCC gcc (GCC) 4.4.2 compiled on May 9 2016 22:28:40' 20160829:17:38:03:035819 gpstate:gpm:gpadmin-[INFO]:-Obtaining Segment details from master... 20160829:17:38:04:035819 gpstate:gpm:gpadmin-[INFO]:-------------------------------------------------------------- 20160829:17:38:04:035819 gpstate:gpm:gpadmin-[INFO]:--Current GPDB mirror list and status 20160829:17:38:04:035819 gpstate:gpm:gpadmin-[INFO]:--Type = Group 20160829:17:38:04:035819 gpstate:gpm:gpadmin-[INFO]:-------------------------------------------------------------- 20160829:17:38:04:035819 gpstate:gpm:gpadmin-[INFO]:- Mirror Datadir Port Status Data Status 20160829:17:38:04:035819 gpstate:gpm:gpadmin-[INFO]:- gps2 /hdfs/data5/gpdata/mirror/gpseg0 50000 Acting as Primary Change Tracking 20160829:17:38:04:035819 gpstate:gpm:gpadmin-[INFO]:- gps2 /hdfs/data6/gpdata/mirror/gpseg1 50001 Acting as Primary Change Tracking 20160829:17:38:04:035819 gpstate:gpm:gpadmin-[INFO]:- gps2 /hdfs/data7/gpdata/mirror/gpseg2 50002 Acting as Primary Change Tracking 20160829:17:38:04:035819 gpstate:gpm:gpadmin-[INFO]:- gps2 /hdfs/data8/gpdata/mirror/gpseg3 50003 Acting as Primary Change Tracking 20160829:17:38:04:035819 gpstate:gpm:gpadmin-[INFO]:- gps3 /hdfs/data5/gpdata/mirror/gpseg4 50000 Acting as Primary Change Tracking 20160829:17:38:04:035819 gpstate:gpm:gpadmin-[INFO]:- gps3 /hdfs/data6/gpdata/mirror/gpseg5 50001 Acting as Primary Change Tracking 20160829:17:38:04:035819 gpstate:gpm:gpadmin-[INFO]:- gps3 /hdfs/data7/gpdata/mirror/gpseg6 50002 Acting as Primary Change Tracking 20160829:17:38:04:035819 gpstate:gpm:gpadmin-[INFO]:- gps3 /hdfs/data8/gpdata/mirror/gpseg7 50003 Acting as Primary Change Tracking 20160829:17:38:04:035819 gpstate:gpm:gpadmin-[WARNING]:-gps1 /hdfs/data5/gpdata/mirror/gpseg8 50000 Failed <<<<<<<< 20160829:17:38:04:035819 gpstate:gpm:gpadmin-[WARNING]:-gps1 /hdfs/data6/gpdata/mirror/gpseg9 50001 Failed <<<<<<<< 20160829:17:38:04:035819 gpstate:gpm:gpadmin-[INFO]:- gps1 /hdfs/data7/gpdata/mirror/gpseg10 50002 Acting as Primary Change Tracking 20160829:17:38:04:035819 gpstate:gpm:gpadmin-[WARNING]:-gps1 /hdfs/data8/gpdata/mirror/gpseg11 50003 Failed <<<<<<<< 20160829:17:38:04:035819 gpstate:gpm:gpadmin-[INFO]:-------------------------------------------------------------- 20160829:17:38:04:035819 gpstate:gpm:gpadmin-[WARNING]:-9 segment(s) configured as mirror(s) are acting as primaries 20160829:17:38:04:035819 gpstate:gpm:gpadmin-[WARNING]:-3 segment(s) configured as mirror(s) have failed 20160829:17:38:04:035819 gpstate:gpm:gpadmin-[WARNING]:-9 mirror segment(s) acting as primaries are in change tracking
gpstate -c 查看primary instance 和 mirror instance 的对应关系
[gpadmin@gpm ~]$ gpstate -c 20160829:17:38:53:035970 gpstate:gpm:gpadmin-[INFO]:-Starting gpstate with args: -c 20160829:17:38:53:035970 gpstate:gpm:gpadmin-[INFO]:-local Greenplum Version: 'postgres (Greenplum Database) 4.3.8.2 build 1' 20160829:17:38:53:035970 gpstate:gpm:gpadmin-[INFO]:-master Greenplum Version: 'PostgreSQL 8.2.15 (Greenplum Database 4.3.8.2 build 1) on x86_64-unknown-linux-gnu, compiled by GCC gcc (GCC) 4.4.2 compiled on May 9 2016 22:28:40' 20160829:17:38:53:035970 gpstate:gpm:gpadmin-[INFO]:-Obtaining Segment details from master... 20160829:17:38:54:035970 gpstate:gpm:gpadmin-[INFO]:-------------------------------------------------------------- 20160829:17:38:54:035970 gpstate:gpm:gpadmin-[INFO]:--Current GPDB mirror list and status 20160829:17:38:54:035970 gpstate:gpm:gpadmin-[INFO]:--Type = Group 20160829:17:38:54:035970 gpstate:gpm:gpadmin-[INFO]:-------------------------------------------------------------- 20160829:17:38:54:035970 gpstate:gpm:gpadmin-[INFO]:- Status Data State Primary Datadir Port Mirror Datadir Port 20160829:17:38:54:035970 gpstate:gpm:gpadmin-[WARNING]:-Mirror Active, Primary Failed Change Tracking gps1 /hdfs/data1/gpdata/primary/gpseg0 40000 gps2 /hdfs/data5/gpdata/mirror/gpseg0 50000 <<<<<<<< 20160829:17:38:54:035970 gpstate:gpm:gpadmin-[WARNING]:-Mirror Active, Primary Failed Change Tracking gps1 /hdfs/data2/gpdata/primary/gpseg1 40001 gps2 /hdfs/data6/gpdata/mirror/gpseg1 50001 <<<<<<<< 20160829:17:38:54:035970 gpstate:gpm:gpadmin-[WARNING]:-Mirror Active, Primary Failed Change Tracking gps1 /hdfs/data3/gpdata/primary/gpseg2 40002 gps2 /hdfs/data7/gpdata/mirror/gpseg2 50002 <<<<<<<< 20160829:17:38:54:035970 gpstate:gpm:gpadmin-[WARNING]:-Mirror Active, Primary Failed Change Tracking gps1 /hdfs/data4/gpdata/primary/gpseg3 40003 gps2 /hdfs/data8/gpdata/mirror/gpseg3 50003 <<<<<<<< 20160829:17:38:54:035970 gpstate:gpm:gpadmin-[WARNING]:-Mirror Active, Primary Failed Change Tracking gps2 /hdfs/data1/gpdata/primary/gpseg4 40000 gps3 /hdfs/data5/gpdata/mirror/gpseg4 50000 <<<<<<<< 20160829:17:38:54:035970 gpstate:gpm:gpadmin-[WARNING]:-Mirror Active, Primary Failed Change Tracking gps2 /hdfs/data2/gpdata/primary/gpseg5 40001 gps3 /hdfs/data6/gpdata/mirror/gpseg5 50001 <<<<<<<< 20160829:17:38:54:035970 gpstate:gpm:gpadmin-[WARNING]:-Mirror Active, Primary Failed Change Tracking gps2 /hdfs/data3/gpdata/primary/gpseg6 40002 gps3 /hdfs/data7/gpdata/mirror/gpseg6 50002 <<<<<<<< 20160829:17:38:54:035970 gpstate:gpm:gpadmin-[WARNING]:-Mirror Active, Primary Failed Change Tracking gps2 /hdfs/data4/gpdata/primary/gpseg7 40003 gps3 /hdfs/data8/gpdata/mirror/gpseg7 50003 <<<<<<<< 20160829:17:38:54:035970 gpstate:gpm:gpadmin-[WARNING]:-Primary Active, Mirror Failed Change Tracking gps3 /hdfs/data1/gpdata/primary/gpseg8 40000 gps1 /hdfs/data5/gpdata/mirror/gpseg8 50000 <<<<<<<< 20160829:17:38:54:035970 gpstate:gpm:gpadmin-[WARNING]:-Primary Active, Mirror Failed Change Tracking gps3 /hdfs/data2/gpdata/primary/gpseg9 40001 gps1 /hdfs/data6/gpdata/mirror/gpseg9 50001 <<<<<<<< 20160829:17:38:54:035970 gpstate:gpm:gpadmin-[WARNING]:-Mirror Active, Primary Failed Change Tracking gps3 /hdfs/data3/gpdata/primary/gpseg10 40002 gps1 /hdfs/data7/gpdata/mirror/gpseg10 50002 <<<<<<<< 20160829:17:38:54:035970 gpstate:gpm:gpadmin-[WARNING]:-Primary Active, Mirror Failed Change Tracking gps3 /hdfs/data4/gpdata/primary/gpseg11 40003 gps1 /hdfs/data8/gpdata/mirror/gpseg11 50003 <<<<<<<< 20160829:17:38:54:035970 gpstate:gpm:gpadmin-[INFO]:-------------------------------------------------------------- 20160829:17:38:54:035970 gpstate:gpm:gpadmin-[WARNING]:-9 segment(s) configured as mirror(s) are acting as primaries 20160829:17:38:54:035970 gpstate:gpm:gpadmin-[WARNING]:-12 segment(s) are in change tracking
二、失败segment修复
经过上述操作，基本了解了GP集群各节点状态，接下来借助Greenplum自带工具进行失败segment修复。gprecoverseg 工具修复失败segment，出现提示是否继续恢复程序时，输出‘Y’。
# 该过程打印信息忘了保存，借用网上的 $ gprecoverseg 20160118:14:08:37:025645 gprecoverseg:gp-master:gpadmin-[INFO]:-Starting gprecoverseg with args: 20160118:14:08:37:025645 gprecoverseg:gp-master:gpadmin-[INFO]:-local Greenplum Version: 'postgres (Greenplum Database) 4.3.6.2 build 1' 20160118:14:08:37:025645 gprecoverseg:gp-master:gpadmin-[INFO]:-master Greenplum Version: 'PostgreSQL 8.2.15 (Greenplum Database 4.3.6.2 build 1) on x86_64-unknown-linux-gnu, compiled by GCC gcc (GCC) 4.4.2 compiled on Nov 12 2015 23:50:28' 20160118:14:08:37:025645 gprecoverseg:gp-master:gpadmin-[INFO]:-Checking if segments are ready 20160118:14:08:37:025645 gprecoverseg:gp-master:gpadmin-[INFO]:-Obtaining Segment details from master... 20160118:14:08:37:025645 gprecoverseg:gp-master:gpadmin-[INFO]:-Obtaining Segment details from master... 20160118:14:08:38:025645 gprecoverseg:gp-master:gpadmin-[INFO]:-Greenplum instance recovery parameters 20160118:14:08:38:025645 gprecoverseg:gp-master:gpadmin-[INFO]:---------------------------------------------------------- 20160118:14:08:38:025645 gprecoverseg:gp-master:gpadmin-[INFO]:-Recovery type = Standard 20160118:14:08:38:025645 gprecoverseg:gp-master:gpadmin-[INFO]:---------------------------------------------------------- 20160118:14:08:38:025645 gprecoverseg:gp-master:gpadmin-[INFO]:-Recovery 1 of 1 20160118:14:08:38:025645 gprecoverseg:gp-master:gpadmin-[INFO]:---------------------------------------------------------- 20160118:14:08:38:025645 gprecoverseg:gp-master:gpadmin-[INFO]:- Synchronization mode = Incremental 20160118:14:08:38:025645 gprecoverseg:gp-master:gpadmin-[INFO]:- Failed instance host = gp-s3 20160118:14:08:38:025645 gprecoverseg:gp-master:gpadmin-[INFO]:- Failed instance address = gp-s3 20160118:14:08:38:025645 gprecoverseg:gp-master:gpadmin-[INFO]:- Failed instance directory = /home/gpadmin/primary/gpseg2 20160118:14:08:38:025645 gprecoverseg:gp-master:gpadmin-[INFO]:- Failed instance port = 40000 20160118:14:08:38:025645 gprecoverseg:gp-master:gpadmin-[INFO]:- Failed instance replication port = 41000 20160118:14:08:38:025645 gprecoverseg:gp-master:gpadmin-[INFO]:- Recovery Source instance host = gp-s1 20160118:14:08:38:025645 gprecoverseg:gp-master:gpadmin-[INFO]:- Recovery Source instance address = gp-s1 20160118:14:08:38:025645 gprecoverseg:gp-master:gpadmin-[INFO]:- Recovery Source instance directory = /home/gpadmin/mirror/gpseg2 20160118:14:08:38:025645 gprecoverseg:gp-master:gpadmin-[INFO]:- Recovery Source instance port = 50000 20160118:14:08:38:025645 gprecoverseg:gp-master:gpadmin-[INFO]:- Recovery Source instance replication port = 51000 20160118:14:08:38:025645 gprecoverseg:gp-master:gpadmin-[INFO]:- Recovery Target = in-place 20160118:14:08:38:025645 gprecoverseg:gp-master:gpadmin-[INFO]:---------------------------------------------------------- Continue with segment recovery procedure Yy|Nn (default=N): > y 20160118:14:08:42:025645 gprecoverseg:gp-master:gpadmin-[INFO]:-1 segment(s) to recover 20160118:14:08:42:025645 gprecoverseg:gp-master:gpadmin-[INFO]:-Ensuring 1 failed segment(s) are stopped 20160118:14:08:43:025645 gprecoverseg:gp-master:gpadmin-[INFO]:-23723: /home/gpadmin/primary/gpseg2 20160118:14:08:51:025645 gprecoverseg:gp-master:gpadmin-[INFO]:-Ensuring that shared memory is cleaned up for stopped segments updating flat files 20160118:14:08:56:025645 gprecoverseg:gp-master:gpadmin-[INFO]:-Updating configuration with new mirrors 20160118:14:08:56:025645 gprecoverseg:gp-master:gpadmin-[INFO]:-Updating mirrors . 20160118:14:08:57:025645 gprecoverseg:gp-master:gpadmin-[INFO]:-Starting mirrors 20160118:14:08:57:025645 gprecoverseg:gp-master:gpadmin-[INFO]:-Commencing parallel primary and mirror segment instance startup, please wait... .. 20160118:14:08:59:025645 gprecoverseg:gp-master:gpadmin-[INFO]:-Process results... 20160118:14:08:59:025645 gprecoverseg:gp-master:gpadmin-[INFO]:-Updating configuration to mark mirrors up 20160118:14:08:59:025645 gprecoverseg:gp-master:gpadmin-[INFO]:-Updating primaries 20160118:14:08:59:025645 gprecoverseg:gp-master:gpadmin-[INFO]:-Commencing parallel primary conversion of 1 segments, please wait... . 20160118:14:09:00:025645 gprecoverseg:gp-master:gpadmin-[INFO]:-Process results... 20160118:14:09:00:025645 gprecoverseg:gp-master:gpadmin-[INFO]:-Done updating primaries 20160118:14:09:00:025645 gprecoverseg:gp-master:gpadmin-[INFO]:-****************************************************************** 20160118:14:09:00:025645 gprecoverseg:gp-master:gpadmin-[INFO]:-Updating segments for resynchronization is completed. 20160118:14:09:00:025645 gprecoverseg:gp-master:gpadmin-[INFO]:-For segments updated successfully, resynchronization will continue in the background. 20160118:14:09:00:025645 gprecoverseg:gp-master:gpadmin-[INFO]:- 20160118:14:09:00:025645 gprecoverseg:gp-master:gpadmin-[INFO]:-Use gpstate -s to check the resynchronization progress. 20160118:14:09:00:025645 gprecoverseg:gp-master:gpadmin-[INFO]:-******************************************************************
2 确认primary和mirror已同步，状态从不正常变为正常的过程:Change Tracking>>Resynchronizing>> Synchronized。
[gpadmin@gpm ~]$ gpstate -m 20160829:18:35:02:048639 gpstate:gpm:gpadmin-[INFO]:-Starting gpstate with args: -m 20160829:18:35:02:048639 gpstate:gpm:gpadmin-[INFO]:-local Greenplum Version: 'postgres (Greenplum Database) 4.3.8.2 build 1' 20160829:18:35:02:048639 gpstate:gpm:gpadmin-[INFO]:-master Greenplum Version: 'PostgreSQL 8.2.15 (Greenplum Database 4.3.8.2 build 1) on x86_64-unknown-linux-gnu, compiled by GCC gcc (GCC) 4.4.2 compiled on May 9 2016 22:28:40' 20160829:18:35:02:048639 gpstate:gpm:gpadmin-[INFO]:-Obtaining Segment details from master... 20160829:18:35:02:048639 gpstate:gpm:gpadmin-[INFO]:-------------------------------------------------------------- 20160829:18:35:02:048639 gpstate:gpm:gpadmin-[INFO]:--Current GPDB mirror list and status 20160829:18:35:02:048639 gpstate:gpm:gpadmin-[INFO]:--Type = Group 20160829:18:35:02:048639 gpstate:gpm:gpadmin-[INFO]:-------------------------------------------------------------- 20160829:18:35:02:048639 gpstate:gpm:gpadmin-[INFO]:- Mirror Datadir Port Status Data Status 20160829:18:35:02:048639 gpstate:gpm:gpadmin-[INFO]:- gps2 /hdfs/data5/gpdata/mirror/gpseg0 50000 Passive Resynchronizing 20160829:18:35:02:048639 gpstate:gpm:gpadmin-[INFO]:- gps2 /hdfs/data6/gpdata/mirror/gpseg1 50001 Passive Resynchronizing 20160829:18:35:02:048639 gpstate:gpm:gpadmin-[INFO]:- gps2 /hdfs/data7/gpdata/mirror/gpseg2 50002 Passive Resynchronizing 20160829:18:35:02:048639 gpstate:gpm:gpadmin-[INFO]:- gps2 /hdfs/data8/gpdata/mirror/gpseg3 50003 Passive Resynchronizing 20160829:18:35:02:048639 gpstate:gpm:gpadmin-[INFO]:- gps3 /hdfs/data5/gpdata/mirror/gpseg4 50000 Passive Resynchronizing 20160829:18:35:02:048639 gpstate:gpm:gpadmin-[INFO]:- gps3 /hdfs/data6/gpdata/mirror/gpseg5 50001 Passive Resynchronizing 20160829:18:35:02:048639 gpstate:gpm:gpadmin-[INFO]:- gps3 /hdfs/data7/gpdata/mirror/gpseg6 50002 Passive Resynchronizing 20160829:18:35:02:048639 gpstate:gpm:gpadmin-[INFO]:- gps3 /hdfs/data8/gpdata/mirror/gpseg7 50003 Passive Resynchronizing 20160829:18:35:02:048639 gpstate:gpm:gpadmin-[INFO]:- gps1 /hdfs/data5/gpdata/mirror/gpseg8 50000 Passive Synchronized 20160829:18:35:02:048639 gpstate:gpm:gpadmin-[INFO]:- gps1 /hdfs/data6/gpdata/mirror/gpseg9 50001 Passive Synchronized 20160829:18:35:02:048639 gpstate:gpm:gpadmin-[INFO]:- gps1 /hdfs/data7/gpdata/mirror/gpseg10 50002 Passive Resynchronizing 20160829:18:35:02:048639 gpstate:gpm:gpadmin-[INFO]:- gps1 /hdfs/data8/gpdata/mirror/gpseg11 50003 Passive Synchronized 20160829:18:35:02:048639 gpstate:gpm:gpadmin-[INFO]:-------------------------------------------------------------- [gpadmin@gpm ~]$ gpstate -m 20160829:18:36:01:048898 gpstate:gpm:gpadmin-[INFO]:-Starting gpstate with args: -m 20160829:18:36:01:048898 gpstate:gpm:gpadmin-[INFO]:-local Greenplum Version: 'postgres (Greenplum Database) 4.3.8.2 build 1' 20160829:18:36:01:048898 gpstate:gpm:gpadmin-[INFO]:-master Greenplum Version: 'PostgreSQL 8.2.15 (Greenplum Database 4.3.8.2 build 1) on x86_64-unknown-linux-gnu, compiled by GCC gcc (GCC) 4.4.2 compiled on May 9 2016 22:28:40' 20160829:18:36:01:048898 gpstate:gpm:gpadmin-[INFO]:-Obtaining Segment details from master... 20160829:18:36:01:048898 gpstate:gpm:gpadmin-[INFO]:-------------------------------------------------------------- 20160829:18:36:01:048898 gpstate:gpm:gpadmin-[INFO]:--Current GPDB mirror list and status 20160829:18:36:01:048898 gpstate:gpm:gpadmin-[INFO]:--Type = Group 20160829:18:36:01:048898 gpstate:gpm:gpadmin-[INFO]:-------------------------------------------------------------- 20160829:18:36:01:048898 gpstate:gpm:gpadmin-[INFO]:- Mirror Datadir Port Status Data Status 20160829:18:36:01:048898 gpstate:gpm:gpadmin-[INFO]:- gps2 /hdfs/data5/gpdata/mirror/gpseg0 50000 Passive Synchronized 20160829:18:36:01:048898 gpstate:gpm:gpadmin-[INFO]:- gps2 /hdfs/data6/gpdata/mirror/gpseg1 50001 Passive Synchronized 20160829:18:36:01:048898 gpstate:gpm:gpadmin-[INFO]:- gps2 /hdfs/data7/gpdata/mirror/gpseg2 50002 Passive Synchronized 20160829:18:36:01:048898 gpstate:gpm:gpadmin-[INFO]:- gps2 /hdfs/data8/gpdata/mirror/gpseg3 50003 Passive Synchronized 20160829:18:36:01:048898 gpstate:gpm:gpadmin-[INFO]:- gps3 /hdfs/data5/gpdata/mirror/gpseg4 50000 Passive Synchronized 20160829:18:36:01:048898 gpstate:gpm:gpadmin-[INFO]:- gps3 /hdfs/data6/gpdata/mirror/gpseg5 50001 Passive Synchronized 20160829:18:36:01:048898 gpstate:gpm:gpadmin-[INFO]:- gps3 /hdfs/data7/gpdata/mirror/gpseg6 50002 Passive Synchronized 20160829:18:36:01:048898 gpstate:gpm:gpadmin-[INFO]:- gps3 /hdfs/data8/gpdata/mirror/gpseg7 50003 Passive Synchronized 20160829:18:36:01:048898 gpstate:gpm:gpadmin-[INFO]:- gps1 /hdfs/data5/gpdata/mirror/gpseg8 50000 Passive Synchronized 20160829:18:36:01:048898 gpstate:gpm:gpadmin-[INFO]:- gps1 /hdfs/data6/gpdata/mirror/gpseg9 50001 Passive Synchronized 20160829:18:36:01:048898 gpstate:gpm:gpadmin-[INFO]:- gps1 /hdfs/data7/gpdata/mirror/gpseg10 50002 Passive Synchronized 20160829:18:36:01:048898 gpstate:gpm:gpadmin-[INFO]:- gps1 /hdfs/data8/gpdata/mirror/gpseg11 50003 Passive Synchronized 20160829:18:36:01:048898 gpstate:gpm:gpadmin-[INFO]:--------------------------------------------------------------
3 恢复所有segment到原有状态
当primary segment宕机，mirror成为了primary segment。执行gprecoverseg命令恢复失败segment后，当前primary segment是原来的mirror，原来失败的primary segment变成了现在的mirror。并没有恢复到系统初始化时候的角色状态，造成一个主机上有多个primary segment，会导致系统处于非平衡状态，消耗更多的系统资源。
gpstate -e命令，能够检查出segment是否是原来的角色
运行gpstate -m，确认所有mirrors是否同步完成(Resynchronized)。
如果mirrors的状态是Resynchronizing，等待完成
运行gprecoverseg -r命令，将所有segments恢复到原有角色。当提示是否继续segment 负载均衡时，输入‘Y’。
[gpadmin@gpm ~]$ gprecoverseg -r 20160829:18:34:09:048215 gprecoverseg:gpm:gpadmin-[INFO]:-Starting gprecoverseg with args: -r 20160829:18:34:09:048215 gprecoverseg:gpm:gpadmin-[INFO]:-local Greenplum Version: 'postgres (Greenplum Database) 4.3.8.2 build 1' 20160829:18:34:09:048215 gprecoverseg:gpm:gpadmin-[INFO]:-master Greenplum Version: 'PostgreSQL 8.2.15 (Greenplum Database 4.3.8.2 build 1) on x86_64-unknown-linux-gnu, compiled by GCC gcc (GCC) 4.4.2 compiled on May 9 2016 22:28:40' 20160829:18:34:09:048215 gprecoverseg:gpm:gpadmin-[INFO]:-Checking if segments are ready 20160829:18:34:09:048215 gprecoverseg:gpm:gpadmin-[INFO]:-Obtaining Segment details from master... 20160829:18:34:09:048215 gprecoverseg:gpm:gpadmin-[INFO]:-Obtaining Segment details from master... 20160829:18:34:09:048215 gprecoverseg:gpm:gpadmin-[INFO]:-Greenplum instance recovery parameters 20160829:18:34:09:048215 gprecoverseg:gpm:gpadmin-[INFO]:---------------------------------------------------------- 20160829:18:34:09:048215 gprecoverseg:gpm:gpadmin-[INFO]:-Recovery type = Rebalance 20160829:18:34:09:048215 gprecoverseg:gpm:gpadmin-[INFO]:---------------------------------------------------------- 20160829:18:34:09:048215 gprecoverseg:gpm:gpadmin-[INFO]:-Unbalanced segment 1 of 18 20160829:18:34:09:048215 gprecoverseg:gpm:gpadmin-[INFO]:---------------------------------------------------------- 20160829:18:34:09:048215 gprecoverseg:gpm:gpadmin-[INFO]:- Unbalanced instance host = gps2 20160829:18:34:09:048215 gprecoverseg:gpm:gpadmin-[INFO]:- Unbalanced instance address = gps2 20160829:18:34:09:048215 gprecoverseg:gpm:gpadmin-[INFO]:- Unbalanced instance directory = /hdfs/data5/gpdata/mirror/gpseg0 20160829:18:34:09:048215 gprecoverseg:gpm:gpadmin-[INFO]:- Unbalanced instance port = 50000 20160829:18:34:09:048215 gprecoverseg:gpm:gpadmin-[INFO]:- Unbalanced instance replication port = 51000 20160829:18:34:09:048215 gprecoverseg:gpm:gpadmin-[INFO]:- Balanced role = Mirror 20160829:18:34:09:048215 gprecoverseg:gpm:gpadmin-[INFO]:- Current role = Primary 20160829:18:34:09:048215 gprecoverseg:gpm:gpadmin-[INFO]:---------------------------------------------------------- 20160829:18:34:09:048215 gprecoverseg:gpm:gpadmin-[INFO]:-Unbalanced segment 2 of 18 20160829:18:34:09:048215 gprecoverseg:gpm:gpadmin-[INFO]:---------------------------------------------------------- 20160829:18:34:09:048215 gprecoverseg:gpm:gpadmin-[INFO]:- Unbalanced instance host = gps1 20160829:18:34:09:048215 gprecoverseg:gpm:gpadmin-[INFO]:- Unbalanced instance address = gps1 20160829:18:34:09:048215 gprecoverseg:gpm:gpadmin-[INFO]:- Unbalanced instance directory = /hdfs/data1/gpdata/primary/gpseg0 20160829:18:34:09:048215 gprecoverseg:gpm:gpadmin-[INFO]:- Unbalanced instance port = 40000 20160829:18:34:09:048215 gprecoverseg:gpm:gpadmin-[INFO]:- Unbalanced instance replication port = 41000 20160829:18:34:09:048215 gprecoverseg:gpm:gpadmin-[INFO]:- Balanced role = Primary 20160829:18:34:09:048215 gprecoverseg:gpm:gpadmin-[INFO]:- Current role = Mirror 20160829:18:34:09:048215 gprecoverseg:gpm:gpadmin-[INFO]:---------------------------------------------------------- 20160829:18:34:09:048215 gprecoverseg:gpm:gpadmin-[INFO]:-Unbalanced segment 3 of 18 20160829:18:34:09:048215 gprecoverseg:gpm:gpadmin-[INFO]:---------------------------------------------------------- 20160829:18:34:09:048215 gprecoverseg:gpm:gpadmin-[INFO]:- Unbalanced instance host = gps2 20160829:18:34:09:048215 gprecoverseg:gpm:gpadmin-[INFO]:- Unbalanced instance address = gps2 20160829:18:34:09:048215 gprecoverseg:gpm:gpadmin-[INFO]:- Unbalanced instance directory = /hdfs/data6/gpdata/mirror/gpseg1 20160829:18:34:09:048215 gprecoverseg:gpm:gpadmin-[INFO]:- Unbalanced instance port = 50001 20160829:18:34:09:048215 gprecoverseg:gpm:gpadmin-[INFO]:- Unbalanced instance replication port = 51001 20160829:18:34:09:048215 gprecoverseg:gpm:gpadmin-[INFO]:- Balanced role = Mirror 20160829:18:34:09:048215 gprecoverseg:gpm:gpadmin-[INFO]:- Current role = Primary 20160829:18:34:09:048215 gprecoverseg:gpm:gpadmin-[INFO]:---------------------------------------------------------- 20160829:18:34:09:048215 gprecoverseg:gpm:gpadmin-[INFO]:-Unbalanced segment 4 of 18 20160829:18:34:09:048215 gprecoverseg:gpm:gpadmin-[INFO]:---------------------------------------------------------- 20160829:18:34:09:048215 gprecoverseg:gpm:gpadmin-[INFO]:- Unbalanced instance host = gps1 20160829:18:34:09:048215 gprecoverseg:gpm:gpadmin-[INFO]:- Unbalanced instance address = gps1 20160829:18:34:09:048215 gprecoverseg:gpm:gpadmin-[INFO]:- Unbalanced instance directory = /hdfs/data2/gpdata/primary/gpseg1 20160829:18:34:09:048215 gprecoverseg:gpm:gpadmin-[INFO]:- Unbalanced instance port = 40001 20160829:18:34:09:048215 gprecoverseg:gpm:gpadmin-[INFO]:- Unbalanced instance replication port = 41001 20160829:18:34:09:048215 gprecoverseg:gpm:gpadmin-[INFO]:- Balanced role = Primary 20160829:18:34:09:048215 gprecoverseg:gpm:gpadmin-[INFO]:- Current role = Mirror 20160829:18:34:10:048215 gprecoverseg:gpm:gpadmin-[INFO]:---------------------------------------------------------- 20160829:18:34:10:048215 gprecoverseg:gpm:gpadmin-[INFO]:-Unbalanced segment 5 of 18 20160829:18:34:10:048215 gprecoverseg:gpm:gpadmin-[INFO]:---------------------------------------------------------- 20160829:18:34:10:048215 gprecoverseg:gpm:gpadmin-[INFO]:- Unbalanced instance host = gps2 20160829:18:34:10:048215 gprecoverseg:gpm:gpadmin-[INFO]:- Unbalanced instance address = gps2 20160829:18:34:10:048215 gprecoverseg:gpm:gpadmin-[INFO]:- Unbalanced instance directory = /hdfs/data7/gpdata/mirror/gpseg2 20160829:18:34:10:048215 gprecoverseg:gpm:gpadmin-[INFO]:- Unbalanced instance port = 50002 20160829:18:34:10:048215 gprecoverseg:gpm:gpadmin-[INFO]:- Unbalanced instance replication port = 51002 20160829:18:34:10:048215 gprecoverseg:gpm:gpadmin-[INFO]:- Balanced role = Mirror 20160829:18:34:10:048215 gprecoverseg:gpm:gpadmin-[INFO]:- Current role = Primary 20160829:18:34:10:048215 gprecoverseg:gpm:gpadmin-[INFO]:---------------------------------------------------------- 20160829:18:34:10:048215 gprecoverseg:gpm:gpadmin-[INFO]:-Unbalanced segment 6 of 18 20160829:18:34:10:048215 gprecoverseg:gpm:gpadmin-[INFO]:---------------------------------------------------------- 20160829:18:34:10:048215 gprecoverseg:gpm:gpadmin-[INFO]:- Unbalanced instance host = gps1 20160829:18:34:10:048215 gprecoverseg:gpm:gpadmin-[INFO]:- Unbalanced instance address = gps1 20160829:18:34:10:048215 gprecoverseg:gpm:gpadmin-[INFO]:- Unbalanced instance directory = /hdfs/data3/gpdata/primary/gpseg2 20160829:18:34:10:048215 gprecoverseg:gpm:gpadmin-[INFO]:- Unbalanced instance port = 40002 20160829:18:34:10:048215 gprecoverseg:gpm:gpadmin-[INFO]:- Unbalanced instance replication port = 41002 20160829:18:34:10:048215 gprecoverseg:gpm:gpadmin-[INFO]:- Balanced role = Primary 20160829:18:34:10:048215 gprecoverseg:gpm:gpadmin-[INFO]:- Current role = Mirror 20160829:18:34:10:048215 gprecoverseg:gpm:gpadmin-[INFO]:---------------------------------------------------------- 20160829:18:34:10:048215 gprecoverseg:gpm:gpadmin-[INFO]:-Unbalanced segment 7 of 18 20160829:18:34:10:048215 gprecoverseg:gpm:gpadmin-[INFO]:---------------------------------------------------------- 20160829:18:34:10:048215 gprecoverseg:gpm:gpadmin-[INFO]:- Unbalanced instance host = gps2 20160829:18:34:10:048215 gprecoverseg:gpm:gpadmin-[INFO]:- Unbalanced instance address = gps2 20160829:18:34:10:048215 gprecoverseg:gpm:gpadmin-[INFO]:- Unbalanced instance directory = /hdfs/data8/gpdata/mirror/gpseg3 20160829:18:34:10:048215 gprecoverseg:gpm:gpadmin-[INFO]:- Unbalanced instance port = 50003 20160829:18:34:10:048215 gprecoverseg:gpm:gpadmin-[INFO]:- Unbalanced instance replication port = 51003 20160829:18:34:10:048215 gprecoverseg:gpm:gpadmin-[INFO]:- Balanced role = Mirror 20160829:18:34:10:048215 gprecoverseg:gpm:gpadmin-[INFO]:- Current role = Primary 20160829:18:34:10:048215 gprecoverseg:gpm:gpadmin-[INFO]:---------------------------------------------------------- 20160829:18:34:10:048215 gprecoverseg:gpm:gpadmin-[INFO]:-Unbalanced segment 8 of 18 20160829:18:34:10:048215 gprecoverseg:gpm:gpadmin-[INFO]:---------------------------------------------------------- 20160829:18:34:10:048215 gprecoverseg:gpm:gpadmin-[INFO]:- Unbalanced instance host = gps1 20160829:18:34:10:048215 gprecoverseg:gpm:gpadmin-[INFO]:- Unbalanced instance address = gps1 20160829:18:34:10:048215 gprecoverseg:gpm:gpadmin-[INFO]:- Unbalanced instance directory = /hdfs/data4/gpdata/primary/gpseg3 20160829:18:34:10:048215 gprecoverseg:gpm:gpadmin-[INFO]:- Unbalanced instance port = 40003 20160829:18:34:10:048215 gprecoverseg:gpm:gpadmin-[INFO]:- Unbalanced instance replication port = 41003 20160829:18:34:10:048215 gprecoverseg:gpm:gpadmin-[INFO]:- Balanced role = Primary 20160829:18:34:10:048215 gprecoverseg:gpm:gpadmin-[INFO]:- Current role = Mirror 20160829:18:34:10:048215 gprecoverseg:gpm:gpadmin-[INFO]:---------------------------------------------------------- 20160829:18:34:10:048215 gprecoverseg:gpm:gpadmin-[INFO]:-Unbalanced segment 9 of 18 20160829:18:34:10:048215 gprecoverseg:gpm:gpadmin-[INFO]:---------------------------------------------------------- 20160829:18:34:10:048215 gprecoverseg:gpm:gpadmin-[INFO]:- Unbalanced instance host = gps3 20160829:18:34:10:048215 gprecoverseg:gpm:gpadmin-[INFO]:- Unbalanced instance address = gps3 20160829:18:34:10:048215 gprecoverseg:gpm:gpadmin-[INFO]:- Unbalanced instance directory = /hdfs/data5/gpdata/mirror/gpseg4 20160829:18:34:10:048215 gprecoverseg:gpm:gpadmin-[INFO]:- Unbalanced instance port = 50000 20160829:18:34:10:048215 gprecoverseg:gpm:gpadmin-[INFO]:- Unbalanced instance replication port = 51000 20160829:18:34:10:048215 gprecoverseg:gpm:gpadmin-[INFO]:- Balanced role = Mirror 20160829:18:34:10:048215 gprecoverseg:gpm:gpadmin-[INFO]:- Current role = Primary 20160829:18:34:10:048215 gprecoverseg:gpm:gpadmin-[INFO]:---------------------------------------------------------- 20160829:18:34:10:048215 gprecoverseg:gpm:gpadmin-[INFO]:-Unbalanced segment 10 of 18 20160829:18:34:10:048215 gprecoverseg:gpm:gpadmin-[INFO]:---------------------------------------------------------- 20160829:18:34:10:048215 gprecoverseg:gpm:gpadmin-[INFO]:- Unbalanced instance host = gps2 20160829:18:34:10:048215 gprecoverseg:gpm:gpadmin-[INFO]:- Unbalanced instance address = gps2 20160829:18:34:10:048215 gprecoverseg:gpm:gpadmin-[INFO]:- Unbalanced instance directory = /hdfs/data1/gpdata/primary/gpseg4 20160829:18:34:10:048215 gprecoverseg:gpm:gpadmin-[INFO]:- Unbalanced instance port = 40000 20160829:18:34:10:048215 gprecoverseg:gpm:gpadmin-[INFO]:- Unbalanced instance replication port = 41000 20160829:18:34:10:048215 gprecoverseg:gpm:gpadmin-[INFO]:- Balanced role = Primary 20160829:18:34:10:048215 gprecoverseg:gpm:gpadmin-[INFO]:- Current role = Mirror 20160829:18:34:10:048215 gprecoverseg:gpm:gpadmin-[INFO]:---------------------------------------------------------- 20160829:18:34:10:048215 gprecoverseg:gpm:gpadmin-[INFO]:-Unbalanced segment 11 of 18 20160829:18:34:10:048215 gprecoverseg:gpm:gpadmin-[INFO]:---------------------------------------------------------- 20160829:18:34:10:048215 gprecoverseg:gpm:gpadmin-[INFO]:- Unbalanced instance host = gps3 20160829:18:34:10:048215 gprecoverseg:gpm:gpadmin-[INFO]:- Unbalanced instance address = gps3 20160829:18:34:10:048215 gprecoverseg:gpm:gpadmin-[INFO]:- Unbalanced instance directory = /hdfs/data6/gpdata/mirror/gpseg5 20160829:18:34:10:048215 gprecoverseg:gpm:gpadmin-[INFO]:- Unbalanced instance port = 50001 20160829:18:34:10:048215 gprecoverseg:gpm:gpadmin-[INFO]:- Unbalanced instance replication port = 51001 20160829:18:34:10:048215 gprecoverseg:gpm:gpadmin-[INFO]:- Balanced role = Mirror 20160829:18:34:10:048215 gprecoverseg:gpm:gpadmin-[INFO]:- Current role = Primary 20160829:18:34:10:048215 gprecoverseg:gpm:gpadmin-[INFO]:---------------------------------------------------------- 20160829:18:34:10:048215 gprecoverseg:gpm:gpadmin-[INFO]:-Unbalanced segment 12 of 18 20160829:18:34:10:048215 gprecoverseg:gpm:gpadmin-[INFO]:---------------------------------------------------------- 20160829:18:34:10:048215 gprecoverseg:gpm:gpadmin-[INFO]:- Unbalanced instance host = gps2 20160829:18:34:10:048215 gprecoverseg:gpm:gpadmin-[INFO]:- Unbalanced instance address = gps2 20160829:18:34:10:048215 gprecoverseg:gpm:gpadmin-[INFO]:- Unbalanced instance directory = /hdfs/data2/gpdata/primary/gpseg5 20160829:18:34:10:048215 gprecoverseg:gpm:gpadmin-[INFO]:- Unbalanced instance port = 40001 20160829:18:34:10:048215 gprecoverseg:gpm:gpadmin-[INFO]:- Unbalanced instance replication port = 41001 20160829:18:34:10:048215 gprecoverseg:gpm:gpadmin-[INFO]:- Balanced role = Primary 20160829:18:34:10:048215 gprecoverseg:gpm:gpadmin-[INFO]:- Current role = Mirror 20160829:18:34:10:048215 gprecoverseg:gpm:gpadmin-[INFO]:---------------------------------------------------------- 20160829:18:34:10:048215 gprecoverseg:gpm:gpadmin-[INFO]:-Unbalanced segment 13 of 18 20160829:18:34:10:048215 gprecoverseg:gpm:gpadmin-[INFO]:---------------------------------------------------------- 20160829:18:34:10:048215 gprecoverseg:gpm:gpadmin-[INFO]:- Unbalanced instance host = gps3 20160829:18:34:10:048215 gprecoverseg:gpm:gpadmin-[INFO]:- Unbalanced instance address = gps3 20160829:18:34:10:048215 gprecoverseg:gpm:gpadmin-[INFO]:- Unbalanced instance directory = /hdfs/data7/gpdata/mirror/gpseg6 20160829:18:34:10:048215 gprecoverseg:gpm:gpadmin-[INFO]:- Unbalanced instance port = 50002 20160829:18:34:10:048215 gprecoverseg:gpm:gpadmin-[INFO]:- Unbalanced instance replication port = 51002 20160829:18:34:10:048215 gprecoverseg:gpm:gpadmin-[INFO]:- Balanced role = Mirror 20160829:18:34:10:048215 gprecoverseg:gpm:gpadmin-[INFO]:- Current role = Primary 20160829:18:34:10:048215 gprecoverseg:gpm:gpadmin-[INFO]:---------------------------------------------------------- 20160829:18:34:10:048215 gprecoverseg:gpm:gpadmin-[INFO]:-Unbalanced segment 14 of 18 20160829:18:34:10:048215 gprecoverseg:gpm:gpadmin-[INFO]:---------------------------------------------------------- 20160829:18:34:10:048215 gprecoverseg:gpm:gpadmin-[INFO]:- Unbalanced instance host = gps2 20160829:18:34:10:048215 gprecoverseg:gpm:gpadmin-[INFO]:- Unbalanced instance address = gps2 20160829:18:34:10:048215 gprecoverseg:gpm:gpadmin-[INFO]:- Unbalanced instance directory = /hdfs/data3/gpdata/primary/gpseg6 20160829:18:34:10:048215 gprecoverseg:gpm:gpadmin-[INFO]:- Unbalanced instance port = 40002 20160829:18:34:10:048215 gprecoverseg:gpm:gpadmin-[INFO]:- Unbalanced instance replication port = 41002 20160829:18:34:10:048215 gprecoverseg:gpm:gpadmin-[INFO]:- Balanced role = Primary 20160829:18:34:10:048215 gprecoverseg:gpm:gpadmin-[INFO]:- Current role = Mirror 20160829:18:34:10:048215 gprecoverseg:gpm:gpadmin-[INFO]:---------------------------------------------------------- 20160829:18:34:10:048215 gprecoverseg:gpm:gpadmin-[INFO]:-Unbalanced segment 15 of 18 20160829:18:34:10:048215 gprecoverseg:gpm:gpadmin-[INFO]:---------------------------------------------------------- 20160829:18:34:10:048215 gprecoverseg:gpm:gpadmin-[INFO]:- Unbalanced instance host = gps3 20160829:18:34:10:048215 gprecoverseg:gpm:gpadmin-[INFO]:- Unbalanced instance address = gps3 20160829:18:34:10:048215 gprecoverseg:gpm:gpadmin-[INFO]:- Unbalanced instance directory = /hdfs/data8/gpdata/mirror/gpseg7 20160829:18:34:10:048215 gprecoverseg:gpm:gpadmin-[INFO]:- Unbalanced instance port = 50003 20160829:18:34:10:048215 gprecoverseg:gpm:gpadmin-[INFO]:- Unbalanced instance replication port = 51003 20160829:18:34:10:048215 gprecoverseg:gpm:gpadmin-[INFO]:- Balanced role = Mirror 20160829:18:34:10:048215 gprecoverseg:gpm:gpadmin-[INFO]:- Current role = Primary 20160829:18:34:10:048215 gprecoverseg:gpm:gpadmin-[INFO]:---------------------------------------------------------- 20160829:18:34:10:048215 gprecoverseg:gpm:gpadmin-[INFO]:-Unbalanced segment 16 of 18 20160829:18:34:10:048215 gprecoverseg:gpm:gpadmin-[INFO]:---------------------------------------------------------- 20160829:18:34:10:048215 gprecoverseg:gpm:gpadmin-[INFO]:- Unbalanced instance host = gps2 20160829:18:34:10:048215 gprecoverseg:gpm:gpadmin-[INFO]:- Unbalanced instance address = gps2 20160829:18:34:10:048215 gprecoverseg:gpm:gpadmin-[INFO]:- Unbalanced instance directory = /hdfs/data4/gpdata/primary/gpseg7 20160829:18:34:10:048215 gprecoverseg:gpm:gpadmin-[INFO]:- Unbalanced instance port = 40003 20160829:18:34:10:048215 gprecoverseg:gpm:gpadmin-[INFO]:- Unbalanced instance replication port = 41003 20160829:18:34:10:048215 gprecoverseg:gpm:gpadmin-[INFO]:- Balanced role = Primary 20160829:18:34:10:048215 gprecoverseg:gpm:gpadmin-[INFO]:- Current role = Mirror 20160829:18:34:10:048215 gprecoverseg:gpm:gpadmin-[INFO]:---------------------------------------------------------- 20160829:18:34:10:048215 gprecoverseg:gpm:gpadmin-[INFO]:-Unbalanced segment 17 of 18 20160829:18:34:10:048215 gprecoverseg:gpm:gpadmin-[INFO]:---------------------------------------------------------- 20160829:18:34:10:048215 gprecoverseg:gpm:gpadmin-[INFO]:- Unbalanced instance host = gps1 20160829:18:34:10:048215 gprecoverseg:gpm:gpadmin-[INFO]:- Unbalanced instance address = gps1 20160829:18:34:10:048215 gprecoverseg:gpm:gpadmin-[INFO]:- Unbalanced instance directory = /hdfs/data7/gpdata/mirror/gpseg10 20160829:18:34:10:048215 gprecoverseg:gpm:gpadmin-[INFO]:- Unbalanced instance port = 50002 20160829:18:34:10:048215 gprecoverseg:gpm:gpadmin-[INFO]:- Unbalanced instance replication port = 51002 20160829:18:34:10:048215 gprecoverseg:gpm:gpadmin-[INFO]:- Balanced role = Mirror 20160829:18:34:10:048215 gprecoverseg:gpm:gpadmin-[INFO]:- Current role = Primary 20160829:18:34:10:048215 gprecoverseg:gpm:gpadmin-[INFO]:---------------------------------------------------------- 20160829:18:34:10:048215 gprecoverseg:gpm:gpadmin-[INFO]:-Unbalanced segment 18 of 18 20160829:18:34:10:048215 gprecoverseg:gpm:gpadmin-[INFO]:---------------------------------------------------------- 20160829:18:34:10:048215 gprecoverseg:gpm:gpadmin-[INFO]:- Unbalanced instance host = gps3 20160829:18:34:10:048215 gprecoverseg:gpm:gpadmin-[INFO]:- Unbalanced instance address = gps3 20160829:18:34:10:048215 gprecoverseg:gpm:gpadmin-[INFO]:- Unbalanced instance directory = /hdfs/data3/gpdata/primary/gpseg10 20160829:18:34:10:048215 gprecoverseg:gpm:gpadmin-[INFO]:- Unbalanced instance port = 40002 20160829:18:34:10:048215 gprecoverseg:gpm:gpadmin-[INFO]:- Unbalanced instance replication port = 41002 20160829:18:34:10:048215 gprecoverseg:gpm:gpadmin-[INFO]:- Balanced role = Primary 20160829:18:34:10:048215 gprecoverseg:gpm:gpadmin-[INFO]:- Current role = Mirror 20160829:18:34:10:048215 gprecoverseg:gpm:gpadmin-[INFO]:---------------------------------------------------------- 20160829:18:34:10:048215 gprecoverseg:gpm:gpadmin-[WARNING]:-This operation will cancel queries that are currently executing. 20160829:18:34:10:048215 gprecoverseg:gpm:gpadmin-[WARNING]:-Connections to the database however will not be interrupted. Continue with segment rebalance procedure Yy|Nn (default=N): > y 20160829:18:34:11:048215 gprecoverseg:gpm:gpadmin-[INFO]:-Getting unbalanced segments 20160829:18:34:11:048215 gprecoverseg:gpm:gpadmin-[INFO]:-Stopping unbalanced primary segments... .. 20160829:18:34:13:048215 gprecoverseg:gpm:gpadmin-[INFO]:-Triggering segment reconfiguration 20160829:18:34:18:048215 gprecoverseg:gpm:gpadmin-[INFO]:-Starting segment synchronization ............. 20160829:18:34:31:048215 gprecoverseg:gpm:gpadmin-[INFO]:-****************************************************************** 20160829:18:34:31:048215 gprecoverseg:gpm:gpadmin-[INFO]:-The rebalance operation has completed successfully. 20160829:18:34:31:048215 gprecoverseg:gpm:gpadmin-[INFO]:-There is a resynchronization running in the background to bring all 20160829:18:34:31:048215 gprecoverseg:gpm:gpadmin-[INFO]:-segments in sync. 20160829:18:34:31:048215 gprecoverseg:gpm:gpadmin-[INFO]:- 20160829:18:34:31:048215 gprecoverseg:gpm:gpadmin-[INFO]:-Use gpstate -e to check the resynchronization progress. 20160829:18:34:31:048215 gprecoverseg:gpm:gpadmin-[INFO]:-******************************************************************
查看同步状态
[gpadmin@gpm ~]$ gpstate -e 20160829:18:34:47:048559 gpstate:gpm:gpadmin-[INFO]:-Starting gpstate with args: -e 20160829:18:34:48:048559 gpstate:gpm:gpadmin-[INFO]:-local Greenplum Version: 'postgres (Greenplum Database) 4.3.8.2 build 1' 20160829:18:34:48:048559 gpstate:gpm:gpadmin-[INFO]:-master Greenplum Version: 'PostgreSQL 8.2.15 (Greenplum Database 4.3.8.2 build 1) on x86_64-unknown-linux-gnu, compiled by GCC gcc (GCC) 4.4.2 compiled on May 9 2016 22:28:40' 20160829:18:34:48:048559 gpstate:gpm:gpadmin-[INFO]:-Obtaining Segment details from master... 20160829:18:34:48:048559 gpstate:gpm:gpadmin-[INFO]:-Gathering data from segments... . 20160829:18:34:49:048559 gpstate:gpm:gpadmin-[INFO]:----------------------------------------------------- 20160829:18:34:49:048559 gpstate:gpm:gpadmin-[INFO]:-Segment Mirroring Status Report 20160829:18:34:49:048559 gpstate:gpm:gpadmin-[INFO]:----------------------------------------------------- 20160829:18:34:49:048559 gpstate:gpm:gpadmin-[INFO]:-Segment Pairs in Resynchronization 20160829:18:34:49:048559 gpstate:gpm:gpadmin-[INFO]:- Current Primary Port Resync mode Est. resync progress Total resync objects Objects to resync Data synced Est. total to sync Est. resync end time Change tracking size Mirror Port 20160829:18:34:49:048559 gpstate:gpm:gpadmin-[INFO]:- gps1 40000 Incremental 100% 0 0 384 kB Sync complete; awaiting config change 128 bytes gps2 50000 20160829:18:34:49:048559 gpstate:gpm:gpadmin-[INFO]:- gps1 40001 Incremental 100% 0 0 384 kB Sync complete; awaiting config change 128 bytes gps2 50001 20160829:18:34:49:048559 gpstate:gpm:gpadmin-[INFO]:- gps1 40002 Incremental 100% 0 0 384 kB Sync complete; awaiting config change 128 bytes gps2 50002 20160829:18:34:49:048559 gpstate:gpm:gpadmin-[INFO]:- gps1 40003 Incremental 100% 0 0 384 kB Sync complete; awaiting config change 128 bytes gps2 50003 20160829:18:34:49:048559 gpstate:gpm:gpadmin-[INFO]:- gps2 40000 Incremental 100% 0 0 384 kB Sync complete; awaiting config change 128 bytes gps3 50000 20160829:18:34:49:048559 gpstate:gpm:gpadmin-[INFO]:- gps2 40001 Incremental 100% 0 0 384 kB Sync complete; awaiting config change 128 bytes gps3 50001 20160829:18:34:49:048559 gpstate:gpm:gpadmin-[INFO]:- gps2 40002 Incremental 100% 0 0 384 kB Sync complete; awaiting config change 128 bytes gps3 50002 20160829:18:34:49:048559 gpstate:gpm:gpadmin-[INFO]:- gps2 40003 Incremental 100% 0 0 384 kB Sync complete; awaiting config change 128 bytes gps3 50003 20160829:18:34:49:048559 gpstate:gpm:gpadmin-[INFO]:- gps3 40002 Incremental 100% 0 0 384 kB Sync complete; awaiting config change 128 bytes gps1 50002
[gpadmin@gpm ~]$ gpstate -m 20160829:18:35:02:048639 gpstate:gpm:gpadmin-[INFO]:-Starting gpstate with args: -m 20160829:18:35:02:048639 gpstate:gpm:gpadmin-[INFO]:-local Greenplum Version: 'postgres (Greenplum Database) 4.3.8.2 build 1' 20160829:18:35:02:048639 gpstate:gpm:gpadmin-[INFO]:-master Greenplum Version: 'PostgreSQL 8.2.15 (Greenplum Database 4.3.8.2 build 1) on x86_64-unknown-linux-gnu, compiled by GCC gcc (GCC) 4.4.2 compiled on May 9 2016 22:28:40' 20160829:18:35:02:048639 gpstate:gpm:gpadmin-[INFO]:-Obtaining Segment details from master... 20160829:18:35:02:048639 gpstate:gpm:gpadmin-[INFO]:-------------------------------------------------------------- 20160829:18:35:02:048639 gpstate:gpm:gpadmin-[INFO]:--Current GPDB mirror list and status 20160829:18:35:02:048639 gpstate:gpm:gpadmin-[INFO]:--Type = Group 20160829:18:35:02:048639 gpstate:gpm:gpadmin-[INFO]:-------------------------------------------------------------- 20160829:18:35:02:048639 gpstate:gpm:gpadmin-[INFO]:- Mirror Datadir Port Status Data Status 20160829:18:35:02:048639 gpstate:gpm:gpadmin-[INFO]:- gps2 /hdfs/data5/gpdata/mirror/gpseg0 50000 Passive Resynchronizing 20160829:18:35:02:048639 gpstate:gpm:gpadmin-[INFO]:- gps2 /hdfs/data6/gpdata/mirror/gpseg1 50001 Passive Resynchronizing 20160829:18:35:02:048639 gpstate:gpm:gpadmin-[INFO]:- gps2 /hdfs/data7/gpdata/mirror/gpseg2 50002 Passive Resynchronizing 20160829:18:35:02:048639 gpstate:gpm:gpadmin-[INFO]:- gps2 /hdfs/data8/gpdata/mirror/gpseg3 50003 Passive Resynchronizing 20160829:18:35:02:048639 gpstate:gpm:gpadmin-[INFO]:- gps3 /hdfs/data5/gpdata/mirror/gpseg4 50000 Passive Resynchronizing 20160829:18:35:02:048639 gpstate:gpm:gpadmin-[INFO]:- gps3 /hdfs/data6/gpdata/mirror/gpseg5 50001 Passive Resynchronizing 20160829:18:35:02:048639 gpstate:gpm:gpadmin-[INFO]:- gps3 /hdfs/data7/gpdata/mirror/gpseg6 50002 Passive Resynchronizing 20160829:18:35:02:048639 gpstate:gpm:gpadmin-[INFO]:- gps3 /hdfs/data8/gpdata/mirror/gpseg7 50003 Passive Resynchronizing 20160829:18:35:02:048639 gpstate:gpm:gpadmin-[INFO]:- gps1 /hdfs/data5/gpdata/mirror/gpseg8 50000 Passive Synchronized 20160829:18:35:02:048639 gpstate:gpm:gpadmin-[INFO]:- gps1 /hdfs/data6/gpdata/mirror/gpseg9 50001 Passive Synchronized 20160829:18:35:02:048639 gpstate:gpm:gpadmin-[INFO]:- gps1 /hdfs/data7/gpdata/mirror/gpseg10 50002 Passive Resynchronizing 20160829:18:35:02:048639 gpstate:gpm:gpadmin-[INFO]:- gps1 /hdfs/data8/gpdata/mirror/gpseg11 50003 Passive Synchronized 20160829:18:35:02:048639 gpstate:gpm:gpadmin-[INFO]:-------------------------------------------------------------- [gpadmin@gpm ~]$ gpstate -m 20160829:18:36:01:048898 gpstate:gpm:gpadmin-[INFO]:-Starting gpstate with args: -m 20160829:18:36:01:048898 gpstate:gpm:gpadmin-[INFO]:-local Greenplum Version: 'postgres (Greenplum Database) 4.3.8.2 build 1' 20160829:18:36:01:048898 gpstate:gpm:gpadmin-[INFO]:-master Greenplum Version: 'PostgreSQL 8.2.15 (Greenplum Database 4.3.8.2 build 1) on x86_64-unknown-linux-gnu, compiled by GCC gcc (GCC) 4.4.2 compiled on May 9 2016 22:28:40' 20160829:18:36:01:048898 gpstate:gpm:gpadmin-[INFO]:-Obtaining Segment details from master... 20160829:18:36:01:048898 gpstate:gpm:gpadmin-[INFO]:-------------------------------------------------------------- 20160829:18:36:01:048898 gpstate:gpm:gpadmin-[INFO]:--Current GPDB mirror list and status 20160829:18:36:01:048898 gpstate:gpm:gpadmin-[INFO]:--Type = Group 20160829:18:36:01:048898 gpstate:gpm:gpadmin-[INFO]:-------------------------------------------------------------- 20160829:18:36:01:048898 gpstate:gpm:gpadmin-[INFO]:- Mirror Datadir Port Status Data Status 20160829:18:36:01:048898 gpstate:gpm:gpadmin-[INFO]:- gps2 /hdfs/data5/gpdata/mirror/gpseg0 50000 Passive Synchronized 20160829:18:36:01:048898 gpstate:gpm:gpadmin-[INFO]:- gps2 /hdfs/data6/gpdata/mirror/gpseg1 50001 Passive Synchronized 20160829:18:36:01:048898 gpstate:gpm:gpadmin-[INFO]:- gps2 /hdfs/data7/gpdata/mirror/gpseg2 50002 Passive Synchronized 20160829:18:36:01:048898 gpstate:gpm:gpadmin-[INFO]:- gps2 /hdfs/data8/gpdata/mirror/gpseg3 50003 Passive Synchronized 20160829:18:36:01:048898 gpstate:gpm:gpadmin-[INFO]:- gps3 /hdfs/data5/gpdata/mirror/gpseg4 50000 Passive Synchronized 20160829:18:36:01:048898 gpstate:gpm:gpadmin-[INFO]:- gps3 /hdfs/data6/gpdata/mirror/gpseg5 50001 Passive Synchronized 20160829:18:36:01:048898 gpstate:gpm:gpadmin-[INFO]:- gps3 /hdfs/data7/gpdata/mirror/gpseg6 50002 Passive Synchronized 20160829:18:36:01:048898 gpstate:gpm:gpadmin-[INFO]:- gps3 /hdfs/data8/gpdata/mirror/gpseg7 50003 Passive Synchronized 20160829:18:36:01:048898 gpstate:gpm:gpadmin-[INFO]:- gps1 /hdfs/data5/gpdata/mirror/gpseg8 50000 Passive Synchronized 20160829:18:36:01:048898 gpstate:gpm:gpadmin-[INFO]:- gps1 /hdfs/data6/gpdata/mirror/gpseg9 50001 Passive Synchronized 20160829:18:36:01:048898 gpstate:gpm:gpadmin-[INFO]:- gps1 /hdfs/data7/gpdata/mirror/gpseg10 50002 Passive Synchronized 20160829:18:36:01:048898 gpstate:gpm:gpadmin-[INFO]:- gps1 /hdfs/data8/gpdata/mirror/gpseg11 50003 Passive Synchronized 20160829:18:36:01:048898 gpstate:gpm:gpadmin-[INFO]:--------------------------------------------------------------
检查恢复结果
可以看到primary 和mirror均没有失败的segment了，恢复成功。
[gpadmin@gpm ~]$ gpstate 20160829:18:36:10:049017 gpstate:gpm:gpadmin-[INFO]:-Starting gpstate with args: 20160829:18:36:10:049017 gpstate:gpm:gpadmin-[INFO]:-local Greenplum Version: 'postgres (Greenplum Database) 4.3.8.2 build 1' 20160829:18:36:10:049017 gpstate:gpm:gpadmin-[INFO]:-master Greenplum Version: 'PostgreSQL 8.2.15 (Greenplum Database 4.3.8.2 build 1) on x86_64-unknown-linux-gnu, compiled by GCC gcc (GCC) 4.4.2 compiled on May 9 2016 22:28:40' 20160829:18:36:10:049017 gpstate:gpm:gpadmin-[INFO]:-Obtaining Segment details from master... 20160829:18:36:11:049017 gpstate:gpm:gpadmin-[INFO]:-Gathering data from segments... . 20160829:18:36:12:049017 gpstate:gpm:gpadmin-[INFO]:-Greenplum instance status summary 20160829:18:36:12:049017 gpstate:gpm:gpadmin-[INFO]:----------------------------------------------------- 20160829:18:36:12:049017 gpstate:gpm:gpadmin-[INFO]:- Master instance = Active 20160829:18:36:12:049017 gpstate:gpm:gpadmin-[INFO]:- Master standby = gps1 20160829:18:36:12:049017 gpstate:gpm:gpadmin-[INFO]:- Standby master state = Standby host passive 20160829:18:36:12:049017 gpstate:gpm:gpadmin-[INFO]:- Total segment instance count from metadata = 24 20160829:18:36:12:049017 gpstate:gpm:gpadmin-[INFO]:----------------------------------------------------- 20160829:18:36:12:049017 gpstate:gpm:gpadmin-[INFO]:- Primary Segment Status 20160829:18:36:12:049017 gpstate:gpm:gpadmin-[INFO]:----------------------------------------------------- 20160829:18:36:12:049017 gpstate:gpm:gpadmin-[INFO]:- Total primary segments = 12 20160829:18:36:12:049017 gpstate:gpm:gpadmin-[INFO]:- Total primary segment valid (at master) = 12 20160829:18:36:12:049017 gpstate:gpm:gpadmin-[INFO]:- Total primary segment failures (at master) = 0 20160829:18:36:12:049017 gpstate:gpm:gpadmin-[INFO]:- Total number of postmaster.pid files missing = 0 20160829:18:36:12:049017 gpstate:gpm:gpadmin-[INFO]:- Total number of postmaster.pid files found = 12 20160829:18:36:12:049017 gpstate:gpm:gpadmin-[INFO]:- Total number of postmaster.pid PIDs missing = 0 20160829:18:36:12:049017 gpstate:gpm:gpadmin-[INFO]:- Total number of postmaster.pid PIDs found = 12 20160829:18:36:12:049017 gpstate:gpm:gpadmin-[INFO]:- Total number of /tmp lock files missing = 0 20160829:18:36:12:049017 gpstate:gpm:gpadmin-[INFO]:- Total number of /tmp lock files found = 12 20160829:18:36:12:049017 gpstate:gpm:gpadmin-[INFO]:- Total number postmaster processes missing = 0 20160829:18:36:12:049017 gpstate:gpm:gpadmin-[INFO]:- Total number postmaster processes found = 12 20160829:18:36:12:049017 gpstate:gpm:gpadmin-[INFO]:----------------------------------------------------- 20160829:18:36:12:049017 gpstate:gpm:gpadmin-[INFO]:- Mirror Segment Status 20160829:18:36:12:049017 gpstate:gpm:gpadmin-[INFO]:----------------------------------------------------- 20160829:18:36:12:049017 gpstate:gpm:gpadmin-[INFO]:- Total mirror segments = 12 20160829:18:36:12:049017 gpstate:gpm:gpadmin-[INFO]:- Total mirror segment valid (at master) = 12 20160829:18:36:12:049017 gpstate:gpm:gpadmin-[INFO]:- Total mirror segment failures (at master) = 0 20160829:18:36:12:049017 gpstate:gpm:gpadmin-[INFO]:- Total number of postmaster.pid files missing = 0 20160829:18:36:12:049017 gpstate:gpm:gpadmin-[INFO]:- Total number of postmaster.pid files found = 12 20160829:18:36:12:049017 gpstate:gpm:gpadmin-[INFO]:- Total number of postmaster.pid PIDs missing = 0 20160829:18:36:12:049017 gpstate:gpm:gpadmin-[INFO]:- Total number of postmaster.pid PIDs found = 12 20160829:18:36:12:049017 gpstate:gpm:gpadmin-[INFO]:- Total number of /tmp lock files missing = 0 20160829:18:36:12:049017 gpstate:gpm:gpadmin-[INFO]:- Total number of /tmp lock files found = 12 20160829:18:36:12:049017 gpstate:gpm:gpadmin-[INFO]:- Total number postmaster processes missing = 0 20160829:18:36:12:049017 gpstate:gpm:gpadmin-[INFO]:- Total number postmaster processes found = 12 20160829:18:36:12:049017 gpstate:gpm:gpadmin-[INFO]:- Total number mirror segments acting as primary segments = 0 20160829:18:36:12:049017 gpstate:gpm:gpadmin-[INFO]:- Total number mirror segments acting as mirror segments = 12 20160829:18:36:12:049017 gpstate:gpm:gpadmin-[INFO]:-----------------------------------------------------
[gpadmin@gpm ~]$ gpstate -e 20160829:19:14:45:008450 gpstate:gpm:gpadmin-[INFO]:-Starting gpstate with args: -e 20160829:19:14:45:008450 gpstate:gpm:gpadmin-[INFO]:-local Greenplum Version: 'postgres (Greenplum Database) 4.3.8.2 build 1' 20160829:19:14:45:008450 gpstate:gpm:gpadmin-[INFO]:-master Greenplum Version: 'PostgreSQL 8.2.15 (Greenplum Database 4.3.8.2 build 1) on x86_64-unknown-linux-gnu, compiled by GCC gcc (GCC) 4.4.2 compiled on May 9 2016 22:28:40' 20160829:19:14:45:008450 gpstate:gpm:gpadmin-[INFO]:-Obtaining Segment details from master... 20160829:19:14:46:008450 gpstate:gpm:gpadmin-[INFO]:-Gathering data from segments... . 20160829:19:14:47:008450 gpstate:gpm:gpadmin-[INFO]:----------------------------------------------------- 20160829:19:14:47:008450 gpstate:gpm:gpadmin-[INFO]:-Segment Mirroring Status Report 20160829:19:14:47:008450 gpstate:gpm:gpadmin-[INFO]:----------------------------------------------------- 20160829:19:14:47:008450 gpstate:gpm:gpadmin-[INFO]:-All segments are running normally
[gpadmin@gpm ~]$ gpstate -c 20160829:19:15:28:008817 gpstate:gpm:gpadmin-[INFO]:-Starting gpstate with args: -c 20160829:19:15:28:008817 gpstate:gpm:gpadmin-[INFO]:-local Greenplum Version: 'postgres (Greenplum Database) 4.3.8.2 build 1' 20160829:19:15:28:008817 gpstate:gpm:gpadmin-[INFO]:-master Greenplum Version: 'PostgreSQL 8.2.15 (Greenplum Database 4.3.8.2 build 1) on x86_64-unknown-linux-gnu, compiled by GCC gcc (GCC) 4.4.2 compiled on May 9 2016 22:28:40' 20160829:19:15:28:008817 gpstate:gpm:gpadmin-[INFO]:-Obtaining Segment details from master... 20160829:19:15:28:008817 gpstate:gpm:gpadmin-[INFO]:-------------------------------------------------------------- 20160829:19:15:28:008817 gpstate:gpm:gpadmin-[INFO]:--Current GPDB mirror list and status 20160829:19:15:28:008817 gpstate:gpm:gpadmin-[INFO]:--Type = Group 20160829:19:15:28:008817 gpstate:gpm:gpadmin-[INFO]:-------------------------------------------------------------- 20160829:19:15:28:008817 gpstate:gpm:gpadmin-[INFO]:- Status Data State Primary Datadir Port Mirror Datadir Port 20160829:19:15:28:008817 gpstate:gpm:gpadmin-[INFO]:- Primary Active, Mirror Available Synchronized gps1 /hdfs/data1/gpdata/primary/gpseg0 40000 gps2 /hdfs/data5/gpdata/mirror/gpseg0 50000 20160829:19:15:28:008817 gpstate:gpm:gpadmin-[INFO]:- Primary Active, Mirror Available Synchronized gps1 /hdfs/data2/gpdata/primary/gpseg1 40001 gps2 /hdfs/data6/gpdata/mirror/gpseg1 50001 20160829:19:15:28:008817 gpstate:gpm:gpadmin-[INFO]:- Primary Active, Mirror Available Synchronized gps1 /hdfs/data3/gpdata/primary/gpseg2 40002 gps2 /hdfs/data7/gpdata/mirror/gpseg2 50002 20160829:19:15:28:008817 gpstate:gpm:gpadmin-[INFO]:- Primary Active, Mirror Available Synchronized gps1 /hdfs/data4/gpdata/primary/gpseg3 40003 gps2 /hdfs/data8/gpdata/mirror/gpseg3 50003 20160829:19:15:28:008817 gpstate:gpm:gpadmin-[INFO]:- Primary Active, Mirror Available Synchronized gps2 /hdfs/data1/gpdata/primary/gpseg4 40000 gps3 /hdfs/data5/gpdata/mirror/gpseg4 50000 20160829:19:15:28:008817 gpstate:gpm:gpadmin-[INFO]:- Primary Active, Mirror Available Synchronized gps2 /hdfs/data2/gpdata/primary/gpseg5 40001 gps3 /hdfs/data6/gpdata/mirror/gpseg5 50001 20160829:19:15:28:008817 gpstate:gpm:gpadmin-[INFO]:- Primary Active, Mirror Available Synchronized gps2 /hdfs/data3/gpdata/primary/gpseg6 40002 gps3 /hdfs/data7/gpdata/mirror/gpseg6 50002 20160829:19:15:28:008817 gpstate:gpm:gpadmin-[INFO]:- Primary Active, Mirror Available Synchronized gps2 /hdfs/data4/gpdata/primary/gpseg7 40003 gps3 /hdfs/data8/gpdata/mirror/gpseg7 50003 20160829:19:15:28:008817 gpstate:gpm:gpadmin-[INFO]:- Primary Active, Mirror Available Synchronized gps3 /hdfs/data1/gpdata/primary/gpseg8 40000 gps1 /hdfs/data5/gpdata/mirror/gpseg8 50000 20160829:19:15:28:008817 gpstate:gpm:gpadmin-[INFO]:- Primary Active, Mirror Available Synchronized gps3 /hdfs/data2/gpdata/primary/gpseg9 40001 gps1 /hdfs/data6/gpdata/mirror/gpseg9 50001 20160829:19:15:28:008817 gpstate:gpm:gpadmin-[INFO]:- Primary Active, Mirror Available Synchronized gps3 /hdfs/data3/gpdata/primary/gpseg10 40002 gps1 /hdfs/data7/gpdata/mirror/gpseg10 50002 20160829:19:15:28:008817 gpstate:gpm:gpadmin-[INFO]:- Primary Active, Mirror Available Synchronized gps3 /hdfs/data4/gpdata/primary/gpseg11 40003 gps1 /hdfs/data8/gpdata/mirror/gpseg11 50003 20160829:19:15:28:008817 gpstate:gpm:gpadmin-[INFO]:--------------------------------------------------------------
三、遗留问题
按以上操作成功恢复了GP集群失败segment，但对恢复工具机制还需进一步了解以上只是部分segment失败、集群仍能正常运行场景的恢复，对其他场景(如GP集群不能正常工作)的恢复，尚未遇到。
参考资料：
http://gpdb.docs.pivotal.io/4320/pdf/GPDB43AdminGuide.pdf
http://blog.chinaunix.net/uid-23284114-id-5598576.html
内容来自用户分享和网络整理，不保证内容的准确性，如有侵权内容，可联系管理员处理 
相关文章推荐
- SQL2005安装错误修复－－对性能监视器计数器注册表值执行系统配置检查失败
- ES 内存使用和GC指标——主节点每30秒会去检查其他节点的状态，如果任何节点的垃圾回收时间超过30秒（Garbage collection duration），则会导致主节点任务该节点脱离集群。
- 从GREENPLUM集群中去除某个SEGMENT
- 事件代码: 4009 事件消息: 视图状态验证失败。 原因: 提供的视图状态未通过完整性检查。
- 开启Healthcheck检查redis连接失败导致注册Eureka状态是DOWN
- Greenplum或DeepGreen中检查资源队列状态
- 如何检查失败的Segment/master
- LVS集群RS健康状态检查
- oracle 11G rac 检查集群组件的状态和配置
- hbase hbck修复集群状态
- greenplum恢复失败的segment的方法
- 检查邮件服务器的postfix-pop3-mysql服务状态的脚本
- JS检查当图片不存在时显示默认图片和键盘大小写键状态
- 验证视图状态 MAC 失败。如果此应用程序由网络场或群集承载,请确保 <machineKey>
- SQLServer Always On FCI 脑裂及可疑状态修复
- SQL 2005 Report Server 请求因 HTTP 状态 400 失败: Bad Request 处理方法 的解决方法
- ORACLE ADO连接失败的修复
- 验证视图状态 MAC 失败。如果此应用程序由网络场或群集承载，请确保 <machineKey> 配置指定了相同的 validationKey 和验证算法。不能在群集中使用 AutoGenerate。
- zabbix通过简单命令监控elasticsearch集群状态
- Tengine笔记3：Nginx的反向代理和健康状态检查