rac重新启动遭遇ORA-01078、ORA-01565、ORA-17503、ORA-12547

今天測试环境server重新启动导致一个节点集群无法重新启动，遭遇ORA-12547错误。详细例如以下：

server重新启动后，rac1集群无法启动，rac2正常启动：

[root@rac1 ~]# ps -ef|grep ora

grid 3975 1 0 16:42 ?

00:00:00 /u01/grid11g/bin/oraagent.bin

root 3998 3692 0 16:42 pts/0 00:00:00 grep ora

[grid@rac2 ~]$ crs_stat -t

Name Type Target State Host

------------------------------------------------------------

ora.ARCH.dg ora....up.type ONLINE ONLINE rac2

ora.DATA.dg ora....up.type ONLINE ONLINE rac2

ora....ER.lsnr ora....er.type ONLINE ONLINE rac2

ora....N1.lsnr ora....er.type ONLINE ONLINE rac2

ora.OCR.dg ora....up.type ONLINE ONLINE rac2

ora.asm ora.asm.type ONLINE ONLINE rac2

ora.cvu ora.cvu.type ONLINE ONLINE rac2

ora.gsd ora.gsd.type OFFLINE OFFLINE

ora....network ora....rk.type ONLINE ONLINE rac2

ora.oc4j ora.oc4j.type ONLINE ONLINE rac2

ora.ons ora.ons.type ONLINE ONLINE rac2

ora.rac1.vip ora....t1.type ONLINE ONLINE rac2

ora....SM2.asm application ONLINE ONLINE rac2

ora....C2.lsnr application ONLINE ONLINE rac2

ora.rac2.gsd application OFFLINE OFFLINE

ora.rac2.ons application ONLINE ONLINE rac2

ora.rac2.vip ora....t1.type ONLINE ONLINE rac2

ora.racdb.db ora....se.type ONLINE ONLINE rac2

ora.scan1.vip ora....ip.type ONLINE ONLINE rac2

[oracle@rac1 ~]$ sqlplus / as sysdba

SQL*Plus: Release 11.2.0.3.0 Production on Tue Dec 16 16:47:58 2014

Copyright (c) 1982, 2011, Oracle. All rights reserved.

ERROR:

ORA-12547: TNS:lost contact

Enter user-name: exit

Enter password:

[oracle@rac1 ~]$

遭遇ORA-12547错误。一般为权限问题，发现集群文件夹权限变了。可能有人为错误导致：

[root@rac1 cdata]# cd /u01/

[root@rac1 u01]# ls -lhrt

total 40K

drwx------ 2 mongodb mongodb 16K Aug 14 2013 lost+found

drwxr-xr-x 8 mongodb mongodb 4.0K Aug 19 2013 grid_anzhuangbao

drwxr-xr-x 66 mongodb mongodb 4.0K Aug 20 2013 grid11g

drwxrwx--- 6 grid oinstall 4.0K Aug 20 2013 oraInventory

drwxr-xr-x 9 mongodb mongodb 4.0K Dec 25 2013 grid

drwxr-xr-x 5 mongodb mongodb 4.0K Jan 6 2014 mongodb

drwxr-xr-x 9 oracle oinstall 4.0K Nov 27 14:45 oracle

更改权限为合适的权限：

[root@rac1 u01]# chown grid:oinstall grid -R

[root@rac1 u01]# chown grid:oinstall grid11g -R

[root@rac1 u01]# ls -lhrt

total 40K

drwx------ 2 mongodb mongodb 16K Aug 14 2013 lost+found

drwxr-xr-x 8 mongodb mongodb 4.0K Aug 19 2013 grid_anzhuangbao

drwxr-xr-x 66 grid oinstall 4.0K Aug 20 2013 grid11g

drwxrwx--- 6 grid oinstall 4.0K Aug 20 2013 oraInventory

drwxr-xr-x 9 grid oinstall 4.0K Dec 25 2013 grid

drwxr-xr-x 5 mongodb mongodb 4.0K Jan 6 2014 mongodb

drwxr-xr-x 9 oracle oinstall 4.0K Nov 27 14:45 oracle

重新启动机器：

[root@rac1 u01]# reboot

集群起来。可是实例没有启动：

[oracle@rac1 ~]$ ps -ef|grep ora

grid 3699 1 0 17:03 ? 00:00:00 /u01/grid11g/bin/oraagent.bin

root 3845 1 0 17:03 ? 00:00:00 /u01/grid11g/bin/orarootagent.bin

grid 4017 1 0 17:03 ? 00:00:00 oracle+ASM1 (DESCRIPTION=(LOCAL=YES)(ADDRESS=(PROTOCOL=beq)))

grid 4032 1 0 17:03 ? 00:00:00 oracle+ASM1_ocr (DESCRIPTION=(LOCAL=YES)(ADDRESS=(PROTOCOL=beq)))

grid 4036 1 0 17:03 ?

00:00:00 oracle+ASM1_asmb_+asm1 (DESCRIPTION=(LOCAL=YES)(ADDRESS=(PROTOCOL=beq)))

grid 4040 1 0 17:03 ? 00:00:00 oracle+ASM1_o000_+asm1 (DESCRIPTION=(LOCAL=YES)(ADDRESS=(PROTOCOL=beq)))

grid 4160 1 0 17:04 ? 00:00:00 /u01/grid11g/bin/oraagent.bin

root 4164 1 0 17:04 ? 00:00:00 /u01/grid11g/bin/orarootagent.bin

grid 4210 1 0 17:04 ? 00:00:00 oracle+ASM1 (DESCRIPTION=(LOCAL=YES)(ADDRESS=(PROTOCOL=beq)))

grid 4212 1 0 17:04 ? 00:00:00 oracle+ASM1 (DESCRIPTION=(LOCAL=YES)(ADDRESS=(PROTOCOL=beq)))

grid 4214 1 0 17:04 ? 00:00:00 oracle+ASM1 (DESCRIPTION=(LOCAL=YES)(ADDRESS=(PROTOCOL=beq)))

root 4286 4052 0 17:04 pts/1 00:00:00 su - oracle

oracle 4287 4286 0 17:04 pts/1 00:00:00 -bash

oracle 4887 4287 0 17:05 pts/1 00:00:00 ps -ef

oracle 4888 4287 0 17:05 pts/1 00:00:00 grep ora

[grid@rac2 u01]$ crs_stat -t -v

Name Type R/RA F/FT Target State Host

----------------------------------------------------------------------

ora.ARCH.dg ora....up.type 0/5 0/ ONLINE ONLINE rac1

ora.DATA.dg ora....up.type 0/5 0/ ONLINE ONLINE rac1

ora....ER.lsnr ora....er.type 0/5 0/ ONLINE ONLINE rac1

ora....N1.lsnr ora....er.type 0/5 0/0 ONLINE ONLINE rac2

ora.OCR.dg ora....up.type 0/5 0/ ONLINE ONLINE rac1

ora.asm ora.asm.type 0/5 0/ ONLINE ONLINE rac1

ora.cvu ora.cvu.type 0/5 0/0 ONLINE ONLINE rac2

ora.gsd ora.gsd.type 0/5 0/ OFFLINE OFFLINE

ora....network ora....rk.type 0/5 0/ ONLINE ONLINE rac1

ora.oc4j ora.oc4j.type 0/1 0/2 ONLINE ONLINE rac2

ora.ons ora.ons.type 0/3 0/ ONLINE ONLINE rac1

ora....SM1.asm application 0/5 0/0 ONLINE ONLINE rac1

ora....C1.lsnr application 0/5 0/0 ONLINE ONLINE rac1

ora.rac1.gsd application 0/5 0/0 OFFLINE OFFLINE

ora.rac1.ons application 0/3 0/0 ONLINE ONLINE rac1

ora.rac1.vip ora....t1.type 0/0 0/0 ONLINE ONLINE rac1

ora....SM2.asm application 0/5 0/0 ONLINE ONLINE rac2

ora....C2.lsnr application 0/5 0/0 ONLINE ONLINE rac2

ora.rac2.gsd application 0/5 0/0 OFFLINE OFFLINE

ora.rac2.ons application 0/3 0/0 ONLINE ONLINE rac2

ora.rac2.vip ora....t1.type 0/0 0/0 ONLINE ONLINE rac2

ora.racdb.db ora....se.type 0/2 0/1 ONLINE ONLINE rac2

ora.scan1.vip ora....ip.type 0/0 0/0 ONLINE ONLINE rac2

手动登录数据库，依旧遭遇ORA-12547，某些文件权限可能依旧有问题：

[oracle@rac1 ~]$ sqlplus / as sysdba

SQL*Plus: Release 11.2.0.3.0 Production on Tue Dec 16 17:06:08 2014

Copyright (c) 1982, 2011, Oracle. All rights reserved.

ERROR:

ORA-12547: TNS:lost contact

Enter user-name: exit

Enter password:

[oracle@rac1 ~]$

[oracle@rac1 11g]$ cd /u01/

[oracle@rac1 u01]$ ls -lhrt

total 40K

drwx------ 2 mongodb mongodb 16K Aug 14 2013 lost+found

drwxr-xr-x 8 mongodb mongodb 4.0K Aug 19 2013 grid_anzhuangbao

drwxr-xr-x 66 grid oinstall 4.0K Aug 20 2013 grid11g

drwxrwx--- 6 grid oinstall 4.0K Aug 20 2013 oraInventory

drwxr-xr-x 9 grid oinstall 4.0K Dec 25 2013 grid

drwxr-xr-x 5 mongodb mongodb 4.0K Jan 6 2014 mongodb

drwxr-xr-x 9 oracle oinstall 4.0K Nov 27 14:45 oracle

[oracle@rac1 u01]$ cd oracle/

[oracle@rac1 oracle]$ ls -lrht

total 28K

drwxrwxr-x 11 mongodb mongodb 4.0K Aug 20 2013 diag

drwxr-xr-x 2 mongodb mongodb 4.0K Aug 20 2013 checkpoints

drwxr-x--- 3 oracle oinstall 4.0K Aug 20 2013 admin

drwxr-x--- 5 mongodb mongodb 4.0K Aug 20 2013 cfgtoollogs

drwxr-xr-x 73 mongodb mongodb 4.0K Nov 5 15:47 11g

drwxr-xr-x 2 mongodb mongodb 4.0K Nov 27 14:59 chtest

drwxr-xr-x 2 mongodb mongodb 4.0K Dec 15 18:45 expdp

ORACLE_HOME文件夹下文件权限有问题。更改权限：

[oracle@rac1 oracle]$ cd ..

[oracle@rac1 u01]$ chown oracle:oinstall -R oracle

重新启动server：

[root@rac1 u01]# reboot

[root@rac1 u01]# ps -ef|grep ora

grid 3674 1 0 17:13 ? 00:00:01 /u01/grid11g/bin/oraagent.bin

root 3956 1 0 17:13 ? 00:00:03 /u01/grid11g/bin/orarootagent.bin

grid 4139 1 0 17:14 ?

00:00:00 oracle+ASM1 (DESCRIPTION=(LOCAL=YES)(ADDRESS=(PROTOCOL=beq)))

grid 4321 1 0 17:14 ?

00:00:00 oracle+ASM1_ocr (DESCRIPTION=(LOCAL=YES)(ADDRESS=(PROTOCOL=beq)))

grid 4325 1 0 17:14 ?

00:00:00 oracle+ASM1_asmb_+asm1 (DESCRIPTION=(LOCAL=YES)(ADDRESS=(PROTOCOL=beq)))

grid 4434 1 0 17:14 ? 00:00:00 /u01/grid11g/bin/oraagent.bin

root 4441 1 0 17:14 ? 00:00:03 /u01/grid11g/bin/orarootagent.bin

grid 4482 1 0 17:14 ? 00:00:00 oracle+ASM1 (DESCRIPTION=(LOCAL=YES)(ADDRESS=(PROTOCOL=beq)))

grid 4487 1 0 17:14 ?

00:00:00 oracle+ASM1 (DESCRIPTION=(LOCAL=YES)(ADDRESS=(PROTOCOL=beq)))

grid 4498 1 0 17:14 ? 00:00:00 oracle+ASM1 (DESCRIPTION=(LOCAL=YES)(ADDRESS=(PROTOCOL=beq)))

root 5906 5595 0 17:24 pts/0 00:00:00 grep ora

奇怪。实例依旧无法重新启动，sqlplus 连接不再报12547，尝试启动实例。报错例如以下：

[oracle@rac1 ~]$ sqlplus / as sysdba

SQL*Plus: Release 11.2.0.3.0 Production on Tue Dec 16 17:48:12 2014

Copyright (c) 1982, 2011, Oracle. All rights reserved.

Connected to an idle instance.

SQL> startup nomount

ORA-01078: failure in processing system parameters

ORA-01565: error in identifying file '+DATA/RACDB/spfileRACDB.ora'

ORA-17503: ksfdopn:2 Failed to open file +DATA/RACDB/spfileRACDB.ora

ORA-12547: TNS:lost contact

SQL> exit

Disconnected

依据网上说，为$GRID_HOME/bin/oracle权限问题引起：

[grid@rac1 ~]$ cd /u01/grid11g/bin/

[grid@rac1 bin]$ ls -lhrt orac

ls: orac: No such file or directory

[grid@rac1 bin]$ ls -lhrt ora

ls: ora: No such file or directory

[grid@rac1 bin]$ pwd

/u01/grid11g/bin

[grid@rac1 bin]$ ls -lhrt oracle

-rwxr-x--x 1 grid oinstall 195M Aug 20 2013 oracle

[grid@rac1 bin]$ pwd

/u01/grid11g/bin

改动权限：

[grid@rac1 bin]$ chmod 6751 oracle

[grid@rac1 bin]$ ls -lhrt oracle

-rwsr-s--x 1 grid oinstall 195M Aug 20 2013 oracle

[grid@rac1 bin]$ exit

logout

[root@rac1 oracleasm]# su - oracle

[oracle@rac1 ~]$ exit

logout

[root@rac1 oracleasm]# su - grid

正常启动实例：

[grid@rac1 ~]$ srvctl start instance -d racdb -i RACDB1

[grid@rac1 ~]$

[grid@rac1 ~]$ crs_stat -t -v

Name Type R/RA F/FT Target State Host

----------------------------------------------------------------------

ora.ARCH.dg ora....up.type 0/5 0/ ONLINE ONLINE rac1

ora.DATA.dg ora....up.type 0/5 0/ ONLINE ONLINE rac1

ora....ER.lsnr ora....er.type 0/5 0/ ONLINE ONLINE rac1

ora....N1.lsnr ora....er.type 0/5 0/0 ONLINE ONLINE rac2

ora.OCR.dg ora....up.type 0/5 0/ ONLINE ONLINE rac1

ora.asm ora.asm.type 0/5 0/ ONLINE ONLINE rac1

ora.cvu ora.cvu.type 0/5 0/0 ONLINE ONLINE rac2

ora.gsd ora.gsd.type 0/5 0/ OFFLINE OFFLINE

ora....network ora....rk.type 0/5 0/ ONLINE ONLINE rac1

ora.oc4j ora.oc4j.type 0/1 0/2 ONLINE ONLINE rac2

ora.ons ora.ons.type 0/3 0/ ONLINE ONLINE rac1

ora....SM1.asm application 0/5 0/0 ONLINE ONLINE rac1

ora....C1.lsnr application 0/5 0/0 ONLINE ONLINE rac1

ora.rac1.gsd application 0/5 0/0 OFFLINE OFFLINE

ora.rac1.ons application 0/3 0/0 ONLINE ONLINE rac1

ora.rac1.vip ora....t1.type 0/0 0/0 ONLINE ONLINE rac1

ora....SM2.asm application 0/5 0/0 ONLINE ONLINE rac2

ora....C2.lsnr application 0/5 0/0 ONLINE ONLINE rac2

ora.rac2.gsd application 0/5 0/0 OFFLINE OFFLINE

ora.rac2.ons application 0/3 0/0 ONLINE ONLINE rac2

ora.rac2.vip ora....t1.type 0/0 0/0 ONLINE ONLINE rac2

ora.racdb.db ora....se.type 0/2 0/1 ONLINE ONLINE rac1

ora.scan1.vip ora....ip.type 0/0 0/0 ONLINE ONLINE rac2

[grid@rac1 ~]$ crsctl stat res -t

--------------------------------------------------------------------------------

NAME TARGET STATE SERVER STATE_DETAILS

--------------------------------------------------------------------------------

Local Resources

--------------------------------------------------------------------------------

ora.ARCH.dg

ONLINE ONLINE rac1

ONLINE ONLINE rac2

ora.DATA.dg

ONLINE ONLINE rac1

ONLINE ONLINE rac2

ora.LISTENER.lsnr

ONLINE ONLINE rac1

ONLINE ONLINE rac2

ora.OCR.dg

ONLINE ONLINE rac1

ONLINE ONLINE rac2

ora.asm

ONLINE ONLINE rac1 Started

ONLINE ONLINE rac2 Started

ora.gsd

OFFLINE OFFLINE rac1

OFFLINE OFFLINE rac2

ora.net1.network

ONLINE ONLINE rac1

ONLINE ONLINE rac2

ora.ons

ONLINE ONLINE rac1

ONLINE ONLINE rac2

--------------------------------------------------------------------------------

Cluster Resources

--------------------------------------------------------------------------------

ora.LISTENER_SCAN1.lsnr

1 ONLINE ONLINE rac2

ora.cvu

1 ONLINE ONLINE rac2

ora.oc4j

1 ONLINE ONLINE rac2

ora.rac1.vip

1 ONLINE ONLINE rac1

ora.rac2.vip

1 ONLINE ONLINE rac2

ora.racdb.db

1 ONLINE ONLINE rac1 Open

2 ONLINE ONLINE rac2 Open

ora.scan1.vip

1 ONLINE ONLINE rac2

[grid@rac1 ~]$

SQL> show parameter name

NAME TYPE VALUE

------------------------------------ ----------- ------------------------------

db_file_name_convert string

db_name string RACDB

db_unique_name string RACDB

global_names boolean FALSE

instance_name string RACDB1

lock_name_space string

log_file_name_convert string

processor_group_name string

service_names string RACDB

SQL> select open_mode from v$database;

OPEN_MODE

--------------------

READ WRITE

SQL>