一次权限修改导致的RAC无法正常启动的问题的解决

关键字RAC，ORA-17503，ORA-12537

测试的Oracle12c RAC数据库出现问题，无法启动了（OEL6.5+ORACLE12C12.1.0.1）

1、RAC中资源状态

[grid@rac1 ~]$ crsctl stat res -t

--------------------------------------------------------------------------------

Name           Target  State        Server                   State details       

--------------------------------------------------------------------------------

Local Resources

--------------------------------------------------------------------------------

ora.ASMNET1LSNR_ASM.lsnr

               ONLINE  ONLINE       rac1                     STABLE

               ONLINE  ONLINE       rac2                     STABLE

ora.DATA.dg

               ONLINE  OFFLINE      rac1                     STABLE

               ONLINE  ONLINE       rac2                     STABLE

ora.FRA.dg

               ONLINE  OFFLINE      rac1                     STABLE

               ONLINE  ONLINE       rac2                     STABLE

ora.LISTENER.lsnr

               ONLINE  ONLINE       rac1                     STABLE

               ONLINE  ONLINE       rac2                     STABLE

ora.OCR_VOTING.dg

               OFFLINE OFFLINE      rac1                     STABLE

               ONLINE  ONLINE       rac2                     STABLE

ora.net1.network

               ONLINE  ONLINE       rac1                     STABLE

               ONLINE  ONLINE       rac2                     STABLE

ora.ons

               ONLINE  ONLINE       rac1                     STABLE

               ONLINE  ONLINE       rac2                     STABLE

ora.proxy_advm

               ONLINE  OFFLINE      rac1                     STABLE

               ONLINE  OFFLINE      rac2                     STABLE

--------------------------------------------------------------------------------

Cluster Resources

--------------------------------------------------------------------------------

ora.LISTENER_SCAN1.lsnr

      1        ONLINE  ONLINE       rac2                     STABLE

ora.asm

      1        ONLINE  OFFLINE                               STABLE

      2        ONLINE  ONLINE       rac2                     STABLE

      3        OFFLINE OFFLINE                               STABLE

ora.cvu

      1        ONLINE  ONLINE       rac2                     STABLE

ora.oc4j

      1        OFFLINE OFFLINE                               STABLE

ora.orarac.db

      1        ONLINE  ONLINE       rac1                     Open,STABLE

      2        ONLINE  OFFLINE                               Instance Shutdown,ST

                                                             ABLE

ora.rac1.vip

      1        ONLINE  ONLINE       rac1                     STABLE

ora.rac2.vip

      1        ONLINE  ONLINE       rac2                     STABLE

ora.scan1.vip

      1        ONLINE  ONLINE       rac2                     STABLE

-------------------------------------------------------------------------------

rac1节点ASM为OFFLINE，rac2节点db为OFFLINE。

2、手动启动两个资源报错

[grid@rac1 ~]$ srvctl start asm -node rac1

PRCR-1013 : Failed to start resource ora.asm

PRCR-1064 : Failed to start resource ora.asm on node rac1

CRS-2674: Start of 'ora.asm' on 'rac1' failed

[grid@rac1 ~]$ srvctl start instance -d orarac -i orarac2

PRCR-1013 : Failed to start resource ora.orarac.db

PRCR-1064 : Failed to start resource ora.orarac.db on node rac2

CRS-5017: The resource action "ora.orarac.db start" encountered the following error:

ORA-01078: failure in processing system parameters

ORA-01565: error in identifying file '+DATA/orarac/spfileorarac.ora'

ORA-17503: ksfdopn:2 Failed to open file +DATA/orarac/spfileorarac.ora

ORA-01034: ORACLE not available

ORA-27123: unable to attach to shared memory segment

Linux-x86_64 Error: 13: Permission denied

Additional information: 2

Additional information: 12255236

. For details refer to "(:CLSN00107:)" in "/u01/app/grid/12.1.0/grid/log/rac2/agent/crsd/oraagent_oracle/oraagent_oracle.log".

CRS-2674: Start of 'ora.orarac.db' on 'rac2' failed

3、原因分析

查看权限设置发现：

[root@rac1 u01]# ll

total 36

drwxrwxr-x 4 oracle oinstall  4096 Mar 10 09:39 app

drwxr-xr-x 3 oracle oinstall  4096 Mar 23 15:55 backup

drwxrwxr-x 9 oracle oinstall  4096 Mar 13 08:34 gridbase

drwxrwxr-x 2 oracle oinstall 16384 Mar 10 09:13 lost+found

drwxrwxr-x 6 oracle oinstall  4096 Mar 12 10:42 oraInventory

drwxrwxr-x 3 oracle oinstall  4096 Mar 12 10:01 software

[root@rac2 u01]# ll

total 44

drwxrwxr-x 4 root   oinstall  4096 Mar 10 09:39 app

drwxr-xr-x 3 oracle oinstall  4096 Mar 23 15:57 backup

-rwxrwxr-x 1 oracle oinstall  8827 Mar 10 10:00 cvuqdisk-1.0.9-1.rpm

drwxrwxr-x 6 grid   oinstall  4096 Mar 11 15:59 gridbase

drwxrwxr-x 2 oracle oinstall 16384 Mar 10 09:35 lost+found

drwxrwxr-x 5 grid   oinstall  4096 Mar 12 10:42 oraInventory

由于大意修改了权限，导致问题的发生。

4、解决rac1上ASM无法启动的问题

修改rac1节点的文件夹权限，便可以启动ASM。

5、解决rac2上db无法启动的问题

CRS-5017: The resource action "ora.orarac.db start" encountered the following error:

ORA-01078: failure in processing system parameters

ORA-01565: error in identifying file '+DATA/orarac/spfileorarac.ora'

ORA-17503: ksfdopn:2 Failed to open file +DATA/orarac/spfileorarac.ora

ORA-01034: ORACLE not available

ORA-27123: unable to attach to shared memory segment

Linux-x86_64 Error: 13: Permission denied

从上面能看出是权限的问题，搜索ora-17503，找到解决办法（重设$GRID_HOME/bin/oracle文件的权限），分别在两个节点上执行：

[root@rac1 grid]# cd bin

[root@rac1 bin]# ls -la oracle

-rwxrwxrwx 1 oracle oinstall 263068997 Mar 11 15:21 oracle

[root@rac1 bin]# chown grid:oinstall oracle

[root@rac1 bin]# chmod 6751 oracle

[root@rac2 u01]# cd /u01/app/grid/12.1.0/grid

[root@rac2 grid]# cd bin

[root@rac2 bin]# ls -la oracle

-rwxrwxrwx 1 grid oinstall 263068997 Mar 11 15:22 oracle

[root@rac2 bin]# chmod 6751 oracle

之后启动正常。

最后还需要在两个节点确认权限(否则通过tns连接时可能出现ora-12537):

chmod 6751 /u01/app/oracle/product/12.1.0/dbhome_1/bin/oracle

6、总结

在操作系统中修改文件夹权限一定要谨慎！