ORACLE,crs大故障 : crs_stop -all 命令后遇到 UNKNOW

crs_stop -all 命令后遇到 UNKNOWN

crs_stop -all 命令后遇到 UNKNOWN

RAC中执行crs_stop -all后提示一些服务停不了，再用crs_stat -t查看状态的时候，停不掉的服务会显示状态是UNKNOWN，

服务在UNKNOWN状态下用 crs_start 不能启动服务

例如：

[oracle@rac1 ~]$ crs_stop -all

Attempting to stop `ora.rac1.gsd` on member `rac1`

Attempting to stop `ora.rac1.ons` on member `rac1`

Attempting to stop `ora.rac2.gsd` on member `rac2`

Attempting to stop `ora.ora9i.db` on member `rac1`

Attempting to stop `ora.rac2.ons` on member `rac2`

Stop of `ora.rac1.gsd` on member `rac1` succeeded.

Stop of `ora.rac2.gsd` on member `rac2` succeeded.

Stop of `ora.rac2.ons` on member `rac2` succeeded.

Stop of `ora.rac1.ons` on member `rac1` succeeded.

Stop of `ora.ora9i.db` on member `rac1` succeeded.

Attempting to stop `ora.rac1.LISTENER_RAC1.lsnr` on member `rac1`

Attempting to stop `ora.rac2.LISTENER_RAC2.lsnr` on member `rac2`

`ora.rac1.LISTENER_RAC1.lsnr` on member `rac1` has experienced an unrecoverable failure.

Human intervention required to resume its availability.

`ora.ora9i.ora9i1.inst` is already OFFLINE.

`ora.rac2.LISTENER_RAC2.lsnr` on member `rac2` has experienced an unrecoverable failure.

Human intervention required to resume its availability.

`ora.ora9i.ora9i2.inst` is already OFFLINE.

Attempting to stop `ora.rac1.ASM1.asm` on member `rac1`

`ora.rac1.ASM1.asm` on member `rac1` has experienced an unrecoverable failure.

Human intervention required to resume its availability.

Attempting to stop `ora.rac2.ASM2.asm` on member `rac2`

Attempting to stop `ora.rac1.LISTENER_RAC1.lsnr` on member `rac1`

`ora.rac2.ASM2.asm` on member `rac2` has experienced an unrecoverable failure.

Human intervention required to resume its availability.

`ora.rac1.LISTENER_RAC1.lsnr` on member `rac1` has experienced an unrecoverable failure.

Human intervention required to resume its availability.

Attempting to stop `ora.rac1.ASM1.asm` on member `rac1`

`ora.rac1.ASM1.asm` on member `rac1` has experienced an unrecoverable failure.

Human intervention required to resume its availability.

Attempting to stop `ora.rac2.LISTENER_RAC2.lsnr` on member `rac2`

Attempting to stop `ora.rac1.vip` on member `rac1`

`ora.rac2.LISTENER_RAC2.lsnr` on member `rac2` has experienced an unrecoverable failure.

Human intervention required to resume its availability.

Attempting to stop `ora.rac2.ASM2.asm` on member `rac2`

`ora.rac2.ASM2.asm` on member `rac2` has experienced an unrecoverable failure.

Human intervention required to resume its availability.

Attempting to stop `ora.rac2.vip` on member `rac2`

Stop of `ora.rac1.vip` on member `rac1` succeeded.

Stop of `ora.rac2.vip` on member `rac2` succeeded.

CRS-0216: Could not stop resource 'ora.ora9i.ora9i1.inst'.

CRS-0216: Could not stop resource 'ora.ora9i.ora9i2.inst'.

CRS-0216: Could not stop resource 'ora.rac1.ASM1.asm'.

CRS-0216: Could not stop resource 'ora.rac1.LISTENER_RAC1.lsnr'.

CRS-0216: Could not stop resource 'ora.rac1.vip'.

CRS-0216: Could not stop resource 'ora.rac2.ASM2.asm'.

CRS-0216: Could not stop resource 'ora.rac2.LISTENER_RAC2.lsnr'.

CRS-0216: Could not stop resource 'ora.rac2.vip'.



在UNKNOWN状态下，使用crs_start -all命令不会把RAC启动起来，启动的时候会报错。

[oracle@rac1 ~]$ crs_start -all

Attempting to start `ora.rac1.vip` on member `rac1`

Attempting to start `ora.rac2.vip` on member `rac2`

Attempting to start `ora.rac2.gsd` on member `rac2`

Attempting to start `ora.rac2.ons` on member `rac2`

Attempting to start `ora.rac1.gsd` on member `rac1`

Attempting to start `ora.rac1.ons` on member `rac1`

`ora.rac2.gsd` on member `rac2` has experienced an unrecoverable failure.

Human intervention required to resume its availability.

`ora.rac2.ons` on member `rac2` has experienced an unrecoverable failure.

Human intervention required to resume its availability.

Attempting to start `ora.ora9i.db` on member `rac1`

`ora.rac1.ons` on member `rac1` has experienced an unrecoverable failure.

Human intervention required to resume its availability.

`ora.rac1.gsd` on member `rac1` has experienced an unrecoverable failure.

Human intervention required to resume its availability.

`ora.ora9i.db` on member `rac1` has experienced an unrecoverable failure.

Human intervention required to resume its availability.

Start of `ora.rac1.vip` on member `rac1` succeeded.

CRS-1028: Dependency analysis failed because of:

'Resource in UNKNOWN state: ora.rac1.ASM1.asm'

Start of `ora.rac2.vip` on member `rac2` succeeded.

CRS-1028: Dependency analysis failed because of:

'Resource in UNKNOWN state: ora.rac2.ASM2.asm'

CRS-0215: Could not start resource 'ora.ora9i.db'.

CRS-0223: Resource 'ora.ora9i.ora9i1.inst' has placement error.

CRS-0223: Resource 'ora.ora9i.ora9i2.inst' has placement error.

CRS-0215: Could not start resource 'ora.rac1.gsd'.

CRS-0215: Could not start resource 'ora.rac1.ons'.

CRS-0215: Could not start resource 'ora.rac2.gsd'.

CRS-0215: Could not start resource 'ora.rac2.ons'.

此时crs_stat -t的状态还是有很多的UNKNOWN

（烦、烦、烦，rac中真奇怪，正常启动关闭服务时，

状态只有online和offline两种（启动与未启动），结果RAC中给你来了个UNKNOWN）：

[oracle@rac1 ~]$ crs_stat -t

Name Type Target State Host

------------------------------------------------------------

ora.ora9i.db application ONLINE UNKNOWN rac1

ora....i1.inst application OFFLINE OFFLINE

ora....i2.inst application OFFLINE OFFLINE

ora....SM1.asm application OFFLINE UNKNOWN rac1

ora....C1.lsnr application OFFLINE UNKNOWN rac1

ora.rac1.gsd application ONLINE UNKNOWN rac1

ora.rac1.ons application ONLINE UNKNOWN rac1

ora.rac1.vip application ONLINE ONLINE rac1

ora....SM2.asm application OFFLINE UNKNOWN rac2

ora....C2.lsnr application OFFLINE UNKNOWN rac2

ora.rac2.gsd application ONLINE UNKNOWN rac2

ora.rac2.ons application ONLINE UNKNOWN rac2

ora.rac2.vip application ONLINE ONLINE rac2

[oracle@rac1 ~]$ sqlplus /nolog

SQL*Plus: Release 10.2.0.1.0 - Production on Thu Aug 7 13:42:36 2008

Copyright (c) 1982, 2005, Oracle. All rights reserved.

SQL> conn /as sysdba

Connected to an idle instance.

显示数据库已经连接不上；

此时可有两种方法来解决：

方法一：先禁用crs服务，后启用crs服务

（相当于从新启动系统，因为crs是随系统自动启动的），需要在两台节点上都执行；

具体步骤：

节点1：

1、[root@rac1 ~]# /etc/init.d/init.crs stop

Shutting down Oracle Cluster Ready Services (CRS):

Stopping resources.

Successfully stopped CRS resources

Stopping CSSD.

Shutting down CSS daemon.

Shutdown request successfully issued.

Shutdown has begun. The daemons should exit soon.

2、[root@rac1 ~]# su - oracle

[oracle@rac1 ~]$ crs_stat -t

CRS-0184: Cannot communicate with the CRS daemon.

3、[oracle@rac1 ~]$ exit

logout

[root@rac1 ~]# /etc/init.d/init.crs start

Startup will be queued to init within 90 seconds.

4、[root@rac1 ~]# su - oracle

[oracle@rac1 ~]$ crs_stat -t

Name Type Target State Host

------------------------------------------------------------

ora.ora9i.db application ONLINE ONLINE rac1

ora....i1.inst application ONLINE ONLINE rac1

ora....i2.inst application ONLINE ONLINE rac2

ora....SM1.asm application ONLINE ONLINE rac1

ora....C1.lsnr application ONLINE ONLINE rac1

ora.rac1.gsd application ONLINE ONLINE rac1

ora.rac1.ons application ONLINE ONLINE rac1

ora.rac1.vip application ONLINE ONLINE rac1

ora....SM2.asm application ONLINE ONLINE rac2

ora....C2.lsnr application ONLINE ONLINE rac2

ora.rac2.gsd application ONLINE ONLINE rac2

ora.rac2.ons application ONLINE ONLINE rac2

ora.rac2.vip application ONLINE ONLINE rac2

[oracle@rac1 ~]$

节点2：

1、[root@rac2 ~]# /etc/init.d/init.crs stop

Shutting down Oracle Cluster Ready Services (CRS):

Stopping resources.

Successfully stopped CRS resources

Stopping CSSD.

Shutting down CSS daemon.

Shutdown request successfully issued.

Shutdown has begun. The daemons should exit soon.

2、[root@rac2 ~]# su - oracle

[oracle@rac2 ~]$ crs_stat -t

CRS-0184: Cannot communicate with the CRS daemon.

[oracle@rac2 ~]$ exit

logout

3、[root@rac2 ~]# /etc/init.d/init.crs start

Startup will be queued to init within 90 seconds.

4、[root@rac2 ~]# su - oracle

[oracle@rac2 ~]$ crs_stat -t

Name Type Target State Host

------------------------------------------------------------

ora.ora9i.db application ONLINE ONLINE rac1

ora....i1.inst application ONLINE ONLINE rac1

ora....i2.inst application ONLINE ONLINE rac2

ora....SM1.asm application ONLINE ONLINE rac1

ora....C1.lsnr application ONLINE ONLINE rac1

ora.rac1.gsd application ONLINE ONLINE rac1

ora.rac1.ons application ONLINE ONLINE rac1

ora.rac1.vip application ONLINE ONLINE rac1

ora....SM2.asm application ONLINE ONLINE rac2

ora....C2.lsnr application ONLINE ONLINE rac2

ora.rac2.gsd application ONLINE ONLINE rac2

ora.rac2.ons application ONLINE ONLINE rac2

ora.rac2.vip application ONLINE ONLINE rac2

[oracle@rac2 ~]$

方法二：可以使用crs_stop -f先强制停止服务，在启动服务（在一台节点上运行即可）；

具体步骤：

[oracle@rac2 ~]$ crs_stop -f ora.rac1.ons

Attempting to stop `ora.rac1.ons` on member `rac1`

Stop of `ora.rac1.ons` on member `rac1` succeeded.

[oracle@rac2 ~]$ crs_start -f ora.rac1.ons

Attempting to start `ora.rac1.ons` on member `rac1`

Start of `ora.rac1.ons` on member `rac1` succeeded.

[oracle@rac2 ~]$ crs_stop -f ora.rac2.ons

Attempting to stop `ora.rac2.ons` on member `rac2`

Stop of `ora.rac2.ons` on member `rac2` succeeded.

[oracle@rac2 ~]$ crs_start -f ora.rac2.ons

Attempting to start `ora.rac2.ons` on member `rac2`

Start of `ora.rac2.ons` on member `rac2` succeeded.

[oracle@rac2 ~]$ crs_stop -f ora.rac1.gsd

Attempting to stop `ora.rac1.gsd` on member `rac1`

Stop of `ora.rac1.gsd` on member `rac1` succeeded.

[oracle@rac2 ~]$ crs_start -f ora.rac1.gsd

Attempting to start `ora.rac1.gsd` on member `rac1`

Start of `ora.rac1.gsd` on member `rac1` succeeded.

[oracle@rac2 ~]$ crs_stop -f ora.rac2.gsd

Attempting to stop `ora.rac2.gsd` on member `rac2`

Stop of `ora.rac2.gsd` on member `rac2` succeeded.

[oracle@rac2 ~]$ crs_start -f ora.rac2.gsd

Attempting to start `ora.rac2.gsd` on member `rac2`

Start of `ora.rac2.gsd` on member `rac2` succeeded.

[oracle@rac2 ~]$ crs_stop -f ora.rac1.LISTENER_RAC1.lsnr

Attempting to stop `ora.rac1.LISTENER_RAC1.lsnr` on member `rac1`

Stop of `ora.rac1.LISTENER_RAC1.lsnr` on member `rac1` succeeded.

[oracle@rac2 ~]$ crs_start -f ora.rac1.LISTENER_RAC1.lsnr

Attempting to start `ora.rac1.LISTENER_RAC1.lsnr` on member `rac1`

Start of `ora.rac1.LISTENER_RAC1.lsnr` on member `rac1` succeeded.

[oracle@rac2 ~]$ crs_stop -f ora.rac2.LISTENER_RAC2.lsnr

Attempting to stop `ora.rac2.LISTENER_RAC2.lsnr` on member `rac2`

Stop of `ora.rac2.LISTENER_RAC2.lsnr` on member `rac2` succeeded.

[oracle@rac2 ~]$ crs_start -f ora.rac2.LISTENER_RAC2.lsnr

Attempting to start `ora.rac2.LISTENER_RAC2.lsnr` on member `rac2`

Start of `ora.rac2.LISTENER_RAC2.lsnr` on member `rac2` succeeded.

[oracle@rac2 ~]$ crs_stop -f ora.rac1.ASM1.asm

Attempting to stop `ora.rac1.ASM1.asm` on member `rac1`

Stop of `ora.rac1.ASM1.asm` on member `rac1` succeeded.

[oracle@rac2 ~]$ crs_start -f ora.rac1.ASM1.asm

Attempting to start `ora.rac1.ASM1.asm` on member `rac1`

Start of `ora.rac1.ASM1.asm` on member `rac1` succeeded.

[oracle@rac2 ~]$ crs_stop -f ora.rac2.ASM2.asm

Attempting to stop `ora.rac2.ASM2.asm` on member `rac2`

Stop of `ora.rac2.ASM2.asm` on member `rac2` succeeded.

[oracle@rac2 ~]$ crs_start -f ora.rac2.ASM2.asm

Attempting to start `ora.rac2.ASM2.asm` on member `rac2`

Start of `ora.rac2.ASM2.asm` on member `rac2` succeeded.

[oracle@rac2 ~]$ crs_stop -f ora.ora9i.ora9i1.inst

`ora.ora9i.ora9i1.inst` is already OFFLINE.

CRS-0216: Could not stop resource 'ora.ora9i.ora9i1.inst'.

[oracle@rac2 ~]$ crs_start -f ora.ora9i.ora9i1.inst

Attempting to start `ora.ora9i.ora9i1.inst` on member `rac1`

Start of `ora.ora9i.ora9i1.inst` on member `rac1` succeeded.

[oracle@rac2 ~]$ crs_stop -f ora.ora9i.ora9i2.inst

`ora.ora9i.ora9i2.inst` is already OFFLINE.

CRS-0216: Could not stop resource 'ora.ora9i.ora9i2.inst'.

[oracle@rac2 ~]$ crs_start -f ora.ora9i.ora9i2.inst

Attempting to start `ora.ora9i.ora9i2.inst` on member `rac2`

Start of `ora.ora9i.ora9i2.inst` on member `rac2` succeeded.

[oracle@rac2 ~]$ crs_stop -f ora.ora9i.db

Attempting to stop `ora.ora9i.db` on member `rac2`

Stop of `ora.ora9i.db` on member `rac2` succeeded.

[oracle@rac2 ~]$ crs_start -f ora.ora9i.db

Attempting to start `ora.ora9i.db` on member `rac1`

Start of `ora.ora9i.db` on member `rac1` succeeded.

[oracle@rac2 ~]$ crs_stat -t

Name Type Target State Host

------------------------------------------------------------

ora.ora9i.db application ONLINE ONLINE rac1

ora....i1.inst application ONLINE ONLINE rac1

ora....i2.inst application ONLINE ONLINE rac2

ora....SM1.asm application ONLINE ONLINE rac1

ora....C1.lsnr application ONLINE ONLINE rac1

ora.rac1.gsd application ONLINE ONLINE rac1

ora.rac1.ons application ONLINE ONLINE rac1

ora.rac1.vip application ONLINE ONLINE rac1

ora....SM2.asm application ONLINE ONLINE rac2

ora....C2.lsnr application ONLINE ONLINE rac2

ora.rac2.gsd application ONLINE ONLINE rac2

ora.rac2.ons application ONLINE ONLINE rac2

ora.rac2.vip application ONLINE ONLINE rac2

总结：在rac环境中如果使用crs_stop -all去停rac服务时，很有可能会出现crs中的一些服务的状态是UNKNOWN，这时我们可以采用两种方法去重新启动crs服务：

1：转换到root用户下用/etc/init.d/init.crs stop先禁用crs，然后再用/etc/init.d/init.crs start去启用crs，启用crs后会自动启动crs的一系列服务，注意此种方法需要在两台节点上都执行；

2：是用crs_stop -f 参数把crs中状态是UNKNOWN的服务关掉，然后再用crs_start -f （加一个-f的参数）启动所有的服务就可以。参考：http://www.rachelp.nl/index_kb.php?menu=articles&actie=show&id=31

以下要分别在两个节点上执行:

在启动的过程中最好检测着crs、ASM和数据库的日志：

crs日志：

[oracle@rac1 ~]$ tail -f /u01/app/oracle/product/10.2.0/crs_1/log/rac1/alertrac1.log

[oracle@rac2 ~]$ tail -f /u01/app/oracle/product/10.2.0/crs_1/log/rac2/alertrac2.log

ASM日志：

[oracle@rac1 ~]$ tail -f /u01/app/oracle/admin/+ASM/bdump/alert_+ASM1.log

[oracle@rac2 ~]$ tail -f /u01/app/oracle/admin/+ASM/bdump/alert_+ASM2.log

数据库日志：

[oracle@rac1 ~]$ tail -f /u01/app/oracle/admin/ora9i/bdump/alert_ora9i1.log

[oracle@rac2 ~]$ tail -f /u01/app/oracle/admin/ora9i/bdump/alert_ora9i2.log