srvctl和crs_start命令无法启动oracle RAC实例, 但sqlplus可以启动

今天遇到一个奇怪问题，发现srvctl和crs_start命令无法启动Oracle RAC实例，但用sqlplus却可以正常启动。最终发现原因是在OCR中数据库的状态变成了disable，将此状态更改为enable后恢复正常。

以下是一个模拟示例：

oracle@his00mlcdrc02:~ $> crs_stat -t
Name           Type           Target    State     Host
------------------------------------------------------------
ora....SM1.asm application    ONLINE    ONLINE    his0...rc01
ora....01.lsnr application    ONLINE    ONLINE    his0...rc01
ora....c01.gsd application    ONLINE    ONLINE    his0...rc01
ora....c01.ons application    ONLINE    ONLINE    his0...rc01
ora....c01.vip application    ONLINE    ONLINE    his0...rc01
ora....SM2.asm application    ONLINE    ONLINE    his0...rc02
ora....02.lsnr application    ONLINE    ONLINE    his0...rc02
ora....c02.gsd application    ONLINE    ONLINE    his0...rc02
ora....c02.ons application    ONLINE    ONLINE    his0...rc02
ora....c02.vip application    ONLINE    ONLINE    his0...rc02
ora....rcdb.db application    OFFLINE   OFFLINE
ora....b1.inst application    OFFLINE   OFFLINE
ora....b2.inst application    OFFLINE   OFFLINE

oracle@his00mlcdrc02:~ $> ps -ef | grep ora_
oracle 23001 16923   0 16:31:27 pts/1       0:00 grep ora_

oracle@his00mlcdrc02:~ $> crs_start ora.hisdrcdb.hisdrcdb2.inst
Attempting to start `ora.hisdrcdb.hisdrcdb2.inst` on member `his00mlcdrc02`
Start of `ora.hisdrcdb.hisdrcdb2.inst` on member `his00mlcdrc02` failed.    (在节点his00mlcdrc02上启动失败）
his00mlcdrc01 : CRS-1019: Resource ora.hisdrcdb.hisdrcdb2.inst (application) cannot run on his00mlcdrc01
(对于象VIP这类可以failover到其它节点的资源，如果在原节点上启动不成功，则Oracle会试图在其它节点上启动。而实例是不能failover的资源，此处提示实例2不能在节点1上启动)
CRS-0215: Could not start resource 'ora.hisdrcdb.hisdrcdb2.inst'.

oracle@his00mlcdrc02:~ $> crs_stat -t
Name           Type           Target    State     Host
------------------------------------------------------------
ora....SM1.asm application    ONLINE    ONLINE    his0...rc01
ora....01.lsnr application    ONLINE    ONLINE    his0...rc01
ora....c01.gsd application    ONLINE    ONLINE    his0...rc01
ora....c01.ons application    ONLINE    ONLINE    his0...rc01
ora....c01.vip application    ONLINE    ONLINE    his0...rc01
ora....SM2.asm application    ONLINE    ONLINE    his0...rc02
ora....02.lsnr application    ONLINE    ONLINE    his0...rc02
ora....c02.gsd application    ONLINE    ONLINE    his0...rc02
ora....c02.ons application    ONLINE    ONLINE    his0...rc02
ora....c02.vip application    ONLINE    ONLINE    his0...rc02
ora....rcdb.db application    OFFLINE   OFFLINE
ora....b1.inst application    OFFLINE   OFFLINE
ora....b2.inst application    ONLINE    OFFLINE
此时可以看到实例2启动并未成功。

oracle@his00mlcdrc02:~ $> srvctl start instance -d hisdrcdb -i hisdrcdb2
用srvctl命令尝试启动实例2，命令很快就返回了。

oracle@his00mlcdrc02:~ $> crs_stat -t
Name           Type           Target    State     Host
------------------------------------------------------------
ora....SM1.asm application    ONLINE    ONLINE    his0...rc01
ora....01.lsnr application    ONLINE    ONLINE    his0...rc01
ora....c01.gsd application    ONLINE    ONLINE    his0...rc01
ora....c01.ons application    ONLINE    ONLINE    his0...rc01
ora....c01.vip application    ONLINE    ONLINE    his0...rc01
ora....SM2.asm application    ONLINE    ONLINE    his0...rc02
ora....02.lsnr application    ONLINE    ONLINE    his0...rc02
ora....c02.gsd application    ONLINE    ONLINE    his0...rc02
ora....c02.ons application    ONLINE    ONLINE    his0...rc02
ora....c02.vip application    ONLINE    ONLINE    his0...rc02
ora....rcdb.db application    OFFLINE   OFFLINE
ora....b1.inst application    OFFLINE   OFFLINE
ora....b2.inst application    ONLINE    OFFLINE
srvctl命令启动也不成功。

oracle@his00mlcdrc02:~ $> sqlplus "/as sysdba"

SQL*Plus: Release 10.2.0.4.0 - Production on Sat Oct 9 16:37:38 2010

Copyright (c) 1982, 2007, Oracle.  All Rights Reserved.

Connected to an idle instance.

SQL> startup;
ORACLE instance started.

Total System Global Area 1610612736 bytes
Fixed Size                  2139192 bytes
Variable Size             399465416 bytes
Database Buffers         1174405120 bytes
Redo Buffers               34603008 bytes
Database mounted.
Database opened.
SQL> show parameter instance_name

NAME                                 TYPE                   VALUE
------------------------------------ ---------------------- ------------------------------
instance_name                        string                 hisdrcdb2
SQL> select open_mode from v$database;

OPEN_MODE
--------------------
READ WRITE
用sqlplus启动实例成功了。

SQL> host crs_stat -t
Name           Type           Target    State     Host
------------------------------------------------------------
ora....SM1.asm application    ONLINE    ONLINE    his0...rc01
ora....01.lsnr application    ONLINE    ONLINE    his0...rc01
ora....c01.gsd application    ONLINE    ONLINE    his0...rc01
ora....c01.ons application    ONLINE    ONLINE    his0...rc01
ora....c01.vip application    ONLINE    ONLINE    his0...rc01
ora....SM2.asm application    ONLINE    ONLINE    his0...rc02
ora....02.lsnr application    ONLINE    ONLINE    his0...rc02
ora....c02.gsd application    ONLINE    ONLINE    his0...rc02
ora....c02.ons application    ONLINE    ONLINE    his0...rc02
ora....c02.vip application    ONLINE    ONLINE    his0...rc02
ora....rcdb.db application    OFFLINE   OFFLINE
ora....b1.inst application    OFFLINE   OFFLINE
ora....b2.inst application    ONLINE    OFFLINE
但crs_stat显示的状态仍然是OFFLINE.

先把实例2停止。
SQL> shutdown immediate;
Database closed.
Database dismounted.
ORACLE instance shut down.
SQL> exit
Disconnected from Oracle Database 10g Enterprise Edition Release 10.2.0.4.0 - 64bit Production
With the Partitioning, Real Application Clusters, OLAP, Data Mining
and Real Application Testing options

oracle@his00mlcdrc02:~ $> ocrdump

查看OCRDUMPFILE

......
[DATABASE.DATABASES.hisdrcdb.ENABLED]
ORATEXT : false
SECURITY : {USER_PERMISSION : PROCR_ALL_ACCESS, GROUP_PERMISSION : PROCR_WRITE, OTHER_PERMISSION : PROCR_READ, USER_NAME : oracle, GROUP_NAME : dba}

[DATABASE.DATABASES.hisdrcdb.POLICY]
ORATEXT : MANUAL
SECURITY : {USER_PERMISSION : PROCR_ALL_ACCESS, GROUP_PERMISSION : PROCR_WRITE, OTHER_PERMISSION : PROCR_READ, USER_NAME : oracle, GROUP_NAME : dba}
......
OCR中显示数据库DATABASE.DATABASES.hisdrcdb.ENABLED属性为false，DATABASE.DATABASES.hisdrcdb.POLICY为manual,而在别的数据库上可以看到这两个属性应为true和automatic.

oracle@his00mlcdrc02:~ $> srvctl status database -d hisdrcdb -f
Instance hisdrcdb1 is disabled
Instance hisdrcdb1 is not running on node his00mlcdrc01
Instance hisdrcdb2 is disabled
Instance hisdrcdb2 is not running on node his00mlcdrc02
srvctl状态表明数据库被disable了。

oracle@his00mlcdrc02:~ $> srvctl enable database -d hisdrcdb
oracle@his00mlcdrc02:~ $> srvctl status database -d hisdrcdb -f
Instance hisdrcdb1 is not running on node his00mlcdrc01
Instance hisdrcdb2 is not running on node his00mlcdrc02
用srvctl命令enable database

oracle@his00mlcdrc02:~ $> ocrdump ocr1
查看ocr1:
......
[DATABASE.DATABASES.hisdrcdb.ENABLED]
ORATEXT : true
SECURITY : {USER_PERMISSION : PROCR_ALL_ACCESS, GROUP_PERMISSION : PROCR_WRITE, OTHER_PERMISSION : PROCR_READ, USER_NAME : oracle, GROUP_NAME : dba}

[DATABASE.DATABASES.hisdrcdb.POLICY]
ORATEXT : AUTOMATIC
SECURITY : {USER_PERMISSION : PROCR_ALL_ACCESS, GROUP_PERMISSION : PROCR_WRITE, OTHER_PERMISSION : PROCR_READ, USER_NAME : oracle, GROUP_NAME : dba}
......
此时这两个属性恢复为true和automatic.

oracle@his00mlcdrc02:~ $> crs_start ora.hisdrcdb.hisdrcdb1.inst
Attempting to start `ora.hisdrcdb.hisdrcdb1.inst` on member `his00mlcdrc01`
Start of `ora.hisdrcdb.hisdrcdb1.inst` on member `his00mlcdrc01` succeeded.
oracle@his00mlcdrc02:~ $>
oracle@his00mlcdrc02:~ $> crs_start ora.hisdrcdb.hisdrcdb2.inst
Attempting to start `ora.hisdrcdb.hisdrcdb2.inst` on member `his00mlcdrc02`
Start of `ora.hisdrcdb.hisdrcdb2.inst` on member `his00mlcdrc02` succeeded.
oracle@his00mlcdrc02:~ $> crs_stat -t
Name           Type           Target    State     Host
------------------------------------------------------------
ora....SM1.asm application    ONLINE    ONLINE    his0...rc01
ora....01.lsnr application    ONLINE    ONLINE    his0...rc01
ora....c01.gsd application    ONLINE    ONLINE    his0...rc01
ora....c01.ons application    ONLINE    ONLINE    his0...rc01
ora....c01.vip application    ONLINE    ONLINE    his0...rc01
ora....SM2.asm application    ONLINE    ONLINE    his0...rc02
ora....02.lsnr application    ONLINE    ONLINE    his0...rc02
ora....c02.gsd application    ONLINE    ONLINE    his0...rc02
ora....c02.ons application    ONLINE    ONLINE    his0...rc02
ora....c02.vip application    ONLINE    ONLINE    his0...rc02
ora....rcdb.db application    ONLINE    ONLINE    his0...rc02
ora....b1.inst application    ONLINE    ONLINE    his0...rc01
ora....b2.inst application    ONLINE    ONLINE    his0...rc02
再次用crs_start命令则能够正常启动实例。

附：srvctl disable disables target, meaning CRS will not consider it for automatic startup, failover, or restart. This option is useful to ensure an object that is down for maintenance is not accidentally automatically restarted.

srvctl enable reenables the specified object.