ORACLE 11.2.0.2 RAC on AIX 6.1 安装报错

听说11.2.0.2版本的bug比较多，不过直到上周还没接触过这个版本。这次客户要求必须是11.2.0.2。没想到第一次安装就遇到bug。

服务器上原来有单实例库，直接卸载。然后重启后进行安装检查，最开始一切顺利，而且预检查也没有任何问题。安装grid在第二个节点上执行root.sh时，最后一步报错：

CRS-4402: The CSS daemon was started in exclusive mode but found an active CSS daemon on node hjracdb1, number 1, and is terminating
An active cluster was found during exclusive startup, restarting to join the cluster
Failed to start Oracle Clusterware stack
Failed to start Cluster Synchorinisation Service in clustered mode at /u01/app/11.2.0/grid/crs/install/<span style="color:#ff0000;">crsconfig_lib.pm line 1020.</span>
/u01/app/11.2.0/grid/perl/bin/perl -I/u01/app/11.2.0/grid/perl/lib -I/u01/app/11.2.0/grid/crs/install /u01/app/11.2.0/grid/crs/install/rootcrs.pl execution failed

第一次报错，猜测可能是原来库卸载、环境检查或某一步有问题。网上查找发现更多的是crsconfig_lib.pm line 1016，不完全相同。

不过进一步查看alert日志：

2014-12-05 14:30:15.109
[ohasd(5374108)]CRS-1301:Oracle High Availability Service started on node ora2.
[client(4653120)]CRS-10001:05-Dec-14 14:30 ACFS-9200: Supported
[client(5701674)]CRS-10001:05-Dec-14 14:30 ACFS-9300: ADVM/ACFS distribution files found.
[client(5701682)]CRS-10001:05-Dec-14 14:30 ACFS-9307: Installing requested ADVM/ACFS software.
[client(5701690)]CRS-10001:05-Dec-14 14:30 ACFS-9359: The AIX ODM entry for 'acfsctl' was successfully added.
[client(5701698)]CRS-10001:05-Dec-14 14:30 ACFS-9359: The AIX ODM entry for 'advmctl' was successfully added.
[client(5701706)]CRS-10001:05-Dec-14 14:30 ACFS-9359: The AIX ODM entry for 'advmvol' was successfully added.
[client(5701714)]CRS-10001:05-Dec-14 14:30 ACFS-9308: Loading installed ADVM/ACFS drivers.
[client(5701732)]CRS-10001:05-Dec-14 14:31 ACFS-9327: Verifying ADVM/ACFS devices.
[client(5701742)]CRS-10001:05-Dec-14 14:31 ACFS-9309: ADVM/ACFS installation correctness verified.
2014-12-05 14:31:04.678
[/u01/app/11.2.0/grid/bin/oraagent.bin(4587526)]CRS-5815:Agent '/u01/app/11.2.0/grid/bin/oraagent_grid' could not find any base type entry points for type 'ora.daemon.type'. Details at (:CRSAGF00108:) {0:1:2} in /u01/app/11.2.0/grid/log/ora8c/agent/ohasd/oraagent_grid/oraagent_grid.log.
2014-12-05 14:31:07.220
[ohasd(5374108)]CRS-2302:Cannot get GPnP profile. Error CLSGPNP_NO_DAEMON (GPNPD daemon is not running).
2014-12-05 14:31:07.372
[gpnpd(3342416)]CRS-2328:GPNPD started on node ora2.
2014-12-05 14:31:10.477
[cssd(5832742)]CRS-1713:CSSD daemon is started in exclusive mode
2014-12-05 14:31:43.730
[cssd(5832742)]CRS-1707:Lease acquisition for node ora2 number 2 completed
2014-12-05 14:31:45.166
[cssd(5832742)]CRS-1605:CSSD voting file is online: /dev/rhdisk11; details in /u01/app/11.2.0/grid/log/ora2/cssd/ocssd.log.
[cssd(5832742)]CRS-1636:The CSS daemon was started in exclusive mode but found an active CSS daemon on node ora7c and is terminating; details at (:CSSNM00006:) in /u01/app/11.2.0/grid/log/ora2/cssd/ocssd.log
2014-12-05 14:31:46.964
[ohasd(5374108)]CRS-2765:Resource 'ora.cssdmonitor' has failed on server 'ora2'.
2014-12-05 14:31:50.851
[gpnpd(3342416)]CRS-2329:GPNPD on node ora2 shutdown.
2014-12-05 14:31:53.220
[mdnsd(5701758)]CRS-5602:mDNS service stopping by request.

再查看ocssd.log日志，报错后一直全部是红色部分日志：

2014-12-05 14:31:15.562: [    CSSD][1]clssnmlalloccx:phyname ora2
2014-12-05 14:31:15.563: [    CSSD][1]clssnmvDiskAvailabilityChange: voting file /dev/rhdisk11 now online
2014-12-05 14:31:18.564: [    CSSD][1]clssnmvDHBValidateNCopy: node 1, , has a disk HB, but no network HB, DHB has rcfg 313251759, wrtcnt, 355, LATS 519517307, lastSeqNo 0, uniqueness 1417760695, timestamp 1417761077/519370756</span>
2014-12-05 14:31:18.565: [    CSSD][1]clssscGetParameterOLR: OLR fetch for parameter node number hint (6) failed with rc 21
2014-12-05 14:31:18.565: [    CSSD][1]clssnmlpickslot:failed to read hint
2014-12-05 14:31:18.570: [   SKGFD][2057]NOTE: No asm libraries found in the system
....
2014-12-05 14:40:00.198: [    CSSD][2577]clssnmvDHBValidateNCopy: node 1, ora7c, has a disk HB, but no network HB, DHB has rcfg 313251759, wrtcnt, 876, LATS 520038938, lastSeqNo 875, uniqueness 1417760695, timestamp 1417761599/519892908
2014-12-05 14:40:01.023: [    CSSD][3348]clssgmWaitOnEventValue: after CmInfo State  val 3, eval 1 waited 0
2014-12-05 14:40:01.200: [    CSSD][2577]clssnmvDHBValidateNCopy: node 1, ora7c, has a disk HB, but no network HB, DHB has rcfg 313251759, wrtcnt, 877, LATS 520039940, lastSeqNo 876, uniqueness 1417760695, timestamp 1417761600/519893908
2014-12-05 14:40:02.024: [    CSSD][3348]clssgmWaitOnEventValue: after CmInfo State  val 3, eval 1 waited 0
2014-12-05 14:40:02.202: [    CSSD][2577]clssnmvDHBValidateNCopy: node 1, ora7c, has a disk HB, but no network HB, DHB has rcfg 313251759, wrtcnt, 878, LATS 520040942, lastSeqNo 877, uniqueness 1417760695, timestamp 1417761601/519894909

根据MOS的Note 1212703.1：/article/7857431.html

初步认为是bug，但还是和文档中描述不完全相同。所以文档中也提供了一个测试是否需要安装此bug的工具：mcasttest

工具下载及使用：

http://download.csdn.net/detail/u010587433/8234231

/article/7857428.html

根据测试结果是不需要安装此补丁的：

###########  testing Multicast on all nodes  ##########

Test for Multicast address 230.0.1.0

Dec  5 14:16:17 | Multicast Succeeded for en0 using address 230.0.1.0:42000
Dec  5 14:16:18 | Multicast Succeeded for en1 using address 230.0.1.0:42001

Test for Multicast address 224.0.0.251

Dec  5 14:16:19 | Multicast Succeeded for en0 using address 224.0.0.251:42002
Dec  5 14:16:20 | Multicast Succeeded for en1 using address 224.0.0.251:42003

但还是报同样的错，没有办法，还是先安装补丁，再试试了。

注：因为是新安装，需在执行root.sh时之前，安装此补丁

如果已执行root.sh且报错，需要执行/u01/app/11.2.0/grid/crs/install/rootcrs.pl-verbose -deconfig -force进行回滚（两个节点都需要执行），然后再安装补丁

下载补丁：p9974223_112020_AIX64-5L.zip（http://pan.baidu.com/s/1mg1Izc0）

分别上传到两台服务器，并解压(root用户)：unzip p9974223_112020_AIX64-5L.zip -d /

授权并使用grid安装（两个节点分别执行，不能使用readme进行安装，只需opatch napply）：

chown -R grid:oinstall /9974223
su - grid
cd /9974223
$ORACLE_HOME/OPatch/opatch napply -local -oh $ORACLE_HOME -id 9974223

输出：

Invoking OPatch 11.2.0.1.1

Oracle Interim Patch Installer version 11.2.0.1.1
Copyright (c) 2009, Oracle Corporation.  All rights reserved.

UTIL session

Oracle Home       : /u01/app/11.2.0/grid
Central Inventory : /u01/app/oraInventory
from           : /etc/oraInst.loc
OPatch version    : 11.2.0.1.1
OUI version       : 11.2.0.2.0
OUI location      : /u01/app/11.2.0/grid/oui
Log file location : /u01/app/11.2.0/grid/cfgtoollogs/opatch/opatch2014-12-05_15-38-05PM.log

Patch history file: /u01/app/11.2.0/grid/cfgtoollogs/opatch/opatch_history.txt

Invoking utility "napply"
Checking conflict among patches...
Checking if Oracle Home has components required by patches...
Checking conflicts against Oracle Home...
OPatch continues with these patches:   9974223

Do you want to proceed? [y|n]
<span style="color:#ff0000;">y</span>
User Responded with: Y

Running prerequisite checks...

You selected -local option, hence OPatch will patch the local system only.

Please shutdown Oracle instances running out of this ORACLE_HOME on the local system.
(Oracle Home = '/u01/app/11.2.0/grid')

Is the local system ready for patching? [y|n]
<span style="color:#ff0000;">y</span>
User Responded with: Y
Backing up files affected by the patch 'NApply' for restore. This might take a while...

Applying patch 9974223...

ApplySession applying interim patch '9974223' to OH '/u01/app/11.2.0/grid'
Backing up files affected by the patch '9974223' for rollback. This might take a while...

Patching component oracle.crs, 11.2.0.2.0...
Copying file to "/u01/app/11.2.0/grid/bin/appagent.bin"
Copying file to "/u01/app/11.2.0/grid/bin/appvipcfg"
Copying file to "/u01/app/11.2.0/grid/bin/appvipcfg.pl"
Copying file to "/u01/app/11.2.0/grid/bin/asmproxy"
Copying file to "/u01/app/11.2.0/grid/bin/cemutlo.bin"
Copying file to "/u01/app/11.2.0/grid/bin/cemutls.bin"
Copying file to "/u01/app/11.2.0/grid/bin/clscfg.bin"
Copying file to "/u01/app/11.2.0/grid/bin/clsecho.bin"
...
Copying file to "/u01/app/11.2.0/grid/oui/lib/aix/libsrvm11.so"
ApplySession adding interim patch '9974223' to inventory

Verifying the update...
Inventory check OK: Patch ID 9974223 is registered in Oracle Home inventory with proper meta-data.
Files check OK: Files from Patch ID 9974223 are present in Oracle Home.
Running make for target install_srvm
Running make for target install

The local system has been patched and can be restarted.

UtilSession: N-Apply done.

OPatch succeeded.

两个节点都成功后，再分别执行root.sh脚本，最后成功安装。

说明：

1.引起此bug的可能操作：11.2.0.2的RAC安装、从11.2.0.2之前版本升级到11.2.0.2、安装11.2.0.3.5/6/7的PSU、安装或升级到12.1.0.1.0

2.报错的行数不一定完全不一致：crsconfig_lib.pm line1020/1016

3.ocssd日志报错基本一致

4.验证组播工具的结果不完全可信