[INS-20802] Oracle Net Configuration Assistant failed. 故障解决

因为项目需要搭建ORACLE11G，版本：11.2.0.3 64位，操作系统版本为：RHEL5.4_x64，在安装GRID的过程中产生如下错误：

根据错误提示，查看安装日志：

[grid@db1 ~]$ tail -n 100 /u01/app/oraInventory/logs/installActions2014-02-20_06-04-03PM.log | more
INFO: ConfigClient.init sOracleHomeLoc=/u01/app/11.2.0/grid, sTopAggId=oracle.crs
INFO: Entering ConfigClient.initClientAndSession method
INFO: ConfigClient.initClientAndSession status : connected to the config framework
INFO: ConfigClient.initClientAndSession session begun oracle.sysman.emCfg.client.CfwSession@368bca43
INFO: ConfigClient.initClientAndSession m_oAggregate=[Loracle.sysman.emCfg.common.IAggregate;@138c10af
INFO: Configuration log directory - /u01/app/11.2.0/grid/cfgtoollogs
INFO: Exiting ConfigClient.init method
INFO: Calling event ConfigSessionInitialized
INFO: ConfigClient.registerOutErrStream method called
INFO: Updating Transient Parameters in the Aggregates
INFO: Updating Transient Parameters for oracle.crs
INFO: Updating Transient Parameters for oracle.assistants.netca.client
INFO: Updating Transient Parameters for OuiConfigVariables
INFO: Updating Transient Parameters for oracle.assistants.asm
INFO: Setting value to transient variable S_ASMMONITORPASSWORD
INFO: Setting value to transient variable S_ASMPASSWORD
INFO: Setting value to transient variable s_asmcaArgs
INFO: Setting value to transient variable s_ASMPasswordFlag
INFO: Setting value to transient variable s_ASMMonitorPasswordFlag
INFO: Successfully updated Transient Parameters in the Aggregates
INFO: Entering ConfigClient.getToolsList method
INFO: Created config job for Update Inventory
INFO: Created config job for Oracle Net Configuration Assistant
INFO: Created config job for Automatic Storage Management Configuration Assistant
INFO: Created config job for Oracle Cluster Verification Utility
INFO: Configuration log directory - /u01/app/11.2.0/grid/cfgtoollogs
INFO:
The Runconfig command constructed is /u01/app/11.2.0/grid/oui/bin/runConfig.sh ORACLE_HOME=/u01/app/11.2.0/grid MODE=perform ACTION=configure RERUN=true $*
INFO: Created a new file /u01/app/11.2.0/grid/cfgtoollogs/configToolAllCommands
INFO: Since the option is to overwrite the existing /u01/app/11.2.0/grid/cfgtoollogs/configToolAllCommands file, backing it up
INFO: The backed up file name is /u01/app/11.2.0/grid/cfgtoollogs/configToolAllCommands.bak
INFO: Entering ConfigClient.executeToolsInAggregate method
INFO: ConfigClient.executeToolsInAggregate oAggregate=oracle.crs:oracle.crs:11.2.0.3.0:common
INFO: Entering ConfigClient.assignAction method
INFO: ConfigClient.assignAction actionref=[Loracle.sysman.emCfg.common.IActionReference;@46c51ce4
INFO: ConfigClient.assignAction action is of type :configuration
INFO: Exiting ConfigClient.assignAction method
INFO: ConfigClient.executeToolsInAggregate action assigned
INFO: Started Plugin named: Update Inventory
INFO: Found associated job
INFO: Starting 'Update Inventory'
INFO: Starting 'Update Inventory'
INFO: /u01/app/11.2.0/grid/oui/bin/runInstaller -nowait -noconsole -waitforcompletion -ignoreSysPrereqs -updateNodeList -silent CRS=true "CLUSTER_NODES={db1,db2}" ORACL
E_HOME=/u01/app/11.2.0/grid
INFO: Starting Oracle Universal Installer...
INFO: Checking swap space: must be greater than 500 MB.   Actual 53199 MB    Passed
INFO: The inventory pointer is located at /etc/oraInst.loc
INFO: The inventory is located at /u01/app/oraInventory
INFO:
WARNING:
INFO: Completed Plugin named: Update Inventory
INFO: Completed 'Update Inventory'
INFO: Completed 'Update Inventory'
INFO: Started Plugin named: Oracle Net Configuration Assistant
INFO: Found associated job
INFO: Starting 'Oracle Net Configuration Assistant'
INFO: Starting 'Oracle Net Configuration Assistant'
INFO: Parsing command line arguments:
INFO:     Parameter "orahome" = /u01/app/11.2.0/grid
INFO:     Parameter "orahnam" = Ora11g_gridinfrahome1
INFO:     Parameter "instype" = typical
INFO:     Parameter "inscomp" = client,oraclenet,javavm,server
INFO:     Parameter "insprtcl" = tcp
INFO:     Parameter "cfg" = local
INFO:     Parameter "authadp" = NO_VALUE
INFO:     Parameter "responsefile" = /u01/app/11.2.0/grid/network/install/netca_typ.rsp
INFO:     Parameter "silent" = true
INFO: Done parsing command line arguments.
INFO: Oracle Net Services Configuration:
INFO: Profile configuration complete.
INFO: Profile configuration complete.
INFO: db1...
INFO: db2...
INFO: Oracle Net Listener Startup:
INFO:     Listener start failed.
INFO: Check the trace file for details: /u01/app/grid/cfgtoollogs/netca/trace_Ora11g_gridinfrahome1-1402206PM4137.log
INFO: Oracle Net Services configuration failed.  The exit code is 1
INFO:
WARNING:
INFO: Completed Plugin named: Oracle Net Configuration Assistant
INFO: Oracle Net Configuration Assistant failed.
INFO: Oracle Net Configuration Assistant failed.
INFO: ConfigClient.executeToolsInAggregate action performed
INFO: Exiting ConfigClient.executeToolsInAggregate method
INFO: Calling event ConfigToolsExecuted
INFO:
The Runconfig command constructed is /u01/app/11.2.0/grid/oui/bin/runConfig.sh ORACLE_HOME=/u01/app/11.2.0/grid MODE=perform ACTION=configure RERUN=false $*
INFO: Created a new file /u01/app/11.2.0/grid/cfgtoollogs/configToolFailedCommands
INFO: Since the option is to overwrite the existing /u01/app/11.2.0/grid/cfgtoollogs/configToolFailedCommands file, backing it up
INFO: The backed up file name is /u01/app/11.2.0/grid/cfgtoollogs/configToolFailedCommands.bak
INFO: ConfigClient.saveSession method called
INFO: Calling event ConfigSessionEnding
INFO: ConfigClient.endSession method called
INFO: Completed Configuration
INFO: Shutting down OUISetupDriver.JobExecutorThread
INFO: Cleaning up, please wait...
INFO: Dispose the install area control object
INFO: Update the state machine to STATE_CLEAN
INFO: All forked task are completed at state setup
INFO: Completed background operations
INFO: Moved to state <setup>

==========================================================================

WARNING:
INFO: Completed Plugin named: Oracle Net Configuration Assistant
INFO: Oracle Net Configuration Assistant failed.
INFO: Oracle Net Configuration Assistant failed.
INFO: ConfigClient.executeToolsInAggregate action performed
INFO: Exiting ConfigClient.executeToolsInAggregate method
INFO: Calling event ConfigToolsExecuted

注意这里的提示，说明grid的listener的安装好的，但是没有启动起来，启动失败，原来我以为是没有创建listenr,后来用netca重新建立的时候发现已经有监听了，于是我就开始手动启动

[grid@db1 trace]$ lsnrctl start
LSNRCTL for Linux: Version 11.2.0.3.0 - Production on 21-FEB-2014 14:22:45
Copyright (c) 1991, 2011, Oracle.  All rights reserved.
Starting /u01/app/11.2.0/grid/bin/tnslsnr: please wait...
TNSLSNR for Linux: Version 11.2.0.3.0 - Production
System parameter file is /u01/app/11.2.0/grid/network/admin/listener.ora
Log messages written to /u01/app/grid/diag/tnslsnr/db1/listener/alert/log.xml
Listening on: (DESCRIPTION=(ADDRESS=(PROTOCOL=ipc)(KEY=LISTENER)))
Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=IPC)(KEY=LISTENER)))
TNS-12547: TNS:lost contact
TNS-12560: TNS:protocol adapter error
TNS-00517: Lost contact
Linux Error: 104: Connection reset by peer

从这里看来，真是有点问题了。

[grid@db1 ~]$ crsctl status resource -t
--------------------------------------------------------------------------------
NAME           TARGET  STATE        SERVER                   STATE_DETAILS
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.GRIDDG.dg
ONLINE  ONLINE       db1
ONLINE  ONLINE       db2
ora.LISTENER.lsnr
ONLINE  OFFLINE      db1
ONLINE  OFFLINE      db2
ora.asm
ONLINE  ONLINE       db1                      Started
ONLINE  ONLINE       db2                      Started
ora.gsd
OFFLINE OFFLINE      db1
OFFLINE OFFLINE      db2
ora.net1.network
ONLINE  ONLINE       db1
ONLINE  ONLINE       db2
ora.ons
ONLINE  OFFLINE      db1
ONLINE  OFFLINE      db2
ora.registry.acfs
ONLINE  ONLINE       db1
ONLINE  ONLINE       db2
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.LISTENER_SCAN1.lsnr
1        ONLINE  OFFLINE
ora.cvu
1        ONLINE  ONLINE       db2
ora.db1.vip
1        ONLINE  ONLINE       db1
ora.db2.vip
1        ONLINE  ONLINE       db2
ora.oc4j
1        ONLINE  OFFLINE
ora.scan1.vip
1        ONLINE  ONLINE       db1

从集群资源来看凡是涉及监听的资源都没有启动起来。

[grid@db1 ~]$ crs_start ora.LISTENER.lsnr
Attempting to start `ora.LISTENER.lsnr` on member `db1`
Attempting to start `ora.LISTENER.lsnr` on member `db2`
CRS-5016: Process "/u01/app/11.2.0/grid/bin/lsnrctl" spawned by agent "/u01/app/11.2.0/grid/bin/oraagent.bin" for action "start" failed: details at "(:CLSN00010:)" in "/u01/app/11.2.0/grid/log/db1/agent/crsd/oraagent_grid/oraagent_grid.log"
CRS-5016: Process "/u01/app/11.2.0/grid/bin/lsnrctl" spawned by agent "/u01/app/11.2.0/grid/bin/oraagent.bin" for action "start" failed: details at "(:CLSN00010:)" in "/u01/app/11.2.0/grid/log/db2/agent/crsd/oraagent_grid/oraagent_grid.log"
CRS-5016: Process "/u01/app/11.2.0/grid/bin/lsnrctl" spawned by agent "/u01/app/11.2.0/grid/bin/oraagent.bin" for action "start" failed: details at "(:CLSN00010:)" in "/u01/app/11.2.0/grid/log/db1/agent/crsd/oraagent_grid/oraagent_grid.log"
CRS-5016: Process "/u01/app/11.2.0/grid/bin/lsnrctl" spawned by agent "/u01/app/11.2.0/grid/bin/oraagent.bin" for action "start" failed: details at "(:CLSN00010:)" in "/u01/app/11.2.0/grid/log/db2/agent/crsd/oraagent_grid/oraagent_grid.log"
Start of `ora.LISTENER.lsnr` on member `db1` failed.
Start of `ora.LISTENER.lsnr` on member `db2` failed.
CRS-0215: Could not start resource 'ora.LISTENER.lsnr db1 1'.
CRS-0215: Could not start resource 'ora.LISTENER.lsnr db2 1'.

用群集管理工具也是无法启动，我有些安奈不住了，从分析日志入手，

tail -n 100 /u01/app/11.2.0/grid/log/db1/agent/crsd/oraagent_grid/oraagent_grid.log | more
2014-02-21 10:53:36.063: [ora.LISTENER.lsnr][1320700224] {1:8446:793} [start] (:CLSN00010:)
2014-02-21 10:53:36.063: [ora.LISTENER.lsnr][1320700224] {1:8446:793} [start] (:CLSN00010:)Copyright (c) 1991, 2011, Oracle.  All rights reserved.
2014-02-21 10:53:36.063: [ora.LISTENER.lsnr][1320700224] {1:8446:793} [start] (:CLSN00010:)
2014-02-21 10:53:36.064: [ora.LISTENER.lsnr][1320700224] {1:8446:793} [start] (:CLSN00010:)Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=IPC)(KEY=LISTENER)))
2014-02-21 10:53:36.064: [ora.LISTENER.lsnr][1320700224] {1:8446:793} [start] (:CLSN00010:)TNS-12541: TNS:no listener
2014-02-21 10:53:36.064: [ora.LISTENER.lsnr][1320700224] {1:8446:793} [start] (:CLSN00010:) TNS-12560: TNS:protocol adapter error
2014-02-21 10:53:36.064: [ora.LISTENER.lsnr][1320700224] {1:8446:793} [start] (:CLSN00010:)  TNS-00511: No listener
2014-02-21 10:53:36.064: [ora.LISTENER.lsnr][1320700224] {1:8446:793} [start] (:CLSN00010:)   Linux Error: 111: Connection refused
2014-02-21 10:53:36.064: [ora.LISTENER.lsnr][1320700224] {1:8446:793} [start] (:CLSN00010:)

日志也没有任何可以参考性，于是从操作系统开始分析，

[grid@db2 ~]$ more /etc/hosts
# Do not remove the following line, or various programs
# that require network functionality will fail.
#127.0.0.1               db1 localhost.localdomain localhost
#::1             localhost6.localdomain6 localhost6
192.169.2.176   db1
192.169.2.177   db2
# Private
10.0.0.1        db1-priv
10.0.0.2        db2-priv
# Virtual
192.169.2.178   db1-vip
192.169.2.179   db2-vip
# SCAN
192.169.2.185   rac-scan

发现loop地址被注释掉，于是去除注释，重启监听服务，

[grid@db1 ~]$ lsnrctl start
LSNRCTL for Linux: Version 11.2.0.3.0 - Production on 21-FEB-2014 14:27:50
Copyright (c) 1991, 2011, Oracle.  All rights reserved.
Starting /u01/app/11.2.0/grid/bin/tnslsnr: please wait...
TNSLSNR for Linux: Version 11.2.0.3.0 - Production
System parameter file is /u01/app/11.2.0/grid/network/admin/listener.ora
Log messages written to /u01/app/grid/diag/tnslsnr/db1/listener/alert/log.xml
Listening on: (DESCRIPTION=(ADDRESS=(PROTOCOL=ipc)(KEY=LISTENER)))
Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=IPC)(KEY=LISTENER)))
STATUS of the LISTENER
------------------------
Alias                     LISTENER
Version                   TNSLSNR for Linux: Version 11.2.0.3.0 - Production
Start Date                21-FEB-2014 14:27:52
Uptime                    0 days 0 hr. 0 min. 0 sec
Trace Level               off
Security                  ON: Local OS Authentication
SNMP                      OFF
Listener Parameter File   /u01/app/11.2.0/grid/network/admin/listener.ora
Listener Log File         /u01/app/grid/diag/tnslsnr/db1/listener/alert/log.xml
Listening Endpoints Summary...
(DESCRIPTION=(ADDRESS=(PROTOCOL=ipc)(KEY=LISTENER)))
The listener supports no services
The command completed successfully

哈哈 监听竟然真的好了！！在重启集群资源

[grid@db1 ~]$ crs_start -all
CRS-5702: Resource 'ora.GRIDDG.dg' is already running on 'db1'
CRS-5702: Resource 'ora.LISTENER.lsnr' is already running on 'db1'
CRS-5702: Resource 'ora.asm' is already running on 'db1'
CRS-5702: Resource 'ora.cvu' is already running on 'db2'
CRS-5702: Resource 'ora.asm' is already running on 'db1'
CRS-5702: Resource 'ora.LISTENER.lsnr' is already running on 'db1'
CRS-2501: Resource 'ora.gsd' is disabled
CRS-5702: Resource 'ora.db1.vip' is already running on 'db1'
CRS-5702: Resource 'ora.asm' is already running on 'db2'
CRS-5702: Resource 'ora.LISTENER.lsnr' is already running on 'db2'
CRS-2501: Resource 'ora.gsd' is disabled
CRS-5702: Resource 'ora.db2.vip' is already running on 'db2'
CRS-2501: Resource 'ora.gsd' is disabled
CRS-5702: Resource 'ora.net1.network' is already running on 'db1'
CRS-5702: Resource 'ora.oc4j' is already running on 'db1'
CRS-5702: Resource 'ora.registry.acfs' is already running on 'db1'
CRS-5702: Resource 'ora.scan1.vip' is already running on 'db2'
Attempting to start `ora.ons` on member `db2`
Attempting to start `ora.ons` on member `db1`
Attempting to start `ora.LISTENER_SCAN1.lsnr` on member `db2`
Start of `ora.ons` on member `db1` succeeded.
CRS-5702: Resource 'ora.ons' is already running on 'db1'
Start of `ora.ons` on member `db2` succeeded.
CRS-5702: Resource 'ora.ons' is already running on 'db2'
Start of `ora.LISTENER_SCAN1.lsnr` on member `db2` succeeded.
CRS-0223: Resource 'ora.GRIDDG.dg' has placement error.
CRS-0223: Resource 'ora.LISTENER.lsnr' has placement error.
CRS-0223: Resource 'ora.asm' has placement error.
CRS-0223: Resource 'ora.cvu' has placement error.
CRS-0223: Resource 'ora.db1.ASM1.asm' has placement error.
CRS-0223: Resource 'ora.db1.LISTENER_DB1.lsnr' has placement error.
CRS-2660: Resource 'ora.db1.gsd' or all of its instances are disabled
CRS-0223: Resource 'ora.db1.vip' has placement error.
CRS-0223: Resource 'ora.db2.ASM2.asm' has placement error.
CRS-0223: Resource 'ora.db2.LISTENER_DB2.lsnr' has placement error.
CRS-2660: Resource 'ora.db2.gsd' or all of its instances are disabled
CRS-0223: Resource 'ora.db2.vip' has placement error.
CRS-2660: Resource 'ora.gsd' or all of its instances are disabled
CRS-0223: Resource 'ora.net1.network' has placement error.
CRS-0223: Resource 'ora.oc4j' has placement error.
CRS-0223: Resource 'ora.registry.acfs' has placement error.
CRS-0223: Resource 'ora.scan1.vip' has placement error.
[grid@db1 ~]$ crs_stat -t
Name           Type           Target    State     Host
------------------------------------------------------------
ora.GRIDDG.dg  ora....up.type ONLINE    ONLINE    db1
ora....ER.lsnr ora....er.type ONLINE    ONLINE    db1
ora....N1.lsnr ora....er.type ONLINE    ONLINE    db2
ora.asm        ora.asm.type   ONLINE    ONLINE    db1
ora.cvu        ora.cvu.type   ONLINE    ONLINE    db2
ora....SM1.asm application    ONLINE    ONLINE    db1
ora....B1.lsnr application    ONLINE    ONLINE    db1
ora.db1.gsd    application    OFFLINE   OFFLINE
ora.db1.ons    application    ONLINE    ONLINE    db1
ora.db1.vip    ora....t1.type ONLINE    ONLINE    db1
ora....SM2.asm application    ONLINE    ONLINE    db2
ora....B2.lsnr application    ONLINE    ONLINE    db2
ora.db2.gsd    application    OFFLINE   OFFLINE
ora.db2.ons    application    ONLINE    ONLINE    db2
ora.db2.vip    ora....t1.type ONLINE    ONLINE    db2
ora.gsd        ora.gsd.type   OFFLINE   OFFLINE
ora....network ora....rk.type ONLINE    ONLINE    db1
ora.oc4j       ora.oc4j.type  ONLINE    ONLINE    db1
ora.ons        ora.ons.type   ONLINE    ONLINE    db1
ora....ry.acfs ora....fs.type ONLINE    ONLINE    db1
ora.scan1.vip  ora....ip.type ONLINE    ONLINE    db2

集群资源正常启动，监听也很正常，到这里我终于发现“我们过于急迫追求结果，往往忽略的过程和细节，到头来，方知细节过程之重要！！