oracle_Listener in INTERMEDIATE status with "Not All Endpoints Registered"

在grid下crsctl stat res -t查询时出现下面状况:
ora.LISTENER.lsnr

               ONLINE  INTERMEDIATE towg1a                   Not All Endpoints Registered           

               ONLINE  ONLINE       towg1b          
 
 

查看官方文档如下：

Applies to:
Oracle Server - Enterprise Edition - Version 11.2.0.1 and later
Information in this document applies to any platform.
Symptoms

2 node RAC cluster, trying to add 2nd instance using dbca, get error:
"Connection to the database cannot be established because the listener could be down. Please make sure that the service is registered with a listener and the listener is up."

$crsctl stat res -t shows the listeners are in INTERMEDIATE status on node 2.
ora.LISTENER.lsnr
       ONLINE   ONLINE               racdb1
       ONLINE   INTERMEDIATE         racdb2   Not All Endpoints Registered
ora.LISTENER_SCAN1.lsnr
   1   ONLINE   INTERMEDIATE         racdb2    Not All Endpoints Registered

 
Cause

The problem is caused by another listener defined statically in listener.ora, using the same port and IP is running from the RDBMS ORACLE_HOME, started manually causing the default listener starting from GRID_HOME can not register its endpoint. Hence the
error reported in dbca.
ps -ef | grep tns:

grid   7222 1 0 Apr26 ? 00:00:13 /u01/app/11.2.0/grid/bin/tnslsnr LISTENER_SCAN1 -inherit
grid   7237 1 0 Apr26 ? 00:00:13 /u01/app/11.2.0/grid/bin/tnslsnr LISTENER -inherit
oracle 7354 1 0 Apr26 ? 00:00:01 /u02/app/oracle/product/11.2.0/db/bin/tnslsnr LISTENER -inherit

  Another possible cause is the listener or scan listener being defined manually in listener.ora, for example:
LISTENER_SCAN3 =                          
 (DESCRIPTION =
   (ADDRESS = (PROTOCOL = TCP)(HOST = racnode1)(PORT = 1523))
 )

LISTENER_SCAN1 =                            
 (DESCRIPTION =
   (ADDRESS = (PROTOCOL = TCP)(HOST = racnode1)(PORT = 1521))

 

 
Solution

From 11.2 onwards, all listeners should be runing from GRID_HOME, listener and listener_scan entry should be added automatically into listener.ora, no manual editing is required for TCP definition.

1. Stop the listener running from RDBMS ORACLE_HOME

$/bin/lsnrctl stop LISTENER

2. stop the listener from GRID_HOME

$/bin/srvctl stop listener -n
$/bin/srvctl stop scan_listener -i

eg:

$/bin/srvctl stop listener -n racnode1
$/bin/srvctl stop scan_listener -i 1

If above command fails to stop the tnslsnr process, please use "kill -9 " to stop the LISTENER and LISTENER_SCAN1 process.

3. remove any manually added LISTENER definition from listener.ora if it exists

4. restart the LISTENER and LISTENER_SCAN1  from GRID_HOME

$/bin/srvctl start listener -n
$/bin/srvctl start scan_listener -i

5. check crsctl stat res -t output, they both should show ONLINE status now.

这台该死的机器上装着RAC的一个节点，就是出问题的节点；还装着一个单实例数据库。先把单实例的监听停掉，然后按照官方文档步骤2，停掉grid下的监听，然后再启动grid下的监听；再启动单实例的监听。草！