terminating the instance due to error 119

RAC安装过程一切正常，关机重启后，GI启动正常，实例没起来。

[root@rac1 network-scripts]# crsctl stat res -t
--------------------------------------------------------------------------------
NAME TARGET STATE SERVER STATE_DETAILS
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.DATA.dg
ONLINE ONLINE rac1
ONLINE ONLINE rac2
ora.FRA.dg
ONLINE ONLINE rac1
ONLINE ONLINE rac2
ora.LISTENER.lsnr
ONLINE ONLINE rac1
ONLINE ONLINE rac2
ora.OCR.dg
ONLINE ONLINE rac1
ONLINE ONLINE rac2
ora.asm
ONLINE ONLINE rac1 Started
ONLINE ONLINE rac2 Started
ora.gsd
OFFLINE OFFLINE rac1
OFFLINE OFFLINE rac2
ora.net1.network
ONLINE ONLINE rac1
ONLINE ONLINE rac2
ora.ons
ONLINE ONLINE rac1
ONLINE ONLINE rac2
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.LISTENER_SCAN1.lsnr
1 ONLINE ONLINE rac1
ora.LISTENER_SCAN2.lsnr
1 ONLINE ONLINE rac2
ora.LISTENER_SCAN3.lsnr
1 ONLINE ONLINE rac2
ora.cvu
1 ONLINE ONLINE rac2
ora.oc4j
1 ONLINE ONLINE rac2
ora.rac.db
1 ONLINE OFFLINE Instance Shutdown
2 ONLINE OFFLINE Instance Shutdown
ora.rac1.vip
1 ONLINE ONLINE rac1
ora.rac2.vip
1 ONLINE ONLINE rac2
ora.scan1.vip
1 ONLINE ONLINE rac1
ora.scan2.vip
1 ONLINE ONLINE rac2
ora.scan3.vip
1 ONLINE ONLINE rac2

马上查数据库日志，有如下发现

[oracle@rac1 trace]$ tail -200f alert_RAC1.log
USER (ospid: 3610): terminating the instance due to error 119
Instance terminated by USER, pid = 3610

强行启动行不行呢？
[root@rac1 ~]# srvctl start database -d RAC
PRCR-1079 : 无法启动资源 ora.rac.db
CRS-5017: The resource action "ora.rac.db start" encountered the following error:
ORA-00119: invalid specification for system parameter REMOTE_LISTENER
ORA-00132: syntax error or unresolved network name 'rac-scan.us.oracle.com:1521'
. For details refer to "(:CLSN00107:)" in "/u01/app/11.2.0/grid/log/rac2/agent/crsd/oraagent_oracle/oraagent_oracle.log".

CRS-5017: The resource action "ora.rac.db start" encountered the following error:
ORA-00119: invalid specification for system parameter REMOTE_LISTENER
ORA-00132: syntax error or unresolved network name 'rac-scan.us.oracle.com:1521'
. For details refer to "(:CLSN00107:)" in "/u01/app/11.2.0/grid/log/rac1/agent/crsd/oraagent_oracle/oraagent_oracle.log".

CRS-2674: Start of 'ora.rac.db' on 'rac2' failed
CRS-2674: Start of 'ora.rac.db' on 'rac1' failed
CRS-2632: There are no more servers to try to place resource 'ora.rac.db' on that would satisfy its placement policy

注意到报错信息中提到ORA-00132: syntax error or unresolved network name 'rac-scan.us.oracle.com:1521'，是不是呢？
反正和网络有关先tnsping一下
[grid@rac1 ~]$ tnsping rac2

TNS Ping Utility for Linux: Version 11.2.0.4.0 - Production on 09-DEC-2015 00:56:12

Copyright (c) 1997, 2013, Oracle. All rights reserved.

Used parameter files:
/u01/app/11.2.0/grid/network/admin/sqlnet.ora

Used EZCONNECT adapter to resolve the alias
Attempting to contact (DESCRIPTION=(CONNECT_DATA=(SERVICE_NAME=))(ADDRESS=(PROTOCOL=TCP)(HOST=192.168.56.12)(PORT=1521)))
OK (0 msec)
看来到监听是没问题的
不放心，再看一眼tnsnames.ora
[root@rac1 ~]# cat /u01/app/oracle/product/11.2.0/dbhome_1/network/admin/tnsnames.ora
# tnsnames.ora Network Configuration File: /u01/app/oracle/product/11.2.0/dbhome_1/network/admin/tnsnames.ora
# Generated by Oracle configuration tools.

RAC =
(DESCRIPTION =
(ADDRESS = (PROTOCOL = TCP)(HOST = rac-scan.us.oracle.com)(PORT = 1521))
(CONNECT_DATA =
(SERVER = DEDICATED)
(SERVICE_NAME = RAC)
)
)
[root@rac1 ~]# ping rac-scan.us.oracle.com
ping: unknown host rac-scan.us.oracle.com
这就有问题了，我在DNS中配置的域名怎么会不可达呢？安装时一切功能都正常的呀！
[root@rac1 ~]# nslookup rac-scan.us.oracle.com
Server: 192.168.1.1
Address: 192.168.1.1#53

** server can't find rac-scan.us.oracle.com: NXDOMAIN
更有问题了，DNS服务器怎么成192.168.1.1了呢？
依次对照所有的DNS配置文件，发现了一个异常的

[root@rac1 ~]# cat /etc/resolv.conf
; generated by /sbin/dhclient-script
nameserver 192.168.1.1
nameserver 192.168.0.1
怎么回到了操作系统最初始的配置了呢？这就要提到另外一个配置文件
[root@rac2 network-scripts]# cat ifcfg-eth0
DEVICE=eth0
TYPE=Ethernet
ONBOOT=yes
NM_CONTROLLED=yes
BOOTPROTO=dhcp
DEFROUTE=yes
PEERDNS=yes
PEERROUTES=yes
IPV4_FAILURE_FATAL=yes
IPV6INIT=no
NAME="System eth0"
USERCTL=no
注意到其中的PEERDNS=yes这一项了吗？
在5.0时代DNS服务器写在 /etc/resolv.conf 文件中，但到了6.0时代DNS可以写在/etc/resolv.conf但是此时需要在 /etc/sysconfig/network-scripts/ifcfg-eth0 文件中添加 PEERDNS=no 配置，不然每次重启网卡就会重写/etc/resolv.conf文件的内容。 http://blog.chinaunix.net/uid-26495963-id-3230810.html