What is Split Brain in Oracle Clusterware and Real Application Cluster (文档 ID 1425586.1)
2017-12-14 21:54
567 查看
What is Split Brain in Oracle Clusterware and Real Application Cluster (文档 ID 1425586.1)
In this Document
Oracle Database - Enterprise Edition - Version 10.1.0.2 and later
Information in this document applies to any platform.
This note is to explain what is split brain in an Oracle Real Application cluster and what errors/consequences are associated with it.
For DBA and Support engineer.
In generic term, split-brain indicates data inconsistencies originating from the maintenance of two separate data sets with overlap in scope, either because of servers in a network design, or a failure
condition based on servers not communicating and unifying their data to each other.
There are two components in Oracle Real Application Cluster implementation could experience split brain.
Cluster nodes maintain their heartbeat via private network and voting disk. When there is a private network disruption, cluster nodes can not communicate to each other via private network for the time
period of miscount setting, split brain will happen. In such case, voting disk will be used to determine which node(s) survive and which node(s) will be evicted. The common voting result will be:
a. The group with more cluster nodes survive
b. The group with lower node member in case of same number of node(s) available in each group
c. Some improvement has been made to ensure node(s) with lower load survive in case the eviction is caused by high system load.
Commonly, one will see messages similar to the followings in ocssd.log when split brain happens:
Above messages indicate the communication from node 2 to node 1 is not working, hence node 2 only sees 1 node, but node 1 is working fine and it can see two nodes in the cluster. To avoid splitbrain,
node 2 aborted itself.
Solution: Please engage network administrator to check private network layer to eliminate any network fault.
To ensure data consistency, each instance of a RAC database needs to keep heartbeat with the other instances. The heartbeat is maintained by background processes like LMON, LMD, LMS and LCK. Any of these
processes experience IPC Send time out will incur communication reconfiguration and instance eviction to avoid split brain. Controlfile is used similarly to voting disk in clusterware layer to determine which instance(s) survive and which instance(s) evict.
The voting result is similar to clusterware voting result. As the result, 1 or more instance(s) will be evicted.
Common messages in instance alert log are similar to:
alert log of instance 1:
---------
Mon Dec 07 19:43:05 2011
IPC Send timeout detected.Sender: ospid 26318
Receiver: inst 2 binc 554466600 ospid 29940
IPC Send timeout to 2.0 inc 8 for msg type 65521 from opid 20
Mon Dec 07 19:43:07 2011
Communications reconfiguration: instance_number 2
Mon Dec 07 19:43:07 2011
Trace dumping is performing id=[cdmp_20091207194307]
Waiting for clusterware split-brain resolution
Mon Dec 07 19:53:07 2011
Evicting instance 2 from cluster
Waiting for instances to leave:
2
...
alert log of instance 2:
---------
Mon Dec 07 19:42:18 2011
IPC Send timeout detected. Receiver ospid 29940
Mon Dec 07 19:42:18 2011
Errors in file
/u01/app/oracle/diag/rdbms/bd/BD2/trace/BD2_lmd0_29940.trc:
Trace dumping is performing id=[cdmp_20091207194307]
Mon Dec 07 19:42:20 2011
Waiting for clusterware split-brain resolution
Mon Dec 07 19:44:45 2011
ERROR: LMS0 (ospid: 29942) detects an idle connection to instance 1
Mon Dec 07 19:44:51 2011
ERROR: LMD0 (ospid: 29940) detects an idle connection to instance 1
Mon Dec 07 19:45:38 2011
ERROR: LMS1 (ospid: 29954) detects an idle connection to instance 1
Mon Dec 07 19:52:27 2011
Errors in file
/u01/app/oracle/diag/rdbms/bd/BD2/trace/PVBD2_lmon_29938.trc
(incident=90153):
ORA-29740: evicted by member 0, group incarnation 10
Incident details in:
/u01/app/oracle/diag/rdbms/bd/BD2/incident/incdir_90153/BD2_lmon_29938_i90153.trc
In above example, instance 2 LMD0 (pid 29940) is the receiver in IPC Send timeout. There could be various reasons causing IPC Send timeout. For example:
a. Network problem
b. Process hang
c. Bug etc
Please see Top 5 issues for Instance Eviction Document 1374110.1 for
more information.
In case of instance eviction, alert log and all background traces need to be checked to determine the root cause.
In this Document
Purpose |
Scope |
Details |
1. Clusterware layer |
2. Real Application Cluster (database) layer |
Known Issues |
References |
APPLIES TO:
Oracle Database - Enterprise Edition - Version 10.1.0.2 and laterInformation in this document applies to any platform.
PURPOSE
This note is to explain what is split brain in an Oracle Real Application cluster and what errors/consequences are associated with it.
SCOPE
For DBA and Support engineer.
DETAILS
In generic term, split-brain indicates data inconsistencies originating from the maintenance of two separate data sets with overlap in scope, either because of servers in a network design, or a failurecondition based on servers not communicating and unifying their data to each other.
There are two components in Oracle Real Application Cluster implementation could experience split brain.
1. Clusterware layer
Cluster nodes maintain their heartbeat via private network and voting disk. When there is a private network disruption, cluster nodes can not communicate to each other via private network for the timeperiod of miscount setting, split brain will happen. In such case, voting disk will be used to determine which node(s) survive and which node(s) will be evicted. The common voting result will be:
a. The group with more cluster nodes survive
b. The group with lower node member in case of same number of node(s) available in each group
c. Some improvement has been made to ensure node(s) with lower load survive in case the eviction is caused by high system load.
Commonly, one will see messages similar to the followings in ocssd.log when split brain happens:
[ CSSD]2011-01-12 23:23:08.090 [1262557536] >TRACE: clssnmCheckDskInfo: Checking disk info... [ CSSD]2011-01-12 23:23:08.090 [1262557536] >ERROR: clssnmCheckDskInfo: Aborting local node to avoid splitbrain. [ CSSD]2011-01-12 23:23:08.090 [1262557536] >ERROR: : my node(2), Leader(2), Size(1) VS Node(1), Leader(1), Size(2) [ CSSD]2011-01-12 23:23:08.090 [1262557536] >ERROR: ################################### [ CSSD]2011-01-12 23:23:08.090 [1262557536] >ERROR: clssscExit: CSSD aborting ###################################
Above messages indicate the communication from node 2 to node 1 is not working, hence node 2 only sees 1 node, but node 1 is working fine and it can see two nodes in the cluster. To avoid splitbrain,
node 2 aborted itself.
Solution: Please engage network administrator to check private network layer to eliminate any network fault.
2. Real Application Cluster (database) layer
To ensure data consistency, each instance of a RAC database needs to keep heartbeat with the other instances. The heartbeat is maintained by background processes like LMON, LMD, LMS and LCK. Any of theseprocesses experience IPC Send time out will incur communication reconfiguration and instance eviction to avoid split brain. Controlfile is used similarly to voting disk in clusterware layer to determine which instance(s) survive and which instance(s) evict.
The voting result is similar to clusterware voting result. As the result, 1 or more instance(s) will be evicted.
Common messages in instance alert log are similar to:
alert log of instance 1:
---------
Mon Dec 07 19:43:05 2011
IPC Send timeout detected.Sender: ospid 26318
Receiver: inst 2 binc 554466600 ospid 29940
IPC Send timeout to 2.0 inc 8 for msg type 65521 from opid 20
Mon Dec 07 19:43:07 2011
Communications reconfiguration: instance_number 2
Mon Dec 07 19:43:07 2011
Trace dumping is performing id=[cdmp_20091207194307]
Waiting for clusterware split-brain resolution
Mon Dec 07 19:53:07 2011
Evicting instance 2 from cluster
Waiting for instances to leave:
2
...
alert log of instance 2:
---------
Mon Dec 07 19:42:18 2011
IPC Send timeout detected. Receiver ospid 29940
Mon Dec 07 19:42:18 2011
Errors in file
/u01/app/oracle/diag/rdbms/bd/BD2/trace/BD2_lmd0_29940.trc:
Trace dumping is performing id=[cdmp_20091207194307]
Mon Dec 07 19:42:20 2011
Waiting for clusterware split-brain resolution
Mon Dec 07 19:44:45 2011
ERROR: LMS0 (ospid: 29942) detects an idle connection to instance 1
Mon Dec 07 19:44:51 2011
ERROR: LMD0 (ospid: 29940) detects an idle connection to instance 1
Mon Dec 07 19:45:38 2011
ERROR: LMS1 (ospid: 29954) detects an idle connection to instance 1
Mon Dec 07 19:52:27 2011
Errors in file
/u01/app/oracle/diag/rdbms/bd/BD2/trace/PVBD2_lmon_29938.trc
(incident=90153):
ORA-29740: evicted by member 0, group incarnation 10
Incident details in:
/u01/app/oracle/diag/rdbms/bd/BD2/incident/incdir_90153/BD2_lmon_29938_i90153.trc
In above example, instance 2 LMD0 (pid 29940) is the receiver in IPC Send timeout. There could be various reasons causing IPC Send timeout. For example:
a. Network problem
b. Process hang
c. Bug etc
Please see Top 5 issues for Instance Eviction Document 1374110.1 for
more information.
In case of instance eviction, alert log and all background traces need to be checked to determine the root cause.
相关文章推荐
- What is Split Brain in Oracle Clusterware and Real Application Cluster (文档 ID 1425586.1)
- What is Split Brain in Oracle Clusterware and Real Application Cluster (Doc ID 1425586.1)
- What Is The Maximum Tablespace Size And Database Limit For An Oracle Database ? (文档 ID 1372905.1)
- How to Check Whether Oracle Binary/Instance is RAC Enabled and Relink Oracle Binary in RAC (文档 ID 28
- How to Check Whether Oracle Binary/Instance is RAC Enabled and Relink Oracle Binary in RAC (文档 ID 28
- How to Check Whether Oracle Binary/Instance is RAC Enabled and Relink Oracle Binary in RAC (文档 ID 28
- How to Delete From or Add Resource to OCR in Oracle Clusterware (文档 ID 1069369.1)
- How to Modify Public Network Information including VIP in Oracle Clusterware (文档 ID 276434.1)
- RAC and Oracle Clusterware Best Practices and Starter Kit (Platform Independent) (文档 ID 810394.1)
- What is a Flash Recovery Area and how to configure it ? (文档 ID 305648.1)
- Oracle Grid Infrastructure: Understanding Split-Brain Node Eviction (文档 ID 1546004.1)
- Oracle Grid Infrastructure: Understanding Split-Brain Node Eviction (文档 ID 1546004.1)
- How to Modify Private Network Information in Oracle Clusterware (文档 ID 283684.1)
- How To Query And Change The Oracle Hidden Parameters In Oracle 10g and 11g (文档 ID 315631.1)
- What is a Flash Recovery Area and how to configure it ? (文档 ID 305648.1)
- 10g Manual Database Creation in Oracle (Single Instance and RAC) (文档 ID 240052.1)
- What is the difference between Views and Materialized Views in Oracle?
- 对于诊断 Oracle Clusterware(CRS 或 GI)和 Real Application Cluster(RAC)问题的数据收集 (文档 ID 2017246.1)
- Procwatcher: Script to Monitor and Examine Oracle DB and Clusterware Processes (文档 ID 459694.1)
- What is a Context in Java and ApplicationContext vs WebApplicationContext