Oracle RAC Study之--10gR2 RAC环境中在线添加、删除Votedisk和在线替换OCR的探讨

Oracle RAC Study之--10gR2 RAC环境中在线添加、删除Votedisk和在线替换OCR的探讨 这里我们不讨论11gR2 RAC在线添加、删除Votedisk和OCR，在11gR2平台可以非常容易的实现。这里也不讨论Votedisk和OCR的还原，还原操作必须在停止Clusterware的情况下完成。下面以实际的例子讨论10gR2 RAC for Linux环境下，Votedisk和OCR的在线添加和删除。
一.Votedisk的在线添加和删除。
1).查看集群资源的状态：

[root@rhel1 bin]# ./crs_stat -t

Name           Type           Target    State     Host
------------------------------------------------------------
ora....L1.lsnr application    ONLINE    ONLINE    rhel1
ora.rhel1.gsd  application    ONLINE    ONLINE    rhel1
ora.rhel1.ons  application    ONLINE    ONLINE    rhel1
ora.rhel1.vip  application    ONLINE    ONLINE    rhel1
ora....L2.lsnr application    ONLINE    ONLINE    rhel2
ora.rhel2.gsd  application    ONLINE    ONLINE    rhel2
ora.rhel2.ons  application    ONLINE    ONLINE    rhel2
ora.rhel2.vip  application    ONLINE    ONLINE    rhel2

2).查看当前votedisk位置及文件名：
[root@rhel1 bin]# ./crsctl query css votedisk

0.     0    /u02/vdisk/vdisk1
1.     0    /u02/vdisk/vdisk2
2.     0    /u02/vdisk/vdisk3
located 3 votedisk(s).

3).查看ocssd进程号：

[root@rhel1 bin]# ps -e | grep -i ocssd
5942 ? 00:00:21 ocssd.bin

4).查看ocssd进程定位的votedisk文件：
[root@rhel1 bin]# cd /proc/5942/fd/

[root@rhel1 fd]# ll
total 0
lr-x------ 1 oracle oinstall 64 Jun 12 18:18 0 -> /dev/null
lrwx------ 1 oracle oinstall 64 Jun 12 18:18 1 -> /u01/app/oracle/crs/log/rhel1/cssd/cssdOUT.log
lrwx------ 1 oracle oinstall 64 Jun 12 18:18 10 -> socket:[13927]
lrwx------ 1 oracle oinstall 64 Jun 12 18:18 11 -> /u02/vdisk/vdisk1
lrwx------ 1 oracle oinstall 64 Jun 12 18:18 12 -> /u02/vdisk/vdisk1
lrwx------ 1 oracle oinstall 64 Jun 12 18:18 13 -> /u02/vdisk/vdisk2
lrwx------ 1 oracle oinstall 64 Jun 12 18:18 14 -> /u02/vdisk/vdisk2
lrwx------ 1 oracle oinstall 64 Jun 12 18:18 15 -> /u02/vdisk/vdisk3
lrwx------ 1 oracle oinstall 64 Jun 12 18:18 16 -> /u02/vdisk/vdisk3
l-wx------ 1 oracle oinstall 64 Jun 12 18:18 17 -> /u01/app/oracle/crs/log/rhel1/alertrhel1.log
lr-x------ 1 oracle oinstall 64 Jun 12 18:18 18 -> /u01/app/oracle/crs/has/mesg/clsdus.msb
lrwx------ 1 oracle oinstall 64 Jun 12 18:18 19 -> socket:[13974]
lrwx------ 1 oracle oinstall 64 Jun 12 18:18 2 -> /u01/app/oracle/crs/log/rhel1/cssd/cssdOUT.log
lrwx------ 1 oracle oinstall 64 Jun 12 18:18 20 -> socket:[13977]
lrwx------ 1 oracle oinstall 64 Jun 12 18:18 21 -> socket:[14446]
lrwx------ 1 oracle oinstall 64 Jun 12 18:18 22 -> socket:[13984]
lrwx------ 1 oracle oinstall 64 Jun 12 18:18 23 -> socket:[13987]
lrwx------ 1 oracle oinstall 64 Jun 12 18:18 24 -> socket:[13989]
lrwx------ 1 oracle oinstall 64 Jun 12 18:18 25 -> socket:[13993]
lrwx------ 1 oracle oinstall 64 Jun 12 18:18 26 -> socket:[13995]
lrwx------ 1 oracle oinstall 64 Jun 12 18:18 27 -> socket:[13996]
lrwx------ 1 oracle oinstall 64 Jun 12 18:18 28 -> socket:[14447]
lrwx------ 1 oracle oinstall 64 Jun 12 18:18 29 -> socket:[14733]
lrwx------ 1 oracle oinstall 64 Jun 12 18:18 3 -> /u01/app/oracle/crs/log/rhel1/cssd/cssdOUT.log
lrwx------ 1 oracle oinstall 64 Jun 12 18:18 30 -> socket:[14449]
lrwx------ 1 oracle oinstall 64 Jun 12 18:18 31 -> socket:[14739]
lr-x------ 1 oracle oinstall 64 Jun 12 18:18 4 -> /u01/app/oracle/crs/srvm/mesg/procus.msb
lr-x------ 1 oracle oinstall 64 Jun 12 18:18 5 -> /u02/ocr/ocr1
lr-x------ 1 oracle oinstall 64 Jun 12 18:18 6 -> /u02/ocr/ocr2
lr-x------ 1 oracle oinstall 64 Jun 12 18:18 7 -> /u01/app/oracle/crs/css/mesg/clssus.msb
l-wx------ 1 oracle oinstall 64 Jun 12 18:18 8 -> /u01/app/oracle/crs/log/rhel1/cssd/ocssd.log
lrwx------ 1 oracle oinstall 64 Jun 12 18:18 9 -> socket:[13926]

文件说明当前ocssd定位的votedisk文件。分别是vdisk1、vdisk2和vdisk3。

5).在线添加Votedisk文件：
[root@rhel1 fd]# cd -

/u01/app/oracle/crs/bin
[root@rhel1 bin]# ./crsctl add css votedisk /u03/vdisk/vdisk4 -force
Now formatting voting disk: /u03/vdisk/vdisk4
successful addition of votedisk /u03/vdisk/vdisk4.

[root@rhel1 bin]# ./crsctl add css votedisk /u03/vdisk/vdisk5 -force
Now formatting voting disk: /u03/vdisk/vdisk5
successful addition of votedisk /u03/vdisk/vdisk5.

[root@rhel1 bin]# ./crsctl add css votedisk /u03/vdisk/vdisk6 -force
Now formatting voting disk: /u03/vdisk/vdisk6
successful addition of votedisk /u03/vdisk/vdisk6.

执行上面的命令在线添加了3个votedisk文件到不同的ocfs2文件系统上，这样做的目的是为了实现存储设备的在线迁移。

6).查看当前的votedisk路径及文件名：
[root@rhel1 bin]# ./crsctl query css votedisk

0.     0    /u02/vdisk/vdisk1
1.     0    /u02/vdisk/vdisk2
2.     0    /u02/vdisk/vdisk3
3.     0    /u03/vdisk/vdisk4
4.     0    /u03/vdisk/vdisk5
5.     0    /u03/vdisk/vdisk6

located 6 votedisk(s).
从上面的输出结果可以看出，3个votedisk文件添加是成功的。

7).查看ocssd进程定位的votedisk文件：
[root@rhel1 bin]# cd /proc/5942/fd/

[root@rhel1 fd]# ll
total 0
lr-x------ 1 oracle oinstall 64 Jun 12 18:18 0 -> /dev/null
lrwx------ 1 oracle oinstall 64 Jun 12 18:18 1 -> /u01/app/oracle/crs/log/rhel1/cssd/cssdOUT.log
lrwx------ 1 oracle oinstall 64 Jun 12 18:18 10 -> socket:[13927]
lrwx------ 1 oracle oinstall 64 Jun 12 18:18 11 -> /u02/vdisk/vdisk1
lrwx------ 1 oracle oinstall 64 Jun 12 18:18 12 -> /u02/vdisk/vdisk1
lrwx------ 1 oracle oinstall 64 Jun 12 18:18 13 -> /u02/vdisk/vdisk2
lrwx------ 1 oracle oinstall 64 Jun 12 18:18 14 -> /u02/vdisk/vdisk2
lrwx------ 1 oracle oinstall 64 Jun 12 18:18 15 -> /u02/vdisk/vdisk3
lrwx------ 1 oracle oinstall 64 Jun 12 18:18 16 -> /u02/vdisk/vdisk3
l-wx------ 1 oracle oinstall 64 Jun 12 18:18 17 -> /u01/app/oracle/crs/log/rhel1/alertrhel1.log
lr-x------ 1 oracle oinstall 64 Jun 12 18:18 18 -> /u01/app/oracle/crs/has/mesg/clsdus.msb
lrwx------ 1 oracle oinstall 64 Jun 12 18:18 19 -> socket:[13974]
lrwx------ 1 oracle oinstall 64 Jun 12 18:18 2 -> /u01/app/oracle/crs/log/rhel1/cssd/cssdOUT.log
lrwx------ 1 oracle oinstall 64 Jun 12 18:18 20 -> socket:[13977]
lrwx------ 1 oracle oinstall 64 Jun 12 18:18 21 -> socket:[14446]
lrwx------ 1 oracle oinstall 64 Jun 12 18:18 22 -> socket:[13984]
lrwx------ 1 oracle oinstall 64 Jun 12 18:18 23 -> socket:[13987]
lrwx------ 1 oracle oinstall 64 Jun 12 18:18 24 -> socket:[13989]
lrwx------ 1 oracle oinstall 64 Jun 12 18:18 25 -> socket:[13993]
lrwx------ 1 oracle oinstall 64 Jun 12 18:18 26 -> socket:[13995]
lrwx------ 1 oracle oinstall 64 Jun 12 18:18 27 -> socket:[13996]
lrwx------ 1 oracle oinstall 64 Jun 12 18:18 28 -> socket:[14447]
lrwx------ 1 oracle oinstall 64 Jun 12 18:18 29 -> socket:[14733]
lrwx------ 1 oracle oinstall 64 Jun 12 18:18 3 -> /u01/app/oracle/crs/log/rhel1/cssd/cssdOUT.log
lrwx------ 1 oracle oinstall 64 Jun 12 18:18 30 -> socket:[14449]
lrwx------ 1 oracle oinstall 64 Jun 12 18:18 31 -> socket:[14739]
lr-x------ 1 oracle oinstall 64 Jun 12 18:18 4 -> /u01/app/oracle/crs/srvm/mesg/procus.msb
lr-x------ 1 oracle oinstall 64 Jun 12 18:18 5 -> /u02/ocr/ocr1
lr-x------ 1 oracle oinstall 64 Jun 12 18:18 6 -> /u02/ocr/ocr2
lr-x------ 1 oracle oinstall 64 Jun 12 18:18 7 -> /u01/app/oracle/crs/css/mesg/clssus.msb
l-wx------ 1 oracle oinstall 64 Jun 12 18:18 8 -> /u01/app/oracle/crs/log/rhel1/cssd/ocssd.log
lrwx------ 1 oracle oinstall 64 Jun 12 18:18 9 -> socket:[13926]

从上面的输出可以看出，ocssd进程定位的votedisk文件并未发生变化，可以猜测，ocssd对votedisk的更新也不会同步到新 添加的3个votedisk文件中，那么在长时间的运行之后，重启服务器时新添加的votedisk过旧。

8).在线删除votedisk文件：
[root@rhel1 fd]# cd -

/u01/app/oracle/crs/bin
[root@rhel1 bin]# ./crsctl delete css votedisk /u02/vdisk/vdisk1 -force
successful deletion of votedisk /u02/vdisk/vdisk1.
[root@rhel1 bin]# ./crsctl delete css votedisk /u02/vdisk/vdisk2 -force
successful deletion of votedisk /u02/vdisk/vdisk2.
[root@rhel1 bin]# ./crsctl delete css votedisk /u02/vdisk/vdisk3 -force
successful deletion of votedisk /u02/vdisk/vdisk3.

9).查看删除votedisk后的votedisk路径及文件名：
[root@rhel1 bin]# ./crsctl query css votedisk

0.     0    /u03/vdisk/vdisk4
1.     0    /u03/vdisk/vdisk5
2.     0    /u03/vdisk/vdisk6
located 3 votedisk(s).

[root@rhel1 bin]# cd /proc/5942/fd/

[root@rhel1 fd]# ll
total 0
lr-x------ 1 oracle oinstall 64 Jun 12 18:18 0 -> /dev/null
lrwx------ 1 oracle oinstall 64 Jun 12 18:18 1 -> /u01/app/oracle/crs/log/rhel1/cssd/cssdOUT.log
lrwx------ 1 oracle oinstall 64 Jun 12 18:18 10 -> socket:[13927]
lrwx------ 1 oracle oinstall 64 Jun 12 18:18 11 -> /u02/vdisk/vdisk1
lrwx------ 1 oracle oinstall 64 Jun 12 18:18 12 -> /u02/vdisk/vdisk1
lrwx------ 1 oracle oinstall 64 Jun 12 18:18 13 -> /u02/vdisk/vdisk2
lrwx------ 1 oracle oinstall 64 Jun 12 18:18 14 -> /u02/vdisk/vdisk2
lrwx------ 1 oracle oinstall 64 Jun 12 18:18 15 -> /u02/vdisk/vdisk3
lrwx------ 1 oracle oinstall 64 Jun 12 18:18 16 -> /u02/vdisk/vdisk3
l-wx------ 1 oracle oinstall 64 Jun 12 18:18 17 -> /u01/app/oracle/crs/log/rhel1/alertrhel1.log
lr-x------ 1 oracle oinstall 64 Jun 12 18:18 18 -> /u01/app/oracle/crs/has/mesg/clsdus.msb
lrwx------ 1 oracle oinstall 64 Jun 12 18:18 19 -> socket:[13974]
lrwx------ 1 oracle oinstall 64 Jun 12 18:18 2 -> /u01/app/oracle/crs/log/rhel1/cssd/cssdOUT.log
lrwx------ 1 oracle oinstall 64 Jun 12 18:18 20 -> socket:[13977]
lrwx------ 1 oracle oinstall 64 Jun 12 18:18 21 -> socket:[14446]
lrwx------ 1 oracle oinstall 64 Jun 12 18:18 22 -> socket:[13984]
lrwx------ 1 oracle oinstall 64 Jun 12 18:18 23 -> socket:[13987]
lrwx------ 1 oracle oinstall 64 Jun 12 18:18 24 -> socket:[13989]
lrwx------ 1 oracle oinstall 64 Jun 12 18:18 25 -> socket:[13993]
lrwx------ 1 oracle oinstall 64 Jun 12 18:18 26 -> socket:[13995]
lrwx------ 1 oracle oinstall 64 Jun 12 18:18 27 -> socket:[13996]
lrwx------ 1 oracle oinstall 64 Jun 12 18:18 28 -> socket:[14447]
lrwx------ 1 oracle oinstall 64 Jun 12 18:18 29 -> socket:[14733]
lrwx------ 1 oracle oinstall 64 Jun 12 18:18 3 -> /u01/app/oracle/crs/log/rhel1/cssd/cssdOUT.log
lrwx------ 1 oracle oinstall 64 Jun 12 18:18 30 -> socket:[14449]
lrwx------ 1 oracle oinstall 64 Jun 12 18:18 31 -> socket:[14739]
lr-x------ 1 oracle oinstall 64 Jun 12 18:18 4 -> /u01/app/oracle/crs/srvm/mesg/procus.msb
lr-x------ 1 oracle oinstall 64 Jun 12 18:18 5 -> /u02/ocr/ocr1
lr-x------ 1 oracle oinstall 64 Jun 12 18:18 6 -> /u02/ocr/ocr2
lr-x------ 1 oracle oinstall 64 Jun 12 18:18 7 -> /u01/app/oracle/crs/css/mesg/clssus.msb
l-wx------ 1 oracle oinstall 64 Jun 12 18:18 8 -> /u01/app/oracle/crs/log/rhel1/cssd/ocssd.log
lrwx------ 1 oracle oinstall 64 Jun 12 18:18 9 -> socket:[13926]

定位未发生任何变化。

下面我们来尝试重启Clusterware，看会发生什么情况。

11).停止Clustereware：
[root@rhel1 fd]# cd -
/u01/app/oracle/crs/bin
[root@rhel1 bin]# ./crsctl stop crs

Stopping resources. This could take several minutes.
Successfully stopped CRS resources.
Stopping CSSD.
Shutting down CSS daemon.
Shutdown request successfully issued.

在所有RAC节点停止Clusterware。

12).启动Clusterware：

[root@rhel1 bin]# ./crsctl start crs
Attempting to start CRS stack
The CRS stack will be started shortly

只在一个节点启动Clusterware。

13).查看CRS组件的状态及votedisk文件：
[root@rhel1 bin]# ./crsctl check crs

CSS appears healthy
CRS appears healthy
EVM appears healthy

[root@rhel1 bin]# ./crsctl query css votedisk

0.     0    0
1.     0    0
2.     0    /u03/vdisk/vdisk6
located 3 votedisk(s).

可以发现，votedisk只有vdisk6显示正常，至少可以说明这种情况是不正常的，可以大胆猜想是由于在线增加votedisk引起的问题。

14).查看ocssd进程定位的votedisk文件：
[root@rhel1 bin]# ps -e | grep -i ocssd
9386 ? 00:00:04 ocssd.bin
[root@rhel1 bin]# cd /proc/9386/fd/

[root@rhel1 fd]# ll
total 0
lr-x------ 1 oracle oinstall 64 Jun 12 18:30 0 -> /dev/null
lrwx------ 1 oracle oinstall 64 Jun 12 18:30 1 -> /u01/app/oracle/crs/log/rhel1/cssd/cssdOUT.log
lrwx------ 1 oracle oinstall 64 Jun 12 18:30 10 -> socket:[44767]
lrwx------ 1 oracle oinstall 64 Jun 12 18:30 11 -> /u03/vdisk/vdisk6
lrwx------ 1 oracle oinstall 64 Jun 12 18:30 12 -> /u03/vdisk/vdisk6
l-wx------ 1 oracle oinstall 64 Jun 12 18:30 13 -> /u01/app/oracle/crs/log/rhel1/alertrhel1.log
lr-x------ 1 oracle oinstall 64 Jun 12 18:30 14 -> /u01/app/oracle/crs/has/mesg/clsdus.msb
lrwx------ 1 oracle oinstall 64 Jun 12 18:30 15 -> socket:[44950]
lrwx------ 1 oracle oinstall 64 Jun 12 18:30 16 -> socket:[44953]
lrwx------ 1 oracle oinstall 64 Jun 12 18:30 17 -> socket:[45584]
lrwx------ 1 oracle oinstall 64 Jun 12 18:30 18 -> socket:[44980]
lrwx------ 1 oracle oinstall 64 Jun 12 18:30 19 -> socket:[44983]
lrwx------ 1 oracle oinstall 64 Jun 12 18:30 2 -> /u01/app/oracle/crs/log/rhel1/cssd/cssdOUT.log
lrwx------ 1 oracle oinstall 64 Jun 12 18:30 20 -> socket:[44986]
lrwx------ 1 oracle oinstall 64 Jun 12 18:30 21 -> socket:[44988]
lrwx------ 1 oracle oinstall 64 Jun 12 18:30 22 -> socket:[44992]
lrwx------ 1 oracle oinstall 64 Jun 12 18:30 23 -> socket:[44993]
lrwx------ 1 oracle oinstall 64 Jun 12 18:30 24 -> socket:[45585]
lrwx------ 1 oracle oinstall 64 Jun 12 18:30 25 -> socket:[45586]
lrwx------ 1 oracle oinstall 64 Jun 12 18:30 26 -> socket:[45587]
lrwx------ 1 oracle oinstall 64 Jun 12 18:30 27 -> socket:[45588]
lrwx------ 1 oracle oinstall 64 Jun 12 18:30 3 -> /u01/app/oracle/crs/log/rhel1/cssd/cssdOUT.log
lr-x------ 1 oracle oinstall 64 Jun 12 18:30 4 -> /u01/app/oracle/crs/srvm/mesg/procus.msb
lr-x------ 1 oracle oinstall 64 Jun 12 18:30 5 -> /u02/ocr/ocr1
lr-x------ 1 oracle oinstall 64 Jun 12 18:30 6 -> /u02/ocr/ocr2
lr-x------ 1 oracle oinstall 64 Jun 12 18:30 7 -> /u01/app/oracle/crs/css/mesg/clssus.msb
l-wx------ 1 oracle oinstall 64 Jun 12 18:30 8 -> /u01/app/oracle/crs/log/rhel1/cssd/ocssd.log
lrwx------ 1 oracle oinstall 64 Jun 12 18:30 9 -> socket:[44766]

从上面的结果可以看出，ocssd进程只定位了vdisk6这一个文件。

15).停止所有节点的Clusterware：
[root@rhel1 fd]# cd -

/u01/app/oracle/crs/bin
[root@rhel1 bin]# ./crsctl stop crs
Stopping resources. This could take several minutes.
Successfully stopped CRS resources.
Stopping CSSD.
Shutting down CSS daemon.
Shutdown request successfully issued.
[root@rhel1 bin]# ps -ef | grep ora
root      2265  2243  0 16:45 ?        00:00:01 hald-addon-storage: polling /dev/hdc
root      8197  8174  0 18:23 pts/3    00:00:00 su - oracle
oracle    8198  8197  0 18:23 pts/3    00:00:00 -bash
oracle   10433  8198  0 18:29 pts/3    00:00:00 tail -f alertrhel1.log
root     13299  3340  0 18:34 pts/0    00:00:00 grep ora
root     30740  9263  0 17:59 pts/2    00:00:00 su - oracle
oracle   30741 30740  0 17:59 pts/2    00:00:00 -bash

16).手动删除没有使用的vdisk4和vdisk5：
[root@rhel1 bin]# cd /u03/vdisk/

[root@rhel1 vdisk]# rm -rf vdisk4 vdisk5

17).在Clusterware脱机的情况下添加vdisk4和vdisk5：
[root@rhel1 vdisk]# cd -

/u01/app/oracle/crs/bin
[root@rhel1 bin]# ./crsctl add css votedisk /u03/vdisk/vdisk4 -force
Now formatting voting disk: /u03/vdisk/vdisk4
successful addition of votedisk /u03/vdisk/vdisk4.
[root@rhel1 bin]# ./crsctl add css votedisk /u03/vdisk/vdisk5 -force
Now formatting voting disk: /u03/vdisk/vdisk5
successful addition of votedisk /u03/vdisk/vdisk5.

17).修改votedisk正确的权限：
[root@rhel1 bin]# cd -

/u03/vdisk
[root@rhel1 vdisk]# ll
total 30000
-rw-r--r-- 1 root   root     10240000 Jun 12 18:37 vdisk4
-rw-r--r-- 1 root   root     10240000 Jun 12 18:37 vdisk5
-rw-r--r-- 1 oracle oinstall 10240000 Jun 12 18:19 vdisk6

[root@rhel1 vdisk]# chown oracle:oinstall *

18).启动Clusterware：
[root@rhel1 vdisk]# cd -

/u01/app/oracle/crs/bin
[root@rhel1 bin]# ./crsctl start crs
Attempting to start CRS stack
The CRS stack will be started shortly
[root@rhel1 bin]# ./crsctl check css
CSS appears healthy
[root@rhel1 bin]# ./crsctl check crs
CSS appears healthy
CRS appears healthy
EVM appears healthy

19).查看votedisk路径及文件名：

[root@rhel1 bin]# ./crsctl query css votedisk

0.     0    /u03/vdisk/vdisk4
1.     0    /u03/vdisk/vdisk5
2.     0    0
3.     0    0
4.     0    /u03/vdisk/vdisk4
located 5 votedisk(s).

可以看出vdisk4和vdisk5被正确添加，vdisk6不见了踪迹，说明10g votedisk在Clusterware的情况下是可以添加的，并且在脱机的情况下添加之后启动ocssd进程会立即定位到这些votedisk文件，所以这并不会导致前面新添加的votedisk未被定位的情况发生。

20).添加vdisk6和清理有问题的votedisk：

[root@rhel1 bin]# ./crsctl stop crs
Stopping resources. This could take several minutes.
Successfully stopped CRS resources.
Stopping CSSD.
Shutting down CSS daemon.
Shutdown request successfully issued.

[root@rhel1 bin]# ps -ef | grep ora
root      2265  2243  0 16:45 ?        00:00:02 hald-addon-storage: polling /dev/hdc
root      8197  8174  0 18:22 pts/3    00:00:00 su - oracle
oracle    8198  8197  0 18:22 pts/3    00:00:00 -bash
oracle   20770  8198  0 18:50 pts/3    00:00:00 tail -f alertrhel1.log
root     21487  3340  0 18:51 pts/0    00:00:00 grep ora
root     30740  9263  0 17:59 pts/2    00:00:00 su - oracle
oracle   30741 30740  0 17:59 pts/2    00:00:00 -bash

[root@rhel1 bin]# ./crsctl add css votedisk /u03/vdisk/vdisk6 -force
Now formatting voting disk: /u03/vdisk/vdisk6
successful addition of votedisk /u03/vdisk/vdisk6.

[root@rhel1 bin]# ./crsctl query css votedisk
0.     0    /u03/vdisk/vdisk4
1.     0    /u03/vdisk/vdisk5
2.     0    /u03/vdisk/vdisk6
3.     0    0
4.     0    0
5.     0    /u03/vdisk/vdisk4
located 6 votedisk(s).

[root@rhel1 bin]# ./crsctl delete css votedisk 0 -force
successful deletion of votedisk 0.

[root@rhel1 bin]# ./crsctl query css votedisk
0.     0    /u03/vdisk/vdisk4
1.     0    /u03/vdisk/vdisk5
2.     0    /u03/vdisk/vdisk6
3.     0    0
4.     0    /u03/vdisk/vdisk4
located 5 votedisk(s).

从上面返回的结果可以看出，虽然votedisk的路径和文件都是0，在删除0的时候也不会被全部删除，只会删除其中的一条。

[root@rhel1 bin]# ./crsctl delete css votedisk 0 -force
successful deletion of votedisk 0.
[root@rhel1 bin]# ./crsctl query css votedisk
0.     0    /u03/vdisk/vdisk4
1.     0    /u03/vdisk/vdisk5
2.     0    /u03/vdisk/vdisk6
3.     0    /u03/vdisk/vdisk4
located 4 votedisk(s).

[root@rhel1 bin]# ./crsctl delete css votedisk /u03/vdisk/vdisk4 -force
successful deletion of votedisk /u03/vdisk/vdisk4.
[root@rhel1 bin]# ./crsctl query css votedisk
0.     0    /u03/vdisk/vdisk5
1.     0    /u03/vdisk/vdisk6
2.     0    /u03/vdisk/vdisk4
located 3 votedisk(s).

删除 /u03/vdisk/vdisk4的votedisk文件也只删除了其中一条记录。

21).启动Clusterware：

[root@rhel1 bin]# ./crsctl start crs
Attempting to start CRS stack
The CRS stack will be started shortly

[root@rhel1 bin]# ./crsctl check crs
CSS appears healthy
CRS appears healthy
EVM appears healthy

[root@rhel1 bin]# ./crsctl query css votedisk
0.     0    /u03/vdisk/vdisk5
1.     0    /u03/vdisk/vdisk6
2.     0    /u03/vdisk/vdisk4
located 3 votedisk(s).

从上面的例子可以说明一点，在10gR2 RAC 的环境中最好不要在Clusterware在线的情况下添加votedisk，虽然可以成功添加，但在重启Clusterware之后可能出现意想不到的问题。

二.OCR的在线替换（添加和删除）。

1).检查OCR的完整性：

[root@rhel1 bin]# ./ocrcheck
Status of Oracle Cluster Registry is as follows :
Version                  :          2
Total space (kbytes)     :     262120
Used space (kbytes)      :       2512
Available space (kbytes) :     259608
ID                       : 2111044529
Device/File Name         : /u02/ocr/ocr1
Device/File integrity check succeeded
Device/File Name         : /u02/ocr/ocr2
Device/File integrity check succeeded
Cluster registry integrity check succeeded

2).提前将需要创建的ocr文件touch：

[root@rhel1 bin]# su - oracle
[oracle@rhel1 ~]$
[oracle@rhel1 ~]$ cd /u03/ocr/
[oracle@rhel1 ocr]$ ls
[oracle@rhel1 ocr]$ touch ocr3
[oracle@rhel1 ocr]$ touch ocr4

上面添加votedisk的时候并未提前touch votedisk文件，但某些情况下votedisk文件也需要提前touch。

3).替换ocrmirror文件，实际是一个迁移ocr的过程：

[root@rhel1 bin]# ./ocrconfig -replace ocrmirror /u03/ocr/ocr4
[root@rhel1 bin]# ./ocrcheck
Status of Oracle Cluster Registry is as follows :
Version                  :          2
Total space (kbytes)     :     262120
Used space (kbytes)      :       2512
Available space (kbytes) :     259608
ID                       : 2111044529
Device/File Name         : /u02/ocr/ocr1
Device/File integrity check succeeded
Device/File Name         : /u03/ocr/ocr4
Device/File integrity check succeeded
Cluster registry integrity check succeeded

迁移后检查ocr的完整性，没问题。

4).查看ocssd进程定位的ocr文件：

ocr文件也是由ocssd进程来维护。
[root@rhel1 bin]# ps -e | grep -i ocssd
23379 ?        00:00:06 ocssd.bin
[root@rhel1 bin]# cd /proc/23379/fd
[root@rhel1 fd]# ll
total 0
lr-x------ 1 oracle oinstall 64 Jun 12 19:00 0 -> /dev/null
lrwx------ 1 oracle oinstall 64 Jun 12 19:00 1 -> /u01/app/oracle/crs/log/rhel1/cssd/cssdOUT.log
lrwx------ 1 oracle oinstall 64 Jun 12 19:00 10 -> socket:[62386]
lrwx------ 1 oracle oinstall 64 Jun 12 19:00 11 -> /u03/vdisk/vdisk5
lrwx------ 1 oracle oinstall 64 Jun 12 19:00 12 -> /u03/vdisk/vdisk5
lrwx------ 1 oracle oinstall 64 Jun 12 19:00 13 -> /u03/vdisk/vdisk6
lrwx------ 1 oracle oinstall 64 Jun 12 19:00 14 -> /u03/vdisk/vdisk6
lrwx------ 1 oracle oinstall 64 Jun 12 19:00 15 -> /u03/vdisk/vdisk4
lrwx------ 1 oracle oinstall 64 Jun 12 19:00 16 -> /u03/vdisk/vdisk4
l-wx------ 1 oracle oinstall 64 Jun 12 19:00 17 -> /u01/app/oracle/crs/log/rhel1/alertrhel1.log
lr-x------ 1 oracle oinstall 64 Jun 12 19:00 18 -> /u01/app/oracle/crs/has/mesg/clsdus.msb
lrwx------ 1 oracle oinstall 64 Jun 12 19:00 19 -> socket:[62607]
lrwx------ 1 oracle oinstall 64 Jun 12 19:00 2 -> /u01/app/oracle/crs/log/rhel1/cssd/cssdOUT.log
lrwx------ 1 oracle oinstall 64 Jun 12 19:00 20 -> socket:[62610]
lrwx------ 1 oracle oinstall 64 Jun 12 19:00 21 -> socket:[62611]
lrwx------ 1 oracle oinstall 64 Jun 12 19:00 22 -> socket:[62627]
lrwx------ 1 oracle oinstall 64 Jun 12 19:00 23 -> socket:[62632]
lrwx------ 1 oracle oinstall 64 Jun 12 19:00 24 -> socket:[62637]
lrwx------ 1 oracle oinstall 64 Jun 12 19:00 25 -> socket:[62640]
lrwx------ 1 oracle oinstall 64 Jun 12 19:00 26 -> socket:[62645]
lrwx------ 1 oracle oinstall 64 Jun 12 19:00 27 -> socket:[62646]
lrwx------ 1 oracle oinstall 64 Jun 12 19:00 28 -> socket:[62650]
lrwx------ 1 oracle oinstall 64 Jun 12 19:00 29 -> socket:[62651]
lrwx------ 1 oracle oinstall 64 Jun 12 19:00 3 -> /u01/app/oracle/crs/log/rhel1/cssd/cssdOUT.log
lrwx------ 1 oracle oinstall 64 Jun 12 19:00 30 -> socket:[62652]
lrwx------ 1 oracle oinstall 64 Jun 12 19:00 31 -> socket:[62653]
lr-x------ 1 oracle oinstall 64 Jun 12 19:00 4 -> /u01/app/oracle/crs/srvm/mesg/procus.msb
lr-x------ 1 oracle oinstall 64 Jun 12 19:00 5 -> /u02/ocr/ocr1
lr-x------ 1 oracle oinstall 64 Jun 12 19:00 6 -> /u02/ocr/ocr2
lr-x------ 1 oracle oinstall 64 Jun 12 19:00 7 -> /u01/app/oracle/crs/css/mesg/clssus.msb
l-wx------ 1 oracle oinstall 64 Jun 12 19:00 8 -> /u01/app/oracle/crs/log/rhel1/cssd/ocssd.log
lrwx------ 1 oracle oinstall 64 Jun 12 19:00 9 -> socket:[62385]

从上面的结果可以看出，ocssd进程定位的ocr文件还是oc1和ocr2，并未随着ocr的迁移发生改变，和votedisk的效果相同。

5).停止Clusterware：
[root@rhel1 fd]# cd -

/u01/app/oracle/crs/bin
[root@rhel1 bin]# ./crsctl stop crs
Stopping resources. This could take several minutes.
Successfully stopped CRS resources.
Stopping CSSD.
Shutting down CSS daemon.
Shutdown request successfully issued.停止所有节点的Clusterware。

[root@rhel1 bin]# ps -ef | grep ora
root      2265  2243  0 16:45 ?        00:00:02 hald-addon-storage: polling /dev/hdc
root      8197  8174  0 18:22 pts/3    00:00:00 su - oracle
oracle    8198  8197  0 18:22 pts/3    00:00:00 -bash
root     26683  3340  0 19:01 pts/0    00:00:00 grep ora
root     30740  9263  0 17:58 pts/2    00:00:00 su - oracle
oracle   30741 30740  0 17:58 pts/2    00:00:00 -bash

6).在Clusterware脱机的情况下迁移ocr文件：

[root@rhel1 bin]# ./ocrconfig -replace ocr /u03/ocr/ocr3
PROT-1: Failed to initialize ocrconfig

由此可以说明OCR文件不能在脱机的情况下迁移。

7).启动Clusterware：

[root@rhel1 bin]# ./crsctl start crs
Attempting to start CRS stack
The CRS stack will be started shortly
[root@rhel1 bin]# ./crsctl check crs
CSS appears healthy
CRS appears healthy
EVM appears healthy
[root@rhel1 bin]# ./ocrcheck
Status of Oracle Cluster Registry is as follows :
Version                  :          2
Total space (kbytes)     :     262120
Used space (kbytes)      :       2512
Available space (kbytes) :     259608
ID                       : 2111044529
Device/File Name         : /u02/ocr/ocr1
Device/File integrity check succeeded
Device/File Name         : /u03/ocr/ocr4
Device/File integrity check succeeded
Cluster registry integrity check succeeded

8).在线替换ocr文件：

[root@rhel1 bin]# ./ocrconfig -replace ocr /u03/ocr/ocr3
[root@rhel1 bin]# ./ocrcheck
Status of Oracle Cluster Registry is as follows :
Version                  :          2
Total space (kbytes)     :     262120
Used space (kbytes)      :       2512
Available space (kbytes) :     259608
ID                       : 2111044529
Device/File Name         : /u03/ocr/ocr3
Device/File integrity check succeeded
Device/File Name         : /u03/ocr/ocr4
Device/File integrity check succeeded
Cluster registry integrity check succeeded

替换成功，OCR文件一致性检查成功。

9).检查ocssd进程定位的ocr文件：

[root@rhel1 bin]# ps -e | grep ocssd
28226 ?        00:00:03 ocssd.bin
[root@rhel1 bin]# cd /proc/28226/fd
[root@rhel1 fd]# ll
total 0
lr-x------ 1 oracle oinstall 64 Jun 12 19:04 0 -> /dev/null
lrwx------ 1 oracle oinstall 64 Jun 12 19:04 1 -> /u01/app/oracle/crs/log/rhel1/cssd/cssdOUT.log
l-wx------ 1 oracle oinstall 64 Jun 12 19:04 10 -> /u01/app/oracle/crs/log/rhel1/cssd/ocssd.log
lrwx------ 1 oracle oinstall 64 Jun 12 19:04 11 -> /u03/vdisk/vdisk5
lrwx------ 1 oracle oinstall 64 Jun 12 19:04 12 -> /u03/vdisk/vdisk5
lrwx------ 1 oracle oinstall 64 Jun 12 19:04 13 -> /u03/vdisk/vdisk6
lrwx------ 1 oracle oinstall 64 Jun 12 19:04 14 -> /u03/vdisk/vdisk6
l-wx------ 1 oracle oinstall 64 Jun 12 19:04 15 -> /u01/app/oracle/crs/log/rhel1/alertrhel1.log
lr-x------ 1 oracle oinstall 64 Jun 12 19:04 16 -> /u01/app/oracle/crs/has/mesg/clsdus.msb
lrwx------ 1 oracle oinstall 64 Jun 12 19:04 17 -> /u03/vdisk/vdisk4
lrwx------ 1 oracle oinstall 64 Jun 12 19:04 18 -> /u03/vdisk/vdisk4
lrwx------ 1 oracle oinstall 64 Jun 12 19:04 19 -> socket:[69253]
lrwx------ 1 oracle oinstall 64 Jun 12 19:04 2 -> /u01/app/oracle/crs/log/rhel1/cssd/cssdOUT.log
lrwx------ 1 oracle oinstall 64 Jun 12 19:04 20 -> socket:[69256]
lrwx------ 1 oracle oinstall 64 Jun 12 19:04 21 -> socket:[69257]
lrwx------ 1 oracle oinstall 64 Jun 12 19:04 22 -> socket:[69285]
lrwx------ 1 oracle oinstall 64 Jun 12 19:04 23 -> socket:[69303]
lrwx------ 1 oracle oinstall 64 Jun 12 19:04 24 -> socket:[69346]
lrwx------ 1 oracle oinstall 64 Jun 12 19:04 25 -> socket:[69353]
lrwx------ 1 oracle oinstall 64 Jun 12 19:04 26 -> socket:[69370]
lrwx------ 1 oracle oinstall 64 Jun 12 19:04 27 -> socket:[69371]
lrwx------ 1 oracle oinstall 64 Jun 12 19:04 28 -> socket:[69379]
lrwx------ 1 oracle oinstall 64 Jun 12 19:04 29 -> socket:[69380]
lrwx------ 1 oracle oinstall 64 Jun 12 19:04 3 -> /u01/app/oracle/crs/log/rhel1/cssd/cssdOUT.log
lrwx------ 1 oracle oinstall 64 Jun 12 19:04 30 -> socket:[69381]
lrwx------ 1 oracle oinstall 64 Jun 12 19:04 31 -> socket:[69382]
lr-x------ 1 oracle oinstall 64 Jun 12 19:04 4 -> /u01/app/oracle/crs/srvm/mesg/procus.msb
lr-x------ 1 oracle oinstall 64 Jun 12 19:04 5 -> /u02/ocr/ocr1
lr-x------ 1 oracle oinstall 64 Jun 12 19:04 6 -> /u03/ocr/ocr4
lr-x------ 1 oracle oinstall 64 Jun 12 19:04 7 -> /u01/app/oracle/crs/css/mesg/clssus.msb
lrwx------ 1 oracle oinstall 64 Jun 12 19:04 8 -> socket:[68925]
lrwx------ 1 oracle oinstall 64 Jun 12 19:04 9 -> socket:[68926]

定位文件未发生变化，需要重启才能定位新替换的OCR文件。

10).重启Clusterware：

[root@rhel1 fd]# cd -
/u01/app/oracle/crs/bin
[root@rhel1 bin]# ./crsctl stop crs
Stopping resources. This could take several minutes.
Successfully stopped CRS resources.
Stopping CSSD.
Shutting down CSS daemon.
Shutdown request successfully issued.
[root@rhel1 bin]# ./crsctl start crs
Attempting to start CRS stack
The CRS stack will be started shortly
[root@rhel1 bin]# ./crsctl check crs
CSS appears healthy
CRS appears healthy
EVM appears healthy

11).查看ocssd进程定位的ocr文件：

[root@rhel1 bin]# ps -e | grep ocssd
30773 ?        00:00:02 ocssd.bin
[root@rhel1 bin]# cd /proc/30773/fd
[root@rhel1 fd]# ll
total 0
lr-x------ 1 oracle oinstall 64 Jun 12 19:05 0 -> /dev/null
lrwx------ 1 oracle oinstall 64 Jun 12 19:05 1 -> /u01/app/oracle/crs/log/rhel1/cssd/cssdOUT.log
lrwx------ 1 oracle oinstall 64 Jun 12 19:05 10 -> socket:[73581]
lrwx------ 1 oracle oinstall 64 Jun 12 19:05 11 -> /u03/vdisk/vdisk5
lrwx------ 1 oracle oinstall 64 Jun 12 19:05 12 -> /u03/vdisk/vdisk5
lrwx------ 1 oracle oinstall 64 Jun 12 19:05 13 -> /u03/vdisk/vdisk6
lrwx------ 1 oracle oinstall 64 Jun 12 19:05 14 -> /u03/vdisk/vdisk6
lrwx------ 1 oracle oinstall 64 Jun 12 19:05 15 -> /u03/vdisk/vdisk4
lrwx------ 1 oracle oinstall 64 Jun 12 19:05 16 -> /u03/vdisk/vdisk4
l-wx------ 1 oracle oinstall 64 Jun 12 19:05 17 -> /u01/app/oracle/crs/log/rhel1/alertrhel1.log
lr-x------ 1 oracle oinstall 64 Jun 12 19:05 18 -> /u01/app/oracle/crs/has/mesg/clsdus.msb
lrwx------ 1 oracle oinstall 64 Jun 12 19:05 19 -> socket:[73764]
lrwx------ 1 oracle oinstall 64 Jun 12 19:05 2 -> /u01/app/oracle/crs/log/rhel1/cssd/cssdOUT.log
lrwx------ 1 oracle oinstall 64 Jun 12 19:05 20 -> socket:[73767]
lrwx------ 1 oracle oinstall 64 Jun 12 19:05 22 -> socket:[73769]
lrwx------ 1 oracle oinstall 64 Jun 12 19:05 23 -> socket:[73772]
lrwx------ 1 oracle oinstall 64 Jun 12 19:05 24 -> socket:[73774]
lrwx------ 1 oracle oinstall 64 Jun 12 19:05 25 -> socket:[73778]
lrwx------ 1 oracle oinstall 64 Jun 12 19:05 26 -> socket:[73780]
lrwx------ 1 oracle oinstall 64 Jun 12 19:05 27 -> socket:[73781]
lrwx------ 1 oracle oinstall 64 Jun 12 19:05 28 -> socket:[73787]
lrwx------ 1 oracle oinstall 64 Jun 12 19:05 29 -> socket:[73788]
lrwx------ 1 oracle oinstall 64 Jun 12 19:05 3 -> /u01/app/oracle/crs/log/rhel1/cssd/cssdOUT.log
lrwx------ 1 oracle oinstall 64 Jun 12 19:05 30 -> socket:[73789]
lrwx------ 1 oracle oinstall 64 Jun 12 19:05 31 -> socket:[73790]
lr-x------ 1 oracle oinstall 64 Jun 12 19:05 4 -> /u01/app/oracle/crs/srvm/mesg/procus.msb
lr-x------ 1 oracle oinstall 64 Jun 12 19:05 5 -> /u03/ocr/ocr3
lr-x------ 1 oracle oinstall 64 Jun 12 19:05 6 -> /u03/ocr/ocr4
lr-x------ 1 oracle oinstall 64 Jun 12 19:05 7 -> /u01/app/oracle/crs/css/mesg/clssus.msb
l-wx------ 1 oracle oinstall 64 Jun 12 19:05 8 -> /u01/app/oracle/crs/log/rhel1/cssd/ocssd.log
lrwx------ 1 oracle oinstall 64 Jun 12 19:05 9 -> socket:[73580]
重启后ocssd进程定位ocr文件成功。

从上面的OCR替换的例子可以看出，OCR的替换（添加和删除）只能在Clusterware在线的情况下才能执行，最好在替换成功，检查OCR一致性正确之后立即重启Clusterware，使得替换的OCR文件生效。
文章出自：http://ms.itpub.net/23135684/viewspace-732734/ ，感谢作者！