虚拟机RAC的ASM磁盘组坏块导致重建DB

2011.11.23虚拟机RAC的ASM磁盘组坏块导致重建DB

刚刚在公司的一台PC机器上用vmware workstation8搭建了一套10gr2的rac环境，用的是裸设备+ASM搭建，在安装成功后，不小心被直接重启了下主机，结果再次启动虚拟机的时候提示到有磁盘损坏，也没有在意。但是在启动RAC的时候出现了问题，一开始的现象是如下几个个资源没办法随着其他资源一起启动：

ora.node1.LISTENER_NODE1.lsnr

ora.node2.LISTENER_NODE2.lsnr

ora.RAC.RAC1.inst

ora.RAC.RAC2.inst

ora.RAC.db

看具体的启动过程：

[oracle@node1 bin]$ crs_stat -t

Name Type Target State Host

------------------------------------------------------------

ora....C1.inst application OFFLINE OFFLINE

ora....C2.inst application OFFLINE OFFLINE

ora.RAC.db application OFFLINE OFFLINE

ora....SM1.asm application OFFLINE OFFLINE

ora....E1.lsnr application OFFLINE OFFLINE

ora.node1.gsd application OFFLINE OFFLINE

ora.node1.ons application OFFLINE OFFLINE

ora.node1.vip application OFFLINE OFFLINE

ora....SM2.asm application OFFLINE OFFLINE

ora....E2.lsnr application OFFLINE OFFLINE

ora.node2.gsd application OFFLINE OFFLINE

ora.node2.ons application OFFLINE OFFLINE

ora.node2.vip application OFFLINE OFFLINE

[oracle@node1 bin]$ crs_start -all

Attempting to start `ora.node1.vip` on member `node1`

Attempting to start `ora.node2.vip` on member `node2`

Start of `ora.node1.vip` on member `node1` succeeded.

Start of `ora.node2.vip` on member `node2` succeeded.

Attempting to start `ora.node1.ASM1.asm` on member `node1`

Attempting to start `ora.node2.ASM2.asm` on member `node2`

Start of `ora.node2.ASM2.asm` on member `node2` succeeded.

Attempting to start `ora.RAC.RAC2.inst` on member `node2`

Start of `ora.RAC.RAC2.inst` on member `node2` failed.

node1 : CRS-1018: Resource ora.node2.vip (application) is already running on node2

node1 : CRS-1018: Resource ora.node2.vip (application) is already running on node2

Start of `ora.node1.ASM1.asm` on member `node1` succeeded.

Attempting to start `ora.RAC.RAC1.inst` on member `node1`

Start of `ora.RAC.RAC1.inst` on member `node1` failed.

node2 : CRS-1018: Resource ora.node1.vip (application) is already running on node1

node2 : CRS-1018: Resource ora.node1.vip (application) is already running on node1

CRS-1002: Resource 'ora.node1.ons' is already running on member 'node1'

CRS-1002: Resource 'ora.node2.ons' is already running on member 'node2'

Attempting to start `ora.node1.gsd` on member `node1`

Attempting to start `ora.RAC.db` on member `node1`

Attempting to start `ora.node2.gsd` on member `node2`

Start of `ora.node1.gsd` on member `node1` succeeded.

Start of `ora.node2.gsd` on member `node2` succeeded.

Start of `ora.RAC.db` on member `node1` failed.

Attempting to start `ora.RAC.db` on member `node2`

Start of `ora.RAC.db` on member `node2` failed.

CRS-1006: No more members to consider

CRS-0215: Could not start resource 'ora.RAC.RAC1.inst'.

CRS-0215: Could not start resource 'ora.RAC.RAC2.inst'.

CRS-0215: Could not start resource 'ora.RAC.db'.

CRS-0223: Resource 'ora.node1.LISTENER_NODE1.lsnr' has placement error.

CRS-0223: Resource 'ora.node1.ons' has placement error.

CRS-0223: Resource 'ora.node2.LISTENER_NODE2.lsnr' has placement error.

CRS-0223: Resource 'ora.node2.ons' has placement error.

[oracle@node1 bin]$ crs_stat -t

Name Type Target State Host

------------------------------------------------------------

ora....C1.inst application ONLINE OFFLINE

ora....C2.inst application ONLINE OFFLINE

ora.RAC.db application ONLINE OFFLINE

ora....SM1.asm application ONLINE ONLINE node1

ora....E1.lsnr application OFFLINE OFFLINE

ora.node1.gsd application ONLINE ONLINE node1

ora.node1.ons application ONLINE ONLINE node1

ora.node1.vip application ONLINE ONLINE node1

ora....SM2.asm application ONLINE ONLINE node2

ora....E2.lsnr application OFFLINE OFFLINE

ora.node2.gsd application ONLINE ONLINE node2

ora.node2.ons application ONLINE ONLINE node2

ora.node2.vip application ONLINE ONLINE node2

尝试先把lsnr起来：

[oracle@node1 bin]$ crs_start ora.node1.LISTENER_NODE1.lsnr

Attempting to start `ora.node1.LISTENER_NODE1.lsnr` on member `node1`

Start of `ora.node1.LISTENER_NODE1.lsnr` on member `node1` succeeded.

[oracle@node1 bin]$ crs_start ora.node2.LISTENER_NODE2.lsnr

Attempting to start `ora.node2.LISTENER_NODE2.lsnr` on member `node2`

Start of `ora.node2.LISTENER_NODE2.lsnr` on member `node2` succeeded.

接着启动两个inst，接着出现问题了，inst无法拉起来：

[oracle@node1 bin]$ crs_start ora.RAC.RAC1.inst

Attempting to start `ora.RAC.RAC1.inst` on member `node1`

Start of `ora.RAC.RAC1.inst` on member `node1` failed.

node2 : CRS-1018: Resource ora.node1.vip (application) is already running on node1

CRS-0215: Could not start resource 'ora.RAC.RAC1.inst'.

检查相关的日志：

首先查看了下asm的日志：

alert_+ASM1.log

ed Nov 23 15:14:12 2011

Starting ORACLE instance (normal)

LICENSE_MAX_SESSION = 0

LICENSE_SESSIONS_WARNING = 0

Interface type 1 eth1 192.168.91.0 configured from OCR for use as a cluster interconnect

Interface type 1 eth0 192.168.88.0 configured from OCR for use as a public interface

Picked latch-free SCN scheme 2

Using LOG_ARCHIVE_DEST_1 parameter default value as /opt/app/product/10.2.0/db_1/dbs/arch

Autotune of undo retention is turned off.

LICENSE_MAX_USERS = 0

SYS auditing is disabled

ksdpec: called for event 13740 prior to event group initialization

Starting up ORACLE RDBMS Version: 10.2.0.1.0.

System parameters with non-default values:

large_pool_size = 12582912

instance_type = asm

cluster_database = TRUE

instance_number = 1

remote_login_passwordfile= EXCLUSIVE

background_dump_dest = /opt/app/admin/+ASM/bdump

user_dump_dest = /opt/app/admin/+ASM/udump

core_dump_dest = /opt/app/admin/+ASM/cdump

asm_diskgroups = DATA1

Cluster communication is configured to use the following interface(s) for this instance

192.168.91.100

Wed Nov 23 15:14:13 2011

cluster interconnect IPC version:Oracle UDP/IP

IPC Vendor 1 proto 2

PMON started with pid=2, OS id=25132

DIAG started with pid=3, OS id=25134

PSP0 started with pid=4, OS id=25136

LMON started with pid=5, OS id=25138

LMD0 started with pid=6, OS id=25140

LMS0 started with pid=7, OS id=25142

MMAN started with pid=8, OS id=25152

DBW0 started with pid=9, OS id=25154

LGWR started with pid=10, OS id=25156

CKPT started with pid=11, OS id=25158

SMON started with pid=12, OS id=25160

RBAL started with pid=13, OS id=25162

GMON started with pid=14, OS id=25164

Wed Nov 23 15:14:13 2011

lmon registered with NM - instance id 1 (internal mem no 0)

Wed Nov 23 15:14:13 2011

Reconfiguration started (old inc 0, new inc 1)

ASM instance

List of nodes:

0 1

Global Resource Directory frozen

Communication channels reestablished

Master broadcasted resource hash value bitmaps

Non-local Process blocks cleaned out

Wed Nov 23 15:14:14 2011

LMS 0: 0 GCS shadows cancelled, 0 closed

Set master node info

Submitted all remote-enqueue requests

Dwn-cvts replayed, VALBLKs dubious

All grantable enqueues granted

Post SMON to start 1st pass IR

Wed Nov 23 15:14:14 2011

LMS 0: 0 GCS shadows traversed, 0 replayed

Wed Nov 23 15:14:14 2011

Submitted all GCS remote-cache requests

Post SMON to start 1st pass IR

Fix write in gcs resources

Reconfiguration complete

LCK0 started with pid=15, OS id=25208

Wed Nov 23 15:14:15 2011

SQL> ALTER DISKGROUP ALL MOUNT

Wed Nov 23 15:14:15 2011

NOTE: cache registered group DATA1 number=1 incarn=0x6f877cd9

* allocate domain 1, invalid = TRUE

freeing rdom 1

Received dirty detach msg from node 1 for dom 1

Wed Nov 23 15:14:22 2011

Loaded ASM Library - Generic Linux, version 2.0.4 (KABI_V2) library for asmlib interface

Wed Nov 23 15:14:22 2011

ORA-15186: ASMLIB error function = [asm_open], error = [1], mesg = [Operation not permitted]

Wed Nov 23 15:14:22 2011

ORA-15186: ASMLIB error function = [asm_open], error = [1], mesg = [Operation not permitted]

Wed Nov 23 15:14:23 2011

NOTE: Hbeat: instance first (grp 1)

Wed Nov 23 15:14:27 2011

NOTE: start heartbeating (grp 1)

NOTE: cache opening disk 0 of grp 1: DATA1_0000 path:/dev/raw/raw3

Wed Nov 23 15:14:27 2011

NOTE: F1X0 found on disk 0 fcn 0.0

NOTE: cache opening disk 1 of grp 1: DATA1_0001 path:/dev/raw/raw4

NOTE: cache mounting (first) group 1/0x6F877CD9 (DATA1)

* allocate domain 1, invalid = TRUE

kjbdomatt send to node 1

Wed Nov 23 15:14:27 2011

NOTE: attached to recovery domain 1

Wed Nov 23 15:14:27 2011

NOTE: starting recovery of thread=1 ckpt=3.315

NOTE: starting recovery of thread=2 ckpt=3.50

WARNING: cache failed to read fn=3 indblk=0 from disk(s): 1

ORA-15196: invalid ASM block header [kfc.c:7910] [endian_kfbh] [3] [2147483648] [0 != 1]

NOTE: a corrupted block was dumped to the trace file

System State dumped to trace file /opt/app/admin/+ASM/udump/+asm1_ora_25219.trc

NOTE: cache initiating offline of disk 1 group 1

WARNING: offlining disk 1.3914828841 (DATA1_0001) with mask 0x3

NOTE: PST update: grp = 1, dsk = 1, mode = 0x6

Wed Nov 23 15:14:27 2011

ERROR: too many offline disks in PST (grp 1)

Wed Nov 23 15:14:27 2011

NOTE: halting all I/Os to diskgroup DATA1

NOTE: active pin found: 0x0x2427ccd0

NOTE: active pin found: 0x0x2427cc64

Abort recovery for domain 1

NOTE: crash recovery signalled OER-15130

ERROR: ORA-15130 signalled during mount of diskgroup DATA1

NOTE: cache dismounting group 1/0x6F877CD9 (DATA1)

Wed Nov 23 15:14:28 2011

kjbdomdet send to node 1

detach from dom 1, sending detach message to node 1

Wed Nov 23 15:14:28 2011

Dirty detach reconfiguration started (old inc 1, new inc 1)

List of nodes:

0 1

Global Resource Directory partially frozen for dirty detach

* dirty detach - domain 1 invalid = TRUE

0 GCS resources traversed, 0 cancelled

Dirty Detach Reconfiguration complete

Wed Nov 23 15:14:28 2011

freeing rdom 1

Wed Nov 23 15:14:28 2011

WARNING: dirty detached from domain 1

Wed Nov 23 15:14:28 2011

ERROR: diskgroup DATA1 was not mounted

Wed Nov 23 15:14:28 2011

WARNING: PST-initiated MANDATORY DISMOUNT of group DATA1 not performed - group not mounted

Wed Nov 23 15:14:28 2011

Errors in file /opt/app/admin/+ASM/bdump/+asm1_b000_25521.trc:

ORA-15001: diskgroup "DATA1" does not exist or is not mounted

[oracle@node1 bdump]$

从下面2段内容可以看到asm在mount diskgroup的时候出现错误了：

。。。。。

WARNING: cache failed to read fn=3 indblk=0 from disk(s): 1

ORA-15196: invalid ASM block header [kfc.c:7910] [endian_kfbh] [3] [2147483648] [0 != 1]

NOTE: a corrupted block was dumped to the trace file

System State dumped to trace file /opt/app/admin/+ASM/udump/+asm1_ora_25219.trc

NOTE: cache initiating offline of disk 1 group 1

WARNING: offlining disk 1.3914828841 (DATA1_0001) with mask 0x3

。。。。

Wed Nov 23 15:14:28 2011

WARNING: dirty detached from domain 1

Wed Nov 23 15:14:28 2011

ERROR: diskgroup DATA1 was not mounted

Wed Nov 23 15:14:28 2011

WARNING: PST-initiated MANDATORY DISMOUNT of group DATA1 not performed - group not mounted

Wed Nov 23 15:14:28 2011

Errors in file /opt/app/admin/+ASM/bdump/+asm1_b000_25521.trc:

ORA-15001: diskgroup "DATA1" does not exist or is not mounted

查看具体的trace：

cat /opt/app/admin/+ASM/udump/+asm1_ora_25219.trc | less

找到如下错误提示

******************************************************

*** 2011-11-23 15:14:28.703

ksedmp: internal or fatal error

ORA-00600: internal error code, arguments: [723], [529336], [529336], [memory leak], [], [], [], []

Current SQL information unavailable - no SGA.

cat /opt/app/admin/+ASM/bdump/+asm1_b000_25521.trc| less

/opt/app/admin/+ASM/bdump/+asm1_b000_25521.trc

Oracle Database 10g Enterprise Edition Release 10.2.0.1.0 - Production

With the Partitioning, Real Application Clusters, OLAP and Data Mining options

ORACLE_HOME = /opt/app/product/10.2.0/db_1

System name: Linux

Node name: node1

Release: 2.6.18-164.el5

Version: #1 SMP Tue Aug 18 15:51:54 EDT 2009

Machine: i686

Instance name: +ASM1

Redo thread mounted by this instance: 0 <none>

Oracle process number: 17

Unix process pid: 25521, image: oracle@node1 (B000)

*** SERVICE NAME:() 2011-11-23 15:14:28.679

*** SESSION ID:(33.1) 2011-11-23 15:14:28.679

ORA-15001: diskgroup "DATA1" does not exist or is not mounted

怎么看都是没有成功mount磁盘组，还是先收工mount下磁盘组看下：

[oracle@node1 bdump]$ sqlplus /nolog

SQL*Plus: Release 10.2.0.1.0 - Production on Wed Nov 23 15:33:36 2011

Copyright (c) 1982, 2005, Oracle. All rights reserved.

SQL> exit

[oracle@node1 bdump]$ export ORACLE_SID=+ASM1

[oracle@node1 bdump]$ sqlplus /nolog

SQL*Plus: Release 10.2.0.1.0 - Production on Wed Nov 23 15:33:41 2011

Copyright (c) 1982, 2005, Oracle. All rights reserved.

SQL> conn /as sysdba

Connected.

SQL> desc v$asm_diskgroup;

Name Null? Type

----------------------------------------- -------- ----------------------------

GROUP_NUMBER NUMBER

NAME VARCHAR2(30)

SECTOR_SIZE NUMBER

BLOCK_SIZE NUMBER

ALLOCATION_UNIT_SIZE NUMBER

STATE VARCHAR2(11)

TYPE VARCHAR2(6)

TOTAL_MB NUMBER

FREE_MB NUMBER

REQUIRED_MIRROR_FREE_MB NUMBER

USABLE_FILE_MB NUMBER

OFFLINE_DISKS NUMBER

UNBALANCED VARCHAR2(1)

COMPATIBILITY VARCHAR2(60)

DATABASE_COMPATIBILITY VARCHAR2(60)

SQL> set linesize 150

SQL> column name format a30;

SQL> column state format a10;

SQL> select name,state from v$asm_diskgroup;

NAME STATE

------------------------------ ----------

DATA1 DISMOUNTED

果然磁盘组没有加载成功，尝试收工mount磁盘组：

SQL> alter diskgroup data1 mount;

alter diskgroup data1 mount

*

ERROR at line 1:

ORA-15032: not all alterations performed

ORA-15130: diskgroup "DATA1" is being dismounted

ORA-15066: offlining disk "DATA1_0001" may result in a data loss

SQL>

报错了，看看日志：

[oracle@node1 bdump]$ tail -50 alert_+ASM1.log

NOTE: F1X0 found on disk 0 fcn 0.0

NOTE: cache opening disk 1 of grp 1: DATA1_0001 path:/dev/raw/raw4

NOTE: cache mounting (first) group 1/0x26277CDE (DATA1)

* allocate domain 1, invalid = TRUE

kjbdomatt send to node 1

Wed Nov 23 15:37:49 2011

NOTE: attached to recovery domain 1

Wed Nov 23 15:37:49 2011

NOTE: starting recovery of thread=1 ckpt=3.315

NOTE: starting recovery of thread=2 ckpt=3.50

WARNING: cache failed to read fn=3 indblk=0 from disk(s): 1

ORA-15196: invalid ASM block header [kfc.c:7910] [endian_kfbh] [3] [2147483648] [0 != 1]

NOTE: a corrupted block was dumped to the trace file

System State dumped to trace file /opt/app/admin/+ASM/udump/+asm1_ora_21931.trc

NOTE: cache initiating offline of disk 1 group 1

WARNING: offlining disk 1.3914828843 (DATA1_0001) with mask 0x3

NOTE: PST update: grp = 1, dsk = 1, mode = 0x6

Wed Nov 23 15:37:49 2011

ERROR: too many offline disks in PST (grp 1)

Wed Nov 23 15:37:49 2011

NOTE: halting all I/Os to diskgroup DATA1

NOTE: active pin found: 0x0x2427ccd0

NOTE: active pin found: 0x0x2427cc64

Abort recovery for domain 1

NOTE: crash recovery signalled OER-15130

ERROR: ORA-15130 signalled during mount of diskgroup DATA1

NOTE: cache dismounting group 1/0x26277CDE (DATA1)

Wed Nov 23 15:37:51 2011

kjbdomdet send to node 1

detach from dom 1, sending detach message to node 1

Wed Nov 23 15:37:51 2011

Dirty detach reconfiguration started (old inc 1, new inc 1)

List of nodes:

0 1

Global Resource Directory partially frozen for dirty detach

* dirty detach - domain 1 invalid = TRUE

0 GCS resources traversed, 0 cancelled

Wed Nov 23 15:37:51 2011

freeing rdom 1

Dirty Detach Reconfiguration complete

Wed Nov 23 15:37:51 2011

WARNING: dirty detached from domain 1

Wed Nov 23 15:37:51 2011

ERROR: diskgroup DATA1 was not mounted

Wed Nov 23 15:37:52 2011

WARNING: PST-initiated MANDATORY DISMOUNT of group DATA1 not performed - group not mounted

Wed Nov 23 15:37:52 2011

Errors in file /opt/app/admin/+ASM/bdump/+asm1_b000_25521.trc:

ORA-15001: diskgroup "DATA1" does not exist or is not mounted

ORA-15001: diskgroup "DATA1" does not exist or is not mounted

致命的ORA-15196: invalid ASM block header ，提示磁盘坏块了。

[oracle@node1 bdump]$ oerr ora 15196

15196, 00000, "invalid ASM block header [%s:%s] [%s] [%s] [%s] [%s != %s]"

// *Cause: ASM encountered an invalid metadata block.

// *Action: Contact Oracle Support Services.

//

[oracle@node1 bdump]$ oerr ora 15001

15001, 00000, "diskgroup \"%s\" does not exist or is not mounted"

// *Cause: An operation failed because the diskgroup specified does not

// exist or is not mounted by the current ASM instance.

// *Action: Verify that the diskgroup name used is valid, that the

// diskgroup exists, and that the diskgroup is mounted by

// the current ASM instance.

//

没辙了，好在是测试环境，重建吧：

先dbca卸载DB，然后重建diskgroup，最后重建db。

在两个在节点上root用户操作，注意raw3和raw4是要创建磁盘组的设备：

dd if=/dev/zero of=/dev/raw/raw3 bs=1024 count=4

dd if=/dev/zero of=/dev/raw/raw4 bs=1024 count=4

接着重建磁盘组：

SQL> column header_status format a15;

SQL> column path format a30;

SQL> select header_status,path from v$asm_disk;

HEADER_STATUS PATH

--------------- ------------------------------

CANDIDATE /dev/raw/raw3

CANDIDATE /dev/raw/raw4

UNKNOWN ORCL:VOL2

FOREIGN /dev/raw/raw1

UNKNOWN ORCL:VOL1

FOREIGN /dev/raw/raw2

6 rows selected.

SQL> create diskgroup datadisk1 external redundancy disk '/dev/raw/raw3' name d1 disk '/dev/raw/raw4' name d2;

Diskgroup created.

最后重新dbca重建db。

重建之后，重启了虚拟机和主机几把，还有再次发现问题，ok。

-The End-