【Oracle ASM数据恢复】ORA-15038: disk ‘XXXXXXX’ mismatch on ‘Time Stamp’ with Target Disk Group错误解析
2014-09-17 15:56
615 查看
如果在mount diskgroup时遇到如下错误,则你有必要读一下这篇文章:
该错误一般是由于OS级别的多路径配置存在问题,导致某个节点上出现大量重复的存储设备而引起的。
对于该类问题,首先从alert.log确认最近一次成功mount时那些磁盘对mount而言是必要的,例如:
接下来可以通过kfed来验证DISK HEADER,使用如下脚本:
Ask_Maclean_liu_Oracle2> Then validate the existing disk create timestamp on node 1 and
also check create timestamp of same disk on node 2 from ASMLib Label level.
Check the timestamp at ASMLib label on node 2 .
kfed read /dev/oracleasm/disks/ | egrep 'kfbh.type|kfdhdb.dskname|kfdhdb.crestmp.hi|kfdhdb.crestmp.lo|kfdhdb.dsknum'
Note down ,"kfdhdb.crestmp.hi" and "kfdhdb.crestmp.lo" from same Label on node 2 and note it down.
For Node 1 ,it will scan unique one ASMLib label check the same ASMlib label on node 1 too.
kfed read /dev/oracleasm/disks/ | egrep 'kfbh.type|kfdhdb.dskname|kfdhdb.crestmp.hi|kfdhdb.crestmp.lo|kfdhdb.dsknum'
And note down the major and minor number of same label from below command on node 1.
ls -ltr /dev/oracleasm/disks/
and get the emcpower device name from /proc/partitions by matching major and minor number .
Then note down the emcpower device name too.
3> Then try to findout which emcpower device is mapped with same ASMLib label using below command from root OS user.
kfed read /dev/emcpower<#>| egrep 'kfbh.type|kfdhdb.driver.provstr|kfdhdb.dskname|kfdhdb.crestmp.hi|kfdhdb.crestmp.lo|kfdhdb.dsknum'
Note down ,emcpower device names on node 1,because you will see lot devices with same ASMLib label for string "kfdhdb.driver.provstr".
Also note down,the value of string "kfdhdb.crestmp.hi" and "kfdhdb.crestmp.lo" all same labeled emcpower pseudo devices .
7> Then mount all those diskgroup after starting up the cluster.
Example ::----------
For DATA2 ,
Node 2,
(grid) grid11gr2 houxpsdbrac02:+ASM2:~> kfed read /dev/oracleasm/disks/DATA2 | egrep
'kfbh.type|kfdhdb.dskname|kfdhdb.crestmp.hi|kfdhdb.crestmp.lo|kfdhdb.dsknum'
kfbh.type: 1 ; 0x002: KFBTYP_DISKHEAD
kfdhdb.dsknum: 11 ; 0x024: 0x000b
kfdhdb.dskname: DATA2 ; 0x028: length=5
kfdhdb.crestmp.hi: 32960211 ; 0x0a8: HOUR=0x13 DAYS=0x16 MNTH=0xb YEAR=0x7db
kfdhdb.crestmp.lo: 2385579008 ; 0x0ac: USEC=0x0 MSEC=0x43 SECS=0x23 MINS=0x23
Node 1,
disknum 11 : /dev/emcpowercf1
kfbh.endian: 1 ; 0x000: 0x01
kfbh.hard: 130 ; 0x001: 0x82
kfbh.type: 1 ; 0x002: KFBTYP_DISKHEAD
kfbh.datfmt: 1 ; 0x003: 0x01
kfbh.block.blk: 0 ; 0x004: T=0 NUMB=0x0
kfbh.block.obj: 2147483659 ; 0x008: TYPE=0x8 NUMB=0xb
kfbh.check: 4029604929 ; 0x00c: 0xf02ee441
kfbh.fcn.base: 0 ; 0x010: 0x00000000
kfbh.fcn.wrap: 0 ; 0x014: 0x00000000
kfbh.spare1: 0 ; 0x018: 0x00000000
kfbh.spare2: 0 ; 0x01c: 0x00000000
kfdhdb.driver.provstr: ORCLDISKDATA2 ; 0x000: length=13 <<<<<<<<<<<<<<<<<<<<<<<<<<
kfdhdb.driver.reserved[0]: 1096040772 ; 0x008: 0x41544144
kfdhdb.driver.reserved
: 50 ; 0x00c: 0x00000032
kfdhdb.driver.reserved
: 0 ; 0x010: 0x00000000
kfdhdb.driver.reserved
: 0 ; 0x014: 0x00000000
kfdhdb.driver.reserved
: 0 ; 0x018: 0x00000000
kfdhdb.driver.reserved
: 0 ; 0x01c: 0x00000000
kfdhdb.compat: 186646528 ; 0x020: 0x0b200000
kfdhdb.dsknum: 11 ; 0x024: 0x000b <<<<<<<<<<<<<<<<<<<<<<<<<<
kfdhdb.grptyp: 1 ; 0x026: KFDGTP_EXTERNAL
kfdhdb.hdrsts: 3 ; 0x027: KFDHDR_MEMBER
kfdhdb.dskname: DATA2 ; 0x028: length=5
kfdhdb.grpname: DATA ; 0x048: length=4
kfdhdb.fgname: DATA2 ; 0x068: length=5
kfdhdb.capname: ; 0x088: length=0
kfdhdb.crestmp.hi: 32960399 ; 0x0a8: HOUR=0xf DAYS=0x1c MNTH=0xb YEAR=0x7db
kfdhdb.crestmp.lo: 31373312 ; 0x0ac: USEC=0x0 MSEC=0x3ae SECS=0x1d MINS=0x0
/dev/emcpowerdb1
kfbh.endian: 1 ; 0x000: 0x01
kfbh.hard: 130 ; 0x001: 0x82
kfbh.type: 1 ; 0x002: KFBTYP_DISKHEAD
kfbh.datfmt: 1 ; 0x003: 0x01
kfbh.block.blk: 0 ; 0x004: T=0 NUMB=0x0
kfbh.block.obj: 2147483659 ; 0x008: TYPE=0x8 NUMB=0xb
kfbh.check: 2125098320 ; 0x00c: 0x7eaa6d50
kfbh.fcn.base: 0 ; 0x010: 0x00000000
kfbh.fcn.wrap: 0 ; 0x014: 0x00000000
kfbh.spare1: 0 ; 0x018: 0x00000000
kfbh.spare2: 0 ; 0x01c: 0x00000000
kfdhdb.driver.provstr: ORCLDISKDATA2 ; 0x000: length
4000
=13 <<<<<<<<<<<<<<<<<<<<<<<<<<
kfdhdb.driver.reserved[0]: 1096040772 ; 0x008: 0x41544144
kfdhdb.driver.reserved
: 50 ; 0x00c: 0x00000032
kfdhdb.driver.reserved
: 0 ; 0x010: 0x00000000
kfdhdb.driver.reserved
: 0 ; 0x014: 0x00000000
kfdhdb.driver.reserved
: 0 ; 0x018: 0x00000000
kfdhdb.driver.reserved
: 0 ; 0x01c: 0x00000000
kfdhdb.compat: 186646528 ; 0x020: 0x0b200000
kfdhdb.dsknum: 11 ; 0x024: 0x000b <<<<<<<<<<<<<<<<<<<<<<<<<<
kfdhdb.grptyp: 1 ; 0x026: KFDGTP_EXTERNAL
kfdhdb.hdrsts: 3 ; 0x027: KFDHDR_MEMBER
kfdhdb.dskname: DATA2 ; 0x028: length=5
kfdhdb.grpname: DATA ; 0x048: length=4
kfdhdb.fgname: DATA2 ; 0x068: length=5
kfdhdb.capname: ; 0x088: length=0
kfdhdb.crestmp.hi: 32960211 ; 0x0a8: HOUR=0x13 DAYS=0x16 MNTH=0xb YEAR=0x7db <<<<<<<<<<<<<<<<<<<<<<<<<<
kfdhdb.crestmp.lo: 2385579008 ; 0x0ac: USEC=0x0 MSEC=0x43 SECS=0x23 MINS=0x23 <<<<<<<<<<<<<<<<<<<<<<<<<<
Note:: If you want to change the asm_diskstring as a workaround ,you have to again find out the right device to open write access to exact device
as at mounting time ASM writes mount timestamp.
SQL> alter diskgroup DATA mount; alter diskgroup DATA mount * ERROR at line 1: ORA-15032: not all alterations performed ORA-15040: diskgroup is incomplete ORA-15042: ASM disk "17" is missing from group number "1" ORA-15042: ASM disk "16" is missing from group number "1" ORA-15042: ASM disk "15" is missing from group number "1" ORA-15042: ASM disk "14" is missing from group number "1" ORA-15042: ASM disk "13" is missing from group number "1" ORA-15038: disk 'ORCL:DATA25' mismatch on 'Time Stamp' with target disk group [1861040353] [1861808156] ORA-15038: disk 'ORCL:DATA24' mismatch on 'Time Stamp' with target disk group [1861040353] [1861808156] ORA-15038: disk 'ORCL:DATA23' mismatch on 'Time Stamp' with target disk group [1861040353] [1861808156] ORA-15038: disk 'ORCL:DATA22' mismatch on 'Time Stamp' with target disk group [1861040353] [1861808156] ORA-15038: disk 'ORCL:DATA21' mismatch on 'Time Stamp' with target disk group [1861040353] [1861808156] [oracle@mlab2 ~]$ oerr ora 15042 15042, 00000, "ASM disk \"%s\" is missing from group number \"%s\" " // *Cause: The specified disk, which is a necessary part of a diskgroup, // could not be found on the system. // *Action: Check the hardware configuration. // [oracle@mlab2 ~]$ oerr ora 15038 15038, 00000, "disk '%s' mismatch on '%s' with target disk group [%s] [%s]" // *Cause: An attempt was made to mount into a disk group a disk whose // recorded allocation unit size, metadata block size, physical // sector size, or creation time stamp was inconsistent with the other // disk group members. // *Action: Check if the system configuration has changed. Verify disk // discovery string. //
该错误一般是由于OS级别的多路径配置存在问题,导致某个节点上出现大量重复的存储设备而引起的。
对于该类问题,首先从alert.log确认最近一次成功mount时那些磁盘对mount而言是必要的,例如:
NOTE: cache opening disk 11 of grp 1: DATA2 label:DATA2 <<<<<<<<<<<<<<<<<< 5 NOTE: cache opening disk 12 of grp 1: DATA20 label:DATA20 > NOTE: cache opening disk 13 of grp 1: DATA3 label:DATA3 <<<<<<<<<<<<<<<<<<<<<< 6 NOTE: cache opening disk 14 of grp 1: DATA4 label:DATA4 NOTE: cache opening disk 15 of grp 1: DATA5 label:DATA5 NOTE: cache opening disk 16 of grp 1: DATA6 label:DATA6 NOTE: cache opening disk 17 of grp 1: DATA7 label:DATA7 NOTE: cache opening disk 18 of grp 1: DATA8 label:DATA8 NOTE: cache opening disk 19 of grp 1: DATA9 label:DATA9
接下来可以通过kfed来验证DISK HEADER,使用如下脚本:
#! /bin/sh rm /tmp/kfed_DH.out /tmp/kfed_FS.out /tmp/kfed_BK.out /tmp/kfed_FD.out /tmp/kfed_DD.out /tmp/kfed_PST.out for i in `ls /dev/asm-disk*` do echo $i >> /tmp/kfed_DH.out kfed read $i >> /tmp/kfed_DH.out echo $i >> /tmp/kfed_FS.out kfed read $i blkn=1 >> /tmp/kfed_FS.out echo $i >> /tmp/kfed_BK.out kfed read $i aun=1 blkn=254 >> /tmp/kfed_BK.out echo $i >> /tmp/kfed_FD.out kfed read $i aun=2 blkn=1 >> /tmp/kfed_FD.out echo $i >> /tmp/kfed_DD.out kfed read $i aun=2 blkn=2 >> /tmp/kfed_DD.out echo $i >> /tmp/kfed_PST.out kfed read $i aun=1 blkn=2 >> /tmp/kfed_PST.out done
====================================================================================================================================
Ask_Maclean_liu_Oracle2> Then validate the existing disk create timestamp on node 1 and
also check create timestamp of same disk on node 2 from ASMLib Label level.
Check the timestamp at ASMLib label on node 2 .
kfed read /dev/oracleasm/disks/ | egrep 'kfbh.type|kfdhdb.dskname|kfdhdb.crestmp.hi|kfdhdb.crestmp.lo|kfdhdb.dsknum'
Note down ,"kfdhdb.crestmp.hi" and "kfdhdb.crestmp.lo" from same Label on node 2 and note it down.
For Node 1 ,it will scan unique one ASMLib label check the same ASMlib label on node 1 too.
kfed read /dev/oracleasm/disks/ | egrep 'kfbh.type|kfdhdb.dskname|kfdhdb.crestmp.hi|kfdhdb.crestmp.lo|kfdhdb.dsknum'
And note down the major and minor number of same label from below command on node 1.
ls -ltr /dev/oracleasm/disks/
and get the emcpower device name from /proc/partitions by matching major and minor number .
Then note down the emcpower device name too.
3> Then try to findout which emcpower device is mapped with same ASMLib label using below command from root OS user.
kfed read /dev/emcpower<#>| egrep 'kfbh.type|kfdhdb.driver.provstr|kfdhdb.dskname|kfdhdb.crestmp.hi|kfdhdb.crestmp.lo|kfdhdb.dsknum'
Note down ,emcpower device names on node 1,because you will see lot devices with same ASMLib label for string "kfdhdb.driver.provstr".
Also note down,the value of string "kfdhdb.crestmp.hi" and "kfdhdb.crestmp.lo" all same labeled emcpower pseudo devices .
7> Then mount all those diskgroup after starting up the cluster.
Example ::----------
For DATA2 ,
Node 2,
(grid) grid11gr2 houxpsdbrac02:+ASM2:~> kfed read /dev/oracleasm/disks/DATA2 | egrep
'kfbh.type|kfdhdb.dskname|kfdhdb.crestmp.hi|kfdhdb.crestmp.lo|kfdhdb.dsknum'
kfbh.type: 1 ; 0x002: KFBTYP_DISKHEAD
kfdhdb.dsknum: 11 ; 0x024: 0x000b
kfdhdb.dskname: DATA2 ; 0x028: length=5
kfdhdb.crestmp.hi: 32960211 ; 0x0a8: HOUR=0x13 DAYS=0x16 MNTH=0xb YEAR=0x7db
kfdhdb.crestmp.lo: 2385579008 ; 0x0ac: USEC=0x0 MSEC=0x43 SECS=0x23 MINS=0x23
Node 1,
disknum 11 : /dev/emcpowercf1
kfbh.endian: 1 ; 0x000: 0x01
kfbh.hard: 130 ; 0x001: 0x82
kfbh.type: 1 ; 0x002: KFBTYP_DISKHEAD
kfbh.datfmt: 1 ; 0x003: 0x01
kfbh.block.blk: 0 ; 0x004: T=0 NUMB=0x0
kfbh.block.obj: 2147483659 ; 0x008: TYPE=0x8 NUMB=0xb
kfbh.check: 4029604929 ; 0x00c: 0xf02ee441
kfbh.fcn.base: 0 ; 0x010: 0x00000000
kfbh.fcn.wrap: 0 ; 0x014: 0x00000000
kfbh.spare1: 0 ; 0x018: 0x00000000
kfbh.spare2: 0 ; 0x01c: 0x00000000
kfdhdb.driver.provstr: ORCLDISKDATA2 ; 0x000: length=13 <<<<<<<<<<<<<<<<<<<<<<<<<<
kfdhdb.driver.reserved[0]: 1096040772 ; 0x008: 0x41544144
kfdhdb.driver.reserved
: 50 ; 0x00c: 0x00000032
kfdhdb.driver.reserved
: 0 ; 0x010: 0x00000000
kfdhdb.driver.reserved
: 0 ; 0x014: 0x00000000
kfdhdb.driver.reserved
: 0 ; 0x018: 0x00000000
kfdhdb.driver.reserved
: 0 ; 0x01c: 0x00000000
kfdhdb.compat: 186646528 ; 0x020: 0x0b200000
kfdhdb.dsknum: 11 ; 0x024: 0x000b <<<<<<<<<<<<<<<<<<<<<<<<<<
kfdhdb.grptyp: 1 ; 0x026: KFDGTP_EXTERNAL
kfdhdb.hdrsts: 3 ; 0x027: KFDHDR_MEMBER
kfdhdb.dskname: DATA2 ; 0x028: length=5
kfdhdb.grpname: DATA ; 0x048: length=4
kfdhdb.fgname: DATA2 ; 0x068: length=5
kfdhdb.capname: ; 0x088: length=0
kfdhdb.crestmp.hi: 32960399 ; 0x0a8: HOUR=0xf DAYS=0x1c MNTH=0xb YEAR=0x7db
kfdhdb.crestmp.lo: 31373312 ; 0x0ac: USEC=0x0 MSEC=0x3ae SECS=0x1d MINS=0x0
/dev/emcpowerdb1
kfbh.endian: 1 ; 0x000: 0x01
kfbh.hard: 130 ; 0x001: 0x82
kfbh.type: 1 ; 0x002: KFBTYP_DISKHEAD
kfbh.datfmt: 1 ; 0x003: 0x01
kfbh.block.blk: 0 ; 0x004: T=0 NUMB=0x0
kfbh.block.obj: 2147483659 ; 0x008: TYPE=0x8 NUMB=0xb
kfbh.check: 2125098320 ; 0x00c: 0x7eaa6d50
kfbh.fcn.base: 0 ; 0x010: 0x00000000
kfbh.fcn.wrap: 0 ; 0x014: 0x00000000
kfbh.spare1: 0 ; 0x018: 0x00000000
kfbh.spare2: 0 ; 0x01c: 0x00000000
kfdhdb.driver.provstr: ORCLDISKDATA2 ; 0x000: length
4000
=13 <<<<<<<<<<<<<<<<<<<<<<<<<<
kfdhdb.driver.reserved[0]: 1096040772 ; 0x008: 0x41544144
kfdhdb.driver.reserved
: 50 ; 0x00c: 0x00000032
kfdhdb.driver.reserved
: 0 ; 0x010: 0x00000000
kfdhdb.driver.reserved
: 0 ; 0x014: 0x00000000
kfdhdb.driver.reserved
: 0 ; 0x018: 0x00000000
kfdhdb.driver.reserved
: 0 ; 0x01c: 0x00000000
kfdhdb.compat: 186646528 ; 0x020: 0x0b200000
kfdhdb.dsknum: 11 ; 0x024: 0x000b <<<<<<<<<<<<<<<<<<<<<<<<<<
kfdhdb.grptyp: 1 ; 0x026: KFDGTP_EXTERNAL
kfdhdb.hdrsts: 3 ; 0x027: KFDHDR_MEMBER
kfdhdb.dskname: DATA2 ; 0x028: length=5
kfdhdb.grpname: DATA ; 0x048: length=4
kfdhdb.fgname: DATA2 ; 0x068: length=5
kfdhdb.capname: ; 0x088: length=0
kfdhdb.crestmp.hi: 32960211 ; 0x0a8: HOUR=0x13 DAYS=0x16 MNTH=0xb YEAR=0x7db <<<<<<<<<<<<<<<<<<<<<<<<<<
kfdhdb.crestmp.lo: 2385579008 ; 0x0ac: USEC=0x0 MSEC=0x43 SECS=0x23 MINS=0x23 <<<<<<<<<<<<<<<<<<<<<<<<<<
Note:: If you want to change the asm_diskstring as a workaround ,you have to again find out the right device to open write access to exact device
as at mounting time ASM writes mount timestamp.
相关文章推荐
- Oracle RAC ASM disk header 备份 恢复 与 重建 示例说明
- Oracle RAC ASM disk header 备份 恢复 与 重建 示例说明
- Oracle ASM数据库故障数据恢复解决方案
- Oracle RAC ASM disk header 备份 恢复 与 重建 示例说明
- oracle中asm disk header 彻底损坏恢复的方法
- Oracle ASM数据库故障数据恢复解决方案
- Oracle 学习之--ASM DISK Header的备份和恢复(2)
- Oracle数据恢复:解决ORA-00600:[4000] ORA-00704: bootstrap process failure错误一例
- Oracle 学习之--ASM DISK Header的备份和恢复(1)
- Oracle技术之ASM上恢复STANDBY数据库出现ORA-15173错误
- Oracle ASM故障数据恢复
- Oracle RAC ASM disk header 备份 恢复 与 重建 示例说明
- Oracle数据恢复:解决ORA-00600:[4000] ORA-00704: bootstrap process failure错误一例
- AUL(MyDUL) Oracle及Oracle ASM数据恢复
- Oracle 11G Rman备份ASM数据恢复到本地磁盘
- Oracle ASM数据库故障数据恢复解决方案
- Oracle 学习之--ASM DISK Header的备份和恢复(1)
- Oracle RAC ASM disk header 备份 恢复 与 重建 示例说明
- Oracle数据恢复:解决ORA-00600:[4000] ORA-00704: bootstrap process failure错误一例
- 意外删除Oracle数据文件(dbf),恢复oralce库的解决办法--Oracle错误代码:ORA-01033