一次ora-600 ktubko_1故障简单分析

===================================================================================================
0. summary
 
    1. 故障现象
    2. 故障分析
    .   2.1 alert log
    .   2.2 block dump trace
    .   2.3 undo header dump trace
    .   2.4 undo block dump trace
    3. 故障处理
  
===================================================================================================
1. 故障现象

数据库open时报ora-600 kcratr_nab_less_than_odr, 这个错误很简单，根据call stack对比和mos描述基本接近，重建控制文件即可。mos参考如下：

Alter database open fails with ORA-00600 kcratr_nab_less_than_odr (文档 ID 1296264.1)

之后库打开后不多就就因为smon触发ora-600 ktubko_1导致实例宕。

===================================================================================================
2. 故障分析

*************************************************
2.1 alert log

Sat Jun 25 14:11:07 2016
Errors in file /u01/app/oracle/diag/rdbms/cwgkdb1/cwgkdb11/trace/cwgkdb11_smon_34408.trc  (incident=549070):
ORA-00600: internal error code, arguments: [ktubko_1], [], [], [], [], [], [], [], [], [], [], []
Incident details in: /u01/app/oracle/diag/rdbms/cwgkdb1/cwgkdb11/incident/incdir_549070/cwgkdb11_smon_34408_i549070.trc
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
ORACLE Instance cwgkdb11 (pid = 32) - Error 600 encountered while recovering transaction (1060, 17) on object 811985.   --- (usn, slot)
Errors in file /u01/app/oracle/diag/rdbms/cwgkdb1/cwgkdb11/trace/cwgkdb11_smon_34408.trc:
ORA-00600: internal error code, arguments: [ktubko_1], [], [], [], [], [], [], [], [], [], [], []
......
Sat Jun 25 14:16:09 2016
Errors in file /u01/app/oracle/diag/rdbms/cwgkdb1/cwgkdb11/trace/cwgkdb11_smon_34408.trc  (incident=549071):
ORA-00600: internal error code, arguments: [ktubko_1], [], [], [], [], [], [], [], [], [], [], []
Incident details in: /u01/app/oracle/diag/rdbms/cwgkdb1/cwgkdb11/incident/incdir_549071/cwgkdb11_smon_34408_i549071.trc
Sat Jun 25 14:16:12 2016
PMON (ospid: 34337): terminating the instance due to error 474
Sat Jun 25 14:16:12 2016
opiodr aborting process unknown ospid (36185) as a result of ORA-1092
Sat Jun 25 14:16:12 2016
System state dump requested by (instance=1, osid=34337 (PMON)), summary=[abnormal instance termination].
System State dumped to trace file /u01/app/oracle/diag/rdbms/cwgkdb1/cwgkdb11/trace/cwgkdb11_diag_34347_20160625141612.trc
Sat Jun 25 14:16:12 2016
ORA-1092 : opitsk aborting process
Dumping diagnostic data in directory=[cdmp_20160625141612], requested by (instance=1, osid=34337 (PMON)), summary=[abnormal instance termination].
Instance terminated by PMON, pid = 34337

*************************************************
2.2 block dump trace

Block header dump:  0x000a55eb
 Object id on Block? Y
 seg/obj: 0xc63d1  csc: 0xd54.3b9c4a10  itc: 3  flg: E  typ: 2 - INDEX                ---- c63d1转化为10进制为811985
     brn: 0  bdba: 0xa55c0 ver: 0x01 opc: 0
     inc: 0  exflg: 0
 
 Itl           Xid                  Uba         Flag  Lck        Scn/Fsc
0x01   0x024a.02f.00000010  0x00c0e031.0010.01  CB--    0  scn 0x0d54.3b08cc0a
0x02   0x0067.03b.0000007a  0x00c08296.003b.49  ----    1  fsc 0x0000.00000000
0x03   0x0424.011.00000005  0x00c2169d.0008.32  C---    0  scn 0x0d54.3b9c4857       ---- 424转换为10进制是1060, 011转换为10进制是17

*************************************************
2.3 undo header dump trace

#### trace undo header dump ####

......
  TRN TBL::
 
  index  state cflags  wrap#    uel         scn            dba            parent-xid    nub     stmt_num    cmt
  ------------------------------------------------------------------------------------------------
   0x00    9    0x00  0x0005  0x0001  0x0d54.3b9b7206  0x00c2169c  0x0000.000.00000000  0x00000001   0x00000000  1466825963
   0x01    9    0x00  0x0005  0x0002  0x0d54.3b9b7c25  0x00c2169c  0x0000.000.00000000  0x00000001   0x00000000  1466825963
   0x02    9    0x00  0x0005  0x0004  0x0d54.3b9b7f1c  0x00c2169c  0x0000.000.00000000  0x00000001   0x00000000  1466825963
......
   0x0d    9    0x00  0x0005  0x000e  0x0d54.3b9c1297  0x00c2169d  0x0000.000.00000000  0x00000002   0x00000000  1466825966
   0x0e    9    0x00  0x0005  0x000f  0x0d54.3b9c189c  0x00c2169d  0x0000.000.00000000  0x00000001   0x00000000  1466825967
   0x0f    9    0x00  0x0005  0x0010  0x0d54.3b9c18f0  0x00c2169d  0x0000.000.00000000  0x00000001   0x00000000  1466825967
   0x10    9    0x00  0x0005  0xffff  0x0d54.3b9c1bae  0x00c2169d  0x0000.000.00000000  0x00000001   0x00000000  1466825967
   0x11   10    0x90  0x0005  0x0008  0x0d54.3b9c1cf9  0x00c2169d  0x0000.000.00000000  0x00000001   0x00000000  0
......

---- state为10表示active

*************************************************
2.4 undo block dump trace

#### trace undo block dump ####

UNDO BLK:  
xid: 0x0424.011.00000005  seq: 0x8   cnt: 0x34  irb: 0x32  icl: 0x0   flg: 0x0000           ----当前undo block所记录的最新事务xid
                                      ---- irb代表如果该事务回滚，那么0x32为起点。icl代表该undo block中undo chain的末端位置
 Rec Offset      Rec Offset      Rec Offset      Rec Offset      Rec Offset
---------------------------------------------------------------------------
0x01 0x3f7c     0x02 0x3f14     0x03 0x3ec0     0x04 0x3e5c     0x05 0x3df4     
0x06 0x3d90     0x07 0x3d4c     0x08 0x3cf8     0x09 0x3ca0     0x0a 0x3c54     
0x0b 0x3c10     0x0c 0x3bbc     0x0d 0x3b64     0x0e 0x3b18     0x0f 0x3ad4     
0x10 0x3a80     0x11 0x3a28     0x12 0x39c4     0x13 0x3980     0x14 0x392c     
0x15 0x38d4     0x16 0x3870     0x17 0x3804     0x18 0x3798     0x19 0x372c     
0x1a 0x36c0     0x1b 0x3654     0x1c 0x35e8     0x1d 0x3560     0x1e 0x350c     
0x1f 0x34ac     0x20 0x3444     0x21 0x33f0     0x22 0x3388     0x23 0x3334     
0x24 0x32d0     0x25 0x3268     0x26 0x321c     0x27 0x31b0     0x28 0x3158     
0x29 0x30f0     0x2a 0x3084     0x2b 0x2ffc     0x2c 0x2fa0     0x2d 0x2f38     
0x2e 0x2ed0     0x2f 0x2e48     0x30 0x2dc0     0x31 0x2d6c     0x32 0x2d0c     
0x33 0x2ca4     0x34 0x2c38     

......
......

*-----------------------------
* Rec #0x32  slt: 0x11  objn: 811985(0x000c63d1)  objd: 811985  tblspc: 31(0x0000001f)
*       Layer:  10 (Index)   opc: 22   rci 0x31   
Undo type:  Regular undo   Last buffer split:  No 
Temp Object:  No 
Tablespace Undo:  No 
rdba: 0x00000000
*-----------------------------
index undo for leaf key operations
KTB Redo 
op: 0x04  ver: 0x01  
compat bit: 4 (post-11) padding: 1
op: L  itl: xid:  0x03c4.026.00000006 uba: 0x00c217bc.000c.9f                ---- record为32的记录对应信息不对
                      flg: C---    lkc:  0     scn: 0x0d54.3b9bd4f6
Dump kdilk : itl=3, kdxlkflg=0x1 sdc=0 indexid=0xa2043 block=0x000a55eb
(kdxlpu): purge leaf row
key :(6):  05 35 04 4e 24 c4

===================================================================================================
3. 故障处理

处理起来很简单，使用10513时间屏蔽回滚，重建问题索引再恢复即可。

Event='10513 trace name context forever, level 2'            # Disable transaction recovery