如何分析解读systemstat dump产生的trc文件

ORACLE数据库的systemstatdump生成trace文件虽然比较简单，但是怎么从trace文件中浩如烟海的信息中提炼有用信息，并作出分析诊断是一件技术活，下面收集、整理如何分析解读systemstatdump产生的trace文件。如果要人工去解读systemstatdump生成的trace文件，真是一件体力活，因为这些trace文件动不动就几百M甚至更大，它产生的跟踪文件包含了系统中所有进程的进程状态等信息。每个进程对应跟踪文件中的一段内容，反映该进程的状态信息，包括进程信息，会话信息，enqueues信息（主要是lock的信息），缓冲区的信息和该进程在SGA区中持有的（held）对象的状态等信息。dumpsystemstate产生的跟踪文件是从dump那一刻开始到dump任务完成之间一段事件内的系统内所有进程的信息。我们需要的是找到导致问题出现的进程的信息，然后根据进程的信息判断导致问题出现的rootcause，并在分析问题后解决问题。幸好网上有人写了这个一个工具ass109.awk，可以节约分析systemstatdump文件或跟踪文件（tracefile）的时间，可以将trace文件中关键信息梳理、整理出来，当然如果了解详细信息，还是必须人工去解读。下面贴上一个例子，是我在学习中的一个案例，本人也是在学习、研究过程中，如有分析不对的地方敬请指出

[oracle@db-serverudump]$awk-fass109.awkscm2_ora_25575.trc

StartingSystemstate1

.................................................

StartingSystemstate2

....................................................

StartingSystemstate3

....................................................

Ass.AwkVersion1.0.9-Processingscm2_ora_25575.trc

SystemState1

~~~~~~~~~~~~~~~~

1:

2:waitingfor'pmontimer'

3:waitingfor'rdbmsipcmessage'

4:waitingfor'rdbmsipcmessage'

5:waitingfor'rdbmsipcmessage'

6:waitingfor'rdbmsipcmessage'

7:waitingfor'rdbmsipcmessage'

8:lastwaitfor'smontimer'

9:waitingfor'rdbmsipcmessage'

10:waitingfor'rdbmsipcmessage'

11:waitingfor'rdbmsipcmessage'

12:waitingfor'rdbmsipcmessage'

13:waitingfor'SQL*Netmessagefromclient'[Latch855675ae0]

Cmd:Select

14:waitingfor'SQL*Netmessagefromclient'[Latch8556759a0]

Cmd:Select

15:waitingfor'SQL*Netmessagefromclient'

Cmd:Select

16:waitingfor'SQL*Netmessagefromclient'[Latch855675720]

17:waitingfor'SQL*Netmessagefromclient'[Latch8556755e0]

Cmd:Insert

18:waitingfor'SQL*Netmessagefromclient'[Latch8556755e0]

Cmd:Select

19:waitingfor'SQL*Netmessagefromclient'[Latch8556755e0]

Cmd:Select

20:waitingfor'SQL*Netmessagefromclient'[Latch8556755e0]

Cmd:Select

21:waitingfor'SQL*Netmessagefromclient'[Latch8556755e0]

Cmd:Insert

22:waitingfor'virtualcircuitstatus'

Cmd:Select

23:

24:

25:waitingfor'virtualcircuitstatus'

Cmd:Select

26:

27:waitingfor'virtualcircuitstatus'

Cmd:Select

28:

29:waitingfor'latch:sharedpool'[Latch8556759a0]

Cmd:Select

30:

31:waitingfor'virtualcircuitstatus'

Cmd:Select

33:waitingfor'jobqslavewait'

34:

35:

36:waitingfor'StreamsAQ:qmnslaveidlewait'

37:waitingfor'rdbmsipcmessage'

38:waitingfor'rdbmsipcmessage'

39:waitingfor'rdbmsipcmessage'

40:waitingfor'rdbmsipcmessage'

41:waitingfor'rdbmsipcmessage'

42:waitingfor'rdbmsipcmessage'

43:waitingfor'rdbmsipcmessage'

44:waitingfor'rdbmsipcmessage'

45:waitingfor'rdbmsipcmessage'

46:waitingfor'rdbmsipcmessage'

47:waitingfor'StreamsAQ:qmncoordinatoridlewait'

49:for'StreamsAQ:waitingfortimemanagementorcleanuptasks'

58:

61:waitingfor'virtualcircuitstatus'

Cmd:Select

Blockers

~~~~~~~~

Aboveisalistofalltheprocesses.Iftheyarewaitingforaresource

thenitwillbegiveninsquarebrackets.Belowisasummaryofthe

waiteduponresources,togetherwiththeholderofthatresource.

Notes:

~~~~~

oAprocessidof'???'impliesthattheholderwasnotfoundinthe

systemstate.

ResourceHolderState

Latch855675ae0???Blocker

Latch8556759a0???Blocker

Latch855675720???Blocker

Latch8556755e0???Blocker

ObjectNames

~~~~~~~~~~~~

Latch855675ae0Childlibrarycache

Latch8556759a0Childlibrarycache

Latch855675720Childlibrarycache

Latch8556755e0Childlibrarycache

SystemState2

~~~~~~~~~~~~~~~~

1:

2:waitingfor'pmontimer'

3:waitingfor'rdbmsipcmessage'

4:waitingfor'rdbmsipcmessage'

5:waitingfor'rdbmsipcmessage'

6:waitingfor'rdbmsipcmessage'

7:waitingfor'rdbmsipcmessage'

8:waitingfor'smontimer'

9:waitingfor'rdbmsipcmessage'

10:waitingfor'rdbmsipcmessage'

11:waitingfor'rdbmsipcmessage'

12:waitingfor'rdbmsipcmessage'

13:waitingfor'SQL*Netmessagefromclient'[Latch855675900]

Cmd:Select

14:waitingfor'SQL*Netmessagefromclient'[Latch855675900]

Cmd:Select

15:waitingfor'SQL*Netmessagefromclient'[Latch855675900]

Cmd:Select

16:waitingfor'SQL*Netmessagefromclient'[Latch855675900]

Cmd:Select

17:waitingfor'SQL*Netmessagefromclient'[Latch855675900]

Cmd:Select

18:waitingfor'SQL*Netmessagefromclient'[Latch855675900]

Cmd:Select

19:waitingfor'SQL*Netmessagefromclient'[Latch855675900]

Cmd:Select

20:waitingfor'SQL*Netmessagefromclient'[Latch855675900]

Cmd:Select

21:waitingfor'SQL*Netmessagefromclient'[Latch855675680]

Cmd:Select

22:

23:

24:

25:

26:

27:

28:waitingfor'virtualcircuitstatus'

29:

30:

31:waitingfor'virtualcircuitstatus'

Cmd:Select

32:waitingfor'jobqslavewait'

33:lastwaitfor'latch:sharedpool'[Latch600f7320]

34:

35:waitingfor'virtualcircuitstatus'

Cmd:Select

36:waitingfor'StreamsAQ:qmnslaveidlewait'

37:waitingfor'rdbmsipcmessage'

38:waitingfor'rdbmsipcmessage'

39:waitingfor'rdbmsipcmessage'

40:waitingfor'rdbmsipcmessage'

41:waitingfor'rdbmsipcmessage'

42:waitingfor'rdbmsipcmessage'

43:waitingfor'rdbmsipcmessage'

44:waitingfor'rdbmsipcmessage'

45:waitingfor'rdbmsipcmessage'

46:waitingfor'rdbmsipcmessage'

47:waitingfor'StreamsAQ:qmncoordinatoridlewait'

48:waitingfor'librarycacheloadlock'

49:for'StreamsAQ:waitingfortimemanagementorcleanuptasks'

50:waitingfor'librarycacheloadlock'

58:

61:waitingfor'virtualcircuitstatus'

Cmd:Select

Blockers

~~~~~~~~

Aboveisalistofalltheprocesses.Iftheyarewaitingforaresource

thenitwillbegiveninsquarebrackets.Belowisasummaryofthe

waiteduponresources,togetherwiththeholderofthatresource.

Notes:

~~~~~

oAprocessidof'???'impliesthattheholderwasnotfoundinthe

systemstate.

ResourceHolderState

Latch855675900???Blocker

Latch855675680???Blocker

Latch600f7320???Blocker

ObjectNames

~~~~~~~~~~~~

Latch855675900Childlibrarycache

Latch855675680Childlibrarycache

Latch600f7320Childsharedpool

SystemState3

~~~~~~~~~~~~~~~~

1:

2:waitingfor'pmontimer'

3:waitingfor'rdbmsipcmessage'

4:waitingfor'rdbmsipcmessage'

5:waitingfor'rdbmsipcmessage'

6:waitingfor'rdbmsipcmessage'

7:waitingfor'rdbmsipcmessage'

8:waitingfor'smontimer'

9:waitingfor'rdbmsipcmessage'

10:waitingfor'rdbmsipcmessage'

11:waitingfor'latch:sharedpool'[Latch600f7320]

12:waitingfor'rdbmsipcmessage'

13:waitingfor'SQL*Netmessagefromclient'[Latch855675540]

Cmd:Select

14:waitingfor'SQL*Netmessagefromclient'[Latch855675540]

Cmd:Select

15:waitingfor'SQL*Netmessagefromclient'[Latch855675b80]

Cmd:Select

16:waitingfor'SQL*Netmessagefromclient'[Latch8556757c0]

Cmd:Select

17:waitingfor'SQL*Netmessagefromclient'[Latch855675680]

Cmd:Select

18:waitingfor'SQL*Netmessagefromclient'[Latch855675680]

Cmd:Select

19:waitingfor'SQL*Netmessagefromclient'[Latch855675680]

Cmd:Select

20:waitingfor'SQL*Netmessagefromclient'

Cmd:Select

21:waitingfor'SQL*Netmessagefromclient'[Latch855675680]

Cmd:Select

22:

23:

24:

25:

26:

27:

28:waitingfor'virtualcircuitstatus'

29:

30:

31:waitingfor'virtualcircuitstatus'

Cmd:Select

32:waitingfor'jobqslavewait'

33:lastwaitfor'latch:sharedpool'[Latch600f7320]

Cmd:Select

34:

35:waitingfor'virtualcircuitstatus'

Cmd:Select

36:waitingfor'StreamsAQ:qmnslaveidlewait'

37:waitingfor'rdbmsipcmessage'

38:waitingfor'rdbmsipcmessage'

39:waitingfor'rdbmsipcmessage'

40:waitingfor'rdbmsipcmessage'

41:waitingfor'rdbmsipcmessage'

42:waitingfor'rdbmsipcmessage'

43:waitingfor'rdbmsipcmessage'

44:waitingfor'rdbmsipcmessage'

45:waitingfor'rdbmsipcmessage'

46:waitingfor'rdbmsipcmessage'

47:waitingfor'StreamsAQ:qmncoordinatoridlewait'

48:waitingfor'SQL*Netmessagefromclient'

49:for'StreamsAQ:waitingfortimemanagementorcleanuptasks'

50:waitingfor'latch:sharedpool'[Latch600f7320]

Cmd:Select

58:

61:waitingfor'virtualcircuitstatus'

Cmd:Select

Blockers

~~~~~~~~

Aboveisalistofalltheprocesses.Iftheyarewaitingforaresource

thenitwillbegiveninsquarebrackets.Belowisasummaryofthe

waiteduponresources,togetherwiththeholderofthatresource.

Notes:

~~~~~

oAprocessidof'???'impliesthattheholderwasnotfoundinthe

systemstate.

ResourceHolderState

Latch600f7320???Blocker

Latch855675540???Blocker

Latch855675b80???Blocker

Latch8556757c0???Blocker

Latch855675680???Blocker

ObjectNames

~~~~~~~~~~~~

Latch600f7320Childsharedpool

Latch855675540Childlibrarycache

Latch855675b80Childlibrarycache

Latch8556757c0Childlibrarycache

Latch855675680Childlibrarycache

从输出信息我们能判断我当时做了3次系统状态转储（实际也是执行了三次oradebugdumpsystemstate266），从SystemState2，我们可以看到有3个Blocker，我们以其中部分信息做分析



其中的一个Blocker的latch是855675900，而且这个latch造成了进程20、17、16、19、14、21、15的waitingfor'SQL*Netmessagefromclient',从下面信息可以看到hold住latch855675900是oracle@xxxx(J000)进程，也就是job的进程。也就说，由于这个j000进程的异常，hold住了855675900的latch。




其实这个案例跟“一个Job运行失败导致数据库挂死”有点类似，最后也发现这个JOB是EMD_MAINTENANCE.EXECUTE_EM_DBMS_JOB_PROCS，当然引起问题的原因更复杂，不在此处讨论。另外，metalink上也有一篇关于如何解读、理解SystemstateDumps的文章：ReadingandUnderstandingSystemstateDumps(文档ID423153.1)，具体内容如下所示。

PURPOSE

[align=left][/align]
Tobeabletoreadasystemstate,ornavigatethroughasystemstateinordertoidentifywhatsessionsaredoingand,
inthecaseofawaitingsession,whichsession(s)holdtheresourceitrequires

SCOPE

ThisdocumentisintendedforDBAs.

DETAILS

[align=left][/align]
Howtousethisdocument

Eachwaitscenariowillbegiven,alongwithkeypointsinthesystemstatewhichcanbeusedtomatchtothecorrespondingentry
inyourownsystemstates.Itwillthengiveyouexamplesofmatchingholders(iewhatyouneedtofindintherestofthesystemstate
tobeabletoidentifywhoisblocking)

WhatisaSystemstate?

Asystemstateismadeupoftheprocessstateofeachprocessintheinstancefoundatthetimethesystemstatewascalledfor.Each
processtateismadeupofSO(StateObjects)whichholddetailsofthestateofcurrentobjectsownedbyeachPROCESS.

HowtoNavigatethroughasystemstate

Whatyouneedtodoisstartbydeterminingwhatmostsessionarewaitingfor(orinthecaseofasessionyouknow
isblocked,thePROCESSnumberoftheprocess).So-youwillnowhaveeitheraPROCESSXXora,forexample,'latchfree'
whichyouneedtobeginwith.Whatyouthenneedtodoisnavigate(throughviorawindowseditor)andfindeither
PROCESSXXorthefirstexampleof'latchfree'.IfyouareusingPROCESSXXthenyounowneedtofindwhattheprocess
iswaitingfor.Youwillnowhave:-

PROCESSXXwaitsforYYYYYYY

Whatyouthenneedtodoisfind,byusingthisguide,theSOfortheresourcethesessioniswaitingforandthenfind(by
searchingbackfromthatpoint)thePROCESSXXofthatsession.Younowhave:-

PROCESSXXwaitsforYYYYYYY
PROCESSYYholdsYYYYYYY

Youthenbeginagain,findingtheresourceitiswaiting
for(ifany)andthatresourcesholder.EventuallyyouwillcometoaprocesswhichisontheCPU(lastwaited)oryouwill
havenavigatedbacktoaPROCESSyoualreadyknowabout.InthecaseoftheprocesswhichisontheCPUyouwillthenneed
togetanerrorstacktodeterminewhyitisblocking.Inthecaseofa'deadlock'youwillnowhaveadependencytreeoftheform:-

PROCESSXXwaitsforYYYYYYY
PROCESSYYholdsYYYYYYYandwaitsforZZZZZZZZ
PROCESSZZholdsZZZZZZZ...etcetc

CommonwaitscenariosandcorrespondingEntries

1-Enqueues

PROCESS41
...
waitingfor'enq:TX-rowlockcontention'blockingsess=0x39b3a5c90seq=152wait_time=0secondssincewait
started=796
name|mode=54580006,usn<54580006issplitintoASCII54+ASCII58(TX)+Mode0006(X)...

Tofindmoredetailsontheenqueue,simplydoasearchforthestring'req:'(searchingDOWN)

SO:39ad80d60,type:5,owner:393cb85e0,flag:INIT/-/-/0x00
(enqueue)TX-00020009-0001FA04DID:0001-0029-00000090
lv:00000000000000000000000000000000flag:0x6
res:39aef20c8,req:X,prv:39aef20e8,own:39b383aa8,sess:39b383aa8,proc:39b7384f0

Nowyouhavetheenqueuenameasastring(TX-00020009-0001FA04)whichyoucanusetosearchfortheHOLDER:-

(enqueue)TX-00020009-0001FA04DID:0001-002E-00000014
lv:00000000000000000000000000000000flag:0x6
res:39aef20c8,mode:X,prv:39aef20d8,own:39b3a5c90,sess:39b3a5c90,proc:39b73ac78

Wecanseeweholdtheenqueue(mode:X)inaincompatiblemodetothereq:Xrequest...

2-Rowcachelocks

PROCESS19:
...
waitingfor'rowcachelock'blockingsess=0x0seq=2174wait_time=0
cacheid=7,mode=0,request=3*
Wedonotholditcurrently(mode=0),butwantitinShared(mode=3)...
--------------------------------------------------------------------------------
SO:7000000c6de7678,type:48,owner:7000000a6c97cf8,flag:INIT/-/-/0x00
rowcacheenqueue:count=1session=7000000a660b8b0object=7000000eedc13a0,request=S*HereweseetherequestisShared(S)
savepoint=2148
rowcacheparentobject:address=7000000eedc13a0cid=7(dc_users)*dc_usersisthecachetypeindicatedby7
hash=2a057ebetyp=9transaction=7000000c42297a0flags=00000002
own=7000000eedc1480[7000000c6de8518,7000000c6de8518]wat=7000000eedc1490[7000000c6de7568,7000000c6deed98]mode=X*Theholderhasitinthismode
status=VALID/-/-/-/-/-/-/-/-
request=Nrelease=TRUEflags=0

TofindtheHOLDER,searchforobject,MODEofholder(ieobject=7000000eedc13a0,mode=X):-

SO:7000000c6de84e8,type:48,owner:7000000c42297a0,flag:INIT/-/-/0x00
rowcacheenqueue:count=1session=7000000a6702710object=7000000eedc13a0,mode=X*ThisconfirmstheModewethoughttheholderhad(X)
savepoint=109
rowcacheparentobject:address=7000000eedc13a0cid=7(dc_users)
hash=2a057ebetyp=9transaction=7000000c42297a0flags=00000002
own=7000000eedc1480[7000000c6de8518,7000000c6de8518]wat=7000000eedc1490[7000000c6de7568,7000000c6df1b08]mode=X
status=VALID/-/-/-/-/-/-/-/-
request=Nrelease=TRUEflags=0
instancelockid=QH0000044000000000
set=0,complete=FALSE
set=1,complete=FALSE
set=2,complete=FALSE
data=
3-LibraryCachePins(10G-Mutexes)

PROCESS16:

waitingfor'cursor:pinSwaitonX'blockingsess=0x0seq=58849wait_time=0secondssincewaitstarted=0
idn=535d1a6c,value=c1600000000,where|sleeps=5003f2428

Tofindmoredetailsusetheidn=XXXXXXtosearchdowninthesystemstate(ieidn=535d1a6c)

KGXAtomicOperationLog7000002e5b9d160
Mutex7000002b8e92268(3094,0)idn535d1a6coperGET_SHRD*Wecansee(a)ThatSID3094holdsit(3094,0)and(b)wewantitinShared(GET_SHRD)
CursorPinuid2489efd0whr5slp58733
opr=2pso=70000028c47def0flg=0
pcs=7000002b8e92268nxt=0flg=34cld=3hd=70000030d6c6eb0par=7000002eefe64d0
ct=31hsh=0unp=0unn=0hvl=b825a4d0nhv=1ses=700000309b42600
hep=7000002b8e922e8flg=80ld=1ob=7000002de49f8a0ptr=70000022cf39db8fex=70000022cf390c8

TofindtheHOLDER,searchforidnXXXXXXXoperuntilyoufindonewhichisheld(ienotGET_XXX)(ieidn535d1a6coper):-

KGXAtomicOperationLog7000002cd934270
Mutex7000002b8e92268(3094,0)idn535d1a6coperEXCL*WecanseeSID3094holdsinExclusive(EXCL)
CursorPinuid3094efd0whr7slp0
opr=3pso=7000002a71c4180flg=0
pcs=7000002b8e92268nxt=0flg=34cld=3hd=70000030d6c6eb0par=7000002eefe64d0
ct=31hsh=0unp=0unn=0hvl=b825a4d0nhv=1ses=700000309b42600
hep=7000002b8e922e8flg=80ld=1ob=7000002de49f8a0ptr=70000022cf39db8fex=70000022cf390c8

4-LibraryCachePins(Pre10G-nonmutex)

PROCESS20:

waitingfor'librarycachepin'blockingsess=0x0seq=575wait_time=0
handleaddress=c00000006c0f8490,pinaddress=c0000000689b19a8,10*mode+namespace=14

Tofindmoredetailsusethehandle=XXXXXXtosearchdowninthesystemstate(iehandle=c00000006c0f8490)andyouwillseea'request'line

SO:c0000000689b19a8,type:34,owner:c00000006cf85e80,flag:INIT/-/-/0x00
LIBRARYOBJECTPIN:pin=c0000000689b19a8handle=c00000006c0f8490request=Slock=c00000006d00e218*WecanseewewantitinShared(S)
user=c00000005eeafeb0session=c00000005eeafeb0count=0mask=0000savepoint=17flags=[00]

TofindtheHOLDER,searchfor'handle=XXXXXXXXXmode'operuntilyoufindonewhichisheldinanincompatiblemode(iehandle=c00000006c0f8490mode):-

SO:c00000006b1f4780,type:34,owner:c0000000699758e8,flag:INIT/-/-/0x00
LIBRARYOBJECTPIN:pin=c00000006b1f4780handle=c00000006c0f8490mode=Xlock=c00000006b6c40a0*WeholditinExclusive(X)
user=c00000005edf0f48session=c00000005edf0f48count=1mask=0001savepoint=49flags=[00]
5-LibraryCacheLock

PROCESS35:

waitingfor'librarycachelock'blockingsess=0x0seq=35844wait_time=0secondssincewaitstarted=14615
handleaddress=70000030de975a8,lockaddress=70000026947e190,100*mode+namespace=12d

Tofindmoredetailsusethehandleaddressintheformhandle=addresstosearchdowninthesystemstate(iehandle=70000030de975a8)

SO:70000026947e190,type:53,owner:700000308d726f0,flag:INIT/-/-/0x00
LIBRARYOBJECTLOCK:lock=70000026947e190handle=70000030de975a8request=X*WewantitinExclusive(X)
callpin=0sessionpin=0hpc=0000hlc=0000
htl=70000026947e210[7000002b333ffe8,7000002b333ffe8]htb=7000002b333ffe8ssga=7000002b333f2a0
user=700000307a7ca68session=700000307a7ca68count=0flags=[0000]savepoint=0x23e411
LIBRARYOBJECTHANDLE:handle=70000030de975a8mtx=70000030de976d8(0)cdp=0
name=ACSELP.POLIZA*Thisistheobjectwearetryingtolock

TofindtheHOLDER,searchfor'handle=XXXXXXXXXXmode='untilyoufindonewhichisheld(butnotinNULL)(iehandle=70000030de975a8mode=):-

SO:700000288b03ae0,type:53,owner:7000002cc697468,flag:INIT/-/-/0x00
LIBRARYOBJECTLOCK:lock=700000288b03ae0handle=70000030de975a8mode=S*WeholdininShared(S)
callpin=0sessionpin=0hpc=0000hlc=0000
htl=700000288b03b60[7000002a179a1a8,7000002b3800878]htb=7000002b3800878ssga=7000002b37ffb30
user=70000030fafab00session=70000030fafab00count=1flags=[0000]savepoint=0x417
LIBRARYOBJECTHANDLE:handle=70000030de975a8mtx=70000030de976d8(0)cdp=0
name=ACSELP.POLIZA*Thisconfirmstheobject

6-Latchfree

PROCESS8:

waitingfor'latchfree'blockingsess=0x0seq=4577wait_time=0
address=99ff60018,number=9d,tries=0*9disthelatch#fromv$latchnameinHEX

IfyoulooktowardsthetopofthePROCESSdumpyouwillseetheexactlatchwearewaitingforandevenwhoholdsit:

waitingfor99ff60018Childlibrarycachelevel=5child#=3
Locationfromwherelatchisheld:kglic:child
Contextsavedfromcall:26
state=busy
possibleholderpid=127ospid=23086*ThistellusPROCESS127(ospid:23086)holdsit
wtr=99ff60018,nextwaiter9993858b8

So-PROCESS127holdsit.IfwenowgotoPROCESS127wewillsee:-

holding99ff60018Childlibrarycachelevel=5child#=3
Locationfromwherelatchisheld:kglic:child
Contextsavedfromcall:26
state=busy

Otherusefulinformation

Ifyouwishtofindwhatobjectahandlereferstothenusethehandle=XXXXXXXXXXuntilyoucomeacrosstheLIBRARYOBJECTHANDLE.iehandle=c00000006c0f8490:-

LIBRARYOBJECTHANDLE:handle=c00000006c0f8490
name=SELECTUSERFROMDUAL*Thisisthenameofthehandle
hash=cd1ceca0timestamp=03-23-200709:00:00
namespace=CRSRflags=RON/TIM/PN0/SML/[12010000]*ItisaCURSOR(CRSR)..butwecantellthatbythename!
另外如果要理解、解读systemstatedump的内容，如何阅读systemstatedump这篇文章不得不细读。这个里面讲述了很多Detail方面的东西。非常受益！
参考资料：
http://www.oracleblog.org/working-case/database-hang-due-to-job-dead/
http://www.askmaclean.com/archives/%E8%BD%AC%E5%A6%82%E4%BD%95%E9%98%85%E8%AF%BBsystemstate-dump.html