系统内存不足导致的一次数据异常关闭

今天数据库突然挂掉了，查看alert日志发现：

Thu Jun 25 01:07:14 2015

Thread 1 advanced to log sequence 262 (LGWR switch)

Current log# 1 seq# 262 mem# 0: /oracle/oradata/sip/redo01.log

Archived Log entry 257 added for thread 1 sequence 261 ID 0x47d3d707 dest 1:

Thu Jun 25 01:08:11 2015

System state dump requested by (instance=1, osid=30314 (PMON)), summary=[abnormal instance termination].

System State dumped to trace file /oracle/diag/rdbms/sip/sip/trace/sip_diag_30332_20150625010811.trc

Thu Jun 25 01:08:11 2015

PMON (ospid: 30314): terminating the instance due to error 822

Thu Jun 25 01:08:15 2015

ORA-1092 : opitsk aborting process

Instance terminated by PMON, pid = 30314

可见之前还正常，突然被终结了。

查看系统日志：

Jun 23 18:04:51 SIP-DB yum[2801]: Installed: ftp-0.17-54.el6.x86_64

Jun 25 01:08:09 SIP-DB kernel: oracle invoked oom-killer: gfp_mask=0x201da, order=0, oom_adj=0, oom_score_adj=0

Jun 25 01:08:09 SIP-DB kernel: oracle cpuset=/ mems_allowed=0

Jun 25 01:08:09 SIP-DB kernel: Pid: 27179, comm: oracle Not tainted 2.6.32-431.23.3.el6.x86_64 #1

Jun 25 01:08:09 SIP-DB kernel: Call Trace:

Jun 25 01:08:09 SIP-DB kernel: [<ffffffff810d0431>] ? cpuset_print_task_mems_allowed+0x91/0xb0

Jun 25 01:08:09 SIP-DB kernel: [<ffffffff81122810>] ? dump_header+0x90/0x1b0

Jun 25 01:08:09 SIP-DB kernel: [<ffffffff8122833c>] ? security_real_capable_noaudit+0x3c/0x70

Jun 25 01:08:09 SIP-DB kernel: [<ffffffff81122c92>] ? oom_kill_process+0x82/0x2a0

Jun 25 01:08:09 SIP-DB kernel: [<ffffffff81122bd1>] ? select_bad_process+0xe1/0x120

Jun 25 01:08:09 SIP-DB kernel: [<ffffffff811230d0>] ? out_of_memory+0x220/0x3c0

Jun 25 01:08:09 SIP-DB kernel: [<ffffffff8112f9ef>] ? __alloc_pages_nodemask+0x89f/0x8d0

Jun 25 01:08:09 SIP-DB kernel: [<ffffffff811678ea>] ? alloc_pages_current+0xaa/0x110

Jun 25 01:08:09 SIP-DB kernel: [<ffffffff8111fc07>] ? __page_cache_alloc+0x87/0x90

Jun 25 01:08:09 SIP-DB kernel: [<ffffffff8111f5ee>] ? find_get_page+0x1e/0xa0

Jun 25 01:08:09 SIP-DB kernel: [<ffffffff81121545>] ? generic_file_aio_read+0x585/0x700

Jun 25 01:08:09 SIP-DB kernel: [<ffffffff81188cda>] ? do_sync_read+0xfa/0x140

Jun 25 01:08:09 SIP-DB kernel: [<ffffffff8109afa0>] ? autoremove_wake_function+0x0/0x40

Jun 25 01:08:09 SIP-DB kernel: [<ffffffff8105de8d>] ? thread_group_times+0x3d/0x120

Jun 25 01:08:09 SIP-DB kernel: [<ffffffff81226d76>] ? security_file_permission+0x16/0x20

Jun 25 01:08:09 SIP-DB kernel: [<ffffffff81189695>] ? vfs_read+0xb5/0x1a0

Jun 25 01:08:09 SIP-DB kernel: [<ffffffff81091d2b>] ? do_sys_times+0xab/0xd0

Jun 25 01:08:09 SIP-DB kernel: [<ffffffff811899c2>] ? sys_pread64+0x82/0xa0

Jun 25 01:08:09 SIP-DB kernel: [<ffffffff8100b072>] ? system_call_fastpath+0x16/0x1b

Jun 25 01:08:09 SIP-DB kernel: Mem-Info:

Jun 25 01:08:09 SIP-DB kernel: Node 0 DMA per-cpu:

Jun 25 01:08:09 SIP-DB kernel: CPU 0: hi: 0, btch: 1 usd: 0

Jun 25 01:08:09 SIP-DB kernel: CPU 1: hi: 0, btch: 1 usd: 0

Jun 25 01:08:09 SIP-DB kernel: CPU 2: hi: 0, btch: 1 usd: 0

Jun 25 01:08:09 SIP-DB kernel: CPU 3: hi: 0, btch: 1 usd: 0

Jun 25 01:08:09 SIP-DB kernel: CPU 4: hi: 0, btch: 1 usd: 0

Jun 25 01:08:09 SIP-DB kernel: CPU 5: hi: 0, btch: 1 usd: 0

Jun 25 01:08:09 SIP-DB kernel: CPU 6: hi: 0, btch: 1 usd: 0

Jun 25 01:08:09 SIP-DB kernel: CPU 7: hi: 0, btch: 1 usd: 0

Jun 25 01:08:09 SIP-DB kernel: Node 0 DMA32 per-cpu:

Jun 25 01:08:09 SIP-DB kernel: CPU 0: hi: 186, btch: 31 usd: 0

Jun 25 01:08:09 SIP-DB kernel: CPU 1: hi: 186, btch: 31 usd: 0

Jun 25 01:08:09 SIP-DB kernel: CPU 2: hi: 186, btch: 31 usd: 0

Jun 25 01:08:09 SIP-DB kernel: CPU 3: hi: 186, btch: 31 usd: 0

Jun 25 01:08:09 SIP-DB kernel: CPU 4: hi: 186, btch: 31 usd: 0

Jun 25 01:08:09 SIP-DB kernel: CPU 5: hi: 186, btch: 31 usd: 60

Jun 25 01:08:09 SIP-DB kernel: CPU 6: hi: 186, btch: 31 usd: 0

Jun 25 01:08:09 SIP-DB kernel: CPU 7: hi: 186, btch: 31 usd: 0

Jun 25 01:08:09 SIP-DB kernel: Node 0 Normal per-cpu:

Jun 25 01:08:09 SIP-DB kernel: CPU 0: hi: 186, btch: 31 usd: 0

Jun 25 01:08:09 SIP-DB kernel: CPU 1: hi: 186, btch: 31 usd: 0

Jun 25 01:08:09 SIP-DB kernel: CPU 2: hi: 186, btch: 31 usd: 0

Jun 25 01:08:09 SIP-DB kernel: CPU 3: hi: 186, btch: 31 usd: 0

Jun 25 01:08:09 SIP-DB kernel: CPU 4: hi: 186, btch: 31 usd: 0

Jun 25 01:08:09 SIP-DB kernel: CPU 5: hi: 186, btch: 31 usd: 0

Jun 25 01:08:09 SIP-DB kernel: CPU 6: hi: 186, btch: 31 usd: 0

Jun 25 01:08:09 SIP-DB kernel: CPU 7: hi: 186, btch: 31 usd: 0

Jun 25 01:08:09 SIP-DB kernel: active_anon:786810 inactive_anon:598780 isolated_anon:0

Jun 25 01:08:09 SIP-DB kernel: active_file:145938 inactive_file:191123 isolated_file:141

Jun 25 01:08:09 SIP-DB kernel: unevictable:0 dirty:190674 writeback:483 unstable:0

Jun 25 01:08:09 SIP-DB kernel: free:24940 slab_reclaimable:25814 slab_unreclaimable:12349

Jun 25 01:08:09 SIP-DB kernel: mapped:684185 shmem:992477 pagetables:203816 bounce:0

Jun 25 01:08:09 SIP-DB kernel: Node 0 DMA free:15728kB min:124kB low:152kB high:184kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15340kB mlocked:0kB dirty:0kB writeback:0kB
mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes

Jun 25 01:08:09 SIP-DB kernel: lowmem_reserve[]: 0 3768 8060 8060

Jun 25 01:08:09 SIP-DB kernel: Node 0 DMA32 free:48152kB min:31532kB low:39412kB high:47296kB active_anon:1500552kB inactive_anon:1113036kB active_file:224128kB inactive_file:369116kB unevictable:0kB isolated(anon):0kB isolated(file):256kB present:3858656kB
mlocked:0kB dirty:367384kB writeback:988kB mapped:1592520kB shmem:2020796kB slab_reclaimable:43944kB slab_unreclaimable:13100kB kernel_stack:1848kB pagetables:263196kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:138688 all_unreclaimable? no

Jun 25 01:08:09 SIP-DB kernel: lowmem_reserve[]: 0 0 4292 4292

Jun 25 01:08:09 SIP-DB kernel: Node 0 Normal free:35880kB min:35920kB low:44900kB high:53880kB active_anon:1646688kB inactive_anon:1282084kB active_file:359624kB inactive_file:395376kB unevictable:0kB isolated(anon):0kB isolated(file):308kB present:4395520kB
mlocked:0kB dirty:395312kB writeback:944kB mapped:1144220kB shmem:1949112kB slab_reclaimable:59312kB slab_unreclaimable:36296kB kernel_stack:2720kB pagetables:552068kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:460416 all_unreclaimable? no

Jun 25 01:08:09 SIP-DB kernel: lowmem_reserve[]: 0 0 0 0

Jun 25 01:08:09 SIP-DB kernel: Node 0 DMA: 2*4kB 1*8kB 2*16kB 2*32kB 2*64kB 1*128kB 0*256kB 0*512kB 1*1024kB 1*2048kB 3*4096kB = 15728kB

Jun 25 01:08:09 SIP-DB kernel: Node 0 DMA32: 5148*4kB 1834*8kB 189*16kB 34*32kB 20*64kB 27*128kB 4*256kB 4*512kB 2*1024kB 0*2048kB 0*4096kB = 49232kB

Jun 25 01:08:09 SIP-DB kernel: Node 0 Normal: 2163*4kB 970*8kB 396*16kB 187*32kB 67*64kB 14*128kB 0*256kB 1*512kB 1*1024kB 0*2048kB 0*4096kB = 36348kB

Jun 25 01:08:09 SIP-DB kernel: 1329619 total pagecache pages

Jun 25 01:08:09 SIP-DB kernel: 0 pages in swap cache

Jun 25 01:08:09 SIP-DB kernel: Swap cache stats: add 0, delete 0, find 0/0

Jun 25 01:08:09 SIP-DB kernel: Free swap = 16777208kB

Jun 25 01:08:09 SIP-DB kernel: Total swap = 16777208kB

Jun 25 01:08:09 SIP-DB kernel: 2097151 pages RAM

Jun 25 01:08:09 SIP-DB kernel: 82637 pages reserved

Jun 25 01:08:09 SIP-DB kernel: 9350672 pages shared

Jun 25 01:08:09 SIP-DB kernel: 991485 pages non-shared

Jun 25 01:08:09 SIP-DB kernel: [ pid ] uid tgid total_vm rss cpu oom_adj oom_score_adj name

Jun 25 01:08:09 SIP-DB kernel: [ 435] 0 435 2670 201 0 -17 -1000 udevd

Jun 25 01:08:09 SIP-DB kernel: [ 1083] 0 1083 62797 309 0 0 0 rsyslogd

Jun 25 01:08:09 SIP-DB kernel: [ 1141] 38 1141 6648 289 0 0 0 ntpd

Jun 25 01:08:09 SIP-DB kernel: [ 1151] 0 1151 29332 282 0 0 0 crond

Jun 25 01:08:09 SIP-DB kernel: [ 1180] 0 1180 7905 17 0 0 0 gshelld

Jun 25 01:08:09 SIP-DB kernel: [ 1197] 0 1197 1016 91 6 0 0 mingetty

Jun 25 01:08:09 SIP-DB kernel: [ 1199] 0 1199 1016 90 2 0 0 mingetty

Jun 25 01:08:09 SIP-DB kernel: [ 1201] 0 1201 1016 90 4 0 0 mingetty

Jun 25 01:08:09 SIP-DB kernel: [ 1203] 0 1203 1016 91 4 0 0 mingetty

Jun 25 01:08:09 SIP-DB kernel: [ 1207] 0 1207 1016 90 6 0 0 mingetty

Jun 25 01:08:09 SIP-DB kernel: [ 1209] 0 1209 1016 91 4 0 0 mingetty

Jun 25 01:08:09 SIP-DB kernel: [ 1230] 0 1230 118751 961 5 0 0 AliYunDunUpdate

Jun 25 01:08:09 SIP-DB kernel: [ 1284] 0 1284 194826 1665 5 0 0 AliYunDun

Jun 25 01:08:09 SIP-DB kernel: [ 1299] 0 1299 209991 1979 4 0 0 AliHids

Jun 25 01:08:09 SIP-DB kernel: [24128] 0 24128 16541 274 0 -17 -1000 sshd

Jun 25 01:08:09 SIP-DB kernel: [24337] 28 24337 157649 289 1 0 0 nscd

Jun 25 01:08:09 SIP-DB kernel: [29259] 500 29259 54234 1895 0 0 0 tnslsnr

Jun 25 01:08:09 SIP-DB kernel: [30314] 500 30314 1085477 8955 0 0 0 oracle

Jun 25 01:08:09 SIP-DB kernel: [30318] 500 30318 1084863 4294 2 0 0 oracle

Jun 25 01:08:09 SIP-DB kernel: [30322] 500 30322 1084863 4104 1 0 0 oracle

Jun 25 01:08:09 SIP-DB kernel: [30328] 500 30328 1084863 4171 1 0 0 oracle

Jun 25 01:08:09 SIP-DB kernel: [30332] 500 30332 1084863 4112 0 0 0 oracle

Jun 25 01:08:09 SIP-DB kernel: [30336] 500 30336 1085318 11623 1 0 0 oracle

Jun 25 01:08:09 SIP-DB kernel: [30340] 500 30340 1087551 13604 2 0 0 oracle

Jun 25 01:08:09 SIP-DB kernel: [30344] 500 30344 1084863 336329 1 0 0 oracle

Jun 25 01:08:09 SIP-DB kernel: [30348] 500 30348 1092190 185397 0 0 0 oracle

Jun 25 01:08:09 SIP-DB kernel: [30352] 500 30352 1089018 8587 0 0 0 oracle

Jun 25 01:08:09 SIP-DB kernel: [30356] 500 30356 1084991 24525 0 0 0 oracle

Jun 25 01:08:09 SIP-DB kernel: [30360] 500 30360 1086546 47594 0 0 0 oracle

Jun 25 01:08:09 SIP-DB kernel: [30364] 500 30364 1085262 8653 5 0 0 oracle

Jun 25 01:08:09 SIP-DB kernel: [30368] 500 30368 1086414 50695 0 0 0 oracle

Jun 25 01:08:09 SIP-DB kernel: [30372] 500 30372 1085286 19614 1 0 0 oracle

Jun 25 01:08:09 SIP-DB kernel: [30376] 500 30376 1085391 4094 2 0 0 oracle

Jun 25 01:08:09 SIP-DB kernel: [30380] 500 30380 1085129 3843 6 0 0 oracle

Jun 25 01:08:09 SIP-DB kernel: [30438] 500 30438 1096943 11318 3 0 0 oracle

Jun 25 01:08:09 SIP-DB kernel: [30442] 500 30442 1092847 10977 5 0 0 oracle

Jun 25 01:08:09 SIP-DB kernel: [30446] 500 30446 1090799 4410 1 0 0 oracle

Jun 25 01:08:09 SIP-DB kernel: [30450] 500 30450 1096943 11759 1 0 0 oracle

Jun 25 01:08:09 SIP-DB kernel: [30458] 500 30458 1084863 4523 5 0 0 oracle

Jun 25 01:08:09 SIP-DB kernel: [30486] 500 30486 1086046 37927 0 0 0 oracle

Jun 25 01:08:09 SIP-DB kernel: [30498] 500 30498 1086259 13676 3 0 0 oracle

Jun 25 01:08:09 SIP-DB kernel: [30766] 500 30766 1086559 6460 5 0 0 oracle

Jun 25 01:08:09 SIP-DB kernel: [10304] 0 10304 25204 669 0 0 0 sshd

Jun 25 01:08:09 SIP-DB kernel: [10310] 0 10310 14585 646 0 0 0 sftp-server

Jun 25 01:08:09 SIP-DB kernel: [12277] 500 12277 1085703 12118 5 0 0 oracle

Jun 25 01:08:09 SIP-DB kernel: [16173] 500 16173 1085550 7933 0 0 0 oracle

Jun 25 01:08:09 SIP-DB kernel: [16177] 500 16177 1085551 8104 0 0 0 oracle

Jun 25 01:08:09 SIP-DB kernel: [16181] 500 16181 1085550 8792 0 0 0 oracle

Jun 25 01:08:09 SIP-DB kernel: [16185] 500 16185 1085561 8483 0 0 0 oracle

Jun 25 01:08:09 SIP-DB kernel: [16189] 500 16189 1085549 8393 0 0 0 oracle

Jun 25 01:08:09 SIP-DB kernel: [16197] 500 16197 1085162 5966 0 0 0 oracle

Jun 25 01:08:09 SIP-DB kernel: [16201] 500 16201 1085293 7335 0 0 0 oracle

Jun 25 01:08:09 SIP-DB kernel: [16205] 500 16205 1085551 7821 0 0 0 oracle

Jun 25 01:08:09 SIP-DB kernel: [16209] 500 16209 1085162 5818 0 0 0 oracle

Jun 25 01:08:09 SIP-DB kernel: [16213] 500 16213 1085162 5804 0 0 0 oracle

Jun 25 01:08:09 SIP-DB kernel: [16247] 500 16247 1085294 7358 0 0 0 oracle

Jun 25 01:08:09 SIP-DB kernel: [16251] 500 16251 1085551 8019 0 0 0 oracle

Jun 25 01:08:09 SIP-DB kernel: [16255] 500 16255 1085294 7220 0 0 0 oracle

Jun 25 01:08:09 SIP-DB kernel: [16259] 500 16259 1085294 7421 0 0 0 oracle

Jun 25 01:08:09 SIP-DB kernel: [16263] 500 16263 1085551 8047 0 0 0 oracle

Jun 25 01:08:09 SIP-DB kernel: [17229] 500 17229 1085555 13604 0 0 0 oracle

Jun 25 01:08:09 SIP-DB kernel: [17233] 500 17233 1085826 18918 0 0 0 oracle

Jun 25 01:08:09 SIP-DB kernel: [17237] 500 17237 1085558 13959 0 0 0 oracle

Jun 25 01:08:09 SIP-DB kernel: [17241] 500 17241 1085430 13661 0 0 0 oracle

Jun 25 01:08:09 SIP-DB kernel: [17245] 500 17245 1085558 39491 0 0 0 oracle

Jun 25 01:08:09 SIP-DB kernel: [17265] 500 17265 1085555 15481 0 0 0 oracle

Jun 25 01:08:09 SIP-DB kernel: [17277] 500 17277 1085817 18358 0 0 0 oracle

Jun 25 01:08:09 SIP-DB kernel: [17305] 500 17305 1085557 12906 0 0 0 oracle

Jun 25 01:08:09 SIP-DB kernel: [17325] 500 17325 1085556 32133 0 0 0 oracle

Jun 25 01:08:09 SIP-DB kernel: [17343] 500 17343 1085557 42001 0 0 0 oracle

Jun 25 01:08:09 SIP-DB kernel: [17347] 500 17347 1085557 12639 0 0 0 oracle

Jun 25 01:08:09 SIP-DB kernel: [17355] 500 17355 1085558 12568 0 0 0 oracle

Jun 25 01:08:09 SIP-DB kernel: [17359] 500 17359 1085557 16684 0 0 0 oracle

Jun 25 01:08:09 SIP-DB kernel: [17365] 500 17365 1085556 14139 0 0 0 oracle

Jun 25 01:08:09 SIP-DB kernel: [17369] 500 17369 1085822 37259 0 0 0 oracle

Jun 25 01:08:09 SIP-DB kernel: [17377] 500 17377 1085557 12854 0 0 0 oracle

Jun 25 01:08:09 SIP-DB kernel: [17389] 500 17389 1085821 15934 0 0 0 oracle

Jun 25 01:08:09 SIP-DB kernel: [17393] 500 17393 1085558 35179 0 0 0 oracle

Jun 25 01:08:09 SIP-DB kernel: [17401] 500 17401 1086071 113320 0 0 0 oracle

Jun 25 01:08:09 SIP-DB kernel: [17405] 500 17405 1086455 80434 0 0 0 oracle

Jun 25 01:08:09 SIP-DB kernel: [17409] 500 17409 1085561 150402 0 0 0 oracle

Jun 25 01:08:09 SIP-DB kernel: [17413] 500 17413 1086245 128282 0 0 0 oracle

Jun 25 01:08:09 SIP-DB kernel: [17417] 500 17417 1086587 115121 0 0 0 oracle

Jun 25 01:08:09 SIP-DB kernel: [17421] 500 17421 1085558 12387 0 0 0 oracle

Jun 25 01:08:09 SIP-DB kernel: [17433] 500 17433 1085556 12449 0 0 0 oracle

Jun 25 01:08:09 SIP-DB kernel: [17441] 500 17441 1085558 42395 0 0 0 oracle

Jun 25 01:08:09 SIP-DB kernel: [17453] 500 17453 1085554 13040 0 0 0 oracle

Jun 25 01:08:09 SIP-DB kernel: [17457] 500 17457 1085558 41053 0 0 0 oracle

Jun 25 01:08:09 SIP-DB kernel: [17467] 500 17467 1085556 13095 0 0 0 oracle

Jun 25 01:08:09 SIP-DB kernel: [17475] 500 17475 1085558 13590 0 0 0 oracle

Jun 25 01:08:09 SIP-DB kernel: [17479] 500 17479 1085557 12963 0 0 0 oracle

Jun 25 01:08:09 SIP-DB kernel: [17491] 500 17491 1085555 11402 0 0 0 oracle

Jun 25 01:08:09 SIP-DB kernel: [17495] 500 17495 1085557 58069 0 0 0 oracle

Jun 25 01:08:09 SIP-DB kernel: [17499] 500 17499 1085558 62538 0 0 0 oracle

Jun 25 01:08:09 SIP-DB kernel: [17503] 500 17503 1085558 44952 0 0 0 oracle

Jun 25 01:08:09 SIP-DB kernel: [17507] 500 17507 1085560 61911 0 0 0 oracle

Jun 25 01:08:09 SIP-DB kernel: [17523] 500 17523 1085558 32448 0 0 0 oracle

Jun 25 01:08:09 SIP-DB kernel: [17531] 500 17531 1085556 30449 0 0 0 oracle

Jun 25 01:08:09 SIP-DB kernel: [17545] 500 17545 1085558 13626 0 0 0 oracle

Jun 25 01:08:09 SIP-DB kernel: [17557] 500 17557 1085813 67773 0 0 0 oracle

Jun 25 01:08:09 SIP-DB kernel: [17561] 500 17561 1085566 40153 1 0 0 oracle

Jun 25 01:08:09 SIP-DB kernel: [17565] 500 17565 1085558 12087 0 0 0 oracle

Jun 25 01:08:09 SIP-DB kernel: [17569] 500 17569 1085557 22287 0 0 0 oracle

Jun 25 01:08:09 SIP-DB kernel: [17575] 500 17575 1085557 12840 0 0 0 oracle

Jun 25 01:08:09 SIP-DB kernel: [17583] 500 17583 1085557 13566 0 0 0 oracle

Jun 25 01:08:09 SIP-DB kernel: [17597] 500 17597 1085556 13167 0 0 0 oracle

Jun 25 01:08:09 SIP-DB kernel: [17605] 500 17605 1085556 13977 0 0 0 oracle

Jun 25 01:08:09 SIP-DB kernel: [17609] 500 17609 1085557 14357 0 0 0 oracle

Jun 25 01:08:09 SIP-DB kernel: [17613] 500 17613 1085556 13650 0 0 0 oracle

Jun 25 01:08:09 SIP-DB kernel: [17617] 500 17617 1085555 13180 0 0 0 oracle

Jun 25 01:08:09 SIP-DB kernel: [17625] 500 17625 1085557 54014 0 0 0 oracle

Jun 25 01:08:09 SIP-DB kernel: [17629] 500 17629 1085558 67111 0 0 0 oracle

Jun 25 01:08:09 SIP-DB kernel: [17633] 500 17633 1085559 112836 4 0 0 oracle

Jun 25 01:08:09 SIP-DB kernel: [17637] 500 17637 1085558 63563 0 0 0 oracle

Jun 25 01:08:09 SIP-DB kernel: [17641] 500 17641 1085819 118554 0 0 0 oracle

Jun 25 01:08:09 SIP-DB kernel: [17645] 500 17645 1086082 66722 0 0 0 oracle

Jun 25 01:08:09 SIP-DB kernel: [17649] 500 17649 1086199 54099 0 0 0 oracle

Jun 25 01:08:09 SIP-DB kernel: [17653] 500 17653 1085557 65583 0 0 0 oracle

Jun 25 01:08:09 SIP-DB kernel: [17657] 500 17657 1086455 93988 0 0 0 oracle

Jun 25 01:08:09 SIP-DB kernel: [17661] 500 17661 1085558 64335 0 0 0 oracle

Jun 25 01:08:09 SIP-DB kernel: [17665] 500 17665 1086078 81023 0 0 0 oracle

Jun 25 01:08:09 SIP-DB kernel: [17669] 500 17669 1085557 12829 0 0 0 oracle

Jun 25 01:08:09 SIP-DB kernel: [17673] 500 17673 1085557 63520 0 0 0 oracle

Jun 25 01:08:09 SIP-DB kernel: [17675] 500 17675 1085563 163201 0 0 0 oracle

Jun 25 01:08:09 SIP-DB kernel: [17693] 500 17693 1085556 12342 0 0 0 oracle

Jun 25 01:08:09 SIP-DB kernel: [17697] 500 17697 1085558 12438 0 0 0 oracle

Jun 25 01:08:09 SIP-DB kernel: [17791] 500 17791 1085558 31065 0 0 0 oracle

Jun 25 01:08:09 SIP-DB kernel: [17795] 500 17795 1085557 31178 0 0 0 oracle

Jun 25 01:08:09 SIP-DB kernel: [17817] 500 17817 1085554 12821 0 0 0 oracle

Jun 25 01:08:09 SIP-DB kernel: [17821] 500 17821 1085556 50532 0 0 0 oracle

Jun 25 01:08:09 SIP-DB kernel: [17839] 500 17839 1085558 37189 0 0 0 oracle

Jun 25 01:08:09 SIP-DB kernel: [17847] 500 17847 1085555 12526 0 0 0 oracle

Jun 25 01:08:09 SIP-DB kernel: [17867] 500 17867 1085429 41098 0 0 0 oracle

Jun 25 01:08:09 SIP-DB kernel: [17873] 500 17873 1085557 30435 0 0 0 oracle

Jun 25 01:08:09 SIP-DB kernel: [17877] 500 17877 1085429 30619 0 0 0 oracle

Jun 25 01:08:09 SIP-DB kernel: [17881] 500 17881 1085556 31063 0 0 0 oracle

Jun 25 01:08:09 SIP-DB kernel: [17895] 500 17895 1085557 13867 0 0 0 oracle

Jun 25 01:08:09 SIP-DB kernel: [17911] 500 17911 1085556 30758 0 0 0 oracle

Jun 25 01:08:09 SIP-DB kernel: [17915] 500 17915 1085555 31021 0 0 0 oracle

Jun 25 01:08:09 SIP-DB kernel: [17927] 500 17927 1085428 13912 0 0 0 oracle

Jun 25 01:08:09 SIP-DB kernel: [17931] 500 17931 1085429 13399 0 0 0 oracle

Jun 25 01:08:09 SIP-DB kernel: [17945] 500 17945 1085558 13490 0 0 0 oracle

Jun 25 01:08:09 SIP-DB kernel: [17953] 500 17953 1085558 13058 0 0 0 oracle

Jun 25 01:08:09 SIP-DB kernel: [18125] 500 18125 1085430 30449 0 0 0 oracle

Jun 25 01:08:09 SIP-DB kernel: [18129] 500 18129 1085428 12266 0 0 0 oracle

Jun 25 01:08:09 SIP-DB kernel: [18137] 500 18137 1085558 12604 0 0 0 oracle

Jun 25 01:08:09 SIP-DB kernel: [18143] 500 18143 1085557 13188 0 0 0 oracle

Jun 25 01:08:09 SIP-DB kernel: [18155] 500 18155 1085555 12551 0 0 0 oracle

Jun 25 01:08:09 SIP-DB kernel: [18159] 500 18159 1085558 12885 0 0 0 oracle

Jun 25 01:08:09 SIP-DB kernel: [18163] 500 18163 1085558 13632 0 0 0 oracle

Jun 25 01:08:09 SIP-DB kernel: [18167] 500 18167 1085557 12442 0 0 0 oracle

Jun 25 01:08:09 SIP-DB kernel: [18177] 500 18177 1085554 40984 0 0 0 oracle

Jun 25 01:08:09 SIP-DB kernel: [18181] 500 18181 1085556 12490 0 0 0 oracle

Jun 25 01:08:09 SIP-DB kernel: [18185] 500 18185 1085557 53961 0 0 0 oracle

Jun 25 01:08:09 SIP-DB kernel: [18209] 500 18209 1085558 31167 0 0 0 oracle

Jun 25 01:08:09 SIP-DB kernel: [18213] 500 18213 1085430 12828 0 0 0 oracle

Jun 25 01:08:09 SIP-DB kernel: [18219] 500 18219 1085428 30720 0 0 0 oracle

Jun 25 01:08:09 SIP-DB kernel: [18223] 500 18223 1085557 12875 0 0 0 oracle

Jun 25 01:08:09 SIP-DB kernel: [18239] 500 18239 1085557 13875 0 0 0 oracle

Jun 25 01:08:09 SIP-DB kernel: [18243] 500 18243 1085556 13570 0 0 0 oracle

Jun 25 01:08:09 SIP-DB kernel: [18247] 500 18247 1085430 13620 0 0 0 oracle

Jun 25 01:08:09 SIP-DB kernel: [18259] 500 18259 1085555 12676 0 0 0 oracle

Jun 25 01:08:09 SIP-DB kernel: [18285] 500 18285 1085556 12979 0 0 0 oracle

Jun 25 01:08:09 SIP-DB kernel: [18289] 500 18289 1085558 13020 0 0 0 oracle

Jun 25 01:08:09 SIP-DB kernel: [18307] 500 18307 1085557 12280 0 0 0 oracle

Jun 25 01:08:09 SIP-DB kernel: [18311] 500 18311 1085558 13073 0 0 0 oracle

Jun 25 01:08:09 SIP-DB kernel: [18333] 500 18333 1085558 12827 0 0 0 oracle

Jun 25 01:08:09 SIP-DB kernel: [18337] 500 18337 1085429 12382 0 0 0 oracle

Jun 25 01:08:09 SIP-DB kernel: [18341] 500 18341 1085430 13503 0 0 0 oracle

Jun 25 01:08:09 SIP-DB kernel: [18351] 500 18351 1085557 13204 0 0 0 oracle

Jun 25 01:08:09 SIP-DB kernel: [18359] 500 18359 1085558 12559 0 0 0 oracle

Jun 25 01:08:09 SIP-DB kernel: [18371] 500 18371 1085556 13008 0 0 0 oracle

Jun 25 01:08:09 SIP-DB kernel: [18383] 500 18383 1085556 13192 0 0 0 oracle

Jun 25 01:08:09 SIP-DB kernel: [18387] 500 18387 1085557 51934 0 0 0 oracle

Jun 25 01:08:09 SIP-DB kernel: [18405] 500 18405 1085556 13051 0 0 0 oracle

Jun 25 01:08:09 SIP-DB kernel: [18409] 500 18409 1085555 13010 0 0 0 oracle

Jun 25 01:08:09 SIP-DB kernel: [18413] 500 18413 1085556 11905 0 0 0 oracle

Jun 25 01:08:09 SIP-DB kernel: [18423] 500 18423 1085557 13839 0 0 0 oracle

Jun 25 01:08:09 SIP-DB kernel: [18435] 500 18435 1085558 13624 0 0 0 oracle

Jun 25 01:08:09 SIP-DB kernel: [18443] 500 18443 1085559 31639 4 0 0 oracle

Jun 25 01:08:09 SIP-DB kernel: [18455] 500 18455 1085556 12889 0 0 0 oracle

Jun 25 01:08:09 SIP-DB kernel: [18461] 500 18461 1085556 12916 0 0 0 oracle

Jun 25 01:08:09 SIP-DB kernel: [18469] 500 18469 1085427 12811 0 0 0 oracle

Jun 25 01:08:09 SIP-DB kernel: [18473] 500 18473 1085556 45938 0 0 0 oracle

Jun 25 01:08:09 SIP-DB kernel: [18481] 500 18481 1085554 12550 0 0 0 oracle

Jun 25 01:08:09 SIP-DB kernel: [18485] 500 18485 1085557 12517 0 0 0 oracle

Jun 25 01:08:09 SIP-DB kernel: [18493] 500 18493 1085427 12406 0 0 0 oracle

Jun 25 01:08:09 SIP-DB kernel: [18503] 500 18503 1085555 12110 0 0 0 oracle

Jun 25 01:08:09 SIP-DB kernel: [18507] 500 18507 1085428 41926 0 0 0 oracle

Jun 25 01:08:09 SIP-DB kernel: [18511] 500 18511 1085429 46833 0 0 0 oracle

Jun 25 01:08:09 SIP-DB kernel: [18523] 500 18523 1085557 12284 0 0 0 oracle

Jun 25 01:08:09 SIP-DB kernel: [18539] 500 18539 1085556 12358 0 0 0 oracle

Jun 25 01:08:09 SIP-DB kernel: [18545] 500 18545 1085556 12891 0 0 0 oracle

Jun 25 01:08:09 SIP-DB kernel: [18549] 500 18549 1085558 30307 0 0 0 oracle

Jun 25 01:08:09 SIP-DB kernel: [18579] 500 18579 1085558 30664 0 0 0 oracle

Jun 25 01:08:09 SIP-DB kernel: [18605] 500 18605 1085555 30209 0 0 0 oracle

Jun 25 01:08:09 SIP-DB kernel: [18609] 500 18609 1085556 12558 0 0 0 oracle

Jun 25 01:08:09 SIP-DB kernel: [18615] 500 18615 1085556 12035 0 0 0 oracle

Jun 25 01:08:09 SIP-DB kernel: [18645] 500 18645 1085558 12461 0 0 0 oracle

Jun 25 01:08:09 SIP-DB kernel: [18650] 500 18650 1085557 41220 0 0 0 oracle

Jun 25 01:08:09 SIP-DB kernel: [18654] 500 18654 1085558 14081 0 0 0 oracle

Jun 25 01:08:09 SIP-DB kernel: [18658] 500 18658 1085557 12905 0 0 0 oracle

Jun 25 01:08:09 SIP-DB kernel: [18662] 500 18662 1085431 12694 0 0 0 oracle

Jun 25 01:08:09 SIP-DB kernel: [18666] 500 18666 1085557 32906 0 0 0 oracle

Jun 25 01:08:09 SIP-DB kernel: [18780] 500 18780 1085429 12761 0 0 0 oracle

Jun 25 01:08:09 SIP-DB kernel: [18784] 500 18784 1085557 41518 0 0 0 oracle

Jun 25 01:08:09 SIP-DB kernel: [18788] 500 18788 1085556 31325 0 0 0 oracle

Jun 25 01:08:09 SIP-DB kernel: [18794] 500 18794 1085555 12123 0 0 0 oracle

Jun 25 01:08:09 SIP-DB kernel: [18798] 500 18798 1085556 12617 0 0 0 oracle

Jun 25 01:08:09 SIP-DB kernel: [18802] 500 18802 1085555 11689 0 0 0 oracle

Jun 25 01:08:09 SIP-DB kernel: [18816] 500 18816 1085427 11444 0 0 0 oracle

Jun 25 01:08:09 SIP-DB kernel: [18820] 500 18820 1085556 12778 0 0 0 oracle

Jun 25 01:08:09 SIP-DB kernel: [18826] 500 18826 1085558 12594 0 0 0 oracle

Jun 25 01:08:09 SIP-DB kernel: [19174] 500 19174 1085429 41253 0 0 0 oracle

Jun 25 01:08:09 SIP-DB kernel: [19178] 500 19178 1085558 13927 0 0 0 oracle

Jun 25 01:08:09 SIP-DB kernel: [19190] 500 19190 1085554 29965 0 0 0 oracle

Jun 25 01:08:09 SIP-DB kernel: [19194] 500 19194 1085556 31096 0 0 0 oracle

Jun 25 01:08:09 SIP-DB kernel: [19198] 500 19198 1085557 30335 0 0 0 oracle

Jun 25 01:08:09 SIP-DB kernel: [19202] 500 19202 1085557 31748 0 0 0 oracle

Jun 25 01:08:09 SIP-DB kernel: [19206] 500 19206 1085558 32826 0 0 0 oracle

Jun 25 01:08:09 SIP-DB kernel: [19210] 500 19210 1085557 31519 0 0 0 oracle

Jun 25 01:08:09 SIP-DB kernel: [19214] 500 19214 1085429 30709 0 0 0 oracle

Jun 25 01:08:09 SIP-DB kernel: [19218] 500 19218 1085557 49490 0 0 0 oracle

Jun 25 01:08:09 SIP-DB kernel: [19224] 500 19224 1085430 12430 0 0 0 oracle

Jun 25 01:08:09 SIP-DB kernel: [19228] 500 19228 1085558 30815 0 0 0 oracle

Jun 25 01:08:09 SIP-DB kernel: [19232] 500 19232 1085558 12180 0 0 0 oracle

Jun 25 01:08:09 SIP-DB kernel: [19236] 500 19236 1085558 13343 0 0 0 oracle

Jun 25 01:08:09 SIP-DB kernel: [19240] 500 19240 1085556 13531 0 0 0 oracle

Jun 25 01:08:09 SIP-DB kernel: [19246] 500 19246 1085556 12964 5 0 0 oracle

Jun 25 01:08:09 SIP-DB kernel: [19250] 500 19250 1085557 12055 0 0 0 oracle

Jun 25 01:08:09 SIP-DB kernel: [19254] 500 19254 1085428 12637 0 0 0 oracle

Jun 25 01:08:09 SIP-DB kernel: [19268] 500 19268 1085555 12174 0 0 0 oracle

Jun 25 01:08:09 SIP-DB kernel: [19272] 500 19272 1085555 11174 0 0 0 oracle

Jun 25 01:08:09 SIP-DB kernel: [19276] 500 19276 1085555 29864 0 0 0 oracle

Jun 25 01:08:09 SIP-DB kernel: [19284] 500 19284 1085555 13576 0 0 0 oracle

Jun 25 01:08:09 SIP-DB kernel: [19288] 500 19288 1085555 12498 0 0 0 oracle

Jun 25 01:08:09 SIP-DB kernel: [19292] 500 19292 1085430 12981 0 0 0 oracle

Jun 25 01:08:09 SIP-DB kernel: [19296] 500 19296 1085554 11902 0 0 0 oracle

Jun 25 01:08:09 SIP-DB kernel: [19300] 500 19300 1085557 12560 0 0 0 oracle

Jun 25 01:08:09 SIP-DB kernel: [19304] 500 19304 1085557 31831 0 0 0 oracle

Jun 25 01:08:09 SIP-DB kernel: [19308] 500 19308 1085556 12863 0 0 0 oracle

Jun 25 01:08:09 SIP-DB kernel: [20142] 500 20142 1085556 41578 0 0 0 oracle

Jun 25 01:08:09 SIP-DB kernel: [20146] 500 20146 1085555 12236 0 0 0 oracle

Jun 25 01:08:09 SIP-DB kernel: [20150] 500 20150 1085557 13443 0 0 0 oracle

Jun 25 01:08:09 SIP-DB kernel: [20154] 500 20154 1085558 12453 0 0 0 oracle

Jun 25 01:08:09 SIP-DB kernel: [20158] 500 20158 1085428 12460 0 0 0 oracle

Jun 25 01:08:09 SIP-DB kernel: [20162] 500 20162 1085555 13408 0 0 0 oracle

Jun 25 01:08:09 SIP-DB kernel: [20168] 500 20168 1085557 31637 0 0 0 oracle

Jun 25 01:08:09 SIP-DB kernel: [20172] 500 20172 1085555 12439 0 0 0 oracle

Jun 25 01:08:09 SIP-DB kernel: [20178] 500 20178 1085556 13113 0 0 0 oracle

Jun 25 01:08:09 SIP-DB kernel: [21264] 500 21264 1085558 41565 0 0 0 oracle

Jun 25 01:08:09 SIP-DB kernel: [21630] 500 21630 1085557 45061 0 0 0 oracle

Jun 25 01:08:09 SIP-DB kernel: [21634] 500 21634 1085985 109203 0 0 0 oracle

Jun 25 01:08:09 SIP-DB kernel: [21638] 500 21638 1085946 135602 0 0 0 oracle

Jun 25 01:08:09 SIP-DB kernel: [21642] 500 21642 1085558 37078 0 0 0 oracle

Jun 25 01:08:09 SIP-DB kernel: [22477] 500 22477 1085557 30752 0 0 0 oracle

Jun 25 01:08:09 SIP-DB kernel: [22495] 500 22495 1085558 12364 0 0 0 oracle

Jun 25 01:08:09 SIP-DB kernel: [22499] 500 22499 1085554 12332 0 0 0 oracle

Jun 25 01:08:09 SIP-DB kernel: [22503] 500 22503 1085557 30919 0 0 0 oracle

Jun 25 01:08:09 SIP-DB kernel: [22563] 500 22563 1085557 30068 0 0 0 oracle

Jun 25 01:08:09 SIP-DB kernel: [22567] 500 22567 1085428 12130 0 0 0 oracle

Jun 25 01:08:09 SIP-DB kernel: [22573] 500 22573 1085555 12743 0 0 0 oracle

Jun 25 01:08:09 SIP-DB kernel: [22577] 500 22577 1085555 41735 0 0 0 oracle

Jun 25 01:08:09 SIP-DB kernel: [22581] 500 22581 1085555 12192 0 0 0 oracle

Jun 25 01:08:09 SIP-DB kernel: [22585] 500 22585 1085558 42835 0 0 0 oracle

Jun 25 01:08:09 SIP-DB kernel: [22589] 500 22589 1085557 31250 0 0 0 oracle

Jun 25 01:08:09 SIP-DB kernel: [22593] 500 22593 1085429 12481 0 0 0 oracle

Jun 25 01:08:09 SIP-DB kernel: [22597] 500 22597 1085555 12491 0 0 0 oracle

Jun 25 01:08:09 SIP-DB kernel: [22601] 500 22601 1085556 13819 0 0 0 oracle

Jun 25 01:08:09 SIP-DB kernel: [22605] 500 22605 1085557 13720 0 0 0 oracle

Jun 25 01:08:09 SIP-DB kernel: [22609] 500 22609 1085554 11855 0 0 0 oracle

Jun 25 01:08:09 SIP-DB kernel: [22613] 500 22613 1085558 12944 0 0 0 oracle

Jun 25 01:08:09 SIP-DB kernel: [22617] 500 22617 1085554 30163 0 0 0 oracle

Jun 25 01:08:09 SIP-DB kernel: [22623] 500 22623 1085557 30428 0 0 0 oracle

Jun 25 01:08:09 SIP-DB kernel: [22627] 500 22627 1085558 12718 0 0 0 oracle

Jun 25 01:08:09 SIP-DB kernel: [22633] 500 22633 1085429 30834 0 0 0 oracle

Jun 25 01:08:09 SIP-DB kernel: [22685] 500 22685 1085558 31137 0 0 0 oracle

Jun 25 01:08:09 SIP-DB kernel: [22689] 500 22689 1085428 12694 0 0 0 oracle

Jun 25 01:08:09 SIP-DB kernel: [22693] 500 22693 1085554 12004 0 0 0 oracle

Jun 25 01:08:09 SIP-DB kernel: [22697] 500 22697 1085556 12623 0 0 0 oracle

Jun 25 01:08:09 SIP-DB kernel: [22703] 500 22703 1085557 36371 0 0 0 oracle

Jun 25 01:08:09 SIP-DB kernel: [22707] 500 22707 1085556 12987 0 0 0 oracle

Jun 25 01:08:09 SIP-DB kernel: [22711] 500 22711 1085557 13038 0 0 0 oracle

Jun 25 01:08:09 SIP-DB kernel: [22715] 500 22715 1085554 41183 0 0 0 oracle

Jun 25 01:08:09 SIP-DB kernel: [22719] 500 22719 1085557 41557 0 0 0 oracle

Jun 25 01:08:09 SIP-DB kernel: [22723] 500 22723 1085557 37651 0 0 0 oracle

Jun 25 01:08:09 SIP-DB kernel: [22727] 500 22727 1085557 14376 0 0 0 oracle

Jun 25 01:08:09 SIP-DB kernel: [22731] 500 22731 1085557 11939 0 0 0 oracle

Jun 25 01:08:09 SIP-DB kernel: [22735] 500 22735 1085555 41536 0 0 0 oracle

Jun 25 01:08:09 SIP-DB kernel: [22739] 500 22739 1085555 12501 0 0 0 oracle

Jun 25 01:08:09 SIP-DB kernel: [22743] 500 22743 1085554 12068 0 0 0 oracle

Jun 25 01:08:09 SIP-DB kernel: [22749] 500 22749 1085557 30880 0 0 0 oracle

Jun 25 01:08:09 SIP-DB kernel: [22753] 500 22753 1085555 14220 0 0 0 oracle

Jun 25 01:08:09 SIP-DB kernel: [22757] 500 22757 1085557 42491 0 0 0 oracle

Jun 25 01:08:09 SIP-DB kernel: [22761] 500 22761 1085429 12769 0 0 0 oracle

Jun 25 01:08:09 SIP-DB kernel: [22767] 500 22767 1085429 32130 0 0 0 oracle

Jun 25 01:08:09 SIP-DB kernel: [22771] 500 22771 1085555 11716 0 0 0 oracle

Jun 25 01:08:09 SIP-DB kernel: [22775] 500 22775 1085557 13544 0 0 0 oracle

Jun 25 01:08:09 SIP-DB kernel: [22779] 500 22779 1085558 43748 0 0 0 oracle

Jun 25 01:08:09 SIP-DB kernel: [22783] 500 22783 1085557 31105 0 0 0 oracle

Jun 25 01:08:09 SIP-DB kernel: [22787] 500 22787 1085556 14735 0 0 0 oracle

Jun 25 01:08:09 SIP-DB kernel: [22791] 500 22791 1085429 12277 0 0 0 oracle

Jun 25 01:08:09 SIP-DB kernel: [22795] 500 22795 1085558 12722 0 0 0 oracle

Jun 25 01:08:09 SIP-DB kernel: [22799] 500 22799 1085558 14014 0 0 0 oracle

Jun 25 01:08:09 SIP-DB kernel: [22803] 500 22803 1085556 12220 0 0 0 oracle

Jun 25 01:08:09 SIP-DB kernel: [22809] 500 22809 1085558 30848 0 0 0 oracle

Jun 25 01:08:09 SIP-DB kernel: [22813] 500 22813 1085170 6899 0 0 0 oracle

Jun 25 01:08:09 SIP-DB kernel: [22817] 500 22817 1085557 12752 0 0 0 oracle

Jun 25 01:08:09 SIP-DB kernel: [22821] 500 22821 1085556 13173 0 0 0 oracle

Jun 25 01:08:09 SIP-DB kernel: [22825] 500 22825 1085554 12238 0 0 0 oracle

Jun 25 01:08:09 SIP-DB kernel: [22829] 500 22829 1085556 30503 4 0 0 oracle

Jun 25 01:08:09 SIP-DB kernel: [22885] 500 22885 1085554 11271 0 0 0 oracle

Jun 25 01:08:09 SIP-DB kernel: [22889] 500 22889 1085557 12351 0 0 0 oracle

Jun 25 01:08:09 SIP-DB kernel: [22901] 500 22901 1085557 43318 0 0 0 oracle

Jun 25 01:08:09 SIP-DB kernel: [22955] 500 22955 1085555 30844 0 0 0 oracle

Jun 25 01:08:09 SIP-DB kernel: [23323] 500 23323 1085558 12256 0 0 0 oracle

Jun 25 01:08:09 SIP-DB kernel: [23327] 500 23327 1085557 13352 0 0 0 oracle

Jun 25 01:08:09 SIP-DB kernel: [23331] 500 23331 1085556 11873 0 0 0 oracle

Jun 25 01:08:09 SIP-DB kernel: [23335] 500 23335 1085555 30041 0 0 0 oracle

Jun 25 01:08:09 SIP-DB kernel: [23339] 500 23339 1085555 12160 0 0 0 oracle

Jun 25 01:08:09 SIP-DB kernel: [23345] 500 23345 1085556 41322 0 0 0 oracle

Jun 25 01:08:09 SIP-DB kernel: [23349] 500 23349 1085558 12080 0 0 0 oracle

Jun 25 01:08:09 SIP-DB kernel: [23353] 500 23353 1085557 12148 0 0 0 oracle

Jun 25 01:08:09 SIP-DB kernel: [23357] 500 23357 1085556 30705 0 0 0 oracle

Jun 25 01:08:09 SIP-DB kernel: [23361] 500 23361 1085558 11996 0 0 0 oracle

Jun 25 01:08:09 SIP-DB kernel: [23365] 500 23365 1085555 30734 0 0 0 oracle

Jun 25 01:08:09 SIP-DB kernel: [23369] 500 23369 1085557 12099 0 0 0 oracle

Jun 25 01:08:09 SIP-DB kernel: [23373] 500 23373 1085555 42331 0 0 0 oracle

Jun 25 01:08:09 SIP-DB kernel: [23379] 500 23379 1085428 11944 0 0 0 oracle

Jun 25 01:08:09 SIP-DB kernel: [23383] 500 23383 1085556 11421 0 0 0 oracle

Jun 25 01:08:09 SIP-DB kernel: [23387] 500 23387 1085557 14116 0 0 0 oracle

Jun 25 01:08:09 SIP-DB kernel: [23391] 500 23391 1085558 31611 0 0 0 oracle

Jun 25 01:08:09 SIP-DB kernel: [23397] 500 23397 1085554 11753 0 0 0 oracle

Jun 25 01:08:09 SIP-DB kernel: [23403] 500 23403 1085556 11849 0 0 0 oracle

Jun 25 01:08:09 SIP-DB kernel: [23407] 500 23407 1085558 12469 0 0 0 oracle

Jun 25 01:08:09 SIP-DB kernel: [23411] 500 23411 1085555 11600 0 0 0 oracle

Jun 25 01:08:09 SIP-DB kernel: [23417] 500 23417 1085554 12683 0 0 0 oracle

Jun 25 01:08:09 SIP-DB kernel: [23421] 500 23421 1085554 29833 0 0 0 oracle

Jun 25 01:08:09 SIP-DB kernel: [23489] 500 23489 1085428 31228 0 0 0 oracle

Jun 25 01:08:09 SIP-DB kernel: [26174] 500 26174 1085569 10892 0 0 0 oracle

Jun 25 01:08:09 SIP-DB kernel: [26178] 500 26178 1085551 8455 0 0 0 oracle

Jun 25 01:08:09 SIP-DB kernel: [26182] 500 26182 1085294 7532 0 0 0 oracle

Jun 25 01:08:09 SIP-DB kernel: [26186] 500 26186 1085294 7714 0 0 0 oracle

Jun 25 01:08:09 SIP-DB kernel: [26190] 500 26190 1085550 9570 0 0 0 oracle

Jun 25 01:08:09 SIP-DB kernel: [ 7399] 500 7399 1085825 34706 0 0 0 oracle

Jun 25 01:08:09 SIP-DB kernel: [20727] 500 20727 1084863 5047 1 0 0 oracle

Jun 25 01:08:09 SIP-DB kernel: [20840] 500 20840 1085816 37158 3 0 0 oracle

Jun 25 01:08:09 SIP-DB kernel: [22402] 500 22402 1086558 7226 0 0 0 oracle

Jun 25 01:08:09 SIP-DB kernel: [23116] 500 23116 1085560 25352 0 0 0 oracle

Jun 25 01:08:09 SIP-DB kernel: [23138] 500 23138 1085554 33359 0 0 0 oracle

Jun 25 01:08:09 SIP-DB kernel: [23182] 500 23182 1085815 45277 0 0 0 oracle

Jun 25 01:08:09 SIP-DB kernel: [23200] 500 23200 1085559 36379 0 0 0 oracle

Jun 25 01:08:09 SIP-DB kernel: [23204] 500 23204 1085556 24560 0 0 0 oracle

Jun 25 01:08:09 SIP-DB kernel: [23322] 500 23322 1085683 32003 0 0 0 oracle

Jun 25 01:08:09 SIP-DB kernel: [23631] 500 23631 1085815 37122 0 0 0 oracle

Jun 25 01:08:09 SIP-DB kernel: [23635] 500 23635 1085816 49491 0 0 0 oracle

Jun 25 01:08:09 SIP-DB kernel: [23639] 500 23639 1085560 53198 0 0 0 oracle

Jun 25 01:08:09 SIP-DB kernel: [23657] 500 23657 1085559 41668 0 0 0 oracle

Jun 25 01:08:09 SIP-DB kernel: [23665] 500 23665 1085815 41192 0 0 0 oracle

Jun 25 01:08:09 SIP-DB kernel: [23741] 500 23741 1085559 42575 0 0 0 oracle

Jun 25 01:08:09 SIP-DB kernel: [23767] 500 23767 1085814 45467 0 0 0 oracle

Jun 25 01:08:09 SIP-DB kernel: [23837] 500 23837 1085559 40316 0 0 0 oracle

Jun 25 01:08:09 SIP-DB kernel: [23863] 500 23863 1085559 41834 0 0 0 oracle

Jun 25 01:08:09 SIP-DB kernel: [23875] 500 23875 1085814 40051 0 0 0 oracle

Jun 25 01:08:09 SIP-DB kernel: [24843] 500 24843 1085162 5817 0 0 0 oracle

Jun 25 01:08:09 SIP-DB kernel: [24847] 500 24847 1085162 5900 0 0 0 oracle

Jun 25 01:08:09 SIP-DB kernel: [24851] 500 24851 1085162 6259 0 0 0 oracle

Jun 25 01:08:09 SIP-DB kernel: [24855] 500 24855 1085162 5886 0 0 0 oracle

Jun 25 01:08:09 SIP-DB kernel: [24859] 500 24859 1085162 5888 0 0 0 oracle

Jun 25 01:08:09 SIP-DB kernel: [24869] 500 24869 1085680 27042 0 0 0 oracle

Jun 25 01:08:09 SIP-DB kernel: [24873] 500 24873 1085162 5959 0 0 0 oracle

Jun 25 01:08:09 SIP-DB kernel: [24877] 500 24877 1085162 5964 0 0 0 oracle

Jun 25 01:08:09 SIP-DB kernel: [24881] 500 24881 1085162 5969 0 0 0 oracle

Jun 25 01:08:09 SIP-DB kernel: [24885] 500 24885 1085162 5981 0 0 0 oracle

Jun 25 01:08:09 SIP-DB kernel: [25069] 500 25069 1085559 34675 0 0 0 oracle

Jun 25 01:08:09 SIP-DB kernel: [25073] 500 25073 1085546 9324 0 0 0 oracle

Jun 25 01:08:09 SIP-DB kernel: [25077] 500 25077 1085555 20105 0 0 0 oracle

Jun 25 01:08:09 SIP-DB kernel: [25087] 500 25087 1085556 24367 0 0 0 oracle

Jun 25 01:08:09 SIP-DB kernel: [27133] 0 27133 35053 437 0 0 0 crond

Jun 25 01:08:09 SIP-DB kernel: [27134] 0 27134 40821 429 0 0 0 su

Jun 25 01:08:09 SIP-DB kernel: [27136] 500 27136 26526 309 2 0 0 rman_backup_lv1

Jun 25 01:08:09 SIP-DB kernel: [27152] 500 27152 34069 14062 1 0 0 rman

Jun 25 01:08:09 SIP-DB kernel: [27169] 500 27169 1093001 25319 6 0 0 oracle

Jun 25 01:08:09 SIP-DB kernel: [27172] 500 27172 1085312 7537 1 0 0 oracle

Jun 25 01:08:09 SIP-DB kernel: [27176] 500 27176 1092998 14943 5 0 0 oracle

Jun 25 01:08:09 SIP-DB kernel: [27179] 500 27179 1092991 12997 0 0 0 oracle

Jun 25 01:08:09 SIP-DB kernel: [27340] 500 27340 1084862 4749 5 0 0 oracle

Jun 25 01:08:09 SIP-DB kernel: Out of memory: Kill process 30344 (oracle) score 54 or sacrifice child

Jun 25 01:08:09 SIP-DB kernel: Killed process 30344, UID 500, (oracle) total-vm:4339452kB, anon-rss:2132kB, file-rss:1343184kB

发现是内存不足导致系统把oracle进程杀掉了。

网上搜了一下oom

oom-killer 触发的原因都是内存不足导致的，而内存不足通常是由于内存超分配导致的。

关于linux内核的超分配是由overcommit_memory参数控制的(/proc/sys/vm/overcommit_memory)，

关于参数的解释大神都有很详细的解释了，我就直接拿来用了:

0：启发式策略，比较严重的Overcommit将不能得逞，比如你突然申请了128TB的内存。

而轻微的Overcommit将被允许。另外，root能Overcommit的值比普通用户要稍微多些。

1：永远允许Overcommit，这种策略适合那些不能承受内存分配失败的应用，比如某些科学计算应用。

2：永远禁止Overcommit，在这个情况下，系统所能分配的内存不会超过swap+RAM*系数

（/proc/sys/vm/overcmmit_ratio，默认50%，你可以调整）， 如果这么多资源

已经用光，那么后面任何尝试申请内存的行为都会返回错误，这通常意味着此时没法运行任何新程序。

查看了一下我的配置
[root@TESTDB ~]# cat /proc/sys/vm/swappiness

0

果然是0，怪不得系统日志里看swap都没用到，就把oracle干掉了。

修改一下在观察观察吧。

echo 60 > /proc/sys/vm/swappiness

另外发现一个帖子也有可能用到：
http://www.xifenfei.com/3400.html