ORA-27300，ORA-27301，ORA-27302，ORA-04030导致crashed database

今天检查数据库的时候发现的alert日志报错，OS:HP UNIX 11.31 DB:Oracle Database 10g Enterprise Edition Release 10.2.0.5.0 – 64bit Prodction RAC
Errors in file /oracle/admin/scrk/bdump/scrk2_psp0_3627.trc:
ORA-27300: OS system dependent operation:fork failed with status: 11
ORA-27301: OS failure message: Resource temporarily unavailable
ORA-27302: failure occurred at: skgpspawn5
ORA-27303: additional information: skgpspawn5
Mon Mar 14 15:23:00 EAT 2011
Process J001 died, see its trace file
Mon Mar 14 15:23:00 EAT 2011
kkjcre1p: unable to spawn jobq slave process
Mon Mar 14 15:23:00 EAT 2011
Errors in file /oracle/admin/scrk/bdump/scrk2_cjq0_3649.trc:

Errors in file /oracle/admin/scrk/bdump/scrk2_psp0_3627.trc:
ORA-27300: OS system dependent operation:fork failed with status: 11
ORA-27301: OS failure message: Resource temporarily unavailable
ORA-27302: failure occurred at: skgpspawn5
ORA-27303: additional information: skgpspawn5
Mon Mar 14 15:46:09 EAT 2011
Process J000 died, see its trace file
Mon Mar 14 15:46:09 EAT 2011
kkjcre1p: unable to spawn jobq slave process
Mon Mar 14 15:46:09 EAT 2011
Errors in file /oracle/admin/scrk/bdump/scrk2_cjq0_3649.trc:

Mon Mar 14 15:47:09 EAT 2011
Process startup failed, error stack:
Mon Mar 14 15:47:09 EAT 2011
Errors in file /oracle/admin/scrk/bdump/scrk2_psp0_3627.trc:
ORA-27300: OS system dependent operation:fork failed with status: 12
ORA-27301: OS failure message: Not enough space
ORA-27302: failure occurred at: skgpspawn3
Mon Mar 14 15:47:10 EAT 2011
Process J000 died, see its trace file
Mon Mar 14 15:47:10 EAT 2011
kkjcre1p: unable to spawn jobq slave process
Mon Mar 14 15:47:10 EAT 2011
Errors in file /oracle/admin/scrk/bdump/scrk2_cjq0_3649.trc:

Process startup failed, error stack:
Mon Mar 14 15:53:07 EAT 2011
Errors in file /oracle/admin/scrk/bdump/scrk2_psp0_3627.trc:
ORA-27300: OS system dependent operation:fork failed with status: 12
ORA-27301: OS failure message: Not enough space
ORA-27302: failure occurred at: skgpspawn3
Mon Mar 14 15:53:08 EAT 2011
Process J001 died, see its trace file
Mon Mar 14 15:53:08 EAT 2011
kkjcre1p: unable to spawn jobq slave process
Mon Mar 14 15:53:08 EAT 2011
Errors in file /oracle/admin/scrk/bdump/scrk2_cjq0_3649.trc:

Process startup failed, error stack:
Mon Mar 14 15:53:23 EAT 2011
Errors in file /oracle/admin/scrk/bdump/scrk2_psp0_3627.trc:
ORA-27300: OS system dependent operation:fork failed with status: 12
ORA-27301: OS failure message: Not enough space
ORA-27302: failure occurred at: skgpspawn3
Mon Mar 14 15:53:24 EAT 2011
Process m000 died, see its trace file
Mon Mar 14 15:53:24 EAT 2011
ksvcreate: Process(m000) creation failed

Errors in file /oracle/admin/scrk/bdump/scrk2_dbw0_3639.trc:
ORA-04030: out of process memory when trying to allocate 1049136 bytes (KSFD vec heap,KS
FD contig I/O buf)
Mon Mar 14 15:53:55 EAT 2011
System state dump is made for local instance
Mon Mar 14 15:53:55 EAT 2011
DBW0: terminating instance due to error 4030
Mon Mar 14 15:53:55 EAT 2011
System State dumped to trace file /oracle/admin/scrk/bdump/scrk2_diag_3620.trc
Mon Mar 14 15:53:58 EAT 2011
Shutting down instance (abort)
License high water mark = 174
Mon Mar 14 15:54:00 EAT 2011
Instance terminated by DBW0, pid = 3639
Mon Mar 14 15:54:03 EAT 2011
Instance terminated by USER, pid = 9322
Mon Mar 14 16:25:27 EAT 2011
Starting ORACLE instance (normal)
通过查询metalink [ID 579365.1] ：
ORA-27300: OS system dependent operation:fork failed with status: 11错误是由于：Maximum number of PROCESSES allowed per user may be too low (for example: maxuproc on AIX, maxuprc on HP and Solaris. On Solaris 10, check project.max-lwps)。不过alert日志的ORA-27303: additional information: 是skgpspawn5不是文档中的skgpspawn3。由于对OS不熟，此问题暂时搁置，以备以后查询。
ORA-27300: OS system dependent operation:fork failed with status: 12 错误是由于：STATUS 12 - ENOMEM Not enough core / memory ;During an exec or a break, the program asked for more memory than the one available by the system. This error also occurs when there are too many segmentation registers which are required for the arrangement of text data or stack segments.
ORA-04030: out of process memory when trying to allocate 1049136 bytes (KSFD vec heap,KS
FD contig I/O buf)是由于dbw0没有请求到足够的内存导致了实力的重启。根据metalink [ID 580552.1]在system log应当还有WARNING: Sorry, no swap space to grow stack for pid ***** (oracle)的警告信息，可是在此时间段内没有发现，此问题暂时搁置以待复查。