ORA-03113: 通信通道的文件结尾 进程 ID: 3949 会话 ID: 1 序列号: 3

一套11.2.0.3的RAC拆分为单节点。更新完最新Bundle&PSU，test了下自动回滚。当回滚成功后。

[oracle@testora dbs]$ sqlplus / as sysdba

SQL*Plus: Release 11.2.0.3.0 Production on 星期一 12月 23 10:51:36 2013

Copyright (c) 1982, 2011, Oracle.  All rights reserved.

已连接到空闲例程。

SQL> startup
ORACLE 例程已经启动。

Total System Global Area  609624064 bytes
Fixed Size		    2230792 bytes
Variable Size		  226493944 bytes
Database Buffers	  377487360 bytes
Redo Buffers		    3411968 bytes
ORA-03113: 通信通道的文件结尾
进程 ID: 3949
会话 ID: 1 序列号: 3

SQL> exit

这个时候dg的状态为挂载了

[grid@testora ~]$ sqlplus / as sysasm

SQL*Plus: Release 11.2.0.3.0 Production on Mon Dec 23 12:48:28 2013

Copyright (c) 1982, 2011, Oracle.  All rights reserved.

Connected to:
Oracle Database 11g Enterprise Edition Release 11.2.0.3.0 - 64bit Production
With the Automatic Storage Management option

SQL> select status from v$instance;

STATUS
------------
STARTED

alter 告警日志：/u01/app/oracle/diag/rdbms/testora/testora/trace/alert_testora.log

Mon Dec 23 10:51:55 2013
Starting ORACLE instance (normal)
****************** Large Pages Information *****************

Total Shared Global Region in Large Pages = 0 KB (0%)

Large Pages used by this instance: 0 (0 KB)
Large Pages unused system wide = 0 (0 KB) (alloc incr 4096 KB)
Large Pages configured system wide = 0 (0 KB)
Large Page size = 2048 KB

RECOMMENDATION:
Total Shared Global Region size is 586 MB. For optimal performance,
prior to the next instance restart increase the number
of unused Large Pages by atleast 293 2048 KB Large Pages (586 MB)
system wide to get 100% of the Shared
Global Region allocated with Large pages
***********************************************************
LICENSE_MAX_SESSION = 0
LICENSE_SESSIONS_WARNING = 0
Picked latch-free SCN scheme 3
Using LOG_ARCHIVE_DEST_1 parameter default value as /u01/app/oracle/product/11.2.0/dbhome_1/dbs/arch
Autotune of undo retention is turned on.
IMODE=BR
ILAT =27
LICENSE_MAX_USERS = 0
SYS auditing is disabled
Starting up:
Oracle Database 11g Enterprise Edition Release 11.2.0.3.0 - 64bit Production
With the Partitioning, OLAP, Data Mining and Real Application Testing options.
ORACLE_HOME = /u01/app/oracle/product/11.2.0/dbhome_1
System name:    Linux
Node name:      testora
Release:        2.6.18-194.el5
Version:        #1 SMP Tue Mar 16 21:52:39 EDT 2010
Machine:        x86_64
Using parameter settings in client-side pfile /u01/app/oracle/product/11.2.0/dbhome_1/dbs/inittestora.ora on machine testora
System parameters with non-default values:
processes                = 150
nls_language             = "SIMPLIFIED CHINESE"
sga_target               = 584M
control_files            = "+DATA/testora/controlfile/current.260.822235297"
db_block_size            = 8192
compatible               = "11.2.0.0.0"
db_create_file_dest      = "+DATA"
db_recovery_file_dest_size= 41222905K
undo_tablespace          = "UNDOTBS1"
remote_login_passwordfile= "EXCLUSIVE"
db_domain                = ""
dispatchers              = "(PROTOCOL=TCP) (SERVICE=testoraXDB)"
audit_file_dest          = "/u01/app/oracle/admin/testora/adump"
audit_trail              = "DB"
db_name                  = "testora"
open_cursors             = 300
diagnostic_dest          = "/u01/app/oracle"
Mon Dec 23 10:51:56 2013
PMON started with pid=2, OS id=3902
Mon Dec 23 10:51:56 2013
PSP0 started with pid=3, OS id=3904
Mon Dec 23 10:51:57 2013
VKTM started with pid=4, OS id=3906 at elevated priority
VKTM running at (1)millisec precision with DBRM quantum (100)ms
Mon Dec 23 10:51:57 2013
GEN0 started with pid=5, OS id=3910
Mon Dec 23 10:51:57 2013
DIAG started with pid=6, OS id=3912
Mon Dec 23 10:51:57 2013
DBRM started with pid=7, OS id=3914
Mon Dec 23 10:51:57 2013
DIA0 started with pid=8, OS id=3916
Mon Dec 23 10:51:57 2013
MMAN started with pid=9, OS id=3918
Mon Dec 23 10:51:57 2013
DBW0 started with pid=10, OS id=3920
Mon Dec 23 10:51:57 2013
LGWR started with pid=11, OS id=3922
Mon Dec 23 10:51:57 2013
CKPT started with pid=12, OS id=3924
Mon Dec 23 10:51:57 2013
SMON started with pid=13, OS id=3926
Mon Dec 23 10:51:57 2013
RECO started with pid=14, OS id=3928
Mon Dec 23 10:51:57 2013
RBAL started with pid=15, OS id=3930
Mon Dec 23 10:51:58 2013
ASMB started with pid=16, OS id=3932
Mon Dec 23 10:51:58 2013
MMON started with pid=17, OS id=3934
NOTE: initiating MARK startup
starting up 1 dispatcher(s) for network address '(ADDRESS=(PARTIAL=YES)(PROTOCOL=TCP))'...
Mon Dec 23 10:51:58 2013
MMNL started with pid=18, OS id=3938
Starting background process MARK
Mon Dec 23 10:51:58 2013
Mon Dec 23 10:30:23 2013
System state dump requested by (instance=1, osid=3733 (PMON)), summary=[abnormal instance termination].
System State dumped to trace file /u01/app/oracle/diag/rdbms/testora/testora/trace/testora_diag_3743.trc
Dumping diagnostic data in directory=[cdmp_20131223103023], requested by (instance=1, osid=3733 (PMON)), summary=[abnormal instance termination].
Instance terminated by PMON, pid = 3733
Mon Dec 23 10:51:55 2013
Starting ORACLE instance (normal)
****************** Large Pages Information *****************

Total Shared Global Region in Large Pages = 0 KB (0%)

Large Pages used by this instance: 0 (0 KB)
Large Pages unused system wide = 0 (0 KB) (alloc incr 4096 KB)
Large Pages configured system wide = 0 (0 KB)
Large Page size = 2048 KB

RECOMMENDATION:
Total Shared Global Region size is 586 MB. For optimal performance,
prior to the next instance restart increase the number
of unused Large Pages by atleast 293 2048 KB Large Pages (586 MB)
system wide to get 100% of the Shared
Global Region allocated with Large pages
***********************************************************
LICENSE_MAX_SESSION = 0
LICENSE_SESSIONS_WARNING = 0
Picked latch-free SCN scheme 3
Using LOG_ARCHIVE_DEST_1 parameter default value as /u01/app/oracle/product/11.2.0/dbhome_1/dbs/arch
Autotune of undo retention is turned on.
IMODE=BR
ILAT =27
LICENSE_MAX_USERS = 0
SYS auditing is disabled
Starting up:
Oracle Database 11g Enterprise Edition Release 11.2.0.3.0 - 64bit Production
With the Partitioning, OLAP, Data Mining and Real Application Testing options.
ORACLE_HOME = /u01/app/oracle/product/11.2.0/dbhome_1
System name:    Linux
Node name:      testora
Release:        2.6.18-194.el5
Version:        #1 SMP Tue Mar 16 21:52:39 EDT 2010
Machine:        x86_64
Using parameter settings in client-side pfile /u01/app/oracle/product/11.2.0/dbhome_1/dbs/inittestora.ora on machine testora
System parameters with non-default values:
processes                = 150
nls_language             = "SIMPLIFIED CHINESE"
sga_target               = 584M
control_files            = "+DATA/testora/controlfile/current.260.822235297"
db_block_size            = 8192
compatible               = "11.2.0.0.0"
db_create_file_dest      = "+DATA"
db_recovery_file_dest_size= 41222905K
undo_tablespace          = "UNDOTBS1"
remote_login_passwordfile= "EXCLUSIVE"
db_domain                = ""
dispatchers              = "(PROTOCOL=TCP) (SERVICE=testoraXDB)"
audit_file_dest          = "/u01/app/oracle/admin/testora/adump"
audit_trail              = "DB"
db_name                  = "testora"
open_cursors             = 300
diagnostic_dest          = "/u01/app/oracle"
Mon Dec 23 10:51:56 2013
PMON started with pid=2, OS id=3902
Mon Dec 23 10:51:56 2013
PSP0 started with pid=3, OS id=3904
Mon Dec 23 10:51:57 2013
VKTM started with pid=4, OS id=3906 at elevated priority
VKTM running at (1)millisec precision with DBRM quantum (100)ms
Mon Dec 23 10:51:57 2013
GEN0 started with pid=5, OS id=3910
Mon Dec 23 10:51:57 2013
DIAG started with pid=6, OS id=3912
Mon Dec 23 10:51:57 2013
DBRM started with pid=7, OS id=3914
Mon Dec 23 10:51:57 2013
DIA0 started with pid=8, OS id=3916
Mon Dec 23 10:51:57 2013
MMAN started with pid=9, OS id=3918
Mon Dec 23 10:51:57 2013
DBW0 started with pid=10, OS id=3920
Mon Dec 23 10:51:57 2013
LGWR started with pid=11, OS id=3922
Mon Dec 23 10:51:57 2013
CKPT started with pid=12, OS id=3924
Mon Dec 23 10:51:57 2013
SMON started with pid=13, OS id=3926
Mon Dec 23 10:51:57 2013
RECO started with pid=14, OS id=3928
Mon Dec 23 10:51:57 2013
RBAL started with pid=15, OS id=3930
Mon Dec 23 10:51:58 2013
ASMB started with pid=16, OS id=3932
Mon Dec 23 10:51:58 2013
MMON started with pid=17, OS id=3934
NOTE: initiating MARK startup
starting up 1 dispatcher(s) for network address '(ADDRESS=(PARTIAL=YES)(PROTOCOL=TCP))'...
Mon Dec 23 10:51:58 2013
MMNL started with pid=18, OS id=3938
Starting background process MARK
Mon Dec 23 10:51:58 2013
ORACLE_BASE from environment = /u01/app/oracle
Mon Dec 23 10:51:58 2013
ALTER DATABASE   MOUNT
Mon Dec 23 10:52:01 2013
Sweep [inc][132146]: completed
Sweep [inc][132145]: completed
Sweep [inc2][132146]: completed
Sweep [inc2][132145]: completed
NOTE: Loaded library: System
ORA-15025: ?????? "/dev/raw/raw1"
ORA-27041: ??????
Linux-x86_64 Error: 13: Permission denied
Additional information: 9
SUCCESS: diskgroup DATA was mounted
NOTE: dependency between database testora and diskgroup resource ora.DATA.dg is established
Errors in file /u01/app/oracle/diag/rdbms/testora/testora/trace/testora_ckpt_3924.trc  (incident=133345):
ORA-00600: internal error code, arguments: [kfioTranslateIO03], [], [], [], [], [], [], [], [], [], [], []
Incident details in: /u01/app/oracle/diag/rdbms/testora/testora/incident/incdir_133345/testora_ckpt_3924_i133345.trc
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
Errors in file /u01/app/oracle/diag/rdbms/testora/testora/trace/testora_ckpt_3924.trc  (incident=133346):
ORA-00600: internal error code, arguments: [17090], [], [], [], [], [], [], [], [], [], [], []
Incident details in: /u01/app/oracle/diag/rdbms/testora/testora/incident/incdir_133346/testora_ckpt_3924_i133346.trc
Dumping diagnostic data in directory=[cdmp_20131223105205], requested by (instance=1, osid=3924 (CKPT)), summary=[incident=133345].
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
ERROR: unrecoverable error ORA-600 raised in ASM I/O path; terminating process 3924
Dumping diagnostic data in directory=[cdmp_20131223105206], requested by (instance=1, osid=3924 (CKPT)), summary=[incident=133346].
Mon Dec 23 10:52:07 2013
PMON (ospid: 3902): terminating the instance due to error 469
Mon Dec 23 10:52:08 2013
System state dump requested by (instance=1, osid=3902 (PMON)), summary=[abnormal instance termination].
System State dumped to trace file /u01/app/oracle/diag/rdbms/testora/testora/trace/testora_diag_3912.trc
Dumping diagnostic data in directory=[cdmp_20131223105208], requested by (instance=1, osid=3902 (PMON)), summary=[abnormal instance termination].
Instance terminated by PMON, pid = 3902

cd $ORACLE_HOME/bin

-rwsr-s--x 1 oracle oinstall oracle （错误）

正常

-rwsr-s--x 1 oracle asmadmin 232656133 Dec 19 11:07 oracle

解决：

chown oracle:asmadmin oracle

chmod 6751 oracle

验证：

[oracle@testora ~]$ sqlplus / as sysdba

SQL*Plus: Release 11.2.0.3.0 Production on 星期一 12月 23 12:57:54 2013

Copyright (c) 1982, 2011, Oracle. All rights reserved.

已连接到空闲例程。

idle> startup

ORACLE 例程已经启动。

Total System Global Area 609624064 bytes

Fixed Size 2230792 bytes

Variable Size 226493944 bytes

Database Buffers 377487360 bytes

Redo Buffers 3411968 bytes

数据库装载完毕。

数据库已经打开。

[oracle@testora ~]$ lsnrctl status

LSNRCTL for Linux: Version 11.2.0.3.0 - Production on 23-12月-2013 12:58:53

Copyright (c) 1991, 2011, Oracle. All rights reserved.

正在连接到 (DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=192.168.1.150)(PORT=1521)))

LISTENER 的 STATUS

------------------------

别名 LISTENER

版本 TNSLSNR for Linux: Version 11.2.0.3.0 - Production

启动日期 23-12月-2013 11:22:54

正常运行时间 0 天 1 小时 35 分 59 秒

跟踪级别 off

安全性 ON: Local OS Authentication

SNMP OFF

监听程序参数文件 /u01/app/grid/product/11.2.0/grid/network/admin/listener.ora

监听程序日志文件 /u01/app/grid/diag/tnslsnr/testora/listener/alert/log.xml

监听端点概要...

(DESCRIPTION=(ADDRESS=(PROTOCOL=ipc)(KEY=EXTPROC1521)))

(DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=testora)(PORT=1521)))

服务摘要..

服务 "+ASM" 包含 1 个实例。

实例 "+ASM", 状态 READY, 包含此服务的 1 个处理程序...

服务 "testora" 包含 1 个实例。

实例 "testora", 状态 READY, 包含此服务的 1 个处理程序...

服务 "testoraXDB" 包含 1 个实例。

实例 "testora", 状态 READY, 包含此服务的 1 个处理程序...

命令执行成功

[oracle@testora ~]$ sqlplus test/test@192.168.1.150:1521/testora

SQL*Plus: Release 11.2.0.3.0 Production on 星期一 12月 23 12:59:30 2013

Copyright (c) 1982, 2011, Oracle. All rights reserved.

连接到:

Oracle Database 11g Enterprise Edition Release 11.2.0.3.0 - 64bit Production

With the Partitioning, Automatic Storage Management, OLAP, Data Mining

and Real Application Testing options

test@TESTORA>