案例:Standby RAC遭遇ORA-1157,1111,1110导致实例crash处理
2017-08-11 22:18
441 查看
案例:Standby RAC遭遇ORA-1157,1111,1110导致实例crash处理
环境:RHEL 6.5 + Oracle RAC 11.2.0.4 + Dataguard
今天在实验环境的Pirmary RAC主库上做了一个增加表空间的操作,结果Standby RAC启动同步后直接crash,具体报错如下:
根据报错查到MOS相关文档:
How to resolve ORA-01111 ORA-01110 ORA-01157 in a physical standby database (文档 ID 1416554.1)
这里应该就可以猜测出原因基本就是由于主库建立一个文件,备库同步日志,没有创建正确。
MOS 解决方案核心内容:
看起来和STANDBY_FILE_MANAGEMENT有关系,进一步查看show parameter STANDBY_FILE_MANAGEMENT
发现果然都是MANUAL。
先处理当前问题,按照MOS的几种方法,选择适合我这里环境的方案:
创建和问题文件相同结构的空文件,然后备库开启应用所有归档文件即可完成恢复。
创建与错误数据文件结构相同的空文件,具体大小从主库查询,不加大小可能遭遇其他问题,详见上面MOS文档:
Alter database create datafile '/opt/app/oracle/product/11.2.0/dbhome_1/dbs/UNNAMED00010' as '+DATA/' size 104857600;
创建后,可以发现v$datafile已经正确:
然后修改STANDBY_FILE_MANAGEMENT=AUTO,正常开启数据库,开启实时应用:
再测试增加数据文件,删除数据文件,都可以正常同步:
备库都会自动的正常添加删除,日志都会记录如下:
总结:DG环境,一定要注意参数STANDBY_FILE_MANAGEMENT=AUTO,否则可能遇到类似情况。
环境:RHEL 6.5 + Oracle RAC 11.2.0.4 + Dataguard
今天在实验环境的Pirmary RAC主库上做了一个增加表空间的操作,结果Standby RAC启动同步后直接crash,具体报错如下:
Fri Aug 11 19:14:20 2017 Completed: ALTER DATABASE MOUNT /* db agent *//* {1:53752:2} */ ALTER DATABASE OPEN /* db agent *//* {1:53752:2} */ AUDIT_TRAIL initialization parameter is changed to OS, as DB is NOT compatible for database opened with read-only access This instance was first to open Fri Aug 11 19:14:23 2017 Using STANDBY_ARCHIVE_DEST parameter default value as USE_DB_RECOVERY_FILE_DEST Beginning Standby Crash Recovery. Serial Media Recovery started Fri Aug 11 19:14:24 2017 Managed Standby Recovery starting Real Time Apply Fri Aug 11 19:14:25 2017 Warning: VKTM detected a time drift. Time drifts can result in an unexpected behavior such as time-outs. Please check trace file for more details. Fri Aug 11 19:14:26 2017 Primary database is in MAXIMUM PERFORMANCE mode Fri Aug 11 19:14:26 2017 Primary database is in MAXIMUM PERFORMANCE mode RFS[1]: Assigned to RFS process 4031 RFS[1]: Selected log 11 for thread 1 sequence 171 dbid -1785877518 branch 919999037 Standby Crash Recovery aborted due to error 1111. Errors in file /opt/app/oracle/diag/rdbms/mynas/jyzhao1/trace/jyzhao1_ora_3923.trc: ORA-01111: name for data file 10 is unknown - rename to correct file ORA-01110: data file 10: '/opt/app/oracle/product/11.2.0/dbhome_1/dbs/UNNAMED00010' ORA-01157: cannot identify/lock data file 10 - see DBWR trace file ORA-01111: name for data file 10 is unknown - rename to correct file ORA-01110: data file 10: '/opt/app/oracle/product/11.2.0/dbhome_1/dbs/UNNAMED00010' Fri Aug 11 19:14:27 2017 RFS[2]: Assigned to RFS process 4029 RFS[2]: Selected log 13 for thread 1 sequence 170 dbid -1785877518 branch 919999037 Fri Aug 11 19:14:28 2017 RFS[3]: Assigned to RFS process 4043 RFS[3]: Selected log 22 for thread 2 sequence 134 dbid -1785877518 branch 919999037 RFS[4]: Assigned to RFS process 4041 RFS[4]: Selected log 23 for thread 2 sequence 135 dbid -1785877518 branch 919999037 Completed Standby Crash Recovery. Fri Aug 11 19:14:31 2017 Abort recovery for domain 0 Errors in file /opt/app/oracle/diag/rdbms/mynas/jyzhao1/trace/jyzhao1_ora_3923.trc: ORA-10458: standby database requires recovery ORA-01157: cannot identify/lock data file 10 - see DBWR trace file ORA-01111: name for data file 10 is unknown - rename to correct file ORA-01110: data file 10: '/opt/app/oracle/product/11.2.0/dbhome_1/dbs/UNNAMED00010' ORA-10458 signalled during: ALTER DATABASE OPEN /* db agent *//* {1:53752:2} */... Fri Aug 11 19:14:33 2017 Shutting down instance (abort) License high water mark = 9 USER (ospid: 4089): terminating the instance Instance terminated by USER, pid = 4089 Fri Aug 11 19:14:34 2017 Instance shutdown complete
根据报错查到MOS相关文档:
How to resolve ORA-01111 ORA-01110 ORA-01157 in a physical standby database (文档 ID 1416554.1)
这里应该就可以猜测出原因基本就是由于主库建立一个文件,备库同步日志,没有创建正确。
MOS 解决方案核心内容:
SQL>ALTER DATABASE RECOVER MANAGED STANDBY DATABASE CANCEL; SQL>ALTER SYSTEM SET STANDBY_FILE_MANAGEMENT=MANUAL SID='*'; Rename the unknown datafile 97. SQL> ALTER DATABASE RENAME FILE '/u01/app/oracle/product/10.2.0.5/db_1/dbs/UNNAMED00097' to '</absolute path/real datafile name>'; Create a empty datafile which same structure as the datafile 97 and it would need all archivelogs from time of creation for recovery Alter database create datafile '/u01/app/oracle/product/10.2.0.5/db_1/dbs/UNNAMED00097' as '+DATA_ACS4/' size <Actual size of datafile on Primary> Please note while giving the above command on ASM you also need to specify the action size of the datafile from the primary (Query v$datafiles for Bytes column) SQL>ALTER SYSTEM SET STANDBY_FILE_MANAGEMENT=AUTO SID='*'; SQL>ALTER DATABASE RECOVER MANAGED STANDBY DATABASE DISCONNECT; SQL>ALTER DATABASE RECOVER MANAGED STANDBY DATABASE USING CURRENT LOGFILE DISCONNECT;
看起来和STANDBY_FILE_MANAGEMENT有关系,进一步查看show parameter STANDBY_FILE_MANAGEMENT
On Primary: SYS@jyzhao1 > show parameter STANDBY_FILE_MANAGEMENT NAME TYPE VALUE ------------------------------------ ----------- ------------------------------ standby_file_management string MANUAL SYS@jyzhao1 > On Standby: SQL> show parameter STANDBY_FILE_MANAGEMENT NAME TYPE VALUE ------------------------------------ ----------- ------------------------------ standby_file_management string MANUAL SQL>
发现果然都是MANUAL。
先处理当前问题,按照MOS的几种方法,选择适合我这里环境的方案:
创建和问题文件相同结构的空文件,然后备库开启应用所有归档文件即可完成恢复。
创建与错误数据文件结构相同的空文件,具体大小从主库查询,不加大小可能遭遇其他问题,详见上面MOS文档:
Alter database create datafile '/opt/app/oracle/product/11.2.0/dbhome_1/dbs/UNNAMED00010' as '+DATA/' size 104857600;
创建后,可以发现v$datafile已经正确:
SQL> select name from v$datafile; NAME -------------------------------------------------------------------------------- +DATA/mynas/datafile/system.258.951608183 +DATA/mynas/datafile/sysaux.257.951608183 +DATA/mynas/datafile/undotbs1.259.951608185 +DATA/mynas/datafile/users.265.951608205 +DATA/mynas/datafile/undotbs2.261.951608185 +DATA/mynas/datafile/dbs_d_jingyu.262.951608185 +DATA/mynas/datafile/dbs_i_jingyu.263.951608185 +DATA/mynas/datafile/test.264.951608185 +DATA/mynas/datafile/test2.260.951608185 /opt/app/oracle/product/11.2.0/dbhome_1/dbs/UNNAMED00010 10 rows selected. SQL> Alter database create datafile '/opt/app/oracle/product/11.2.0/dbhome_1/dbs/UNNAMED00010' as '+DATA/' size 104857600; Database altered. SQL> select name from v$datafile; NAME -------------------------------------------------------------------------------- +DATA/mynas/datafile/system.258.951608183 +DATA/mynas/datafile/sysaux.257.951608183 +DATA/mynas/datafile/undotbs1.259.951608185 +DATA/mynas/datafile/users.265.951608205 +DATA/mynas/datafile/undotbs2.261.951608185 +DATA/mynas/datafile/dbs_d_jingyu.262.951608185 +DATA/mynas/datafile/dbs_i_jingyu.263.951608185 +DATA/mynas/datafile/test.264.951608185 +DATA/mynas/datafile/test2.260.951608185 +DATA/mynas/datafile/dbs_d_hank.273.951774293 10 rows selected.
然后修改STANDBY_FILE_MANAGEMENT=AUTO,正常开启数据库,开启实时应用:
SQL>ALTER SYSTEM SET STANDBY_FILE_MANAGEMENT=AUTO SID='*'; SQL>ALTER DATABASE OPEN; SQL>ALTER DATABASE RECOVER MANAGED STANDBY DATABASE USING CURRENT LOGFILE DISCONNECT;
再测试增加数据文件,删除数据文件,都可以正常同步:
SYS@jyzhao1 >create tablespace dbs_only_test; SYS@jyzhao1 >drop tablespace dbs_only_test including contents and datafiles;
备库都会自动的正常添加删除,日志都会记录如下:
Fri Aug 11 21:49:11 2017 Media Recovery Waiting for thread 1 sequence 175 (in transit) Recovery of Online Redo Log: Thread 1 Group 11 Seq 175 Reading mem 0 Mem# 0: +FRA/mynas/standbylog/standby_group_11.log Fri Aug 11 21:51:15 2017 Successfully added datafile 11 to media recovery Datafile #11: '+DATA/mynas/datafile/dbs_only_test.275.951774665' Fri Aug 11 21:53:20 2017 Recovery deleting file #11:'+DATA/mynas/datafile/dbs_only_test.275.951774665' from controlfile. Deleted Oracle managed file +DATA/mynas/datafile/dbs_only_test.275.951774665 Recovery dropped tablespace 'DBS_ONLY_TEST'
总结:DG环境,一定要注意参数STANDBY_FILE_MANAGEMENT=AUTO,否则可能遇到类似情况。
相关文章推荐
- Standby RAC遭遇ORA-1157,1111,1110导致实例crash处理
- Ora-1157 ora-1110错误解决案例一枚
- 【案例分享】RAC中可能由于节点cache temp space导致ora-1652错误
- Ora-1157 ora-1110错误解决案例一枚
- ORA-1157, ORA-1110的错误解决方法
- RAC 实例不能启动 ORA-1589 signalled during ALTER DATABASE OPEN
- RAC 安装patch 后启动实例 报错 ORA-00439 feature not enabled- Real Application Clusters 解决方法
- oracle11g rac 实例1不能启动 ORA-01078
- 11gR2启动ASM实例时遭遇ORA-29701
- oracle技术之利用STANDBY将单实例数据库升级为RAC环境(二)
- Oracle 11gR2单实例转换RAC-----ORA-25152
- rac重启遭遇ORA-01078、ORA-01565、ORA-17503、ORA-12547
- 最常见的5个导致 RAC 实例崩溃的问题
- RAC环境STANDBY数据库只能有一个实例在进行日志恢复工作
- RAC 实例不能启动 手工启动实例报错 ORA-03113 end-of-file on communication channel
- 11.2.0.3 ASM实例出现ORA-4031错误导致数据库归档失败
- standby database 中的登录触发器 导致 ORA-00604 ORA-16000
- rac重新启动遭遇ORA-01078、ORA-01565、ORA-17503、ORA-12547
- 11gR2启动ASM实例时遭遇ORA-29701
- RMAN异机恢复遭遇ORA-01547、ORA-01152、ORA-01110错误案例