11gR2 dataguard 备库文件损坏处理

环境模拟：
主库：

SQL> select DATABASE_ROLE,open_mode from v$database;
DATABASE_ROLE OPEN_MODE
---------------- --------------------
PRIMARY  READ WRITE

备库：

SQL> select DATABASE_ROLE,open_mode from v$database;
DATABASE_ROLE OPEN_MODE
---------------- --------------------
PHYSICAL STANDBY READ ONLY WITH APPLY

2.主库中断日志传输，备库停止日志应用，然后再主库更新数据。

SQL>  ALTER SYSTEM SET log_archive_dest_state_2='reset' SCOPE=BOTH;
System altered.

alert日志可以看到，不再向备库传输日志：

ALTER SYSTEM SET log_archive_dest_state_2='reset' SCOPE=BOTH;
Wed Oct 21 10:41:05 2015
Thread 1 advanced to log sequence 355 (LGWR switch)
Current log# 1 seq# 355 mem# 0: +DATA/phub/onlinelog/group_1.262.890480943
Current log# 1 seq# 355 mem# 1: +DATA/phub/onlinelog/group_1.263.890480945
Wed Oct 21 10:41:05 2015
Archived Log entry 707 added for thread 1 sequence 354 ID 0x1fffdaed dest 1:
Thread 1 cannot allocate new log, sequence 356
Checkpoint not complete
Current log# 1 seq# 355 mem# 0: +DATA/phub/onlinelog/group_1.262.890480943
Current log# 1 seq# 355 mem# 1: +DATA/phub/onlinelog/group_1.263.890480945
Thread 1 advanced to log sequence 356 (LGWR switch)
Current log# 2 seq# 356 mem# 0: +DATA/phub/onlinelog/group_2.264.890480945
Current log# 2 seq# 356 mem# 1: +DATA/phub/onlinelog/group_2.265.890480945
Wed Oct 21 10:41:06 2015
Archived Log entry 708 added for thread 1 sequence 355 ID 0x1fffdaed dest 1:

备库停止日志应用：

SQL> ALTER DATABASE recover managed standby DATABASE cancel;
Database altered.
SQL> select open_mode from v$database;
OPEN_MODE
--------------------
READ ONLY

测试数据：

主库，

SQL> conn scott/scott;
Connected.
SQL> select count(*) from test;
COUNT(*)
----------
87065

备库：
SQL> select count(*) from test;
COUNT(*)
----------
87065
主库更新test表数据：

SQL> delete from test where rownum<1000;
999 rows deleted.
SQL> update test set owner='SCOTT' where object_id<10000;
8812 rows updated.
SQL> insert into test select * from test;
86066 rows created.
SQL> commit;
Commit complete.

SQL> alter system switch logfile;
System altered.
SQL> /
System altered.
SQL> /
System altered.

删除最新归档日志：

ASMCMD [+data/PHUB/ARCHIVELOG/2015_10_21] > rm -rf thread_1_seq_359.544.893674539
ASMCMD [+data/PHUB/ARCHIVELOG/2015_10_21] > rm -rf thread_1_seq_358.543.893674457
ASMCMD [+data/PHUB/ARCHIVELOG/2015_10_21] > rm -rf thread_1_seq_357.542.893674453
ASMCMD [+data/PHUB/ARCHIVELOG/2015_10_21] >

启用主库日志传输：
SQL> ALTER system SET log_archive_dest_state_2 = 'enable';
System altered.
启用备库日志应用：

SQL> conn / as sysdba
Connected.
SQL> ALTER DATABASE recover managed standby DATABASE disconnect FROM SESSION;
Database altered.

查看备库alert日志：
[oracle@dg trace]$ tail -f alert_MECBS.log
Serial Media Recovery started
Managed Standby Recovery not using Real Time Apply
Waiting for all non-current ORLs to be archived...
All non-current ORLs have been archived.
Completed: ALTER DATABASE recover managed standby DATABASE disconnect FROM SESSION
Media Recovery Log +DATA/mecbs/archivelog/2015_10_21/thread_1_seq_354.606.893674715
Media Recovery Log +DATA/mecbs/archivelog/2015_10_21/thread_1_seq_355.607.893674715
Media Recovery Log +DATA/mecbs/archivelog/2015_10_21/thread_1_seq_356.605.893674715
Media Recovery Waiting for thread 1 sequence 357
Fetching gap sequence in thread 1, gap sequence 357-359
Wed Oct 21 11:01:34 2015
FAL[client]: Failed to request gap sequence
GAP - thread 1 sequence 357-359
DBID 536511065 branch 890484819
FAL[client]: All defined FAL servers have been attempted.
------------------------------------------------------------
Check that the CONTROL_FILE_RECORD_KEEP_TIME initialization
parameter is defined to a value that's sufficiently large
enough to maintain adequate log switch information to resolve
archivelog gaps.
------------------------------------------------------------
解决办法：
查询主库scn：

SQL> SELECT SEQUENCE#,FIRST_CHANGE#,NEXT_CHANGE# FROM v$archived_log WHERE SEQUENCE# > 356 ORDER BY 1;
SEQUENCE# FIRST_CHANGE# NEXT_CHANGE#
---------- ------------- ------------
357 5501419      5501424
358 5501424      5501430
359 5501430      5501524
360 5501524      5501782
360 5501524      5501782

2)根据scn，进行rman增量备份

RMAN> backup device type disk incremental from scn 5501419 database format '/home/oracle/data_%U.bak';
Starting backup at 21-OCT-15
using target database control file instead of recovery catalog
allocated channel: ORA_DISK_1
channel ORA_DISK_1: SID=400 device type=DISK
allocated channel: ORA_DISK_2
channel ORA_DISK_2: SID=613 device type=DISK
channel ORA_DISK_1: starting full datafile backup set
channel ORA_DISK_1: specifying datafile(s) in backup set
input datafile file number=00006 name=+DATA/phub/datafile/llc01.dbf
input datafile file number=00003 name=+DATA/phub/datafile/undotbs1.260.891340857
input datafile file number=00004 name=+DATA/phub/datafile/users.269.891340843
channel ORA_DISK_1: starting piece 1 at 21-OCT-15
channel ORA_DISK_2: starting full datafile backup set
channel ORA_DISK_2: specifying datafile(s) in backup set
input datafile file number=00002 name=+DATA/phub/datafile/sysaux.272.891340857
input datafile file number=00007 name=+DATA/phub/datafile/idx01.dbf
input datafile file number=00001 name=+DATA/phub/datafile/system.271.891340857
input datafile file number=00005 name=+DATA/phub/datafile/example.287.891340843
channel ORA_DISK_2: starting piece 1 at 21-OCT-15
channel ORA_DISK_1: finished piece 1 at 21-OCT-15
piece handle=/home/oracle/data_5lqk8pt4_1_1.bak tag=TAG20151021T111028 comment=NONE
channel ORA_DISK_1: backup set complete, elapsed time: 00:01:06
channel ORA_DISK_1: starting full datafile backup set
channel ORA_DISK_1: specifying datafile(s) in backup set
channel ORA_DISK_2: finished piece 1 at 21-OCT-15
piece handle=/home/oracle/data_5mqk8pt4_1_1.bak tag=TAG20151021T111028 comment=NONE
channel ORA_DISK_2: backup set complete, elapsed time: 00:01:05
including current control file in backup set
channel ORA_DISK_1: starting piece 1 at 21-OCT-15
channel ORA_DISK_1: finished piece 1 at 21-OCT-15
piece handle=/home/oracle/data_5nqk8pv6_1_1.bak tag=TAG20151021T111028 comment=NONE
channel ORA_DISK_1: backup set complete, elapsed time: 00:00:01
Finished backup at 21-OCT-15

上传备份文件到备库：

[oracle@cwogg ~]$ scp data_5* 172.16.30.228:/home/oracle/
oracle@172.16.30.228's password:
data_5lqk8pt4_1_1.bak                                                           100% 1248KB   1.2MB/s   00:00
data_5mqk8pt4_1_1.bak                                                           100% 7128KB   7.0MB/s   00:00
data_5nqk8pv6_1_1.bak                                                           100%   10MB  10.2MB/s   00:00

备库上进行recover：

SQL> ALTER DATABASE recover managed standby DATABASE cancel;
Database altered

[oracle@dg ~]$ rman target /
Recovery Manager: Release 11.2.0.4.0 - Production on Wed Oct 21 11:19:03 2015
Copyright (c) 1982, 2011, Oracle and/or its affiliates. All rights reserved.
connected to target database: PHUB (DBID=536511065)
RMAN> catalog start with '/home/oracle/backup/';
using target database control file instead of recovery catalog
searching for all files that match the pattern /home/oracle/backup/
List of Files Unknown to the Database
=====================================
File Name: /home/oracle/backup/data_5lqk8pt4_1_1.bak
File Name: /home/oracle/backup/data_5mqk8pt4_1_1.bak
File Name: /home/oracle/backup/data_5nqk8pv6_1_1.bak
Do you really want to catalog the above files (enter YES or NO)? yes
cataloging files...
cataloging done
List of Cataloged Files
=======================
File Name: /home/oracle/backup/data_5lqk8pt4_1_1.bak
File Name: /home/oracle/backup/data_5mqk8pt4_1_1.bak
File Name: /home/oracle/backup/data_5nqk8pv6_1_1.bak

恢复备库：
[oracle@dg ~]$ sqlplus / as sysdba

SQL*Plus: Release 11.2.0.4.0 Production on Wed Oct 21 11:22:44 2015

Copyright (c) 1982, 2013, Oracle. All rights reserved.

Connected to:
Oracle Database 11g Enterprise Edition Release 11.2.0.4.0 - 64bit Production
With the Partitioning, Automatic Storage Management, OLAP, Data Mining
and Real Application Testing options

SQL> shutdown immediate;
Database closed.
Database dismounted.
ORACLE instance shut down.
SQL> startup mount;
ORACLE instance started.

Total System Global Area 835104768 bytes
Fixed Size 2257840 bytes
Variable Size 541068368 bytes
Database Buffers 289406976 bytes
Redo Buffers 2371584 bytes
Database mounted.
SQL> exit
Disconnected from Oracle Database 11g Enterprise Edition Release 11.2.0.4.0 - 64bit Production
With the Partitioning, Automatic Storage Management, OLAP, Data Mining
and Real Application Testing options
[oracle@dg ~]$ rman target /
Recovery Manager: Release 11.2.0.4.0 - Production on Wed Oct 21 11:23:39 2015
Copyright (c) 1982, 2011, Oracle and/or its affiliates. All rights reserved.
connected to target database: PHUB (DBID=536511065, not open)
RMAN> recover database noredo;
Starting recover at 21-OCT-15
using target database control file instead of recovery catalog
allocated channel: ORA_DISK_1
channel ORA_DISK_1: SID=36 device type=DISK
allocated channel: ORA_DISK_2
channel ORA_DISK_2: SID=37 device type=DISK
channel ORA_DISK_1: starting incremental datafile backup set restore
channel ORA_DISK_1: specifying datafile(s) to restore from backup set
destination for restore of datafile 00001: +DATA/mecbs/datafile/system.259.891103927
destination for restore of datafile 00002: +DATA/mecbs/datafile/sysaux.260.891104071
destination for restore of datafile 00005: +DATA/mecbs/datafile/example.261.891104187
destination for restore of datafile 00007: +DATA/mecbs/datafile/idx.410.891688925
channel ORA_DISK_1: reading from backup piece /home/oracle/backup/data_5mqk8pt4_1_1.bak
channel ORA_DISK_2: starting incremental datafile backup set restore
channel ORA_DISK_2: specifying datafile(s) to restore from backup set
destination for restore of datafile 00003: +DATA/mecbs/datafile/undotbs1.262.891104243
destination for restore of datafile 00004: +DATA/mecbs/datafile/users.263.891104267
destination for restore of datafile 00006: +DATA/mecbs/datafile/llc.258.891103925
channel ORA_DISK_2: reading from backup piece /home/oracle/backup/data_5lqk8pt4_1_1.bak
channel ORA_DISK_1: piece handle=/home/oracle/backup/data_5mqk8pt4_1_1.bak tag=TAG20151021T111028
channel ORA_DISK_1: restored backup piece 1
channel ORA_DISK_1: restore complete, elapsed time: 00:00:03
channel ORA_DISK_2: piece handle=/home/oracle/backup/data_5lqk8pt4_1_1.bak tag=TAG20151021T111028
channel ORA_DISK_2: restored backup piece 1
channel ORA_DISK_2: restore complete, elapsed time: 00:00:03
Finished recover at 21-OCT-15
查看备库alert日志：
Wed Oct 21 11:23:45 2015
Incremental restore complete of datafile 3 +DATA/mecbs/datafile/undotbs1.262.891104243
checkpoint is 5502788
last deallocation scn is 1354205
Wed Oct 21 11:23:45 2015
Incremental restore complete of datafile 5 +DATA/mecbs/datafile/example.261.891104187
checkpoint is 5502790
last deallocation scn is 1114995
Incremental restore complete of datafile 4 +DATA/mecbs/datafile/users.263.891104267
checkpoint is 5502788
last deallocation scn is 3
Incremental restore complete of datafile 1 +DATA/mecbs/datafile/system.259.891103927
checkpoint is 5502790
last deallocation scn is 1095967
Incremental restore complete of datafile 6 +DATA/mecbs/datafile/llc.258.891103925
checkpoint is 5502788
last deallocation scn is 1099825
Incremental restore complete of datafile 7 +DATA/mecbs/datafile/idx.410.891688925
checkpoint is 5502790
Incremental restore complete of datafile 2 +DATA/mecbs/datafile/sysaux.260.891104071
checkpoint is 5502790
last deallocation scn is 994406
开启日志应用：
SQL> ALTER DATABASE recover managed standby DATABASE disconnect FROM SESSION;
Database altered.
主库切换日志：

SQL> ALTER system switch logfile;
System altered.
SQL> /
System altered.
SQL> SELECT MAX(al.SEQUENCE#) "Last Seq Recieved", MAX(lh.SEQUENCE#) "Last Seq Applied" FROM v$archived_log al, v$log_history lh;
Last Seq Recieved Last Seq Applied
----------------- ----------------
363       363

备库：

SQL> SELECT MAX(al.SEQUENCE#) "Last Seq Recieved", MAX(lh.SEQUENCE#) "Last Seq Applied" FROM v$archived_log al, v$log_history lh;

Last Seq Recieved Last Seq Applied
----------------- ----------------
363 356
此时备库仍然显示有gap：

Completed: ALTER DATABASE recover managed standby DATABASE disconnect FROM SESSION
Media Recovery Waiting for thread 1 sequence 357
Fetching gap sequence in thread 1, gap sequence 357-359
Wed Oct 21 11:27:29 2015
RFS[1]: Selected log 4 for thread 1 sequence 363 dbid 536511065 branch 890484819
Wed Oct 21 11:27:29 2015
Archived Log entry 333 added for thread 1 sequence 362 ID 0x1fffdaed dest 1:
Wed Oct 21 11:27:49 2015
RFS[1]: Selected log 5 for thread 1 sequence 364 dbid 536511065 branch 890484819
Wed Oct 21 11:27:49 2015
Archived Log entry 334 added for thread 1 sequence 363 ID 0x1fffdaed dest 1:
Wed Oct 21 11:27:52 2015
FAL[client]: Failed to request gap sequence
GAP - thread 1 sequence 357-359
DBID 536511065 branch 890484819
FAL[client]: All defined FAL servers have been attempted.
------------------------------------------------------------
Check that the CONTROL_FILE_RECORD_KEEP_TIME initialization
parameter is defined to a value that's sufficiently large
enough to maintain adequate log switch information to resolve
archivelog gaps.
------------------------------------------------------------

停止备库日志应用，在重启，
两边数据以及同步了：
SQL> archive log list;
Database log mode Archive Mode
Automatic archival Enabled
Archive destination +DATA
Oldest online log sequence 365
Next log sequence to archive 0
Current log sequence 367

SQL> archive log list;
Database log mode Archive Mode
Automatic archival Enabled
Archive destination +DATA
Oldest online log sequence 365
Next log sequence to archive 367
Current log sequence 367
但是备库alertlog仍然报错：找不到那3个归档文件：

FAL[client]: Failed to request gap sequence
GAP - thread 1 sequence 357-359
DBID 536511065 branch 890484819
FAL[client]: All defined FAL servers have been attempted.
重建备库的控制文件：

SQL> ALTER DATABASE CREATE standby controlfile AS '/tmp/standby.ctl';
Database altered.
[oracle@cwogg tmp]$ scp standby.ctl 172.16.30.228:/home/oracle/
oracle@172.16.30.228's password:
standby.ctl                                                                     100%   10MB  10.1MB/s   00:00

[oracle@dg ~]$ sqlplus / as sysdba
SQL*Plus: Release 11.2.0.4.0 Production on Wed Oct 21 11:40:43 2015
Copyright (c) 1982, 2013, Oracle. All rights reserved.
Connected to an idle instance.
SQL> startup nomount;
ORACLE instance started.
Total System Global Area 835104768 bytes
Fixed Size 2257840 bytes
Variable Size 541068368 bytes
Database Buffers 289406976 bytes
Redo Buffers 2371584 bytes
RMAN> restore controlfile from '/home/oracle/standby.ctl';
Starting restore at 21-OCT-15
using target database control file instead of recovery catalog
allocated channel: ORA_DISK_1
channel ORA_DISK_1: SID=24 device type=DISK
channel ORA_DISK_1: copied control file copy
output file name=+DATA/mecbs/controlfile/control01.ctl
output file name=+DATA/mecbs/controlfile/control02.ctl
Finished restore at 21-OCT-15
RMAN> startup mount;
database is already started
database mounted
released channel: ORA_DISK_1
待续，，，