ORACLE 使用rman备份通过restore、recover恢复standby库ORA-10877实战

1、备库standby异常报错

昨天凌晨磁盘空间突然暴涨导致oracle备库异常，报警，后了过来清理掉磁盘的备份文件，去了之后，归档日志能同步过来了，但是启动备库standby发现mrp没有启动，后台报错如下，google了，说这种情况要重新再做备库（不知道是否还有其它修复办法呢？）：错误日志报错信息如下：

Completed: alter database recover managed standby database using current logfile disconnect from session
Wed May 31 10:44:48 2017
Dumping diagnostic data in directory=[cdmp_20170531104448], requested by (instance=1, osid=43411 (PR00)), summary=[incident=600239].
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
Errors with log /oracle/app/oracle/archivelogs/1_18799_940127349.dbf
MRP0: Background Media Recovery terminated with error 600
Errors in file /oracle/app/oracle/diag/rdbms/powerdes_s1/powerdes/trace/powerdes_pr00_43411.trc:
ORA-00600: internal error code, arguments: [2619], [18799], [], [], [], [], [], [], [], [], [], []
Managed Standby Recovery not using Real Time Apply
Recovery interrupted!
Errors in file /oracle/app/oracle/diag/rdbms/powerdes_s1/powerdes/trace/powerdes_pr00_43411.trc:
ORA-00600: internal error code, arguments: [2619], [18799], [], [], [], [], [], [], [], [], [], []
Errors in file /oracle/app/oracle/diag/rdbms/powerdes_s1/powerdes/trace/powerdes_mrp0_43408.trc  (incident=600231):
ORA-00600: internal error code, arguments: [2619], [18799], [], [], [], [], [], [], [], [], [], []
ORA-10877: error signaled in parallel recovery slave
ORA-10877: error signaled in parallel recovery slave
ORA-10877: error signaled in parallel recovery slave
ORA-10877: error signaled in parallel recovery slave
ORA-10877: error signaled in parallel recovery slave

2、使用duplicate target 进行恢复备库

duplicate target database for standby nofilenamecheck dorecover;

Starting Duplicate Db at 05-JUN-17
release channel c1;
release channel c2;
using target database control file instead of recovery catalog
allocated channel: ORA_AUX_DISK_1
channel ORA_AUX_DISK_1: SID=4708 device type=DISK

contents of Memory Script:
{
set until scn  14430447592;
restore clone standby controlfile;
}
executing Memory Script

executing command: SET until clause

Starting restore at 05-JUN-17
using channel ORA_AUX_DISK_1

channel ORA_AUX_DISK_1: starting datafile backup set restore
channel ORA_AUX_DISK_1: restoring control file
channel ORA_AUX_DISK_1: reading from backup piece /oracle/backup/data/controlfiles/c-3391761643-20170605-00
channel ORA_AUX_DISK_1: ORA-19870: error while restoring backup piece /oracle/backup/data/controlfiles/c-3391761643-20170605-00
ORA-19505: failed to identify file "/oracle/backup/data/controlfiles/c-3391761643-20170605-00"
ORA-27037: unable to obtain file status
Linux-x86_64 Error: 2: No such file or directory
Additional information: 3

这里恢复失败，google了下，没有找到合适的办法，咋办呢？需要换种思路去恢复了，duplicate此路不通了。

3、尝试使用最原始的rman备份直接restore、recover的方式

copy控制文件、备份文件，备份文件已经有了，所以不需要再次生成，但是控制文件需要再次生成一下，于是在主库上生成最新的备份的控制文件：

RMAN> backup current controlfile for standby format '/home/oracle/ctlfile4.bak';

Starting backup at 05-JUN-17
using target database control file instead of recovery catalog
allocated channel: ORA_DISK_1
channel ORA_DISK_1: SID=948 device type=DISK
channel ORA_DISK_1: starting full datafile backup set
channel ORA_DISK_1: specifying datafile(s) in backup set
including standby control file in backup set
channel ORA_DISK_1: starting piece 1 at 05-JUN-17
channel ORA_DISK_1: finished piece 1 at 05-JUN-17
piece handle=/home/oracle/ctlfile4.bak tag=TAG20170605T223329 comment=NONE
channel ORA_DISK_1: backup set complete, elapsed time: 00:00:01
Finished backup at 05-JUN-17

Starting Control File and SPFILE Autobackup at 05-JUN-17
piece handle=/oracle/backup/data/controlfiles/c-3391761643-20170605-02 comment=NONE
Finished Control File and SPFILE Autobackup at 05-JUN-17

RMAN>

scp将控制文件和备份文件copy到备库上去

scp -r 2017-06-05 101.251.31.131:/oracle/
scp /home/oracle/ctlfile4.bak 101.251.31.131:/home/oracle/

接下来基本全在备库上进行操作，进行restore standby controlfile恢复控制文件

RMAN> restore standby controlfile from '/home/oracle/ctlfile4.bak';

Starting restore at 05-JUN-17
using target database control file instead of recovery catalog
allocated channel: ORA_DISK_1
channel ORA_DISK_1: SID=4708 device type=DISK

channel ORA_DISK_1: restoring control file
channel ORA_DISK_1: restore complete, elapsed time: 00:00:01
output file name=/oracle/app/oracle/oradata/powerdes/control01.ctl
output file name=/oracle/app/oracle/fast_recovery_area/powerdes/control02.ctl
Finished restore at 05-JUN-17

RMAN>

启动到mount状态

RMAN> alter database mount;

database mounted
released channel: ORA_DISK_1

RMAN

注册备份集合

RMAN> catalog start with '/oracle/2017-06-05';

Starting implicit crosscheck backup at 05-JUN-17
allocated channel: ORA_DISK_1
channel ORA_DISK_1: SID=4709 device type=DISK
Crosschecked 147 objects
Finished implicit crosscheck backup at 05-JUN-17

Starting implicit crosscheck copy at 05-JUN-17
using channel ORA_DISK_1
Crosschecked 2 objects
Finished implicit crosscheck copy at 05-JUN-17

searching for all files in the recovery area
cataloging files...
no files cataloged

searching for all files that match the pattern /oracle/2017-06-05

List of Files Unknown to the Database
=====================================
File Name: /oracle/2017-06-05/arch_POWERDES_20170605_9901.bak
File Name: /oracle/2017-06-05/arch_POWERDES_20170605_9899.bak
File Name: /oracle/2017-06-05/arch_POWERDES_20170605_9910.bak
File Name: /oracle/2017-06-05/arch_POWERDES_20170605_9912.bak
File Name: /oracle/2017-06-05/rman_backup.log
File Name: /oracle/2017-06-05/arch_POWERDES_20170605_9908.bak
File Name: /oracle/2017-06-05/arch_POWERDES_20170605_9904.bak
File Name: /oracle/2017-06-05/arch_POWERDES_20170605_9905.bak
File Name: /oracle/2017-06-05/arch_POWERDES_20170605_9906.bak
File Name: /oracle/2017-06-05/arch_POWERDES_20170605_9909.bak
File Name: /oracle/2017-06-05/arch_POWERDES_20170605_9903.bak
File Name: /oracle/2017-06-05/arch_POWERDES_20170605_9902.bak
File Name: /oracle/2017-06-05/full_POWERDES_20170605_9911.bak
File Name: /oracle/2017-06-05/arch_POWERDES_20170605_9900.bak
File Name: /oracle/2017-06-05/arch_POWERDES_20170605_9907.bak

Do you really want to catalog the above files (enter YES or NO)? YES
cataloging files...
cataloging done

List of Cataloged Files
=======================
File Name: /oracle/2017-06-05/arch_POWERDES_20170605_9901.bak
File Name: /oracle/2017-06-05/arch_POWERDES_20170605_9899.bak
File Name: /oracle/2017-06-05/arch_POWERDES_20170605_9910.bak
File Name: /oracle/2017-06-05/arch_POWERDES_20170605_9912.bak
File Name: /oracle/2017-06-05/arch_POWERDES_20170605_9908.bak
File Name: /oracle/2017-06-05/arch_POWERDES_20170605_9904.bak
File Name: /oracle/2017-06-05/arch_POWERDES_20170605_9905.bak
File Name: /oracle/2017-06-05/arch_POWERDES_20170605_9906.bak
File Name: /oracle/2017-06-05/arch_POWERDES_20170605_9909.bak
File Name: /oracle/2017-06-05/arch_POWERDES_20170605_9903.bak
File Name: /oracle/2017-06-05/arch_POWERDES_20170605_9902.bak
File Name: /oracle/2017-06-05/full_POWERDES_20170605_9911.bak
File Name: /oracle/2017-06-05/arch_POWERDES_20170605_9900.bak
File Name: /oracle/2017-06-05/arch_POWERDES_20170605_9907.bak

List of Files Which Where Not Cataloged
=======================================
File Name: /oracle/2017-06-05/rman_backup.log
RMAN-07517: Reason: The file header is corrupted

RMAN>

博客作者黄杉（mchdba），博客源地址：http://blog.csdn.net/mchdba/article/details/72874642，谢绝转载。

开始restore database操作

RMAN> restore database;

Starting restore at 05-JUN-17
using channel ORA_DISK_1

channel ORA_DISK_1: starting datafile backup set restore
channel ORA_DISK_1: specifying datafile(s) to restore from backup set
channel ORA_DISK_1: restoring datafile 00001 to /home/oradata/powerdes/system01.dbf
channel ORA_DISK_1: restoring datafile 00002 to /home/oradata/powerdes/sysaux01.dbf
channel ORA_DISK_1: restoring datafile 00003 to /home/oradata/powerdes/undotbs01.dbf
channel ORA_DISK_1: restoring datafile 00004 to /home/oradata/powerdes/users01.dbf
channel ORA_DISK_1: restoring datafile 00005 to /home/oradata/powerdes/powerdesk01.dbf
channel ORA_DISK_1: restoring datafile 00006 to /home/oradata/powerdes/plas01.dbf
channel ORA_DISK_1: restoring datafile 00007 to /home/oradata/powerdes/pl01.dbf
channel ORA_DISK_1: restoring datafile 00008 to /home/oradata/powerdes/help01.dbf
channel ORA_DISK_1: restoring datafile 00009 to /home/oradata/powerdes/adobelc01.dbf
channel ORA_DISK_1: restoring datafile 00010 to /home/oradata/powerdes/sms01.dbf
channel ORA_DISK_1: restoring datafile 00011 to /home/oradata/powerdes/plcrm01.dbf
channel ORA_DISK_1: restoring datafile 00012 to /home/oradata/powerdes/powerdesk02.dbf
channel ORA_DISK_1: restoring datafile 00013 to /home/oradata/powerdes/datagm01.dbf
channel ORA_DISK_1: restoring datafile 00014 to /home/oradata/powerdes/plimp01.DBF
channel ORA_DISK_1: restoring datafile 00015 to /home/oradata/powerdes/dwetl01.DBF
channel ORA_DISK_1: restoring datafile 00016 to /home/oradata/powerdes/dw02.DBF
channel ORA_DISK_1: restoring datafile 00017 to /home/oradata/powerdes/timdba01.DBF
channel ORA_DISK_1: restoring datafile 00018 to /home/oradata/powerdes/users02.dbf
channel ORA_DISK_1: restoring datafile 00019 to /home/oradata/powerdes/system02.dbf
channel ORA_DISK_1: restoring datafile 00020 to /home/oradata/powerdes/powerdesk03.dbf
channel ORA_DISK_1: restoring datafile 00021 to /home/oradata/powerdes/timdba02.dbf
channel ORA_DISK_1: reading from backup piece /oracle/2017-06-05/full_POWERDES_20170605_9911.bak

channel ORA_DISK_1: piece handle=/oracle/2017-06-05/full_POWERDES_20170605_9911.bak tag=TAG20170605T033303
channel ORA_DISK_1: restored backup piece 1
channel ORA_DISK_1: restore complete, elapsed time: 01:29:37
Finished restore at 05-JUN-17

RMAN>

开始rocover database操作

RMAN> recover database;

Starting recover at 05-JUN-17
using channel ORA_DISK_1

starting media recovery

archived log for thread 1 with sequence 20421 is already on disk as file /oracle/app/oracle/archivelogs/1_20421_940127349.dbf
archived log for thread 1 with sequence 20422 is already on disk as file /oracle/app/oracle/archivelogs/1_20422_940127349.dbf
archived log for thread 1 with sequence 20423 is already on disk as file /oracle/app/oracle/archivelogs/1_20423_940127349.dbf
archived log for thread 1 with sequence 20424 is already on disk as file /oracle/app/oracle/archivelogs/1_20424_940127349.dbf
archived log for thread 1 with sequence 20425 is already on disk as file /oracle/app/oracle/archivelogs/1_20425_940127349.dbf
archived log for thread 1 with sequence 20426 is already on disk as file /oracle/app/oracle/archivelogs/1_20426_940127349.dbf
archived log for thread 1 with sequence 20427 is already on disk as file /oracle/app/oracle/archivelogs/1_20427_940127349.dbf
archived log for thread 1 with sequence 20428 is already on disk as file /oracle/app/oracle/archivelogs/1_20428_940127349.dbf
archived log for thread 1 with sequence 20429 is already on disk as file /oracle/app/oracle/archivelogs/1_20429_940127349.dbf
archived log for thread 1 with sequence 20430 is already on disk as file /oracle/app/oracle/archivelogs/1_20430_940127349.dbf
archived log for thread 1 with sequence 20431 is already on disk as file /oracle/app/oracle/archivelogs/1_20431_940127349.dbf
archived log for thread 1 with sequence 20432 is already on disk as file /oracle/app/oracle/archivelogs/1_20432_940127349.dbf
archived log for thread 1 with sequence 20433 is already on disk as file /oracle/app/oracle/archivelogs/1_20433_940127349.dbf
archived log for thread 1 with sequence 20434 is already on disk as file /oracle/app/oracle/archivelogs/1_20434_940127349.dbf
channel ORA_DISK_1: starting archived log restore to default destination
channel ORA_DISK_1: restoring archived log
archived log thread=1 sequence=20357
channel ORA_DISK_1: restoring archived log
archived log thread=1 sequence=20358
channel ORA_DISK_1: restoring archived log
archived log thread=1 sequence=20359
channel ORA_DISK_1: restoring archived log
archived log thread=1 sequence=20360
channel ORA_DISK_1: reading from backup piece /oracle/2017-06-05/arch_POWERDES_20170605_9912.bak
channel ORA_DISK_1: piece handle=/oracle/2017-06-05/arch_POWERDES_20170605_9912.bak tag=TAG20170605T051030
channel ORA_DISK_1: restored backup piece 1
channel ORA_DISK_1: restore complete, elapsed time: 00:00:15
archived log file name=/oracle/app/oracle/archivelogs/1_20357_940127349.dbf thread=1 sequence=20357
archived log file name=/oracle/app/oracle/archivelogs/1_20358_940127349.dbf thread=1 sequence=20358
archived log file name=/oracle/app/oracle/archivelogs/1_20359_940127349.dbf thread=1 sequence=20359
archived log file name=/oracle/app/oracle/archivelogs/1_20360_940127349.dbf thread=1 sequence=20360
unable to find archived log
archived log thread=1 sequence=20361
RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-03002: failure of recover command at 06/05/2017 22:17:07
RMAN-06054: media recovery requesting unknown archived log for thread 1 with sequence 20361 and starting SCN of 14425991740

RMAN>  recover database until scn 14425991740;

Starting recover at 05-JUN-17
using channel ORA_DISK_1

starting media recovery
Oracle Error:
ORA-01547: warning: RECOVER succeeded but OPEN RESETLOGS would get error below
ORA-01152: file 1 was not restored from a sufficiently old backup
ORA-01110: data file 1: '/home/oradata/powerdes/system01.dbf'

media recovery complete, elapsed time: 00:00:01

Finished recover at 05-JUN-17

RMAN>

最后开始启动传输并且应用归档日志

SQL> alter database recover managed standby database using current logfile disconnect from session;

Database altered.

SQL>

4、开始检查主库备库数据一致性

查看归档日志应用情况：

SQL> select sequence#,applied from v$archived_log order by sequence# asc;

SEQUENCE# APPLIED
---------- ---------
20379 YES
20380 YES
20381 YES
20382 YES
20383 YES
20384 YES
20385 YES
20386 YES
20387 YES
20388 YES
20389 YES

SEQUENCE# APPLIED
---------- ---------
20390 YES
20391 YES
20392 YES
20393 YES
20394 YES
20395 YES
20396 YES
20397 YES
20398 NO
20399 NO
20400 NO

查看后台alert日志，正在进行应用

Media Recovery Log /oracle/app/oracle/archivelogs/1_20391_940127349.dbf
Mon Jun 05 22:24:00 2017
Media Recovery Log /oracle/app/oracle/archivelogs/1_20392_940127349.dbf
Mon Jun 05 22:24:10 2017
Media Recovery Log /oracle/app/oracle/archivelogs/1_20393_940127349.dbf
Mon Jun 05 22:24:22 2017
Media Recovery Log /oracle/app/oracle/archivelogs/1_20394_940127349.dbf
Mon Jun 05 22:24:33 2017
Media Recovery Log /oracle/app/oracle/archivelogs/1_20395_940127349.dbf
Mon Jun 05 22:24:47 2017
Media Recovery Log /oracle/app/oracle/archivelogs/1_20396_940127349.dbf
Mon Jun 05 22:25:00 2017
Media Recovery Log /oracle/app/oracle/archivelogs/1_20397_940127349.dbf

查看备库主库log信息是保持一致的，如下所示：

SQL> archive log list;
Database log mode          Archive Mode
Automatic archival         Enabled
Archive destination        /oracle/app/oracle/archivelogs
Oldest online log sequence     20430
Next log sequence to archive   20435
Current log sequence           20435
SQL>
SQL> archive log list;
Database log mode          Archive Mode
Automatic archival         Enabled
Archive destination        /oracle/app/oracle/archivelogs
Oldest online log sequence     20430
Next log sequence to archive   0
Current log sequence           20435
SQL>

至此，利用rman备份直接restore、recover的方式恢复了dataguard的standby库。

仔细思考下，这个问题以前一直没有遇到过，磁盘满了后，按照道理来说，清理磁盘后，可以直接恢复得，但是却不能恢复，原因猜测在于启用了实时应用归档日志导致。这样就导致了重新再次传输加载的时候，混乱了。去查看以前的dataguard搭建记录http://blog.csdn.net/mchdba/article/details/71036947，发现有类似的操作记录：

（2）添加current logfile参数，使得应用当前正在读写，还没有完成归档的日志

SQL> alter database recover managedstandby database using current logfile disconnect from session;