Work around ORA-10567,ORA-10561, ORA-600: [3020], [3 ] - Oracle 10gr2 RAC on Standby

Standby alertlog file Error -

 

Media Recovery Log /u03/fra/BMSPROD_PD/archivelog/2013_09_11/o1_mf_2_59982_931w1f3d_.arc

Media Recovery Log /u03/fra/BMSPROD_PD/archivelog/2013_09_11/o1_mf_1_60283_931w1o8z_.arc

Wed Sep 11 16:27:34 2013

Errors in file /u01/app/oracle/admin/bmsprod/bdump/bmsprod_p011_7930.trc:

ORA-00600: internal error code, arguments: [3020], [31], [1424770], [131448194], [], [], [], []

ORA-10567: Redo is inconsistent with data block (file# 31, block# 1424770)

ORA-10564: tablespace ARSYSTEM

ORA-01110: data file 31: '/u03/oradata/BMSPROD_PD/datafile/ARSYSTEM_31.dbf'

ORA-10561: block type 'TRANSACTION MANAGED DATA BLOCK', data object# 212538

Wed Sep 11 16:27:36 2013

Errors in file /u01/app/oracle/admin/bmsprod/bdump/bmsprod_p011_7930.trc:

ORA-00600: internal error code, arguments: [3020], [31], [1424770], [131448194], [], [], [], []

ORA-10567: Redo is inconsistent with data block (file# 31, block# 1424770)

ORA-10564: tablespace ARSYSTEM

Take the RMAN backup of particular datafile.

On Primary -

$ rman target /

Recovery Manager: Release 10.2.0.3.0 - Production on Thu Sep 12 00:12:02 2013

Copyright (c) 1982, 2005, Oracle.  All rights reserved.

connected to target database: BMSPROD (DBID=3750301684)

RMAN> run {

allocate channel t1 device type disk;

allocate channel t2 device type disk;

backup datafile 31  FORMAT '/u04/backup/bmsprod/incr/datafile_31_%U';

}

2> 3> 4> 5>

using target database control file instead of recovery catalog

allocated channel: t1

channel t1: sid=402 instance=bmsprod1 devtype=DISK

allocated channel: t2

channel t2: sid=351 instance=bmsprod1 devtype=DISK

Starting backup at 12-SEP-13

channel t1: starting full datafile backupset

channel t1: specifying datafile(s) in backupset

input datafile fno=00031 name=+DATA/bmsprod_od/datafile/arsystem.724.808399285

channel t1: starting piece 1 at 12-SEP-13

channel t1: finished piece 1 at 12-SEP-13

piece handle=/u04/backup/bmsprod/incr/datafile_31_4iojkdb0_1_1 tag=TAG20130912T001216 comment=NONE

channel t1: backup set complete, elapsed time: 00:06:05

Finished backup at 12-SEP-13

released channel: t1

released channel: t2

RMAN> exit

Copy the backup file to standby.

On standby -

$ rman target /

Recovery Manager: Release 10.2.0.3.0 - Production on Thu Sep 12 01:18:34 2013

Copyright (c) 1982, 2005, Oracle.  All rights reserved.

connected to target database: BMSPROD (DBID=3750301684, not open)

RMAN> catalog backuppiece '/u03/incre/datafile_31_4iojkdb0_1_1';

using target database control file instead of recovery catalog

cataloged backuppiece

backup piece handle=/u03/incre/datafile_31_4iojkdb0_1_1 recid=6521 stamp=825902332

RMAN> RESTORE DATAFILE 31

2> ;

Starting restore at 12-SEP-13

allocated channel: ORA_DISK_1

channel ORA_DISK_1: sid=140 devtype=DISK

channel ORA_DISK_1: starting datafile backupset restore

channel ORA_DISK_1: specifying datafile(s) to restore from backup set

restoring datafile 00031 to /u03/oradata/BMSPROD_PD/datafile/ARSYSTEM_31.dbf

channel ORA_DISK_1: reading from backup piece /u03/incre/datafile_31_4iojkdb0_1_1

channel ORA_DISK_1: restored backup piece 1

piece handle=/u03/incre/datafile_31_4iojkdb0_1_1 tag=TAG20130912T001216

channel ORA_DISK_1: restore complete, elapsed time: 00:13:06

Finished restore at 12-SEP-13

RMAN> exit

Recovery Manager complete.

$ sqlplus "/as sysdba"

SQL*Plus: Release 10.2.0.3.0 - Production on Thu Sep 12 01:35:06 2013

Copyright (c) 1982, 2006, Oracle.  All Rights Reserved.

Connected to:

Oracle Database 10g Enterprise Edition Release 10.2.0.3.0 - 64bit Production

With the Partitioning, OLAP and Data Mining options

idle> ALTER DATABASE RECOVER MANAGED STANDBY DATABASE DISCONNECT FROM SESSION;

Database altered.

Archive log will start applying on standby Site.

 

In Standby alertlog file –

Thu Sep 12 01:33:59 2013

Full restore complete of datafile 31 /u03/oradata/BMSPROD_PD/datafile/ARSYSTEM_31.dbf.  Elapsed time: 0:13:02

  checkpoint is 12761509036160

  last deallocation scn is 12761472787611

Thu Sep 12 01:35:24 2013

ALTER DATABASE RECOVER MANAGED STANDBY DATABASE DISCONNECT FROM SESSION

Thu Sep 12 01:35:24 2013