Hi all,

We have a 6 node RAC system on Red Hat Linux AS 4. We have a 5 node DRS system on the same O/S. All DRS instances are in standby mode. Only one node is actually performing managed recovery on the DRS. All the instances are ASM manaaged. All dbs are on 10.2.0.3.

On Monday 22nd the standby database was about 500 logs out of
synch with the LIVE database. This had happened because of a combination of the TEMP tablespace reaching its capacity on the LIVE system and the creation of tablespaces for future partitions on the system. The tablespaces where
successfully created on LIVE however the changes were not propagated to the
standby. The standby was complaining that it could not create the tablespaces
whilst it was recovering. This in normal circumstances should happen
automatically. Hence, I created the tablespaces manually and applied the logs.
This was an iterative process as the tablespaces were created on LIVE over 2/3
days. Once the databases were in synch I switched the mode of tablespace
creation on the STANDBY to AUTOMATC (like how it was originally). I then tested
this by creating a TEST tablespace on LIVE and saw that the change was
propagated properly to the standby.

On Wednesday (yesterday) the standby was again about 80 logs behind. Upon investigation, I found that the standby
instance performing the recovery had been terminated by its ASM instance at
2.30 in the morning (Wednesdat). This is the same time the scripts (for
tablespace creation for future partitions) ran on LIVE. Hence, I started the
standby up again and started performing recovery. However, this time the
recovery could not proceed and the alert log complained that it was that it was trying to fetch logs to apply and could not
find them on LIVE. It turns out that these logs are very old (and not on the
system) and the database is already much further ahead in time. Hence, we could not understand why it needed these logs?

Below is a snippet of the alert which shows log 8nnn being applied for thread 2:

Code:
Mon Oct 22 19:06:50 2007
Media Recovery Log +LOG/dbn/archivelog/2007_10_22/thread_1_seq_7428.5860.636662985
Media Recovery Log +LOG/dbn/archivelog/2007_10_22/thread_2_seq_8075.5920.636661525
Media Recovery Log +LOG/dbn/archivelog/2007_10_22/thread_3_seq_7153.5933.636661279
Media Recovery Log +LOG/dbn/archivelog/2007_10_22/thread_4_seq_4319.5936.636661167
Media Recovery Log +LOG/dbn/archivelog/2007_10_22/thread_5_seq_4048.5893.636662001
Media Recovery Log +LOG/dbn/archivelog/2007_10_22/thread_6_seq_4359.5888.636662003
Media Recovery Log +LOG/dbn/archivelog/2007_10_22/thread_4_seq_4320.5862.636662977
Media Recovery Log +LOG/dbn/archivelog/2007_10_22/thread_3_seq_7154.5855.636662985
Mon Oct 22 19:07:11 2007
Media Recovery Log +LOG/dbn/archivelog/2007_10_22/thread_2_seq_8076.5887.636662003
Media Recovery Log +LOG/dbn/archivelog/2007_10_22/thread_2_seq_8077.5853.636662985
Media Recovery Log +LOG/dbn/archivelog/2007_10_22/thread_5_seq_4049.5852.636663785
-----------------------------------------------------------------------------------
Recovery of Online Redo Log: Thread 2 Group 44 Seq 8083 Reading mem 0
  Mem# 0: +LOG/dbn/onlinelog/group_44.1497.627140621
Mon Oct 22 19:57:30 2007
Media Recovery Waiting for thread 3 sequence 7157 (in transit)
Mon Oct 22 19:57:30 2007
Recovery of Online Redo Log: Thread 3 Group 49 Seq 7157 Reading mem 0
  Mem# 0: +LOG/dbn/onlinelog/group_49.1501.627140669
Media Recovery Waiting for thread 4 sequence 4323 (in transit)
Mon Oct 22 19:57:30 2007
Recovery of Online Redo Log: Thread 4 Group 55 Seq 4323 Reading mem 0
  Mem# 0: +LOG/dbn/onlinelog/group_55.1475.627140727
Media Recovery Waiting for thread 6 sequence 4362 (in transit)
Mon Oct 22 19:57:31 2007
Recovery of Online Redo Log: Thread 6 Group 64 Seq 4362 Reading mem 0
  Mem# 0: +LOG/dbn/onlinelog/group_64.1377.627644627
Media Recovery Waiting for thread 5 sequence 4054 (in transit)
Mon Oct 22 19:57:32 2007
Recovery of Online Redo Log: Thread 5 Group 60 Seq 4054 Reading mem 0
  Mem# 0: +LOG/dbn/onlinelog/group_60.1455.627140773
Mon Oct 22 19:58:09 2007
Media Recovery Waiting for thread 1 sequence 7431 (in transit)
Mon Oct 22 19:58:09 2007
Recovery of Online Redo Log: Thread 1 Group 40 Seq 7431 Reading mem 0
  Mem# 0: +LOG/dbn/onlinelog/group_40.1812.627643647
Mon Oct 22 20:14:47 2007
Primary database is in MAXIMUM PERFORMANCE mode
RFS[6]: Successfully opened standby log 45: '+LOG/dbn/onlinelog/group_45.792.627140631'
Mon Oct 22 20:15:09 2007
Media Recovery Waiting for thread 2 sequence 8084 (in transit)
Mon Oct 22 20:15:09 2007
Recovery of Online Redo Log: Thread 2 Group 45 Seq 8084 Reading mem 0
  Mem# 0: +LOG/dbn/onlinelog/group_45.792.627140631
Mon Oct 22 20:35:09 2007
Primary database is in MAXIMUM PERFORMANCE mode
RFS[6]: Successfully opened standby log 44: '+LOG/dbn/onlinelog/group_44.1497.627140621'
Mon Oct 22 20:37:15 2007
Media Recovery Waiting for thread 2 sequence 8085 (in transit)
Mon Oct 22 20:37:15 2007
Recovery of Online Redo Log: Thread 2 Group 44 Seq 8085 Reading mem 0
  Mem# 0: +LOG/dbn/onlinelog/group_44.1497.627140621
Mon Oct 22 20:45:44 2007
Primary database is in MAXIMUM PERFORMANCE mode
RFS[11]: Successfully opened standby log 50: '+LOG/dbn/onlinelog/group_50.1493.627140679'
Mon Oct 22 20:45:46 2007
Primary database is in MAXIMUM PERFORMANCE mode
RFS[6]: Successfully opened standby log 45: '+LOG/dbn/onlinelog/group_45.792.627140631'
Mon Oct 22 20:45:47 2007
Below is the alert log error which is complaining about a gap in the sequence once i started managed recovery(these logs are not in present)

Code:
Thu Oct 25 11:11:19 2007
Completed: alter database recover managed standby database disconnect
Thu Oct 25 11:11:22 2007
Media Recovery Log +LOG/dbn/archivelog/2007_10_24/thread_5_seq_4093.25762.636785
793
Media Recovery Log +LOG/dbn/archivelog/2007_10_24/thread_1_seq_7470.25859.636783
907
Media Recovery Waiting for thread 2 sequence 3226
Fetching gap sequence in thread 2, gap sequence 3226-3325
Thu Oct 25 11:12:01 2007
FAL[client, MRP0]: Error 16099 fetching archived redo log from DBH2
Thu Oct 25 11:12:01 2007
Errors in file /opt/oracle/admin/dbn/bdump/nadc1_mrp0_22853.trc:
ORA-16099: Message 16099 not found; No message file for product=RDBMS, facility=
ORA
Thu Oct 25 11:12:35 2007
FAL[client, MRP0]: Error 16099 fetching archived redo log from DBH2
Thu Oct 25 11:12:35 2007
Errors in file /opt/oracle/admin/dbn/bdump/nadc1_mrp0_22853.trc:
ORA-16099: Message 16099 not found; No message file for product=RDBMS, facility=
ORA
Thu Oct 25 11:13:10 2007
FAL[client, MRP0]: Error 16099 fetching archived redo log from DBH2
Thu Oct 25 11:13:10 2007
Errors in file /opt/oracle/admin/dbn/bdump/nadc1_mrp0_22853.trc:
ORA-16099: Message 16099 not found; No message file for product=RDBMS, facility=
ORA
Thu Oct 25 11:13:58 2007
FAL[client, MRP0]: Error 3113 fetching archived redo log from DBH2
Thu Oct 25 11:13:58 2007
Errors in file /opt/oracle/admin/dbn/bdump/nadc1_mrp0_22853.trc:
ORA-03113: Message 3113 not found; No message file for product=RDBMS, facility=ORA
ORA-03113: Message 3113 not found; No message file for product=RDBMS, facility=ORA
Thu Oct 25 11:15:06 2007
FAL[client, MRP0]: Error 16099 fetching archived redo log from DBH3
Thu Oct 25 11:15:06 2007
Errors in file /opt/oracle/admin/dbn/bdump/nadc1_mrp0_22853.trc:
ORA-16099: Message 16099 not found; No message file for product=RDBMS, facility=ORA
Thu Oct 25 11:16:34 2007
FAL[client, MRP0]: Error 3113 fetching archived redo log from DBH3
Thu Oct 25 11:16:34 2007
Errors in file /opt/oracle/admin/dbn/bdump/nadc1_mrp0_22853.trc:
ORA-03113: Message 3113 not found; No message file for product=RDBMS, facility=ORA
Thu Oct 25 11:19:29 2007
FAL[client]: Failed to request gap sequence
 GAP - thread 2 sequence 3226-3325
 DBID 1445841364 branch 626885844
FAL[client]: All defined FAL servers have been attempted.
-------------------------------------------------------------
Check that the CONTROL_FILE_RECORD_KEEP_TIME initialization
parameter is defined to a value that is sufficiently large
enough to maintain adequate log switch information to resolve
archivelog gaps.
-------------------------------------------------------------
Any advice on this would be highly appreciated.

Thanks in advance
Chucks