RMAN incremental backup failures - TSM TDP issues
I get the below errors every once in a while when doing a "level 1 incremental backup" with 2 channels. I have little idea why. It doesn't fail at any consistent point..the failure will sometimes occur at the beginning and other times half way through the backup. I'm at a loss here. Ideas anyone? Below are the RMAN and TSM logs.
Pay attention to "09/22/04 22:10:34 ANE4994S (Session: 41266, Node: DWDEV01_TDPORA) TDP Oracle SUN ANU0599 ANU2602E The object /dwdev//dwdev_incr1_datafiles_537573609_5630_1 was not found on the TSM Server(SESSION: 41266)" in the TSM log.
The backup piece name being written is unique! What the heck is a "not found" supposed to mean! Beats me.
RMAN logfile :
Recovery Manager: Release 9.2.0.4.0 - 64bit Production
Copyright (c) 1995, 2002, Oracle Corporation. All rights reserved.
connected to recovery catalog database
2> run
3> {
4> allocate channel t1 type 'sbt_tape' parms
5>
5> 'ENV=(TDPO_OPTFILE=/opt/tivoli/tsm/client/oracle/bin64/DWDEV/tdpo.opt)';
6> allocate channel t2 type 'sbt_tape' parms
7>
7> 'ENV=(TDPO_OPTFILE=/opt/tivoli/tsm/client/oracle/bin64/DWDEV/tdpo.opt)';
8> BACKUP INCREMENTAL LEVEL = 1 CUMULATIVE
9> tag='DWDEV LVL 1 CUM DB BKUP'
10> format 'dwdev_incr1_datafiles_%t_%s_%p'
11> filesperset 5
12> DATABASE;
13> sql 'alter system archive log current';
14> backup
15> format 'dwdev_incr1_archlog_%t_%s_%p'
16> ARCHIVELOG ALL not backed up 2 times TAG='DWDEV Archivelog bkup';
17> BACKUP format 'dwdev_ctrl_file_%t_%s_%p'
18> current controlfile
19> TAG='DWDEV curr controlfile bkup';
20> release channel t1;
21> release channel t2;
22> }
23>
connected to target database: DWDEV (DBID=294984608)
allocated channel: t1
channel t1: sid=17 devtype=SBT_TAPE
channel t1: Tivoli Data Protection for Oracle: version 5.2.0.0
allocated channel: t2
channel t2: sid=16 devtype=SBT_TAPE
channel t2: Tivoli Data Protection for Oracle: version 5.2.0.0
Starting backup at 22-sep-2004 22:00:06
channel t1: starting incremental level 1 datafile backupset channel t1: specifying datafile(s) in backupset input datafile fno=00050 name=/u14/oradata/DWDEV/opeagg_dat1_03.dbf
input datafile fno=00052 name=/u10/oradata/DWDEV/opeagg_dat1_01.dbf
input datafile fno=00010 name=/u03/oradata/DWDEV/development01.dbf
input datafile fno=00019 name=/u09/oradata/DWDEV/rbsbig1_01.dbf
input datafile fno=00049 name=/u16/oradata/DWDEV/opeagg_dat1_04.dbf
channel t1: starting piece 1 at 22-sep-2004 22:00:09
channel t2: starting incremental level 1 datafile backupset channel t2: specifying datafile(s) in backupset input datafile fno=00051 name=/u13/oradata/DWDEV/opeagg_dat1_02.dbf
input datafile fno=00057 name=/u05/oradata/DWDEV/opeagg_idx1_01.dbf
input datafile fno=00016 name=/u11/oradata/DWDEV/prodrbs1_01.dbf
input datafile fno=00053 name=/u14/oradata/DWDEV/opeagg_dat2_02.dbf
input datafile fno=00061 name=/u10/oradata/DWDEV/ope_dat1_03.dbf
channel t2: starting piece 1 at 22-sep-2004 22:00:09
channel t2: finished piece 1 at 22-sep-2004 22:06:44
piece handle=dwdev_incr1_datafiles_537573609_5631_1 comment=API Version 2.0,MMS Version 5.2.0.0 channel t2: backup set complete, elapsed time: 00:06:35 channel t2: starting incremental level 1 datafile backupset channel t2: specifying datafile(s) in backupset input datafile fno=00055 name=/u02/oradata/DWDEV/opeagg_dat2_03.dbf
input datafile fno=00031 name=/u15/oradata/DWDEV/category_dat1_01.dbf
input datafile fno=00047 name=/u15/oradata/DWDEV/opeaggarch_dat1_01.dbf
input datafile fno=00058 name=/u03/oradata/DWDEV/opeagg_idx1_03.dbf
input datafile fno=00045 name=/u09/oradata/DWDEV/opeagg2004_dat1_01.dbf
channel t2: starting piece 1 at 22-sep-2004 22:06:44
released channel: t1
released channel: t2
RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-03009: failure of backup command on t1 channel at 09/22/2004 22:07:29
ORA-27206: requested file not found in media management catalog
ORA-19502: write error on file "dwdev_incr1_datafiles_537573609_5630_1", blockno 513 (blocksize=512)
ORA-27030: skgfwrt: sbtwrite2 returned error
ORA-19511: Error received from media manager layer, error text:
ANS1017E (RC-50) Session rejected: TCP/IP connection failure
Recovery Manager complete.
----------------------
TSM logs
09/22/04 22:10:34 ANR0406I Session 41265 started for node DWDEV01_TDPORA (TDP Oracle SUN) (Tcp/Ip 172.16.0.25(35081)). (SESSION: 41265)
09/22/04 22:10:34 ANR0403I Session 41265 ended for node DWDEV01_TDPORA (TDP Oracle SUN). (SESSION: 41265)
09/22/04 22:10:34 ANR0406I Session 41266 started for node DWDEV01_TDPORA (TDP Oracle SUN) (Tcp/Ip 172.16.0.25(35082)). (SESSION: 41266)
09/22/04 22:10:34 ANE4994S (Session: 41266, Node: DWDEV01_TDPORA) TDP Oracle SUN ANU0599 ANU2602E The object /dwdev//dwdev_incr1_datafiles_537573609_5630_1 was not found on the TSM Server(SESSION: 41266)
09/22/04 22:10:34 ANR0403I Session 41266 ended for node DWDEV01_TDPORA (TDP Oracle SUN). (SESSION: 41266)
09/22/04 22:10:34 ANR0403I Session 41260 ended for node DWDEV01_TDPORA (TDP Oracle SUN). (SESSION: 41260)
