database session terminated unexpectedly RMAN 10038

[ndisang]

Hi out there;
I am on OPS. AIX 4.3 with tivoli/tsm as storage manager.
I want to restore from cluster A to B. I have the backup of A. No problem. Now when I run the restore it fails about 3/4 through with one infamous error "database session terminated unexpectedly for channel ....". No ORA- message. The listener is fine. I turned RMAN debugging on and below is the output as well.
Can someone help?



ERROR message
################
restoring datafile 00395 to /dev/rprod_1024_004
=====================================================
=============== ERROR MESSAGE STACK FOLLOWS ==========
=======================================================
fatal error in recovery manager
fatal error during execution of command
database session for channel t1 terminated unexpectedly





#################################################
RESTORE SCRIPT (NB fails about halfway through)
#################################################
run
{
allocate channel t1 type 'sbt_tape' connect internal/xxxx@xxxx parms
'ENV=(
#DSMI_DIR=/usr/tivoli/tsm/client/ba/bin,
#DSMI_ORC_CONFIG=/tsm/cfg/ccs1_dsm.opt,
#DSMI_ORC_CONFIG=/usr/tivoli/tsm/client/ba/bin/ccs1_oracle.opt,
DSMO_NODE=ccsp1,
DSMO_DEBUG=49,
DSMO_PSWDPATH=/usr/tivoli/tsm/client/oracle/bin)';
#DSMI_LOG=/usr/tivoli/tsm/client/ba/bin)';
restore controlfile;
sql 'alter database mount shared';
restore database;
#recover database;
#sql 'alter database open';
release channel t1;
}

###############################################
DEBUG Trace from RMAN at time session ends
################################################

EXEC SQL AT TARGET declare vsn varchar2 ( 20 ) ; begin vsn := dbms_rcvman . getP
ackageVersion ; kg_vsnkg_vsn_i := vsn ; if vsn is not null then kg_vsnub4
:= to_number ( substr ( vsn , 1 , 2 ) || substr ( vsn , 4 , 2 ) || substr ( vsn
, 7 , 2 ) ) ; end if ; end ;
sqlcode=0
krmxr: callback returned TRUE, skipping sleep
krmxpoq: xc=539433176, action="0000876 STARTED", col_l=15, ind=0, sid=10
krmxr: sleeping for 1 seconds
krmxpoq: xc=539433176, action="0000876 STARTED", col_l=15, ind=0, sid=10
krmxr: sleeping for 2 seconds
krmxpoq: xc=539433176, action="0000876 STARTED", col_l=15, ind=0, sid=10
krmxr: sleeping for 4 seconds
krmxpoq: xc=539433176, action="0000876 STARTED", col_l=15, ind=0, sid=10
krmxr: sleeping for 8 seconds
krmxpoq: xc=539433176, action="0000876 STARTED", col_l=15, ind=0, sid=10
krmxr: sleeping for 10 seconds
krmxpoq: xc=539433176, action="0000876 STARTED", col_l=15, ind=0, sid=10
krmxr: sleeping for 10 seconds
krmxpoq: xc=539433176, action="0000876 STARTED", col_l=15, ind=0, sid=10
krmxr: sleeping for 10 seconds
krmxpoq: xc=539433176, action="0000876 STARTED", col_l=15, ind=0, sid=10

[sambavan]

Here are some questions on this?

Is your ORACLE_HOME environment configured correctly. LD_LIBRARY_PATH, and etc.

IS the libraries like libobk linked properly to the /usr/lib/libobk.so

Did you check your alert_SID.log for any ORA messages or any trace generations

Can you run the rman in the debug mode

    rman target ... catalog ... debug trace /path/filename.log

and check for what exactly happens with the RMAN.

Did you check the sbtio.log normally generated under $ORACLE_BASE/admin/SID/udump

Consult tivoli, oracle will not help you with the third party backup softwares.

I notice that you have posted a few messages at oracle support but hope it doesn't go on vein Contact Tivoli first and they would help you to figure out. At times you have to be patient to get the level 2 support woh would be most informed about the issues. Take it easy and contact Tivoli.

Sam

[ndisang]

thanks
out there. My libraries PATH is fine. and everything else is OK.
There are some suggestions from Oracle site that this may have to do with the ADSM server running out of shared memory. Any comment?
The piece of junk code in my post is the debug file of RMAN (with the 'sleeping seconds'). It quite big so I just pasted the last lines here.
Posted below are some excepts from the sbtio.log file
Thanks
Tony

(64622) OBK-sbt:<12/14/2001:13:22:21> ReadPswdFromFile():/usr/tivoli/tsm/client/
oracle/bin/TDPO.ccsp1 DSMO_FS=tsmob ANS0282E (RC168) Password file is not a
vailable.

(64622) OBK-sbt:<12/14/2001:13:22:21> odsmSess(): psReadPswd(): oer = 7011, errn
o = 168.
(64622) OBK-sbt:<12/14/2001:13:22:21> sbtopen(): sbtopen exit. odsmSess failed.
rc = -1
(23080) OBK-sbt:<12/14/2001:13:31:44> odsmSess(): # of dsmInit retries = 1
(23080) OBK-sbt:<12/14/2001:13:31:44> queryObj(): ANS1302E (RC2) No objects
on server match query

(23080) OBK-sbt:<12/14/2001:13:31:44> sbtopen(): ANS1302E (RC2) No objects o
n server match query
140308) OBK-sbt:<12/24/2001:14:47:25> odsmSess(): # of dsmInit retries = 1
140308) OBK-sbt:<12/24/2001:14:49:25> sbtread(): End of file reached. oer = 706
, errno = 2505.
140308) OBK-sbt:<12/24/2001:14:49:31> odsmSess(): # of dsmInit retries = 1
140308) OBK-sbt:<12/24/2001:14:52:41> sbtread(): End of file reached. oer = 706
, errno = 2505.
140308) OBK-sbt:<12/24/2001:14:52:47> odsmSess(): # of dsmInit retries = 1
140308) OBK-sbt:<12/24/2001:14:55:31> sbtread(): End of file reached. oer = 706
, errno = 2505.

[sambavan]

First check your password file on the two servers ...

(64622) OBK-sbt:<12/14/2001:13:22:21> ReadPswdFromFile():/usr/tivoli/tsm/client/
oracle/bin/TDPO.ccsp1 DSMO_FS=tsmob ANS0282E (RC168) Password file is not available.

Then contact Tivoli. Oracle will not be able to help you with this ...

Sam

[ndisang]

Sam,
I will follow your suggestions and see where I get.
Thanks
Tony