Unable to start crs in a 10g RAC

Hi all,

I have a 2 node 10g RAC running on AIX 5.3 with ASM. Recently i noticed the size (originally 33GB) of my /u01 directory growing very fast & reaching 100%. My backups started failing & after some researching found a trace file over 20G in size. I deleted this file & bounced the database. However, when i did a quick check of my database, i found out that the crs was down. For the past two days i've been unable to start the crs. A query of all my RAC resources using both srvctl & crs_stat (shown below) shows that both listener, ASM & database are down on this server. However sqlplus tells me the contrary (shown below).

(oracle@pgoratt1) /home/oracle > srvctl status db -d pegatt
Instance pegatt1 is not running on node pgoratt1
Instance pegatt2 is running on node pgoratt2
(oracle@pgoratt1) /home/oracle >

(oracle@pgoratt1) /u01/crs/oracle/product/10.2.0/bin > crsstat
HA Resource Target State
----------- ------ -----
ora.pegatt.db ONLINE ONLINE on pgoratt2
ora.pegatt.pegatt1.inst ONLINE OFFLINE
ora.pegatt.pegatt2.inst ONLINE ONLINE on pgoratt2
ora.pgoratt1.ASM1.asm ONLINE OFFLINE
ora.pgoratt1.LISTENER_PGORATT1.lsnr ONLINE OFFLINE
ora.pgoratt1.gsd ONLINE OFFLINE
ora.pgoratt1.ons ONLINE OFFLINE
ora.pgoratt1.vip ONLINE ONLINE on pgoratt1
ora.pgoratt2.ASM2.asm ONLINE ONLINE on pgoratt2
ora.pgoratt2.LISTENER_PGORATT2.lsnr ONLINE ONLINE on pgoratt2
ora.pgoratt2.gsd ONLINE ONLINE on pgoratt2
ora.pgoratt2.ons ONLINE ONLINE on pgoratt2
ora.pgoratt2.vip ONLINE ONLINE on pgoratt2
(oracle@pgoratt1) /u01/crs/oracle/product/10.2.0/bin >

(oracle@pgoratt1) /u01/crs/oracle/product/10.2.0/bin > crsctl start resources
Starting resources.
Error while starting resources. Possible cause: CRSD is down.

SQL> select instance_name,status from gv$instance;

INSTANCE_NAME STATUS
---------------- ------------
pegatt1 OPEN
pegatt2 OPEN


SQL> select instance_name,status from gv$instance;

INSTANCE_NAME STATUS
---------------- ------------
+ASM1 STARTED
+ASM2 STARTED


I'm really confused here because the /u01 directory started increasing in size again, for a reason i can't seem to find since there are no large trace files this time around (increased from 47% to 74% in four days)

I'll be very grateful for any positive input to help me resolve this issue.

Thanks,

Is CRSD process running on box pgoratt1?

No. I don't think so

(oracle@pgoratt1) /u01/crs/oracle/product/10.2.0/bin > crsctl start resources
Starting resources.
Error while starting resources. Possible cause: CRSD is down.

(oracle@pgoratt1) /home/oracle > ps -ef | grep d.bin
oracle 274630 1 0 Jun 20 - 0:34 /u01/crs/oracle/product/10.2.0/bin/evmd.bin
oracle 418030 446698 0 Jun 20 - 34:51 /u01/crs/oracle/product/10.2.0/bin/ocssd.bin
root 979036 991268 0 08:26:58 - 0:21 /u01/crs/oracle/product/10.2.0/bin/crsd.bin restart

Hi PAVB,

I tried starting CRSD using the following commansds, but it wouldn't start:

pgoratt1:/ >/etc/init.crs start
Startup will be queued to init within 30 seconds.
pgoratt1:/ >

pgoratt1:/ > /u01/crs/oracle/product/10.2.0/bin/crsctl start crs
Attempting to start CRS stack
The CRS stack will be started shortly
pgoratt1:/ >

You may need to login as root to start crs. ;)

PS: Also ORACLE_HOME, PATH and LD_LIBRARY_PATH must all point to the crs home!

As you can see above, crs was started as root. ORACLE_HOME, PATH and LD_LIBRARY_PATH are all perfect...

Hi

Anything in the CRS logs ?

regards
Hrishy

Hi,

There's nothing in /u01/crs/oracle/product/10.2.0/log/crs.
There're several core files in /u01/crs/oracle/product/10.2.0/log/pgoratt1/crsd.
Attached is the /u01/crs/oracle/product/10.2.0/log/pgoratt1/crsd/crsd.log file.

Maybe it's time you open an SR w/Oracle. :rolleyes:

Just opened an Sr with Oracle; will keep you posted on results, when issue is resolved...