Hi all,

I have a 2 node 10g RAC running on AIX 5.3 with ASM. Recently i noticed the size (originally 33GB) of my /u01 directory growing very fast & reaching 100%. My backups started failing & after some researching found a trace file over 20G in size. I deleted this file & bounced the database. However, when i did a quick check of my database, i found out that the crs was down. For the past two days i've been unable to start the crs. A query of all my RAC resources using both srvctl & crs_stat (shown below) shows that both listener, ASM & database are down on this server. However sqlplus tells me the contrary (shown below).

(oracle@pgoratt1) /home/oracle > srvctl status db -d pegatt
Instance pegatt1 is not running on node pgoratt1
Instance pegatt2 is running on node pgoratt2
(oracle@pgoratt1) /home/oracle >

(oracle@pgoratt1) /u01/crs/oracle/product/10.2.0/bin > crsstat
HA Resource Target State
----------- ------ -----
ora.pegatt.db ONLINE ONLINE on pgoratt2
ora.pegatt.pegatt1.inst ONLINE OFFLINE
ora.pegatt.pegatt2.inst ONLINE ONLINE on pgoratt2
ora.pgoratt1.ASM1.asm ONLINE OFFLINE
ora.pgoratt1.gsd ONLINE OFFLINE
ora.pgoratt1.ons ONLINE OFFLINE
ora.pgoratt1.vip ONLINE ONLINE on pgoratt1
ora.pgoratt2.ASM2.asm ONLINE ONLINE on pgoratt2
ora.pgoratt2.LISTENER_PGORATT2.lsnr ONLINE ONLINE on pgoratt2
ora.pgoratt2.gsd ONLINE ONLINE on pgoratt2
ora.pgoratt2.ons ONLINE ONLINE on pgoratt2
ora.pgoratt2.vip ONLINE ONLINE on pgoratt2
(oracle@pgoratt1) /u01/crs/oracle/product/10.2.0/bin >

(oracle@pgoratt1) /u01/crs/oracle/product/10.2.0/bin > crsctl start resources
Starting resources.
Error while starting resources. Possible cause: CRSD is down.

SQL> select instance_name,status from gv$instance;

---------------- ------------
pegatt1 OPEN
pegatt2 OPEN

SQL> select instance_name,status from gv$instance;

---------------- ------------

I'm really confused here because the /u01 directory started increasing in size again, for a reason i can't seem to find since there are no large trace files this time around (increased from 47% to 74% in four days)

I'll be very grateful for any positive input to help me resolve this issue.