Unable to start crs in a 10g RAC
DBAsupport.com Forums - Powered by vBulletin
Results 1 to 10 of 10

Thread: Unable to start crs in a 10g RAC

Hybrid View

  1. #1
    Join Date
    Nov 2006
    Posts
    158

    Unable to start crs in a 10g RAC

    Hi all,

    I have a 2 node 10g RAC running on AIX 5.3 with ASM. Recently i noticed the size (originally 33GB) of my /u01 directory growing very fast & reaching 100%. My backups started failing & after some researching found a trace file over 20G in size. I deleted this file & bounced the database. However, when i did a quick check of my database, i found out that the crs was down. For the past two days i've been unable to start the crs. A query of all my RAC resources using both srvctl & crs_stat (shown below) shows that both listener, ASM & database are down on this server. However sqlplus tells me the contrary (shown below).

    (oracle@pgoratt1) /home/oracle > srvctl status db -d pegatt
    Instance pegatt1 is not running on node pgoratt1
    Instance pegatt2 is running on node pgoratt2
    (oracle@pgoratt1) /home/oracle >

    (oracle@pgoratt1) /u01/crs/oracle/product/10.2.0/bin > crsstat
    HA Resource Target State
    ----------- ------ -----
    ora.pegatt.db ONLINE ONLINE on pgoratt2
    ora.pegatt.pegatt1.inst ONLINE OFFLINE
    ora.pegatt.pegatt2.inst ONLINE ONLINE on pgoratt2
    ora.pgoratt1.ASM1.asm ONLINE OFFLINE
    ora.pgoratt1.LISTENER_PGORATT1.lsnr ONLINE OFFLINE
    ora.pgoratt1.gsd ONLINE OFFLINE
    ora.pgoratt1.ons ONLINE OFFLINE
    ora.pgoratt1.vip ONLINE ONLINE on pgoratt1
    ora.pgoratt2.ASM2.asm ONLINE ONLINE on pgoratt2
    ora.pgoratt2.LISTENER_PGORATT2.lsnr ONLINE ONLINE on pgoratt2
    ora.pgoratt2.gsd ONLINE ONLINE on pgoratt2
    ora.pgoratt2.ons ONLINE ONLINE on pgoratt2
    ora.pgoratt2.vip ONLINE ONLINE on pgoratt2
    (oracle@pgoratt1) /u01/crs/oracle/product/10.2.0/bin >

    (oracle@pgoratt1) /u01/crs/oracle/product/10.2.0/bin > crsctl start resources
    Starting resources.
    Error while starting resources. Possible cause: CRSD is down.

    SQL> select instance_name,status from gv$instance;

    INSTANCE_NAME STATUS
    ---------------- ------------
    pegatt1 OPEN
    pegatt2 OPEN


    SQL> select instance_name,status from gv$instance;

    INSTANCE_NAME STATUS
    ---------------- ------------
    +ASM1 STARTED
    +ASM2 STARTED


    I'm really confused here because the /u01 directory started increasing in size again, for a reason i can't seem to find since there are no large trace files this time around (increased from 47% to 74% in four days)

    I'll be very grateful for any positive input to help me resolve this issue.

    Thanks,
    Regards,

    divroro12

  2. #2
    Join Date
    Mar 2007
    Location
    Ft. Lauderdale, FL
    Posts
    3,554
    Is CRSD process running on box pgoratt1?
    Pablo (Paul) Berzukov

    Author of Understanding Database Administration available at amazon and other bookstores.

    Disclaimer: Advice is provided to the best of my knowledge but no implicit or explicit warranties are provided. Since the advisor explicitly encourages testing any and all suggestions on a test non-production environment advisor should not held liable or responsible for any actions taken based on the given advice.

  3. #3
    Join Date
    Nov 2006
    Posts
    158
    No. I don't think so

    (oracle@pgoratt1) /u01/crs/oracle/product/10.2.0/bin > crsctl start resources
    Starting resources.
    Error while starting resources. Possible cause: CRSD is down.

    (oracle@pgoratt1) /home/oracle > ps -ef | grep d.bin
    oracle 274630 1 0 Jun 20 - 0:34 /u01/crs/oracle/product/10.2.0/bin/evmd.bin
    oracle 418030 446698 0 Jun 20 - 34:51 /u01/crs/oracle/product/10.2.0/bin/ocssd.bin
    root 979036 991268 0 08:26:58 - 0:21 /u01/crs/oracle/product/10.2.0/bin/crsd.bin restart
    Regards,

    divroro12

  4. #4
    Join Date
    Nov 2006
    Posts
    158
    Hi PAVB,

    I tried starting CRSD using the following commansds, but it wouldn't start:

    pgoratt1:/ >/etc/init.crs start
    Startup will be queued to init within 30 seconds.
    pgoratt1:/ >

    pgoratt1:/ > /u01/crs/oracle/product/10.2.0/bin/crsctl start crs
    Attempting to start CRS stack
    The CRS stack will be started shortly
    pgoratt1:/ >
    Regards,

    divroro12

  5. #5
    Join Date
    Jul 2002
    Location
    Lake Worth, FL
    Posts
    1,478

    Talking

    You may need to login as root to start crs.

    PS: Also ORACLE_HOME, PATH and LD_LIBRARY_PATH must all point to the crs home!
    "The person who says it cannot be done should not interrupt the person doing it." --Chinese Proverb

  6. #6
    Join Date
    Nov 2006
    Posts
    158
    As you can see above, crs was started as root. ORACLE_HOME, PATH and LD_LIBRARY_PATH are all perfect...
    Regards,

    divroro12

  7. #7
    Join Date
    Jan 2001
    Posts
    2,828
    Hi

    Anything in the CRS logs ?

    regards
    Hrishy

  8. #8
    Join Date
    Nov 2006
    Posts
    158

    Unable to start crs in a 10g RAC

    Hi,

    There's nothing in /u01/crs/oracle/product/10.2.0/log/crs.
    There're several core files in /u01/crs/oracle/product/10.2.0/log/pgoratt1/crsd.
    Attached is the /u01/crs/oracle/product/10.2.0/log/pgoratt1/crsd/crsd.log file.
    Attached Files Attached Files
    Regards,

    divroro12

  9. #9
    Join Date
    Jul 2002
    Location
    Lake Worth, FL
    Posts
    1,478

    Wink To open SR or not open SR...that is the question!



    Maybe it's time you open an SR w/Oracle.

    "The person who says it cannot be done should not interrupt the person doing it." --Chinese Proverb

  10. #10
    Join Date
    Nov 2006
    Posts
    158
    Just opened an Sr with Oracle; will keep you posted on results, when issue is resolved...
    Regards,

    divroro12

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  



Click Here to Expand Forum to Full Width