10g R1 RAC nodeapps issue - Urgent!

[suhasd75]

All,

I have a 2-node RAC on HP hardware, Linux 4 and Clarion CX500 SAN with Oracle 10g R1 (10.1.0.4.0).

It was working fine till now, and all of a sudden, the node applications seem to have a problem.
I get the following error when I run srvctl:

$srvctl status nodeapps -n
VIP is running on node:
GSD is running on node:
Listener is running on node:
ONS daemon is not running on node:

Iam not sure how ONS did not start.
I rebooted the servers after graceful shutdowns and still the same status!

I did use crs_stat utility also just to double-check with the same result.

Iam at a loss now on how this could happen... Nothing was changed since I installed the RAC and it was working fine till now.

Any help in this matter will be highly helpful and I would be much obliged.

Thanks in advance!

Suhas

[Tarry]

Try restarting stuff manually.

I've done it to simulate the scenario:

[oracle@vmora01rh4 bin]$ crs_stop -all


Attempting to stop `ora.vmora01rh4.ons` on member `vmora01rh4`
Attempting to stop `ora.vmora02rh4.ons` on member `vmora02rh4`
Stop of `ora.vmora01rh4.ons` on member `vmora01rh4` succeeded.
.
.
.
.

Do this to check status:

[oracle@vmora01rh4 bin]$ crs_stat -t
Name           Type           Target    State     Host
------------------------------------------------------------
ora.fokerac.db application    OFFLINE   OFFLINE
ora....c1.inst application    OFFLINE   OFFLINE
ora....c2.inst application    OFFLINE   UNKNOWN   vmora02rh4
ora....serv.cs application    ONLINE    UNKNOWN   vmora02rh4
ora....ac1.srv application    OFFLINE   OFFLINE
ora....ac2.srv application    OFFLINE   OFFLINE
ora....SM1.asm application    OFFLINE   OFFLINE
ora....H4.lsnr application    OFFLINE   OFFLINE
ora....rh4.gsd application    ONLINE    UNKNOWN   vmora01rh4
ora....rh4.ons application    OFFLINE   OFFLINE
ora....rh4.vip application    OFFLINE   OFFLINE
ora....SM2.asm application    OFFLINE   UNKNOWN   vmora02rh4
ora....H4.lsnr application    OFFLINE   UNKNOWN   vmora02rh4
ora....rh4.gsd application    ONLINE    UNKNOWN   vmora02rh4
ora....rh4.ons application    OFFLINE   UNKNOWN   vmora02rh4
ora....rh4.vip application    OFFLINE   OFFLINE

If necessary use srvctl utility to stop all databases...

[oracle@vmora01rh4 bin]$ srvctl stop instance -d fokerac -i fokerac1
[oracle@vmora01rh4 bin]$ srvctl stop instance -d fokerac -i fokerac2

If that doesn't help either (remember it IS possible that he complains about a human intervention...

Stop individual daemons by doing this...

[oracle@vmora01rh4 bin]$ crs_stop ora.vmora01rh4.ons

Do a...

crsctl -t

again.

and if all are stop start it again...

crs_start all

Tell us if it worked.

[suhasd75]

Hey Tarry,
Thanks a lot for sending out this info.
I did manually try restarting instances, services node apps etc, but it didnt work.
I also rebooted the server itself, to no avail.
I'll try to use onsctl and start only ONS daemons sometime tmrw. Users are already logged in and I dont wish to touch the servers.
Again, thanks very much for the timely advice.
Regards,
Suhas

[Tarry]

Ok Suhas,

Let us know if the ONS started weith onsctl.

[suhasd75]

No Tarry,

I tried using onsctl. Though the onsctl ping gives output as "ons is running ...", when I run either crs_stat -t or srvctl, ONS still says OFFLINE/daemon not running, respectively.

Of course, this is not a "mandatory" node app, but Iam a bit intrigued on how it could go down, and not come up even on restart/reboot.

I'll keep you posted.

Thanks a lot for the help, again.

Regards
Suhas