10gR2 RAC nodeapps issue - ONS

[suhasd75]

Hello RAC Gurus,

I have a two-node cluster on RHES 4, Oracle 10.2.0.3.
Unfortunately, I did not install this particular system, but the maintenance has been assigned to me... :(

Issue:
ONS daemon is running on first node, but not on the second node.
I tried starting ons using: $ORACLE_HOME/bin/onsctl start and onsctl ping command returns successfully mentioning ons is started.

However, when I issue: srvctl status nodeapps -n , it still says ONS daemon is not running on node:

Am I missing something here?
Anyone, any ideas/thoughts?

Thanks in advance!

Regards,
Suhas

[pando]

checked the logs?

[suhasd75]

which logs?
$ORACE_HOME/opmn/logs/ons.log ?

[pando]

$ORA_CRS_HOME/log/$HOSTNAME/racg

[suhasd75]

Hello Pando,

Here's a relevant excerpt from the ora..ons.log:

2007-07-22 02:31:21.114: [ RACG][2540684544] [17186][2540684544][ora.evg60lx-sscout-ora-rac2.ons]: clsrcexecut: cmd = /opt/oracle/crs/bin/racgeut -e _USR_ORA_DEBUG=0 540 /opt/oracle/crs/bin/onsctl ping

2007-07-22 02:31:21.114: [ RACG][2540684544] [17186][2540684544][ora.evg60lx-sscout-ora-rac2.ons]: clsrcexecut: rc = 1, time = 0.200s

2007-07-22 02:31:21.114: [ RACG][2540684544] [17186][2540684544][ora.evg60lx-sscout-ora-rac2.ons]: end for resource = ora.evg60lx-sscout-ora-rac2.ons, action = start, status = 1, time = 1.340s

2007-07-22 02:31:21.361: [ RACG][2540684544] [17341][2540684544][ora.evg60lx-sscout-ora-rac2.ons]: onsctl: shutting down ons daemon ...
libclntsh.so dlopen failed:
libclntsh.so: cannot open shared object file: No such file or directory
Unable to open libclntsh.so
onsctl: shutdown of ons failed!

2007-07-22 02:31:21.361: [ RACG][2540684544] [17341][2540684544][ora.evg60lx-sscout-ora-rac2.ons]: clsrcexecut: env ORACLE_CONFIG_HOME=/opt/oracle/crs

2007-07-22 02:31:21.362: [ RACG][2540684544] [17341][2540684544][ora.evg60lx-sscout-ora-rac2.ons]: clsrcexecut: cmd = /opt/oracle/crs/bin/racgeut -e _USR_ORA_DEBUG=0 540 /opt/oracle/crs/bin/onsctl stop

2007-07-22 02:31:21.362: [ RACG][2540684544] [17341][2540684544][ora.evg60lx-sscout-ora-rac2.ons]: clsrcexecut: rc = 3, time = 0.210s

and that's the end; no log entries since 07/22

Please advice.

Regards,
Suhas

[hrishy]

Hi

I am not sure but looks like you need to do

What are the contents of
$ORACLE_HOME/opmn/logs/ons.log file

Check the file permissions on the node where it is working

ls -ltr $ORACLE_HOME/opmn/lib/lib*.so

You might have to do on the other node this

$ORACLE_HOME/opmn/bin/opmnctl stopall
chmod o+rx $ORACLE_HOME/opmn/lib/lib*.so
$ORACLE_HOME/opmn/bin/opmnctl start

Please be careful as i am just speculating about this here as i am not completely sure

regards
Hrishy

[suhasd75]

Hrishy,

What are the contents of
$ORACLE_HOME/opmn/logs/ons.log file
A: 07/08/29 16:39:30 [2] Connection 0,172.18.0.212,6202 message out of sync:
This is repeated 3-4 times.

ls -ltr $ORACLE_HOME/opmn/lib/lib*.so
A: there's only one file in this folder: ons.jar

$ORACLE_HOME/opmn/bin/opmnctl stopall
A: There's no such executable in this folder; only "ons"

Let me know if you can think of anything.

Thanks!
Suhas

[hrishy]

Hi

What about on the node where this is working fine

what does this return
ls -ltr $ORACLE_HOME/opmn/lib/lib*.so

regards
Hrishy

[suhasd75]

Hrishy,

There's only the one jar file on both nodes.

Regards,
Suhas

[hrishy]

Hi

Do you see anywhere libclntsh.so in the following path

$ORACLE_CONFIG_HOME

find . -name "libclntsh.so" $ORACLE_CONFIG_HOME

the onsctl utility should be there in
/opt/oracle/crs/bin/onsctl

I am extremly sorry that i asked you to search in Oracle home instead of CR home

regards
Hrishy