10gR2 RAC nodeapps issue - ONS

Hello RAC Gurus,

I have a two-node cluster on RHES 4, Oracle 10.2.0.3.
Unfortunately, I did not install this particular system, but the maintenance has been assigned to me... :(

Issue:
ONS daemon is running on first node, but not on the second node.
I tried starting ons using: $ORACLE_HOME/bin/onsctl start and onsctl ping command returns successfully mentioning ons is started.

However, when I issue: srvctl status nodeapps -n , it still says ONS daemon is not running on node:

Am I missing something here?
Anyone, any ideas/thoughts?

Thanks in advance!

Regards,
Suhas

checked the logs?

which logs?
$ORACE_HOME/opmn/logs/ons.log ?

$ORA_CRS_HOME/log/$HOSTNAME/racg

Hello Pando,

Here's a relevant excerpt from the ora..ons.log:

2007-07-22 02:31:21.114: [ RACG][2540684544] [17186][2540684544][ora.evg60lx-sscout-ora-rac2.ons]: clsrcexecut: cmd = /opt/oracle/crs/bin/racgeut -e _USR_ORA_DEBUG=0 540 /opt/oracle/crs/bin/onsctl ping

2007-07-22 02:31:21.114: [ RACG][2540684544] [17186][2540684544][ora.evg60lx-sscout-ora-rac2.ons]: clsrcexecut: rc = 1, time = 0.200s

2007-07-22 02:31:21.114: [ RACG][2540684544] [17186][2540684544][ora.evg60lx-sscout-ora-rac2.ons]: end for resource = ora.evg60lx-sscout-ora-rac2.ons, action = start, status = 1, time = 1.340s

2007-07-22 02:31:21.361: [ RACG][2540684544] [17341][2540684544][ora.evg60lx-sscout-ora-rac2.ons]: onsctl: shutting down ons daemon ...
libclntsh.so dlopen failed:
libclntsh.so: cannot open shared object file: No such file or directory
Unable to open libclntsh.so
onsctl: shutdown of ons failed!

2007-07-22 02:31:21.361: [ RACG][2540684544] [17341][2540684544][ora.evg60lx-sscout-ora-rac2.ons]: clsrcexecut: env ORACLE_CONFIG_HOME=/opt/oracle/crs

2007-07-22 02:31:21.362: [ RACG][2540684544] [17341][2540684544][ora.evg60lx-sscout-ora-rac2.ons]: clsrcexecut: cmd = /opt/oracle/crs/bin/racgeut -e _USR_ORA_DEBUG=0 540 /opt/oracle/crs/bin/onsctl stop

2007-07-22 02:31:21.362: [ RACG][2540684544] [17341][2540684544][ora.evg60lx-sscout-ora-rac2.ons]: clsrcexecut: rc = 3, time = 0.210s

and that's the end; no log entries since 07/22

Please advice.

Regards,
Suhas

Hi

I am not sure but looks like you need to do

What are the contents of
$ORACLE_HOME/opmn/logs/ons.log file

Check the file permissions on the node where it is working

ls -ltr $ORACLE_HOME/opmn/lib/lib*.so

You might have to do on the other node this

$ORACLE_HOME/opmn/bin/opmnctl stopall
chmod o+rx $ORACLE_HOME/opmn/lib/lib*.so
$ORACLE_HOME/opmn/bin/opmnctl start

Please be careful as i am just speculating about this here as i am not completely sure

regards
Hrishy

Hrishy,

What are the contents of
$ORACLE_HOME/opmn/logs/ons.log file
A: 07/08/29 16:39:30 [2] Connection 0,172.18.0.212,6202 message out of sync:
This is repeated 3-4 times.

ls -ltr $ORACLE_HOME/opmn/lib/lib*.so
A: there's only one file in this folder: ons.jar

$ORACLE_HOME/opmn/bin/opmnctl stopall
A: There's no such executable in this folder; only "ons"

Let me know if you can think of anything.

Thanks!
Suhas

Hi

What about on the node where this is working fine

what does this return
ls -ltr $ORACLE_HOME/opmn/lib/lib*.so

regards
Hrishy

Hrishy,

There's only the one jar file on both nodes.

Regards,
Suhas

Hi

Do you see anywhere libclntsh.so in the following path

$ORACLE_CONFIG_HOME

find . -name "libclntsh.so" $ORACLE_CONFIG_HOME

the onsctl utility should be there in
/opt/oracle/crs/bin/onsctl

I am extremly sorry that i asked you to search in Oracle home instead of CR home

regards
Hrishy

Hi

I noticed in the first post that you started

I tried starting ons using: $ORACLE_HOME/bin/onsctl start and onsctl ping command returns successfully mentioning ons is started.

You need to stop this as you started it from the wrong home.

Well according to me you should start ONS from the cluster services home

That means
export ORACLE_CONFIG_HOME=/opt/oracle/crs
and then do
$ORACLE_CONFIG_HOME/bin/onsctl start

regards
Hrishy

Hrishy,

The libclntsh.so file resides in $ORACLE_HOME/lib as well as $CRS_HOME/lib folder as a symbolic link to libclntsh.so.10.1 (same folder), which is of the same size and same permissions (oracle:oinstall) on "both" nodes.

Also, I had used the $CRS_HOME/bin/onsctl utility to start ons. It does say ons started, but daemon does not start (srvctl status nodeapps -n / crs_stat -t)

Any ideas?

Regards,
Suhas

Hi

what does this return

ls -ltr $CRS_HOME/opmn/lib/lib*.so

also when you do opmnctl start

what does the logfile
$CRS_HOME/opmn/logs/ons.log

what does the config file of ons look like

regards
Hrishy

try $ORA_CRS_HOME/bin/onsctl debug

in your working node, racg log shows something like this?

Code:

---


2007-08-31 06:46:32.653: [    RACG][2540702976] [8219][2540702976][ora.oex2.ons]: Number of onsconfiguration retrieved, numcfg = 3
onscfg[0]
  {node = oex1.risc.com, port = 6201}
Adding remote host oex1.risc.com:6201
onscfg[1]
  {node = oex2.risc.com, port = 6201}
Adding remote host oex2.risc.com:6201
onscfg[2]
  {node = oex3.risc.c
2007-08-31 06:46:32.687: [    RACG][2540702976] [8219][2540702976][ora.oex2.ons]: om, port = 6201}
Adding remote host oex3.risc.com:6201
Number of onsconfiguration retrieved, numcfg = 3
onscfg[0]
  {node = oex1.risc.com, port = 6201}
Adding remote host oex1.risc.com:6201
onscfg[1]
  {node = oex2.risc.com, port = 6201}
Adding remote h
2007-08-31 06:46:32.687: [    RACG][2540702976] [8219][2540702976][ora.oex2.ons]: ost oex2.risc.com:6201
onscfg[2]
  {node = oex3.risc.com, port = 6201}
Adding remote host oex3.risc.com:6201
onsctl: ons started

---