Oracle gave me a patch that allowed us to format the OCR and voting disk. Now the problem may just be I need to run root.sh on node1 or wipe both clean and start fresh.

Devices formatted, now we need to get the CRS daemon up and running on both nodes..

[root@dr2db2 ~]# dd if=/dev/zero of=/dev/raw/raw2 bs=1048576 count=1000
1000+0 records in
1000+0 records out
[root@dr2db2 ~]# /orahome/app/oracle/product/10.1.2.0.2/CRS/root.sh
WARNING: directory '/orahome/app/oracle/product/10.1.2.0.2' is not owned by root
WARNING: directory '/orahome/app/oracle/product' is not owned by root
WARNING: directory '/orahome/app/oracle' is not owned by root
WARNING: directory '/orahome/app' is not owned by root
Checking to see if Oracle CRS stack is already configured

Setting the permissions on OCR backup directory
Setting up NS directories
Oracle Cluster Registry configuration upgraded successfully
WARNING: directory '/orahome/app/oracle/product/10.1.2.0.2' is not owned by root
WARNING: directory '/orahome/app/oracle/product' is not owned by root
WARNING: directory '/orahome/app/oracle' is not owned by root
WARNING: directory '/orahome/app' is not owned by root
Successfully accumulated necessary OCR keys.
Using ports: CSS=49895 CRS=49896 EVMC=49898 and EVMR=49897.
node :
node 1: dr2db2 dr2db2-eth2 dr2db2
node 2: dr2db1 dr2db1-eth2 dr2db1
Creating OCR keys for user 'root', privgrp 'root'..
Operation successful.
Now formatting voting device: /dev/raw/raw6
Format of 1 voting devices complete.
Startup will be queued to init within 90 seconds.
Adding daemons to inittab
Expecting the CRS daemons to be up within 600 seconds.
CSS is active on these nodes.
dr2db2
CSS is inactive on these nodes.
dr2db1
Local node checking complete.
Run root.sh on remaining nodes to start CRS daemons.

[root@dr2db2 ~]# ps -ef | grep ora
root 6572 2526 0 13:39 ? 00:00:00 sshd: oracle [priv]
oracle 6574 6572 0 13:39 ? 00:00:00 sshd: oracle@pts/1
oracle 6575 6574 0 13:39 pts/1 00:00:00 -bash
root 14172 2526 0 17:54 ? 00:00:00 sshd: oracle [priv]
oracle 14176 14172 0 17:55 ? 00:00:00 sshd: oracle@pts/2
oracle 14177 14176 0 17:55 pts/2 00:00:00 -bash
root 14682 1 0 17:57 ? 00:00:00 /bin/su -l oracle -c sh -c 'ulimit -c unlimited; cd /orahome/app/oracle/product/10.1.2.0.2/CRS/log/dr2db2/evmd; exec /orahome/app/oracle/product/10.1.2.0.2/CRS/bin/evmd '
root 14686 1 0 17:57 ? 00:00:00 /orahome/app/oracle/product/10.1.2.0.2/CRS/bin/crsd.bin reboot
oracle 14959 14682 0 17:58 ? 00:00:00 /orahome/app/oracle/product/10.1.2.0.2/CRS/bin/evmd.bin
root 15018 14942 0 17:58 ? 00:00:00 /bin/su -l oracle -c /bin/sh -c 'ulimit -c unlimited; cd /orahome/app/oracle/product/10.1.2.0.2/CRS/log/dr2db2/cssd; /orahome/app/oracle/product/10.1.2.0.2/CRS/bin/ocssd || exit $?'
oracle 15021 15018 0 17:58 ? 00:00:00 /bin/sh -c ulimit -c unlimited; cd /orahome/app/oracle/product/10.1.2.0.2/CRS/log/dr2db2/cssd; /orahome/app/oracle/product/10.1.2.0.2/CRS/bin/ocssd || exit $?
oracle 15057 15021 0 17:58 ? 00:00:00 /orahome/app/oracle/product/10.1.2.0.2/CRS/bin/ocssd.bin
root 15996 14206 0 17:59 pts/2 00:00:00 grep ora


crs_setperm crs_setperm.bin crs_start crs_start.bin crs_stat crs_stat.bin crs_stop crs_stop.bin
[root@dr2db2 ~]# /orahome/app/oracle/product/10.1.2.0.2/CRS/bin/crs_stat -t
CRS-0184: Cannot communicate with the CRS daemon.


on next node

[root@dr2db1 ~]# /orahome/app/oracle/product/10.1.2.0.2/CRS/root.sh
WARNING: directory '/orahome/app/oracle/product/10.1.2.0.2' is not owned by root
WARNING: directory '/orahome/app/oracle/product' is not owned by root
WARNING: directory '/orahome/app/oracle' is not owned by root
WARNING: directory '/orahome/app' is not owned by root
Checking to see if Oracle CRS stack is already configured
/etc/oracle does not exist. Creating it now.

Setting the permissions on OCR backup directory
Setting up NS directories
Oracle Cluster Registry configuration upgraded successfully
WARNING: directory '/orahome/app/oracle/product/10.1.2.0.2' is not owned by root
WARNING: directory '/orahome/app/oracle/product' is not owned by root
WARNING: directory '/orahome/app/oracle' is not owned by root
WARNING: directory '/orahome/app' is not owned by root
/orahome/app/oracle/product/10.1.2.0.2/CRS/bin/crsctl.bin: error while loading shared libraries: libstdc++.so.5: cannot open shared object file: No such file or directory
Failure initializing entries in /etc/oracle/scls_scr/dr2db1.


I think whats needed is a clean install.