Hi DBA's,

We have a two node RAC database and recently we have a patch upgrade in the application end, and after that we started hitting ORA-04031, so in-order to resolve the error we try to manually shutdown the the first node and during that time the first node was evicted. Each node has 4 database instances, and we are trying to shutdown only one database instance.

I would like to know whether the node eviction was done by oracle or because of some other reason? because there is nothing updated in the ocssd log on this.

[ CSSD]2013-01-13 11:12:59.337 [1801] >TRACE: clssnmReadDskHeartbeat: node 1, wmdbp1, has a disk HHB, but no network HB, DHB has rcfg 176896393, wrtcnt, 74343812, LATS 1045382822, lastSeqNo 74343812, tiimestamp 1358046779/3054389986
[ CSSD]2013-01-13 11:12:59.343 [1287] >TRACE: clssnmReadDskHeartbeat: node 1, wmdbp1, has a disk HHB, but no network HB, DHB has rcfg 176896393, wrtcnt, 74343812, LATS 1045382828, lastSeqNo 74343812, tiimestamp 1358046779/3054389986

...
LOG:-
-----

[ CSSD]2013-01-13 20:34:41.957 [1287] >TRACE: clssnmReadDskHeartbeat: node 1, wmdbp1, has a disk HHB, but no network HB, DHB has rcfg 176896395, wrtcnt, 74376699, LATS 1079085574, lastSeqNo 74376699, tiimestamp 1358080481/3088092000
[ CSSD]2013-01-13 20:34:42.362 [1544] >TRACE: clssnmReadDskHeartbeat: node 1, wmdbp1, has a disk HHB, but no network HB, DHB has rcfg 176896395, wrtcnt, 74376699, LATS 1079085979, lastSeqNo 74376699, tiimestamp 1358080481/3088092000
[ CSSD]2013-01-13 20:34:42.588 [1801] >TRACE: clssnmReadDskHeartbeat: node 1, wmdbp1, has a disk HHB, but no network HB, DHB has rcfg 176896395, wrtcnt, 74376700, LATS 1079086205, lastSeqNo 74376700, tiimestamp 1358080482/3088093033