Can't bring up database!sculkget: failed to lock

[areeb]

Hi,
I am trying to bring up the db and its kind of hung ,on checking alert log its showing this error.Help me to resolve it.
"sculkget: failed to lock /u01/app/oracle/product/10.2.0/db_1/dbs/lkinstMSBD1 exclusive"

Thanks

[PAVB]

You are probably getting an ORA-09968 error along side identifying the offending PID; the one holding the lock on your affected file.

[areeb]

This is part of alertlog ,i dont see ORA-09968 in it.

Tue Aug 21 15:57:53 2007
LMS 0: 0 GCS shadows traversed, 0 replayed
Tue Aug 21 15:57:53 2007
Submitted all GCS remote-cache requests
Post SMON to start 1st pass IR
Fix write in gcs resources
Reconfiguration complete
LCK0 started with pid=58, OS id=5979
Tue Aug 21 16:00:02 2007
sculkget: failed to lock /u01/app/oracle/product/10.2.0/db_1/dbs/lkinstMSBD1 exc
lusive
sculkget: lock held by PID: 4323
Oracle Instance Shutdown operation failed. Another process may be attempting to
startup or shutdown this Instance.
Failed to acquire instance startup/shutdown serialization primitive
Tue Aug 21 16:00:08 2007
Shutting down instance (abort)
License high water mark = 1
Instance terminated by USER, pid = 10092

[areeb]

oracle RAC 10202 runing on HP-UX 11.23v

[dbasan]

sculkget: lock held by PID: 4323
Oracle Instance Shutdown operation failed. Another process may be attempting to
startup or shutdown this Instance.
Failed to acquire instance startup/shutdown serialization primitive

Are you working on different sessions?

Lock held by 4323 has to be released.

[PAVB]

As pointed by dbasan you have to get rid of the lock held by PID 4323

Can you identify PID 4323? what kind of process is it?

Be sure everything is down, then try again.

Did you said RAC? what's the status of the other node/s? Shouldn't you being using crs_stop to shutdown your instance?

[areeb]

All the other instances in this cluster are running fine except the one that is down ,i tried to bring it up from different sessions.?grepping for the PID did't give anything ,how to find what is the process?

crs_stat -t command sometimes just hangs and at times work and shows that the instances are up ,where as it is actually down
Tried stopping the db with srvctl but its just hanging doing nothing.

/bdump $ ps -ef|grep 4323
oracle 14323 1 0 Aug 16 ? 0:00 oracleWxx2D1 (LOCAL=NO)
oracle 3530 28173 0 20:03:18 pts/6 0:00 grep 4323

[PAVB]

Your ps -ef | grep 4323 is showing a different PID... 14323, pretty similar but not the one that was holding the lock.

If you run crsstat.sh; what's the status of this database node (instance)?

Could you share with us the command you are using to bring the instance back up?

[areeb]

crs_stat -t command sometimes just hangs and at times work and shows that the instances are up ,where as it is actually down .
Tried stopping the db with srvctl but its just hanging doing nothing.
even check on crs is hanging.

commands using
$ORA_CRS_HOME/bin/crs_stat -t
srvctl stop database -d dbname-as both instance are down
pbrun crsctl check crs

[dbasan]

Tue Aug 21 16:00:08 2007
Shutting down instance (abort)
License high water mark = 1
Instance terminated by USER, pid = 10092

What's after the termination in the alert log?

Instance is aborted by user 10092. But still you can see some oracle processes running. Check
> ps -ef | grep ora
> top -- to see if any oracle process is still running.

Now you can delete the file /u01/app/oracle/product/10.2.0/db_1/dbs/lkinstMSBD1
If no oracle process is running try to start the db by force -if still unsuccessfull then go for reboot server should be the final option as this would clear all the locks.