Database Instance unable to start

[tspoonfr]

Hi,

I am trying to start up my database but I receive the following errors, any ideas why? I have posted the init.ora and the beginning of the database startup

SQL> startup pfile='/tmp/terry.ora'
ORACLE instance started.

Total System Global Area 6747725824 bytes
Fixed Size                  2213976 bytes
Variable Size            6241126312 bytes
Database Buffers          469762048 bytes
Redo Buffers               34623488 bytes
ORA-03113: end-of-file on communication channel
Process ID: 17491
Session ID: 195 Serial number: 1

SQL>

Here is my init.ora file

[oracle@mondbop001 {SOCPROD1} tmp]$ more terry.ora
SOCPROD2.__db_cache_size=469762048
SOCPROD1.__db_cache_size=469762048
SOCPROD2.__java_pool_size=67108864
SOCPROD1.__java_pool_size=134217728
SOCPROD1.__large_pool_size=67108864
SOCPROD2.__large_pool_size=67108864
SOCPROD2.__oracle_base='/u01/app/grid'#ORACLE_BASE set from environment
SOCPROD1.__oracle_base='/u01/app/grid'#ORACLE_BASE set from environment
SOCPROD1.__pga_aggregate_target=5167382528
SOCPROD2.__pga_aggregate_target=5167382528
SOCPROD1.__sga_target=1610612736
SOCPROD2.__sga_target=1610612736
SOCPROD1.__shared_io_pool_size=0
SOCPROD2.__shared_io_pool_size=0
SOCPROD2.__shared_pool_size=872415232
SOCPROD1.__shared_pool_size=872415232
SOCPROD1.__streams_pool_size=0
SOCPROD2.__streams_pool_size=67108864
*.archive_lag_target=0
*.audit_file_dest='/u01/app/oracle/admin/SOCPROD/adump'
*.audit_trail='NONE'
*.cluster_database=TRUE
*.compatible='11.2.0.0.0'
*.control_files='+DAT1/socprod/controlfile/current.260.729791749'
*.db_block_size=8192
*.db_create_file_dest='+DAT1'
*.db_domain=''
*.db_flashback_retention_target=720
*.db_name='SOCPROD'
*.db_recovery_file_dest_size=188978561024
*.db_unique_name='SOCPRODCAV'
*.dg_broker_config_file1='+DAT1/socprodcav/dr1.dat'
*.dg_broker_start=TRUE
*.diagnostic_dest='/u01/app/oracle'
*.dispatchers='(PROTOCOL=TCP) (SERVICE=SOCPRODXDB)'
*.fal_server='SOCPRODCRO'
SOCPROD1.instance_number=1
SOCPROD2.instance_number=2
*.job_queue_processes=1000
*.log_archive_config='dg_config=(SOCPRODCAV,SOCPRODCRO)'
*.log_archive_dest_2='service="SOCPRODCRO"','LGWR SYNC AFFIRM delay=0 optional compression=disable max_failure=0 max_connections=1 reopen=300 db_unique_name=
"SOCPRODCRO" net_timeout=30','valid_for=(all_logfiles,primary_role)'
*.log_archive_dest_3=''
*.log_archive_dest_state_2='ENABLE'
*.log_archive_dest_state_3='ENABLE'
*.log_archive_format='%t_%s_%r.dbf'
SOCPROD2.log_archive_format='%t_%s_%r.dbf'
SOCPROD1.log_archive_format='%t_%s_%r.dbf'
*.log_archive_max_processes=5
*.log_archive_min_succeed_dest=1
SOCPROD2.log_archive_trace=0
SOCPROD1.log_archive_trace=0
*.memory_target=6730809344
*.nls_language='ENGLISH'
*.nls_territory='UNITED KINGDOM'
*.open_cursors=300
*.pga_aggregate_target=5114953728
*.processes=1000
*.recyclebin='OFF'
*.remote_listener='caversham1-scan.prod.mon.bbc.co.uk:1521'
*.remote_login_passwordfile='exclusive'
*.session_cached_cursors=100
*.sga_target=1610612736
*.smtp_out_server='10.182.216.104:25'
*.standby_file_management='AUTO'
SOCPROD2.thread=2
SOCPROD1.thread=1
*.trace_enabled=FALSE
SOCPROD2.undo_tablespace='UNDOTBS2'
SOCPROD1.undo_tablespace='UNDOTBS1'
[oracle@mondbop001 {SOCPROD1} tmp]$

Starting up:
Oracle Database 11g Enterprise Edition Release 11.2.0.1.0 - 64bit Production
With the Real Application Clusters option.
Using parameter settings in server-side pfile /u01/app/oracle/product/11.2.0/dbhome_1/dbs/initSOCPROD1.ora

My question is why does it attempt to use initSOCPROD1.ora and not the pfile I specified at startup?

Many thanks

[PAVB]

What do you see on Alert Log?

[tspoonfr]

Contents of alert log

1 2 (myinst: 1)
Global Resource Directory frozen
* allocate domain 0, invalid = TRUE
Communication channels reestablished
* domain 0 not valid according to instance 2
* domain 0 valid = 0 according to instance 2
Master broadcasted resource hash value bitmaps
Non-local Process blocks cleaned out
LMS 1: 0 GCS shadows cancelled, 0 closed, 0 Xw survived
LMS 0: 0 GCS shadows cancelled, 0 closed, 0 Xw survived
Set master node info
Submitted all remote-enqueue requests
Dwn-cvts replayed, VALBLKs dubious
All grantable enqueues granted
Post SMON to start 1st pass IR
Submitted all GCS remote-cache requests
Post SMON to start 1st pass IR
Fix write in gcs resources
Reconfiguration complete
Mon Nov 07 09:49:27 2011
LCK0 started with pid=30, OS id=4992
Starting background process RSMN
Mon Nov 07 09:49:27 2011
RSMN started with pid=31, OS id=4996
ORACLE_BASE from environment = /u01/app/oracle
Mon Nov 07 09:49:27 2011
DMON started with pid=32, OS id=5005
Mon Nov 07 09:49:27 2011
ALTER DATABASE MOUNT
This instance was first to mount
Mon Nov 07 09:49:29 2011
ALTER SYSTEM SET local_listener='(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=TCP)(HOST=mondbop001-vip)(PORT=1521))))' SCOPE=MEMORY SID='SOCPROD1';
NOTE: Loaded library: System
SUCCESS: diskgroup DAT1 was mounted
NOTE: dependency between database SOCPRODCAV and diskgroup resource ora.DAT1.dg is established
USER (ospid: 5006): terminating the instance
Mon Nov 07 09:49:35 2011
System state dump is made for local instance
System State dumped to trace file /u01/app/oracle/diag/rdbms/socprodcav/SOCPROD1/trace/SOCPROD1_diag_4864.trc
Mon Nov 07 09:49:36 2011
ORA-1092 : opitsk aborting process
Instance terminated by USER, pid = 5006
Mon Nov 07 09:49:46 2011
Starting ORACLE instance (normal)
LICENSE_MAX_SESSION = 0
LICENSE_SESSIONS_WARNING = 0
Interface type 1 bond1 192.168.1.0 configured from GPnP Profile for use as a cluster interconnect
Interface type 1 bond0 10.182.213.0 configured from GPnP Profile for use as a public interface
Picked latch-free SCN scheme 3
Autotune of undo retention is turned on.
LICENSE_MAX_USERS = 0
SYS auditing is disabled
NOTE: Volume support enabled
Starting up:
Oracle Database 11g Enterprise Edition Release 11.2.0.1.0 - 64bit Production
With the Real Application Clusters option.
Using parameter settings in server-side pfile /u01/app/oracle/product/11.2.0/dbhome_1/dbs/initSOCPROD1.ora
System parameters with non-default values:
processes = 1000
spfile = "+DAT1/socprodcav/spfilesocprod.ora"
nls_language = "ENGLISH"
nls_territory = "UNITED KINGDOM"
sga_target = 1536M
memory_target = 6464M
control_files = "+DAT1/socprod/controlfile/current.260.729791749"
control_files = "+FRA1/socprod/controlfile/current.256.729791749"
db_block_size = 8192
compatible = "11.2.0.0.0"
log_archive_dest_1 = "LOCATION=+FRA1"
log_archive_dest_2 = "service="SOCPRODCRO""
log_archive_dest_2 = "LGWR SYNC AFFIRM delay=0 optional compression=disable max_failure=0 max_connections=1 reopen=300 db_unique_name="SOCPRODCRO" net_timeout=30"
log_archive_dest_2 = "valid_for=(all_logfiles,primary_role)"
log_archive_dest_3 = ""
log_archive_dest_state_2 = "ENABLE"
log_archive_dest_state_3 = "ENABLE"
log_archive_min_succeed_dest= 1
fal_server = "SOCPRODCRO"
log_archive_trace = 0
log_archive_config = "dg_config=(SOCPRODCAV,SOCPRODCRO)"
log_archive_format = "%t_%s_%r.dbf"
log_archive_max_processes= 5
archive_lag_target = 0
cluster_database = TRUE
db_create_file_dest = "+DAT1"
db_recovery_file_dest = "+FRA1"
db_recovery_file_dest_size= 176G
standby_file_management = "AUTO"
thread = 1
db_flashback_retention_target= 720
undo_tablespace = "UNDOTBS1"
instance_number = 1
recyclebin = "OFF"
remote_login_passwordfile= "EXCLUSIVE"
db_domain = ""
dispatchers = "(PROTOCOL=TCP) (SERVICE=SOCPRODXDB)"
remote_listener = "caversham1-scan.prod.mon.bbc.co.uk:1521"
session_cached_cursors = 100
smtp_out_server = "10.182.216.104:25"
job_queue_processes = 1000
audit_file_dest = "/u01/app/oracle/admin/SOCPROD/adump"
audit_trail = "NONE"
db_name = "SOCPROD"
db_unique_name = "SOCPRODCAV"
open_cursors = 300
sql_trace = FALSE
pga_aggregate_target = 4878M
dg_broker_start = TRUE
dg_broker_config_file1 = "+DAT1/socprodcav/dr1.dat"
dg_broker_config_file2 = "+FRA1/socprodcav/dr2.dat"
diagnostic_dest = "/u01/app/oracle"
trace_enabled = FALSE
Deprecated system parameters with specified values:
sql_trace
End of deprecated system parameter listing
Cluster communication is configured to use the following interface(s) for this instance
192.168.1.100
cluster interconnect IPC version:Oracle UDP/IP (generic)
IPC Vendor 1 proto 2
Mon Nov 07 09:49:49 2011
ksuapc : ORA-1033 foreground process starts before PMON
Mon Nov 07 09:49:49 2011
PMON started with pid=2, OS id=5508
Mon Nov 07 09:49:49 2011
VKTM started with pid=3, OS id=5511 at elevated priority
VKTM running at (10)millisec precision with DBRM quantum (100)ms
Mon Nov 07 09:49:49 2011
GEN0 started with pid=4, OS id=5515
Mon Nov 07 09:49:49 2011
DIAG started with pid=5, OS id=5518
Mon Nov 07 09:49:49 2011
DBRM started with pid=6, OS id=5520
Mon Nov 07 09:49:49 2011
PING started with pid=7, OS id=5522
Mon Nov 07 09:49:49 2011
PSP0 started with pid=8, OS id=5524
Mon Nov 07 09:49:49 2011
ACMS started with pid=9, OS id=5528
Mon Nov 07 09:49:49 2011
DIA0 started with pid=10, OS id=5530
Mon Nov 07 09:49:49 2011
LMON started with pid=11, OS id=5532
Mon Nov 07 09:49:52 2011
LMD0 started with pid=12, OS id=5534
Mon Nov 07 09:49:52 2011
LMS0 started with pid=13, OS id=5574 at elevated priority
Mon Nov 07 09:49:52 2011
LMS1 started with pid=14, OS id=5578 at elevated priority
Mon Nov 07 09:49:52 2011
RMS0 started with pid=15, OS id=5582
Mon Nov 07 09:49:52 2011
LMHB started with pid=16, OS id=5584
Mon Nov 07 09:49:52 2011
MMAN started with pid=17, OS id=5588
Mon Nov 07 09:49:52 2011
DBW0 started with pid=18, OS id=5590
Mon Nov 07 09:49:52 2011
LGWR started with pid=19, OS id=5592
Mon Nov 07 09:49:52 2011
CKPT started with pid=20, OS id=5594
Mon Nov 07 09:49:52 2011
SMON started with pid=21, OS id=5596
Mon Nov 07 09:49:52 2011
RECO started with pid=22, OS id=5598
Mon Nov 07 09:49:52 2011
RBAL started with pid=23, OS id=5602
Mon Nov 07 09:49:52 2011
ASMB started with pid=24, OS id=5604
Mon Nov 07 09:49:52 2011
MMON started with pid=25, OS id=5607
starting up 1 dispatcher(s) for network address '(ADDRESS=(PARTIAL=YES)(PROTOCOL=TCP))'...
Mon Nov 07 09:49:52 2011
MMNL started with pid=26, OS id=5611
NOTE: initiating MARK startup
starting up 1 shared server(s) ...
Starting background process MARK
Mon Nov 07 09:49:52 2011
MARK started with pid=28, OS id=5615
NOTE: MARK has subscribed
lmon registered with NM - instance number 1 (internal mem no 0)
Reconfiguration started (old inc 0, new inc 204)
List of instances:
1 2 (myinst: 1)
Global Resource Directory frozen
* allocate domain 0, invalid = TRUE
Communication channels reestablished
* domain 0 not valid according to instance 2
* domain 0 valid = 0 according to instance 2
Master broadcasted resource hash value bitmaps
Non-local Process blocks cleaned out
LMS 0: 0 GCS shadows cancelled, 0 closed, 0 Xw survived
LMS 1: 0 GCS shadows cancelled, 0 closed, 0 Xw survived
Set master node info
Submitted all remote-enqueue requests
Dwn-cvts replayed, VALBLKs dubious
All grantable enqueues granted
Post SMON to start 1st pass IR
Submitted all GCS remote-cache requests
Post SMON to start 1st pass IR
Fix write in gcs resources
Reconfiguration complete
Mon Nov 07 09:49:53 2011
LCK0 started with pid=30, OS id=5635
Starting background process RSMN
Mon Nov 07 09:49:53 2011
RSMN started with pid=31, OS id=5640
ORACLE_BASE not set in environment. It is recommended
that ORACLE_BASE be set in the environment
Reusing ORACLE_BASE from an earlier startup = /u01/app/oracle
Mon Nov 07 09:49:54 2011
DMON started with pid=32, OS id=5648
Mon Nov 07 09:49:54 2011
ALTER DATABASE MOUNT
This instance was first to mount
NOTE: Loaded library: System
SUCCESS: diskgroup DAT1 was mounted
SUCCESS: diskgroup FRA1 was mounted
ORA-00210: cannot open the specified control file
ORA-00202: control file: '+FRA1/socprod/controlfile/current.256.729791749'
ORA-17503: ksfdopn:2 Failed to open file +FRA1/socprod/controlfile/current.256.729791749
ORA-15012: ASM file '+FRA1/socprod/controlfile/current.256.729791749' does not exist
ORA-205 signalled during: ALTER DATABASE MOUNT...
NOTE: dependency between database SOCPRODCAV and diskgroup resource ora.DAT1.dg is established
Mon Nov 07 09:49:56 2011
Shutting down instance (abort)
License high water mark = 2
USER (ospid: 5707): terminating the instance
Instance terminated by USER, pid = 5707
Mon Nov 07 09:49:57 2011
Instance shutdown complete
[oracle@mondbop001 {SOCPROD1} trace]$
[/CODE]

[tspoonfr]

Alert log posted

Also when I enter the ASM utility something strange happens when I attempt to access the FRA disk - see below:-

[grid@mondbop001 ~]$ asmcmd
ASMCMD> lsdg
State Type Rebal Sector Block AU Total_MB Free_MB Req_mir_free_MB Usable_file_MB Offline_disks Voting_files Name
MOUNTED EXTERN N 512 4096 1048576 4094 3696 0 3696 0 N CRS/
MOUNTED EXTERN N 512 4096 1048576 282612 165221 0 165221 0 N DAT1/
MOUNTED NORMAL N 512 4096 1048576 284446 284258 0 142129 0 N FRA1/
ASMCMD> cd FRA1
ASMCMD> ls -ltr
WARNINGption 'r' is deprecated for 'ls'
please use 'reverse'

ASMCMD> ls -ll
ASMCMD> cd ..
ASMCMD> pwd
+
ASMCMD> cd CRS
ASMCMD> ls -ltr
WARNINGption 'r' is deprecated for 'ls'
please use 'reverse'

Type Redund Striped Time Sys Name
Y caversham1/
ASMCMD> cd ..
ASMCMD> cd DAT1
ASMCMD> ls -ltr
WARNINGption 'r' is deprecated for 'ls'
please use 'reverse'

Type Redund Striped Time Sys Name
Y SOCCAV/
Y SOCPROD/
Y SOCPRODCAV/
Y SOCTESTCAV/
ASMCMD>

As you can see I can access the DAT & CRS disks but not the FRA disk - and this is where the controlfile resides

Many thanks