Every morning for the last 3 days around 5 AM we have production Instance crash due to following errors from Alert log:
ORA-00221: error on write to controlfile
ORA-00206: error in writing (block 3, # blocks 1) of controlfile
ORA-00202: controlfile: '/u05/oracle/gvr1/gvr1/control03.ctl'
ORA-27072: File I/O error
IBM AIX RISC System/6000 Error: 22: Invalid argument
Additional information: 7
Mon Jun 12 06:14:01 2006
CKPT: terminating instance due to error 221
Instance terminated by CKPT, pid = 6472600
We had no recent changes in Oracle or on OS level. Database was up for at least 7-8 month without any big issues.
If anybody could give me some help on this would be great!
it tells you exactly what the problem is, what file it was touching and the os error = what more info could you want?
I guess my question is what trigger this errors and how I can fix it
Thanks for reply
Check your OS messages for I/O errors.
Check for an automated admin operation going on at that time? ie maybe your using a SAN or other device.??? and it is going down?
Just spoke with my UNIX admin and they do not see any OS I/o messages
Got to be some problem!
Your DB is complaining that it cannot write! There should be some OS/issue flags.
It must be able to write to the control file. Each and every copy of the control file indicated in the pfile/spfile.
Can you do an ls -l on the file? What are the permissions? Are these being changed?
Who owns the file?
Are there any other files in the same directory? Do you get any other write errors, in the alert log?
Here is an idea:
Copy control01.ctl (I'm guessing you have 3 ctl files) to control04.ctl
Leave control03.ctl alone. (don't delete rename)
Modify your pfile/spfile to utilize control04.ctl instead of 03.
Then see if you get the same issue.
I will try
There is a reported bug with the same error for HP Tru64. Check if there is any patch for the bug in metalink for AIX.
"What is past is PROLOGUE"
It has to be on the OS side, almost sounds like they implemented something that is temporarily locking that file. Especially since it happens at the same time every day.
I remember when this place was cool.
We use RMAN for backup (hot backup), However I find out today that we use TSM software to backup UNIX server. It does backup of Oracle home directory which is on /u01/app.. Could this caused this errors? Should I stop this back up job?
Click Here to Expand Forum to Full Width