Instance crashes

Printable View

06-12-2006, 02:25 PM
Eberman

Instance crashes

Hi,
Every morning for the last 3 days around 5 AM we have production Instance crash due to following errors from Alert log:

ORA-00221: error on write to controlfile
ORA-00206: error in writing (block 3, # blocks 1) of controlfile
ORA-00202: controlfile: '/u05/oracle/gvr1/gvr1/control03.ctl'
ORA-27072: File I/O error
IBM AIX RISC System/6000 Error: 22: Invalid argument
Additional information: 7
Mon Jun 12 06:14:01 2006
CKPT: terminating instance due to error 221
Instance terminated by CKPT, pid = 6472600

We had no recent changes in Oracle or on OS level. Database was up for at least 7-8 month without any big issues.

If anybody could give me some help on this would be great!

Eugene.
06-12-2006, 02:30 PM
davey23uk

it tells you exactly what the problem is, what file it was touching and the os error = what more info could you want?
06-12-2006, 02:33 PM
Eberman

I guess my question is what trigger this errors and how I can fix it

Thanks for reply

Eugene
06-12-2006, 02:48 PM
ixion

Hardware issue?

Check your OS messages for I/O errors.

Check for an automated admin operation going on at that time? ie maybe your using a SAN or other device.??? and it is going down?

Just guessing.
06-12-2006, 02:57 PM
Eberman

Just spoke with my UNIX admin and they do not see any OS I/o messages
06-12-2006, 03:04 PM
ixion

Got to be some problem!
Your DB is complaining that it cannot write! There should be some OS/issue flags.

It must be able to write to the control file. Each and every copy of the control file indicated in the pfile/spfile.

Can you do an ls -l on the file? What are the permissions? Are these being changed?
Who owns the file?
Are there any other files in the same directory? Do you get any other write errors, in the alert log?

Here is an idea:
Copy control01.ctl (I'm guessing you have 3 ctl files) to control04.ctl
Leave control03.ctl alone. (don't delete rename)
Modify your pfile/spfile to utilize control04.ctl instead of 03.

Then see if you get the same issue.
06-12-2006, 03:06 PM
Eberman

Thanks!

I will try

Eugene.
06-12-2006, 03:09 PM
dbasan

There is a reported bug with the same error for HP Tru64. Check if there is any patch for the bug in metalink for AIX.
06-12-2006, 03:41 PM
Mr.Hanky

It has to be on the OS side, almost sounds like they implemented something that is temporarily locking that file. Especially since it happens at the same time every day.
06-12-2006, 03:48 PM
Eberman

We use RMAN for backup (hot backup), However I find out today that we use TSM software to backup UNIX server. It does backup of Oracle home directory which is on /u01/app.. Could this caused this errors? Should I stop this back up job?

Eugene.
06-12-2006, 04:09 PM
matthew00

I was going to say, your back or third party software. Basically, you have some processes that tried to access to the controlfiles and the controlfiles was updated at the same time, the file locked up so the DB crashed.
06-12-2006, 04:47 PM
Mr.Hanky

Quote:

Originally Posted by Eberman

We use RMAN for backup (hot backup), However I find out today that we use TSM software to backup UNIX server. It does backup of Oracle home directory which is on /u01/app.. Could this caused this errors? Should I stop this back up job?

Eugene.

Are you in charge of the OS or does you shop have a sytem admin for that.

It would seem strange that something that has been running with no problems would suddenly start spitting out errors. Has anything changed on the backup for the OS?

Can you just skip the control file mount points without cancelling the entire backup?
06-13-2006, 05:39 AM
KenCunningham

If your controlfile is in the /u05/oracle filesystem and you're just backing up /u01/app, I doubt it. What time is the backup scheduled?
06-13-2006, 07:46 AM
Eberman

Backup Between 5 and AM. Happend this morning again. Even without OS Backup (we removed it from Scheduler)

Also once I restart Oracke everything works Ok and this control file is being updated fine. Until 5 AM?!

Eugene.
06-13-2006, 10:40 AM
ixion

Eugene,

Its unlikely that your backup software is causing this. You don't really need to backup the Oracle control files as part of the OS backup. These are part of the DB.

This sounds like an oracle bug. What version of oracle?
I'd file a bug with Oracle.

Ken
06-13-2006, 10:47 AM
abhaysk

Not really a bug.. we faced the write errors on some of the datafiles and was due to the space on that mount point being full.

Check the free space

Abhay.
06-13-2006, 10:47 AM
Eberman

Oracle version is 10.1.0.3.0

I opened SR with Oracle and they suggest to download Remote Diagnostic Agent and run and then send output report to them

E.
06-13-2006, 10:51 AM
Mr.Hanky

Quote:

Originally Posted by abhaysk

Not really a bug.. we faced the write errors on some of the datafiles and was due to the space on that mount point being full.

Check the free space
Abhay.

We are thinking along the same lines but when it happens at the same time every day it should not be free space, although it is certainly worth investigating.

You know, it could be that they are running a batch file that is blowing up that mount point at the same time every day. Where do you have your TEMP and UNDO files located?
Cause if they swell up during a batch job, that could cause the error. Then by the time you get in they shrink down and everything looks fine.
06-13-2006, 10:52 AM
ixion

Quote:

Originally Posted by abhaysk

Not really a bug.. we faced the write errors on some of the datafiles and was due to the space on that mount point being full.

Check the free space

Abhay.

Ahh,

But then you'd get errors in your OS messages saying your file system was full. And you'd get a different error in your alert log. He had no OS messages.
06-13-2006, 10:56 AM
Eberman

The only space issue we found on Oracle Home drive. It was 85% full. We just cleaned up to 50%.

However control file drive has plenty of space

E.
06-13-2006, 10:59 AM
ixion

Eugene,
what is your hardware configuration on the disk where the control file is stored?
What is you fs type and mounting options?
06-13-2006, 11:03 AM
abhaysk

Well any n/w outages at 5am daily?.. and that ur control file is remote to the server?
06-13-2006, 11:15 AM
Eberman

Hardware config:

aix- jsf2
all oracle filessystems are mirroed
/dev/u03lv -- /u03 jfs2 125829120 rbr,rbw,dio,rw no no

Also no network issues.

Would UNIX Server reboot solve this issue? We had not reboot this box for long time

E.
06-13-2006, 12:34 PM
ixion

A reboot probably won't help, but it wouldn't hurt either. Plus if there are hardware issues the startup diagnostics should help pick these up. The startup FSCK should also pick up any file system issues. Its worth a try.

What is the storage device? How is it attached? IS it an EMC or some other SAN?
Can you relocate control03 to say an internal disk (temporarily)?

Just a thought. I wonder if you're getting intermittent write errors on the mirrored device? hardware mirroring or software?
06-13-2006, 01:54 PM
Eberman

All I know its an ibm fast-t900

We will reboot Unix Server at night time and I will move control files just like suggested yesterday. See if it helps us.

Eugene.
06-13-2006, 03:12 PM
ixion

Eugene..

I looked. The hardware supports internal mirroring. It is a FCAL (fibre chanel arbitrated loop) device. There are basically two connections to each physical drive on seperate loops.

I'd get you sys admin too look and see if there are any hardware errors and run diagnostics on this as well during your reboot/downtime.

Is this located on site? I once had a strange problem with db crashing. The cleaning personel were unplugging the disk subsystem to plug in their vacum.
06-14-2006, 03:00 AM
Thomasps

But, why the crash is happeneing only at 5AM..? If there is any general H/w issue, the DB can crash at any point not restricted to 5AM?
06-14-2006, 12:28 PM
Eberman

Last nigth we stop Unix server OS backup and stop RMAN backup as well. No downtime!!

We will bring back RMAN only to see if this Unix Server OS backup casing trouble.

Eugene
06-14-2006, 01:14 PM
ixion

There You go... Its one of those then. Process of elimination from here out.

Let us know which one caused the problem.
Ken
06-18-2006, 03:49 AM
nguyenjl

first mount the db and issue an recover automatic database may solve the problem if not recreate the controlfile ..I think you know the rest.
06-18-2006, 08:09 AM
davey23uk

Quote:

Originally Posted by nguyenjl

first mount the db and issue an recover automatic database may solve the problem if not recreate the controlfile ..I think you know the rest.

what are you talking about - he isnt doing any sort of recovery
06-19-2006, 05:31 AM
reydp

Quote:

Originally Posted by Eberman

The only space issue we found on Oracle Home drive. It was 85% full. We just cleaned up to 50%.

However control file drive has plenty of space

E.

If I were to troubleshoot this problem. I start with creating new controlfile with different file system and drop the controlfile which got problem. And see it from there.
06-19-2006, 11:53 AM
Mr.Hanky

Was this problem ever resolved?
06-19-2006, 12:11 PM
marist89

Every day at 05:00? It's something that is scheduled.
06-19-2006, 12:13 PM
Mr.Hanky

I usually schedule my instance crashes for Friday afternoon, around 4:30 pm.
06-19-2006, 12:19 PM
marist89

Suckin up that OT, eh?
06-20-2006, 02:42 AM
reydp

Quote:

Originally Posted by marist89

Every day at 05:00? It's something that is scheduled.

maybe there is a command: adcron -e to see crashdb sched every 5am
06-20-2006, 11:00 AM
ixion

I think this was resolved: See thread originators post on page 3.

All times are GMT -4. The time now is 04:29 AM.