Error on SHUTDOWN ABORT

Seems the DB is hanging and getting errors:



We are doing the abort option because we have an EMC DMX 800 and Timefinder(EMC) software is doing a 'cold' snap at this time.DB goes down,snap is taken,DB comes backup. We started having this probelm after some swithces were rebooted that connect to the SAN which in turn talks to our host(AIX 5.3 box). We did have several file systems on the UNIX box that did not mount and we had to run fsck's on them and mount them manually.



from the Oracle alert log:

Mon Jun 27 04:04:43 2005

Thread 1 advanced to log sequence 13672

Mon Jun 27 04:04:43 2005

Current log# 4 seq# 13672 mem# 0: /u03/oradata/PROD/redo_PROD_04.log

Current log# 4 seq# 13672 mem# 1: /u30/oradata/PROD/redo_PROD_04.log

Mon Jun 27 04:04:43 2005

ARC1: Evaluating archive log 1 thread 1 sequence 13671

Mon Jun 27 04:04:44 2005

ARC1: Beginning to archive log 1 thread 1 sequence 13671

Creating archive destination LOG_ARCHIVE_DEST_1: '/u04/oradata/PROD/arch/S70_PROD_0000013671.arc'

Mon Jun 27 04:04:47 2005

Shutting down instance (abort)

License high water mark = 53

Mon Jun 27 04:04:51 2005

ARC1: Completed archiving log 1 thread 1 sequence 13671

Mon Jun 27 04:09:48 2005

PMON: terminating instance due to error 1092

Mon Jun 27 04:09:49 2005

DBW0: terminating instance due to error 1092

Mon Jun 27 04:09:50 2005

LGWR: terminating instance due to error 1092

Mon Jun 27 04:09:50 2005

CKPT: terminating instance due to error 1092

Mon Jun 27 04:09:52 2005

Instance terminated by PMON, pid = 206804

Mon Jun 27 05:13:55 2005

Starting ORACLE instance (normal)

Disable cache advisory with old cache parameters

LICENSE_MAX_SESSION = 750

LICENSE_SESSIONS_WARNING = 500

SCN scheme 3

LICENSE_MAX_USERS = 0

SYS auditing is disabled

Starting up ORACLE RDBMS Version: 9.2.0.4.0.

System parameters with non-default values:

processes = 1800

sessions = 1985

license_max_sessions = 750

license_sessions_warning = 500

event = 10932 trace name context level 128

shared_pool_size = 637534208

sga_max_size = 1585351824

control_files = /u01/oradata/PROD/cont_PROD_01.ctl, /u02/oradata/PROD/cont_PROD_02.ctl, /u03/oradata/PROD/cont_PROD_03.ctl

db_block_buffers = 50000

db_block_size = 8192

db_writer_processes = 1

compatible = 9.2.0.0.0

log_archive_start = TRUE

log_archive_dest = /u04/oradata/PROD/arch

log_archive_format = S70_PROD_%S.arc

log_buffer = 327680

log_checkpoint_interval = 10000

db_files = 200

db_file_multiblock_read_count= 16

dml_locks = 4800

undo_management = AUTO

undo_tablespace = UNDO1

undo_retention = 18000

max_enabled_roles = 50

remote_login_passwordfile= EXCLUSIVE

db_domain = WORLD

utl_file_dir = /u02/sct/banner/general/UTEP_rpt_mods/, /dpaustin

background_dump_dest = /u01/app/oracle/admin/PROD/bdump

user_dump_dest = /u01/app/oracle/admin/PROD/udump

core_dump_dest = /u01/app/oracle/admin/PROD/cdump

sort_area_size = 200000

db_name = PROD

open_cursors = 4096

ifile = /u01/app/oracle/admin/PROD/pfile/configPROD.ora

os_authent_prefix = OPS$

optimizer_mode = rule

PMON started with pid=2

DBW0 started with pid=3

LGWR started with pid=4

CKPT started with pid=5

SMON started with pid=6

RECO started with pid=7

Mon Jun 27 05:13:57 2005

ARCH: STARTING ARCH PROCESSES

ARC0 started with pid=8

ARC0: Archival started

ARC1 started with pid=9

ARC1: Archival started

Mon Jun 27 05:13:58 2005

ARCH: STARTING ARCH PROCESSES COMPLETE

Mon Jun 27 05:13:58 2005

ARC0: Thread not mounted

Mon Jun 27 05:13:58 2005

ARC1: Thread not mounted

Mon Jun 27 05:13:58 2005

ALTER DATABASE MOUNT

Mon Jun 27 05:14:02 2005

Successful mount of redo thread 1, with mount id 4268844950.

Mon Jun 27 05:14:02 2005

Database mounted in Exclusive Mode.

Completed: ALTER DATABASE MOUNT

Mon Jun 27 05:14:02 2005

ALTER DATABASE OPEN

Mon Jun 27 05:14:02 2005

Beginning crash recovery of 1 threads

Mon Jun 27 05:14:03 2005

Started first pass scan

Mon Jun 27 05:14:03 2005

Completed first pass scan

0 redo blocks read, 0 data blocks need recovery

Mon Jun 27 05:14:03 2005

Started recovery at

Thread 1: logseq 13672, block 11, scn 0.4085255294

Recovery of Online Redo Log: Thread 1 Group 4 Seq 13672 Reading mem 0

Mem# 0 errs 0: /u03/oradata/PROD/redo_PROD_04.log

Mem# 1 errs 0: /u30/oradata/PROD/redo_PROD_04.log

Mon Jun 27 05:14:03 2005

Completed redo application

Mon Jun 27 05:14:03 2005

Ended recovery at

Thread 1: logseq 13672, block 11, scn 0.4085275295

0 data blocks read, 0 data blocks written, 0 redo blocks read

Crash recovery completed successfully

Mon Jun 27 05:14:03 2005

LGWR: Primary database is in CLUSTER CONSISTENT mode

Thread 1 advanced to log sequence 13673

Thread 1 opened at log sequence 13673

Current log# 2 seq# 13673 mem# 0: /u03/oradata/PROD/redo_PROD_02.log

Current log# 2 seq# 13673 mem# 1: /u30/oradata/PROD/redo_PROD_02.log

Successful open of redo thread 1.

Mon Jun 27 05:14:03 2005

MTTR advisory is disabled because FAST_START_MTTR_TARGET is not set

Mon Jun 27 05:14:03 2005

SMON: enabling cache recovery

Mon Jun 27 05:14:03 2005

ARC0: Evaluating archive log 4 thread 1 sequence 13672

ARC0: Beginning to archive log 4 thread 1 sequence 13672

Creating archive destination LOG_ARCHIVE_DEST_1: '/u04/oradata/PROD/arch/S70_PROD_0000013672.arc'

ARC0: Completed archiving log 4 thread 1 sequence 13672

Mon Jun 27 05:14:04 2005

Undo Segment 1 Onlined

Undo Segment 2 Onlined

Undo Segment 3 Onlined

Undo Segment 4 Onlined

Undo Segment 5 Onlined

Undo Segment 6 Onlined

Undo Segment 7 Onlined

Undo Segment 8 Onlined

Undo Segment 9 Onlined

Undo Segment 10 Onlined

Successfully onlined Undo Tablespace 1.

Mon Jun 27 05:14:04 2005

SMON: enabling tx recovery

Mon Jun 27 05:14:04 2005

Database Characterset is US7ASCII

replication_dependency_tracking turned off (no async multimaster replication found)

Mon Jun 27 05:14:07 2005

Instance terminated by USER, pid = 190418

Completed: ALTER DATABASE OPEN

Mon Jun 27 05:14:08 2005

Starting ORACLE instance (normal)

Disable cache advisory with old cache parameters

LICENSE_MAX_SESSION = 750

LICENSE_SESSIONS_WARNING = 500

SCN scheme 3

LICENSE_MAX_USERS = 0

SYS auditing is disabled

Starting up ORACLE RDBMS Version: 9.2.0.4.0.

System parameters with non-default values:

processes = 1800

sessions = 1985

license_max_sessions = 750

license_sessions_warning = 500

event = 10932 trace name context level 128

shared_pool_size = 637534208

sga_max_size = 1585351824

control_files = /u01/oradata/PROD/cont_PROD_01.ctl, /u02/oradata/PROD/cont_PROD_02.ctl, /u03/oradata/PROD/cont_PROD_03.ctl

db_block_buffers = 50000

db_block_size = 8192

db_writer_processes = 1

compatible = 9.2.0.0.0

log_archive_start = TRUE

log_archive_dest = /u04/oradata/PROD/arch

log_archive_format = S70_PROD_%S.arc

log_buffer = 327680

log_checkpoint_interval = 10000

db_files = 200

db_file_multiblock_read_count= 16

dml_locks = 4800

undo_management = AUTO

undo_tablespace = UNDO1

undo_retention = 18000

max_enabled_roles = 50

remote_login_passwordfile= EXCLUSIVE

db_domain = WORLD

utl_file_dir = /u02/sct/banner/general/UTEP_rpt_mods/, /dpaustin

background_dump_dest = /u01/app/oracle/admin/PROD/bdump

user_dump_dest = /u01/app/oracle/admin/PROD/udump

core_dump_dest = /u01/app/oracle/admin/PROD/cdump

sort_area_size = 200000

db_name = PROD

open_cursors = 4096

ifile = /u01/app/oracle/admin/PROD/pfile/configPROD.ora

os_authent_prefix = OPS$

optimizer_mode = rule

PMON started with pid=2

DBW0 started with pid=3

LGWR started with pid=4

CKPT started with pid=5

SMON started with pid=6

RECO started with pid=7

Mon Jun 27 05:14:10 2005

ARCH: STARTING ARCH PROCESSES

ARC0 started with pid=8

ARC0: Archival started

ARC1 started with pid=9

ARC1: Archival started



Any help is appreciated.

Mike

Sorry, Oracle RDBMS 9.2.0.4 O/S AIX 5.3

The answer is right in the log...

Code:

---

Mon Jun 27 04:04:44 2005
ARC1: Beginning to archive log 1 thread 1 sequence 13671
Creating archive destination LOG_ARCHIVE_DEST_1: '/u04/oradata/PROD/arch/S70_PROD_0000013671.arc'
Mon Jun 27 04:04:47 2005
Shutting down instance (abort)
License high water mark = 53
Mon Jun 27 04:04:51 2005
ARC1: Completed archiving log 1 thread 1 sequence 13671
Mon Jun 27 04:09:48 2005
PMON: terminating instance due to error 1092

---

The archiver started archiving log 13671 at 04:44, you shutdown abort at 04:47 and the archiver finished at 04:51.

Okay so any suggestions on how to fix this?

Mike

Just curious why you didn't check metalink or asktom? (both mention this issue)