10g recovery failed

[Chucks_k]

DB: 10.2.0.1 using ASM
OS: RHEL4

Hi all,

I am trying to create a single physical standby database of a RAC database. I am getting errors in recovery of the standby database:

channel ORA_DISK_1: reading from backup piece /backuparea2/rman/arch_653263073_30_1
ORA-19870: error reading backup piece /backuparea2/rman/arch_653263073_30_1
ORA-19504: failed to create file "+MMGDRS1"
ORA-17502: ksfdcre:4 Failed to create file +MMGDRS1
ORA-15041: diskgroup space exhausted
RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-03002: failure of recover command at 05/01/2008 18:31:11
RMAN-20506: no backup of archivelog found
RMAN-06053: unable to perform media recovery because of missing log
RMAN-06025: no backup of log thread 2 seq 5856 lowscn 1515363109 found to restore
RMAN-06025: no backup of log thread 2 seq 5852 lowscn 1515090680 found to restore
RMAN-06025: no backup of log thread 2 seq 5849 lowscn 1514881200 found to restore

The above errors point to 2 things:

1/ ASM diskgroup disk space exhausted
2/ RMAN cannot find logs

However, i checked in RMAN and the logs are actually in the backup:

BS Key  Size       Device Type Elapsed Time Completion Time
------- ---------- ----------- ------------ ---------------
26      6.35G      DISK        00:00:00     28-APR-08
        BP Key: 26   Status: AVAILABLE  Compressed: YES  Tag: TAG20080426T191624
        Piece Name: /backuparea2/rman/arch_653263073_30_1

  List of Archived Logs in backup set 26
  Thrd Seq     Low SCN    Low Time  Next SCN   Next Time
  ---- ------- ---------- --------- ---------- ---------

 2    5849    1514881200 11-APR-08 1514949499 11-APR-08
  2    5850    1514949499 11-APR-08 1515021811 11-APR-08
  2    5851    1515021811 11-APR-08 1515090680 11-APR-08
  2    5852    1515090680 11-APR-08 1515153750 11-APR-08
  2    5853    1515153750 11-APR-08 1515223399 11-APR-08
  2    5854    1515223399 11-APR-08 1515293087 11-APR-08
  2    5855    1515293087 11-APR-08 1515363109 11-APR-08
  2    5856    1515363109 11-APR-08 1515433658 11-APR-08

In my recovery log i can see several instances of, altough is seems restore of logs continues until i hit the error above:

ORA-19870: error reading backup piece /backuparea2/rman/arch_653244088_28_1
ORA-19504: failed to create file "+MMGDRS1"
ORA-17502: ksfdcre:4 Failed to create file +MMGDRS1
ORA-15041: diskgroup space exhausted
channel ORA_DISK_1: starting archive log restore to default destination
channel ORA_DISK_1: restoring archive log

I checked the diskspace in the asm dgroup:

SQL> select name,total_mb,free_mb from v$asm_disk;

NAME                             TOTAL_MB    FREE_MB
------------------------------ ---------- ----------
MMGDRS1_0000                       355328     104440
MMGDRS1_0001                      1120256     328785
MMGDRS1_0002                       533504     156752
MMGDRS1_0003                         1024          2
MMGDRS1_0004                         1024          3
MMGDRS1_0005                       560128     164389

6 rows selected.

So from above it seems there is some space so is the above really a case of disk space running out or something else?

Any advise will be greatly appreciated,
Thanks in advance,
Chucks

[abhaysk]

Can you attach complete log file?

[PAVB]

Error message is chrystal clear... ORA-15041: diskgroup space exhausted...
Two of your diskgroups are full...
Where is file "+MMGDRS1 supposed to be created?

[hrishy]

Hi

Whats the output of

SELECT name, free_mb, total_mb, free_mb/total_mb*100 "%" FROM v$asm_diskgroup;

what is the redundancy level used

regards
Hrishy

[Chucks_k]

Hi guys,

Thanks for all the input. I had space in some other disks hence could not understand why space wasnt being used. What i didnt realise is that as ASM stripes everything evenly across all disks, it was was getting stuck at at the disks with no space left. Hence I dropped those particular disks and after the rebalance the recovery was able to continue. The lesson here for me is to have evenly sized disks in the ASM disk group next time!

Thanks again guys,

Chucks