ASM Disk Group Problem

Hi all,

We are running 10.2.03 on Redhat 3.

We have ASM installed and are using EMC San for storage. We had a hardware panic and 3 (these were part of the failover group) of the 6 disks available went offline but reappeared as blank group name:

Code:

---

PATH                          NAME                          STATE
------------------------------ ------------------------------ --------
/dev/raw/raw6                                                NORMAL
/dev/raw/raw5                                                NORMAL
/dev/raw/raw4                                                NORMAL
                              DGGROUP_0005                  FORCING
                              DGGROUP_0004                  FORCING
                              DGGROUP_0003                  FORCING
/dev/raw/raw3                  DGGROUP_0002                  NORMAL
/dev/raw/raw2                  DGGROUP_0001                  NORMAL
/dev/raw/raw1                  DGGROUP_0000                  NORMAL

---

As can from above, raw6,5,4 now appear with no name but are online, the blanks are all offline. I did a alter diskgroup check all and this did not throw up any errors. Now how do i get the raw6,5,4 to be part of a failover group?

Code:

---

SQL> select failgroup, name from v$asm_disk;

FAILGROUP                      NAME
------------------------------ ------------------------------



                              DGGROUP_0005
                              DGGROUP_0004
                              DGGROUP_0003
PRIMARY                        DGGROUP_0002
PRIMARY                        DGGROUP_0001
PRIMARY                        DGGROUP_0000

---

Also does this mean the even though these disks are not part of the failure group, normal redundancy is still availabe as all 6 are still online? Any help will be greatly appreciated

Thanks in advance
Chucks

Have your sa to check if they came back with the same device id

Hi,

Thanks for that. My SA did mention that the disks have 'multiple' paths, not sure if thats of any relevance. What happens if the device id is different?

Thank you in advance

I am not an SA, but I think it may not be recognized as a same device.

Right, but in terms of Oracle ASM does this mean that we dont have redundacy. It looks as if 6 disks are online as per the output above which leads me to believe the redundancy is there albiet in an unconvential manner?

you should have used asmlib because you get device persistent name with it and not with raw devices

raw4,5 and 6 are probably not the ASM disks which you think they are

show us the output of device names

Hi Pando,

Thanks for that, I got not get ASMLib to work. ASM could not discover the disks available for some reason, hence that is why i went down the raw device route. But the ouput of device names do you mean config of the /etc/sysconfig/rawdevices file?

In /etc/sysconfig/rawdevices, I bound the devices as follows:

/dev/raw/raw1 /dev/emcpowerd1
/dev/raw/raw2 /dev/emcpowerh1
/dev/raw/raw3 /dev/emcpowerg1
/dev/raw/raw4 /dev/emcpowerc1
/dev/raw/raw5 /dev/emcpowere1
/dev/raw/raw6 /dev/emcpowerf1

Thanks in advance

show us the output of

select HEADER_STATUS,MODE_STATUS,STATE,LABEL from v$asm_disk;


you could not configure asmlib probably because you need to modify /etc/sysconfig/oracleasm and asmlib v2.0

Hi,

This is the output u requested:

Code:

---

SQL> select HEADER_STATUS,MODE_STATUS,STATE,LABEL from v$asm_disk;

HEADER_STATU MODE_ST STATE    LABEL
------------ ------- -------- -------------------------------
MEMBER      ONLINE  NORMAL
MEMBER      ONLINE  NORMAL
MEMBER      ONLINE  NORMAL
CANDIDATE    OFFLINE FORCING
CANDIDATE    OFFLINE FORCING
CANDIDATE    OFFLINE FORCING
MEMBER      ONLINE  NORMAL
MEMBER      ONLINE  NORMAL
MEMBER      ONLINE  NORMAL

9 rows selected

---

Thanks,
Chucks

check ASM alert and see if it shows any errors mounting the disks

another way is use kfed and look the disk headers which shows ASM metadata such as group name and disk name

to build kfed run

cd $ORACLE_HOME/lib
make -f ins_rdbms.mk ikfed

kfed read /dev/raw/raw4

Hi,

Thanks for that. I had a look at the alert log and it obviously complained around the time when the hardware panick happened, it tried to take some of the disks offline. Below is a snippet from when the problem started:

Code:

---

Thu Dec 20 14:00:42 2007
WARNING: offlining disk 4.4042410621 (DGGROUP_0004) with mask 0x3
NOTE: PST update: grp = 1, dsk = 4, mode = 0x6
Thu Dec 20 14:00:42 2007
Errors in file /opt/oracle/product/10.2.0/asm/admin/+ASM/bdump/+asm_gmon_8677.trc:
ORA-27091: unable to queue I/O
ORA-27072: File I/O error
Linux Error: 5: Input/output error
Additional information: 4
Additional information: 4088
Additional information: -1
Thu Dec 20 14:00:42 2007
WARNING: offlining disk 4.4042410621 (DGGROUP_0004) with mask 0x3
NOTE: PST update: grp = 1, dsk = 4, mode = 0x6
Thu Dec 20 14:00:42 2007
WARNING: offlining disk 4.4042410621 (DGGROUP_0004) with mask 0x3
NOTE: PST update: grp = 1, dsk = 4, mode = 0x6
Thu Dec 20 14:00:42 2007
WARNING: offlining disk 3.4042410622 (DGGROUP_0003) with mask 0x3
NOTE: PST update: grp = 1, dsk = 3, mode = 0x6
Thu Dec 20 14:00:43 2007
WARNING: offlining disk 3.4042410622 (DGGROUP_0003) with mask 0x3
NOTE: PST update: grp = 1, dsk = 3, mode = 0x6
Thu Dec 20 14:00:43 2007
Errors in file /opt/oracle/product/10.2.0/asm/admin/+ASM/bdump/+asm_gmon_8677.trc:
ORA-27091: unable to queue I/O
ORA-27072: File I/O error
Linux Error: 5: Input/output error

---

Apart from that i cannot see anything untoward, i also did a check all on the diskgroup and it returned without any errors.

Will dismounting the diskgroup and mounting it again help?

This kfed utility, i have not heard of before, will it essentially provide the same information we are getting from v$asm_disk?

Thanks very much for your help so far.

Chucks

kfed is a lower level utility, try it with raw4, 5 and 6