We created a diskgroup with normal redundancy, and were wasting half the space on the mirror as a result. We are completely striped and mirrored in EMC, so a mirror buys us no additional protection.
We wanted to recreate the diskgroup with external redundancy rather than creating a new diskgroup and copying all the data to it. This has the additional complexity that the cluster registry and voting disk are also stored in the diskgroup being recreated.
Below is what we did to accomplish this.
STEPS
Ensure you have a good backup of all databases, including the spfile if stored in ASM.
Shutdown applications using and databases in the cluster
Verify cluster status
[root@expressdb1 etc]# /home/oracle/crsstat.sh
NAME TARGET STATE SERVER STATE_DETAILS
------------------------- ---------- ---------- ------------ ----------------------------
ora.DATA.dg ONLINE ONLINE expressdb1
ora.asm ONLINE ONLINE expressdb1 Started
ora.net1.network ONLINE ONLINE expressdb1
ora.ons ONLINE ONLINE expressdb1
ora.LISTENER_SCAN1.lsnr ONLINE ONLINE expressdb1
ora.LISTENER_SCAN2.lsnr ONLINE ONLINE expressdb1
ora.LISTENER_SCAN3.lsnr ONLINE ONLINE expressdb1
ora.cvu ONLINE ONLINE expressdb1
ora.expressdb1.vip ONLINE ONLINE expressdb1
ora.oc4j ONLINE ONLINE expressdb1
ora.scan1.vip ONLINE ONLINE expressdb1
ora.scan2.vip ONLINE ONLINE expressdb1
ora.scan3.vip ONLINE ONLINE expressdb1
Backup OCR and voting disk
[root@expressdb1 etc]# /u01/app/11.2.0/grid/bin/ocrconfig -manualbackup
expressdb1 2013/12/30 16:22:53 /u01/app/11.2.0/grid/cdata/expresscrs/backup_20131230_162253.ocr
Backup ASM spfile
[root@expressdb1 etc]# su - grid
[grid@expressdb1 ~]$ profile
1) +ASM1
2) express
Please select an ORACLE environment from the list above: 1
*************************************************
USER...............grid
ORACLE_SID.........+ASM1
ORACLE_HOME......../u01/app/11.2.0/grid
ORACLE_BASE......../u01/app/oracle
HOST...............expressdb1 192.168.3.50
*************************************************
expressdb1:grid:+ASM1:/home/grid# asmcmd
ASMCMD> spget
+DATA/expresscrs/asmparameterfile/registry.253.835546595
ASMCMD> spcopy +DATA/expresscrs/asmparameterfile/registry.253.835546595 /tmp/asm.ora
ASMCMD> exit
Stop cluster
expressdb1:grid:+ASM1:/home/grid# exit
logout
[root@expressdb1 etc]# /u01/app/11.2.0/grid/bin/crsctl stop crs -f
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'expressdb1'
CRS-2673: Attempting to stop 'ora.crsd' on 'expressdb1'
CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on 'expressdb1'
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'expressdb1' has completed
CRS-4133: Oracle High Availability Services has been stopped.
Zero out ASM diskgroup disks
[root@expressdb1 etc]# dd if=/dev/zero bs=10M count=10 of=/dev/mapper/oracle1p1
10+0 records in
10+0 records out
104857600 bytes (105 MB) copied, 9.35223 s, 11.2 MB/s
[root@expressdb1 etc]# dd if=/dev/zero bs=10M count=10 of=/dev/mapper/oracle2p1
10+0 records in
10+0 records out
104857600 bytes (105 MB) copied, 9.50856 s, 11.0 MB/s
[root@expressdb1 etc]# dd if=/dev/zero bs=10M count=10 of=/dev/mapper/oracle3p1
10+0 records in
10+0 records out
104857600 bytes (105 MB) copied, 9.50625 s, 11.0 MB/s
Restart cluster in exclusive mode
[root@expressdb1 etc]# /u01/app/11.2.0/grid/bin/crsctl start crs -excl -nocrs
CRS-4123: Oracle High Availability Services has been started.
CRS-2672: Attempting to start 'ora.mdnsd' on 'expressdb1'
CRS-2676: Start of 'ora.asm' on 'expressdb1' succeeded
Recreate diskgroup with external redundancy
[root@expressdb1 etc]# su - grid
[grid@expressdb1 ~]$ profile
1) +ASM1
2) express
Please select an ORACLE environment from the list above: 1
*************************************************
USER...............grid
ORACLE_SID.........+ASM1
ORACLE_HOME......../u01/app/11.2.0/grid
ORACLE_BASE......../u01/app/oracle
HOST...............expressdb1 192.168.3.50
*************************************************
expressdb1:grid:+ASM1:/home/grid# sqlplus / as sysasm
SQL*Plus: Release 11.2.0.3.0 Production on Mon Dec 30 16:48:57 2013
Copyright (c) 1982, 2011, Oracle. All rights reserved.
Connected to:
Oracle Database 11g Enterprise Edition Release 11.2.0.3.0 - 64bit Production
With the Real Application Clusters and Automatic Storage Management options
SQL> create diskgroup data external redundancy disk '/dev/mapper/oracle1p1','/dev/mapper/oracle2p1','/dev/mapper/oracle3p1';
Diskgroup created.
SQL> alter diskgroup data set attribute 'compatible.asm' = '11.2.0.0.0';
Diskgroup altered.
SQL> exit
Disconnected from Oracle Database 11g Enterprise Edition Release 11.2.0.3.0 - 64bit Production
With the Real Application Clusters and Automatic Storage Management options
Copy backup spfile into ASM
expressdb1:grid:+ASM1:/home/grid# asmcmd
ASMCMD> spcopy /tmp/asm.ora +DATA/expresscrs/asmparameterfile/asm.ora
ASMCMD> spset +DATA/expresscrs/asmparameterfile/asm.ora
ASMCMD> exit
Restart cluster
expressdb1:grid:+ASM1:/home/grid# exit
logout
[root@expressdb1 etc]# /u01/app/11.2.0/grid/bin/crsctl stop crs -f
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'expressdb1'
CRS-4133: Oracle High Availability Services has been stopped.
[root@expressdb1 etc]# /u01/app/11.2.0/grid/bin/crsctl start crs -excl -nocrs
CRS-4123: Oracle High Availability Services has been started.
CRS-2676: Start of 'ora.asm' on 'expressdb1' succeeded
Restore the OCR and voting disk
[root@expressdb1 etc]# /u01/app/11.2.0/grid/bin/ocrconfig -restore /u01/app/11.2.0/grid/cdata/expresscrs/backup_20131230_162253.ocr
[root@expressdb1 etc]# /u01/app/11.2.0/grid/bin/crsctl replace votedisk +DATA
Successful addition of voting disk 4ee46a25dc614ff9bfc94362cc62cf82.
Successfully replaced voting disk group with +DATA.
CRS-4266: Voting file(s) successfully replaced
Verify restored files
[root@expressdb1 etc]# su - grid
[grid@expressdb1 ~]$ profile
1) +ASM1
2) express
Please select an ORACLE environment from the list above: 1
*************************************************
USER...............grid
ORACLE_SID.........+ASM1
ORACLE_HOME......../u01/app/11.2.0/grid
ORACLE_BASE......../u01/app/oracle
HOST...............expressdb1 192.168.3.50
*************************************************
expressdb1:grid:+ASM1:/home/grid# /u01/app/11.2.0/grid/bin/crsctl query css votedisk
## STATE File Universal Id File Name Disk group
-- ----- ----------------- --------- ---------
1. ONLINE 4ee46a25dc614ff9bfc94362cc62cf82 (/dev/mapper/oracle1p1) [DATA]
Located 1 voting disk(s).
expressdb1:grid:+ASM1:/home/grid#
Restart clusterware in normal mode, restore and recover database