一、需求
Oracle 11gR1 RAC存储计划更换，数据库版本为11.1.0.7，无停机维护窗口。二、环境准备
1、主机环境、OS环境2、安装11.1.0.6.0版Clusterware（linux.x64_11gR1_clusterware.zip）3、安装11.1.0.6.0版RDBMS SOFTWARE（linux.x64_11gR1_database_1013.zip）4、升级到11.1.0.7.0版本（分别升级Clusterware和RDBMS）（p6890831_111070_Linux-x86-64.zip）5、APPLY 3个PSU，UPDATE到11.1.0.7.14p14739378_111070_Linux-x86-64.zip、p11724936_111070_Linux-x86-64.zip、p11724953_111070_Linux-x86-64.zip、p6880880_111000_Linux-x86-64.zip（Opatch）6、创建监听器、创建库SQL> select * from dba_registry_history;# crs_stat -t三、存储配置
因无停机时间，必须通过滚动方式，否则修改存储可能导致节点重启1、节点11）先停节点1实例# srvctl stop instance -d ora11g -i ora11g1# srvctl stop asm -n rac1# crs_stat -t2）连接新存储，重启iscsi识别存储设备# service iscsi restart3）fdisk格式化新增存储，并修改rawdevices文件让系统启动时挂载裸设备# vi /etc/sysconfig/rawdevices--添加如下：/dev/raw/raw1 /dev/sdb1...../dev/raw/raw5 /dev/sdf14）裸设备映射# vi /etc/udev/rules.d/60-raw.rules ACTION=="add", KERNEL=="/dev/sdb1",RUN+="/bin/raw /dev/raw/raw1 %N"ACTION=="add", ENV{MAJOR}=="8",ENV{MINOR}=="17",RUN+="/bin/raw /dev/raw/raw1 %M %m"......ACTION=="add", KERNEL=="/dev/sdf1",RUN+="/bin/raw /dev/raw/raw5 %N"ACTION=="add", ENV{MAJOR}=="8",ENV{MINOR}=="81",RUN+="/bin/raw /dev/raw/raw5 %M %m"KERNEL=="raw[1-5]", OWNER="oracle", GROUP="oinstall", MODE="640"5）重启udev# start_udev6）重启rawdevices服务# service rawdevices restart7）查看raw设备# ls -l /dev/raw8）创建ASM磁盘# /etc/init.d/oracleasm createdisk newasmdsk /dev/sdo1# oracleasm listdisks9）启动监听、实例# srvctl start listener -n rac1# srvctl start asm -n rac1# srvctl start instance -d ora11g -i ora11g1# crs_stat -t2、节点2
与节点1操作基本相同，仅因节点1已创建ASM磁盘，则节点2在节点1的步骤8）扫描即可# oracleasm scandisks# oracleasm listdisks通过如上操作，成功将新的存储连接并识别到划分好的lun，这期间是采用滚动方式的，所以业务不会中断。
四、OCR迁移
1、查看ocr存储位置以及其他信息# ocrcheck或查看/etc/oracle/ocr.loc/dev/raw/raw10、/dev/raw/raw11位ocr盘，且/dev/raw/raw10上是ocr的镜像2、确保crs健康运行# crsctl check crs3、通过ocrconfig命令可以在线迁移ocr# ocrconfig -replace ocr /dev/raw/raw1# ocrconfig -replace ocrmirror /dev/raw/raw2注：一个是ocr，另一个是ocrmirror，分别对应/etc/oracle/ocr.loc的ocrconfig_loc和ocrmirrorconfig_loc所指的位置4、检查是否迁移成功# ocrcheck扩展：Oracle 11g R2版本，ocrconfig命令增强了一些，更改ocr位置的命令也有所改变，如下：ocrconfig -replace <current filename> -replacement <new filename>五、迁移表决盘
1、查看表决磁盘当前存放路径# crsctl query css votedisk2、11g R2之前版本，只能采用添加删除的方式进行迁移。首先删除一个，因为当前有3个存放路径，所以删除一个并不产生影响# crsctl delete css votedisk /dev/raw/raw9# crsctl add css votedisk /dev/raw/raw5# crsctl delete css votedisk /dev/raw/raw8# crsctl add css votedisk /dev/raw/raw4注意：当试图删除表决磁盘首个存放路径的时候报错：# crsctl delete css votedisk /dev/raw/raw7Failure 8 with Cluster Synchronization Services while deleting voting disk.添加force选项强制删除# crsctl delete css votedisk /dev/raw/raw7 -force# crsctl query css votedisk# crsctl add css votedisk /dev/raw/raw33、查看表决盘是否迁移成功# crsctl query css votedisk扩展：Oracle 11g R2中crsctl命令增强很多，其中针对votedisk的操作也增多了replace功能：crsctl replace – replaces the location of voting files可见，在这个版本上DBA可以在线替换存储位置。六、ASM存储迁移
1、登录asm实例，查看当前磁盘组信息$ export ORACLE_SID=+ASM1$ sqlplus / as sysasmSQL> select NAME, ALLOCATION_UNIT_SIZE, STATE, TOTAL_MB, FREE_MB USABLE_FILE_MB from gv$asm_diskgroup;SQL> select failgroup, name from v$asm_disk where group_number=(select group_number from v$asm_diskgroup where name = 'DATADG');2、为DATADG磁盘中添加新的ASM磁盘SQL> alter diskgroup DATADG add disk 'ORCL:NEWASMDSK' name new_asmdsk REBALANCE POWER 10;注：添加多个asm磁盘方式：alter diskgroup DATADG add disk 'ORCL:ASM1', 'ORCL:ASM2', 'ORCL:ASM3' rebalance power n;3、查看asm磁盘rebalance进度，完成后v$asm_operation记录将消失SQL> select * from v$asm_operation;4、添加ASM磁盘后的状态SQL> select name, allocation_unit_size, state, total_mb, free_mb usable_file_mb from gv$asm_diskgroup;SQL> select failgroup, name, path from v$asm_disk where group_number=(select group_number from v$asm_diskgroup where name = 'DATADG');5、删除旧的ASM磁盘SQL> alter diskgroup DATADG drop disk OLDASMDSK rebalance power 5;SQL> select * from v$asm_operation;6、检查数据库状态以及数据情况至此，Oracle数据库已全部使用新存储了，可以在OS级别上卸载旧存储了。七、卸载旧存储
同样采用滚动方式卸载旧磁盘。1、节点1# srvctl stop instance -d ora11g -i ora11g1# srvctl stop asm -n rac1# crs_stat -t# oracleasm listdisks# /etc/init.d/oracleasm deletedisk OLDASMDSK修改/etc/sysconfig/rawdevices和/etc/udev/rules.d/60-raw.rules，清除旧磁盘信息，并重启udev和rawdevices服务。# crs_stat -t# srvctl start listener -n rac1# srvctl start asm -n rac1# srvctl start instance -d ora11g -i ora11g1# crs_stat -t2、节点2
# srvctl stop instance -d ora11g -i ora11g2# srvctl stop asm -n rac2# crs_stat -t# oracleasm scandisks# oracleasm listdisks修改/etc/sysconfig/rawdevices和/etc/udev/rules.d/60-raw.rules，清除旧磁盘信息，并重启udev和rawdevices服务。# crs_stat -t# srvctl start listener -n rac2# srvctl start asm -n rac2# srvctl start instance -d ora11g -i ora11g2# crs_stat -t 注：确认系统可用性（生产环境下可能不允许），手动重启OS[root@rac1 ~]# srvctl stop database -d ora11g[root@rac1 ~]# srvctl stop asm -n rac1[root@rac1 ~]# srvctl stop asm -n rac2[root@rac1 ~]# srvctl stop nodeapps -n rac1[root@rac1 ~]# srvctl stop nodeapps -n rac2[root@rac1 ~]# crsctl stop crs[root@rac1 ~]# shutdown -r now[root@rac2 ~]# crsctl stop crs
[root@rac2 ~]# shutdown -r now重启之后：# raw -qa# ocrcheck# crsctl query css votedisk# 检查数据文件与数据状态