Oracle 19c RAC ASM 密码文件恢复方案四：创建新密码文件覆盖恢复
简介
问题描述：在Oracle19cRAC中启动CRS服务，其中一个节点的ASM实例无法自动启动，需要手动执行startup命令，且启动实例后集群服务正常。
这种情况就有可能是ASM密码文件错误造成的，当ASM的密码文件丢失、损坏，或手动更替后，就会出现这种问题。
为解决这个错误，我将通过四个测试，介绍四种不同的恢复ASM密码文件的方法。
1、通过asmcmd --nocp credfix命令恢复。
2、通过密码文件备份恢复。
3、没有密码文件备份，同时版本低于19.8，通过升级补丁后执行asmcmd --nocp credfix命令恢复。
4、直接创建密码文件恢复
测试四：
直接创建密码文件恢复
测试环境：
Oracle19c双节点RAC集群
rac1 节点一
rac2 节点二
适用场景：
无ASM密码文件备份
版本低于19.8，无法使用credfix
需从 OCR 中提取 CRSUSER__ASM_00X 密码
修复方法：
创建ASM密码文件，从OCR中找到 CRSUSER__ASM_00X的密码，使用找到的密码创建 CRSUSER__ASM_00X 用户。
1.查看集群状态和配置
bash
[grid@rac1 ~]$ crsctl stat res -t
--------------------------------------------------------------------------------
Name Target State Server State details
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.LISTENER.lsnr
ONLINE ONLINE rac1 STABLE
ONLINE ONLINE rac2 STABLE
ora.chad
ONLINE ONLINE rac1 STABLE
ONLINE ONLINE rac2 STABLE
ora.net1.network
ONLINE ONLINE rac1 STABLE
ONLINE ONLINE rac2 STABLE
ora.ons
ONLINE ONLINE rac1 STABLE
ONLINE ONLINE rac2 STABLE
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.ASMNET1LSNR_ASM.lsnr(ora.asmgroup)
1 ONLINE ONLINE rac1 STABLE
2 ONLINE ONLINE rac2 STABLE
3 ONLINE OFFLINE STABLE
ora.DATANEW.dg(ora.asmgroup)
1 ONLINE ONLINE rac1 STABLE
2 ONLINE ONLINE rac2 STABLE
3 OFFLINE OFFLINE STABLE
ora.DG_DATA.dg(ora.asmgroup)
1 ONLINE ONLINE rac1 STABLE
2 ONLINE ONLINE rac2 STABLE
3 OFFLINE OFFLINE STABLE
ora.DG_FRA.dg(ora.asmgroup)
1 ONLINE ONLINE rac1 STABLE
2 ONLINE ONLINE rac2 STABLE
3 OFFLINE OFFLINE STABLE
ora.DG_OCR.dg(ora.asmgroup)
1 ONLINE ONLINE rac1 STABLE
2 ONLINE ONLINE rac2 STABLE
3 OFFLINE OFFLINE STABLE
ora.LISTENER_SCAN1.lsnr
1 ONLINE ONLINE rac1 STABLE
ora.asm(ora.asmgroup)
1 ONLINE ONLINE rac1 Started,STABLE
2 ONLINE ONLINE rac2 Started,STABLE
3 OFFLINE OFFLINE STABLE
ora.asmnet1.asmnetwork(ora.asmgroup)
1 ONLINE ONLINE rac1 STABLE
2 ONLINE ONLINE rac2 STABLE
3 OFFLINE OFFLINE STABLE
ora.cvu
1 ONLINE ONLINE rac1 STABLE
ora.emrep.db
1 ONLINE ONLINE rac1 Open,HOME=/u01/app/o
racle/product/19.3.0
/dbhome_1,STABLE
2 ONLINE ONLINE rac2 Open,HOME=/u01/app/o
racle/product/19.3.0
/dbhome_1,STABLE
ora.qosmserver
1 ONLINE ONLINE rac1 STABLE
ora.rac1.vip
1 ONLINE ONLINE rac1 STABLE
ora.rac2.vip
1 ONLINE ONLINE rac2 STABLE
ora.scan1.vip
1 ONLINE ONLINE rac1 STABLE
--------------------------------------------------------------------------------
bash
[grid@rac1 ~]$ asmcmd lspwusr
Username sysdba sysoper sysasm
SYS TRUE TRUE TRUE
ASMSNMP TRUE FALSE FALSE
CRSUSER__ASM_007 TRUE FALSE TRUE
[grid@rac1 ~]$ srvctl config asm
ASM home: <CRS home>
Password file: +DG_OCR/orapwasm
Backup of Password file: +DG_OCR/orapwASM_backup
ASM listener: LISTENER
ASM instance count: 3
Cluster ASM listener: ASMNET1LSNR_ASM2.删除ASM密码文件，模拟故障
bash
ASMCMD> cd dg_ocr
ASMCMD> ls
ASM/
orapwasm
rac-cluster/
ASMCMD> rm -rf orapwasm3.重启CRS观察异常状况
重启CRS
bash
[root@rac1 ~]# crsctl stop crs
[root@rac2 ~]# crsctl stop crs
[root@rac1 ~]# crsctl start crs
[root@rac2 ~]# crsctl start crs节点一
bash
[grid@rac1 ~]$ crsctl stat res -t
CRS-4535: Cannot communicate with Cluster Ready Services
CRS-4000: Command Status failed, or completed with errors.
[grid@rac1 ~]$ crsctl stat res -t -init
--------------------------------------------------------------------------------
Name Target State Server State details
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.asm
1 ONLINE ONLINE rac1 STABLE
ora.cluster_interconnect.haip
1 ONLINE ONLINE rac1 STABLE
ora.crf
1 ONLINE ONLINE rac1 STABLE
ora.crsd
1 ONLINE OFFLINE STABLE
ora.cssd
1 ONLINE ONLINE rac1 STABLE
ora.cssdmonitor
1 ONLINE ONLINE rac1 STABLE
ora.ctssd
1 ONLINE ONLINE rac1 OBSERVER,STABLE
ora.diskmon
1 OFFLINE OFFLINE STABLE
ora.evmd
1 ONLINE ONLINE rac1 STABLE
ora.gipcd
1 ONLINE ONLINE rac1 STABLE
ora.gpnpd
1 ONLINE ONLINE rac1 STABLE
ora.mdnsd
1 ONLINE ONLINE rac1 STABLE
ora.storage
1 ONLINE OFFLINE rac1 STARTING
--------------------------------------------------------------------------------节点二
bash
[grid@rac2 ~]$ crsctl stat res -t
--------------------------------------------------------------------------------
Name Target State Server State details
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.LISTENER.lsnr
ONLINE ONLINE rac2 STABLE
ora.chad
ONLINE ONLINE rac2 STABLE
ora.net1.network
ONLINE ONLINE rac2 STABLE
ora.ons
ONLINE ONLINE rac2 STABLE
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.ASMNET1LSNR_ASM.lsnr(ora.asmgroup)
1 ONLINE OFFLINE STABLE
2 ONLINE ONLINE rac2 STABLE
3 ONLINE OFFLINE STABLE
ora.DATANEW.dg(ora.asmgroup)
1 OFFLINE OFFLINE STABLE
2 ONLINE ONLINE rac2 STABLE
3 OFFLINE OFFLINE STABLE
ora.DG_DATA.dg(ora.asmgroup)
1 OFFLINE OFFLINE STABLE
2 ONLINE ONLINE rac2 STABLE
3 OFFLINE OFFLINE STABLE
ora.DG_FRA.dg(ora.asmgroup)
1 OFFLINE OFFLINE STABLE
2 ONLINE ONLINE rac2 STABLE
3 OFFLINE OFFLINE STABLE
ora.DG_OCR.dg(ora.asmgroup)
1 OFFLINE OFFLINE STABLE
2 ONLINE ONLINE rac2 STABLE
3 OFFLINE OFFLINE STABLE
ora.LISTENER_SCAN1.lsnr
1 ONLINE ONLINE rac2 STABLE
ora.asm(ora.asmgroup)
1 ONLINE OFFLINE STABLE
2 ONLINE ONLINE rac2 Started,STABLE
3 OFFLINE OFFLINE STABLE
ora.asmnet1.asmnetwork(ora.asmgroup)
1 ONLINE OFFLINE STABLE
2 ONLINE ONLINE rac2 STABLE
3 OFFLINE OFFLINE STABLE
ora.cvu
1 ONLINE ONLINE rac2 STABLE
ora.emrep.db
1 ONLINE OFFLINE STABLE
2 ONLINE OFFLINE rac2 Instance Shutdown,ST
ARTING
ora.qosmserver
1 ONLINE ONLINE rac2 STABLE
ora.rac1.vip
1 ONLINE INTERMEDIATE rac2 FAILED OVER,STABLE
ora.rac2.vip
1 ONLINE ONLINE rac2 STABLE
ora.scan1.vip
1 ONLINE ONLINE rac2 STABLE
--------------------------------------------------------------------------------4.手动启动ASM实例
bash
[grid@rac1 ~]$ sqlplus / as sysasm
SQL> startup5.创建密码文件
创建密码文件
bash
[grid@rac1 ~]$ orapwd file='+dg_ocr/orapwasm' asm=y force=y password=Password123*添加监控用户，密码随意
bash
[grid@rac1 ~]$ asmcmd orapwusr --add ASMSNMP
Enter password: ******
[grid@rac1 ~]$ asmcmd lspwusr
Username sysdba sysoper sysasm
SYS TRUE TRUE FALSE
ASMSNMP FALSE FALSE FALSE 6.获取CRSUSER__ASM_00X用户的密码
先查看CRSUSER__ASM_00X用户的密码。
生成dump文件
bash
[grid@rac1 ~]$ /u01/app/19.3.0/grid/bin/ocrdump /tmp/ocr.dmp
PROT-310: Not all keys were dumped due to permissions.//用grid执行有权限报错，用root执行就不会有报错，但是用grid就够了，需要的内容已经dump出来了。
打开dump出来的文件，搜索之前asmcmd lspwusr查到的CRSUSER__ASM_00X用户。
如果之前没有执行过credfix，那么这个用户通常是CRSUSER__ASM_001。
然后将查到的ORATEXT一行的数字和字母组成的一长串内容复制出来。
我之前是CRSUSER__ASM_007用户，所以搜索CRSUSER__ASM_007用户。
bash
[grid@rac1 ~]$ vi /tmp/ocr.dmp
根据我查到的内容，就是复制94fe808e905aef90bf335f143d8fa6f5这一段。
执行命令，获取CRSUSER__ASM_00X用户的密码。
bash
[grid@rac1 ~]$ crsctl get credmaint -path /ASM/Self/94fe808e905aef90bf335f143d8fa6f5 -credtype userpass -id 0 -attr passwd -local
q19CaqwsMFZV1rW2kx3bTRWuazidD将ORATEXT的一串内容替换在/ASM/Self/后面这里，最后得到的q19CaqwsMFZV1rW2kx3bTRWuazidD就是CRSUSER__ASM_007的密码。
//我的dump文件中有多个CRSUSER__ASM_00X用户，它们ORATEXT的内容都是相同的，所以如果不出意外，一个数据库的所有CRSUSER__ASM_00X用户，默认的密码都是一样的。
7.创建CRSUSER__ASM_00X用户并赋权
bash
[grid@rac1 ~]$ asmcmd orapwusr --add CRSUSER__ASM_007
Enter password: q19CaqwsMFZV1rW2kx3bTRWuazidD给CRSUSER__ASM_007用户，和之前的用户都赋权。
bash
[grid@rac1 ~]$ asmcmd lspwusr
Username sysdba sysoper sysasm
SYS TRUE TRUE FALSE
ASMSNMP FALSE FALSE FALSE
CRSUSER__ASM_007 FALSE FALSE FALSE
[grid@rac1 ~]$ asmcmd orapwusr --grant sysasm SYS
[grid@rac1 ~]$ asmcmd orapwusr --grant sysdba ASMSNMP
[grid@rac1 ~]$ asmcmd orapwusr --grant sysdba CRSUSER__ASM_007
[grid@rac1 ~]$ asmcmd orapwusr --grant sysasm CRSUSER__ASM_007
[grid@rac1 ~]$ asmcmd lspwusr
Username sysdba sysoper sysasm
SYS TRUE TRUE TRUE
ASMSNMP TRUE FALSE FALSE
CRSUSER__ASM_007 TRUE FALSE TRUE8.重启CRS验证恢复
重启CRS
bash
[root@rac1 ~]# crsctl stop crs
[root@rac2 ~]# crsctl stop crs
[root@rac1 ~]# crsctl start crs
[root@rac2 ~]# crsctl start crs查看状态
bash
[grid@rac1 ~]$ crsctl stat res -t
--------------------------------------------------------------------------------
Name Target State Server State details
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.LISTENER.lsnr
ONLINE ONLINE rac1 STABLE
ONLINE ONLINE rac2 STABLE
ora.chad
ONLINE ONLINE rac1 STABLE
ONLINE ONLINE rac2 STABLE
ora.net1.network
ONLINE ONLINE rac1 STABLE
ONLINE ONLINE rac2 STABLE
ora.ons
ONLINE ONLINE rac1 STABLE
ONLINE ONLINE rac2 STABLE
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.ASMNET1LSNR_ASM.lsnr(ora.asmgroup)
1 ONLINE ONLINE rac1 STABLE
2 ONLINE ONLINE rac2 STABLE
3 ONLINE OFFLINE STABLE
ora.DATANEW.dg(ora.asmgroup)
1 ONLINE ONLINE rac1 STABLE
2 ONLINE ONLINE rac2 STABLE
3 OFFLINE OFFLINE STABLE
ora.DG_DATA.dg(ora.asmgroup)
1 ONLINE ONLINE rac1 STABLE
2 ONLINE ONLINE rac2 STABLE
3 OFFLINE OFFLINE STABLE
ora.DG_FRA.dg(ora.asmgroup)
1 ONLINE ONLINE rac1 STABLE
2 ONLINE ONLINE rac2 STABLE
3 OFFLINE OFFLINE STABLE
ora.DG_OCR.dg(ora.asmgroup)
1 ONLINE ONLINE rac1 STABLE
2 ONLINE ONLINE rac2 STABLE
3 OFFLINE OFFLINE STABLE
ora.LISTENER_SCAN1.lsnr
1 ONLINE ONLINE rac2 STABLE
ora.asm(ora.asmgroup)
1 ONLINE ONLINE rac1 Started,STABLE
2 ONLINE ONLINE rac2 Started,STABLE
3 OFFLINE OFFLINE STABLE
ora.asmnet1.asmnetwork(ora.asmgroup)
1 ONLINE ONLINE rac1 STABLE
2 ONLINE ONLINE rac2 STABLE
3 OFFLINE OFFLINE STABLE
ora.cvu
1 ONLINE ONLINE rac2 STABLE
ora.emrep.db
1 ONLINE ONLINE rac1 Open,HOME=/u01/app/o
racle/product/19.3.0
/dbhome_1,STABLE
2 ONLINE ONLINE rac2 Open,HOME=/u01/app/o
racle/product/19.3.0
/dbhome_1,STABLE
ora.qosmserver
1 ONLINE ONLINE rac2 STABLE
ora.rac1.vip
1 ONLINE ONLINE rac1 STABLE
ora.rac2.vip
1 ONLINE ONLINE rac2 STABLE
ora.scan1.vip
1 ONLINE ONLINE rac2 STABLE
--------------------------------------------------------------------------------集群正常启动，恢复成功。