问题：
监控软件连不上TEST集群节点1，发现监听中不存在IP 1.80.检查巡检数据发现5/18日正常。
排查：
1.根据监控软件报错时间排查，监听日志无异常，监听状态中不存在IP 1.80物理IP。
2.进一步发现集群状态异常，多个资源状态显示OFFLINE。
排查发现为CRSD服务异常，其它集群资源正常，所以数据库没有宕机。
3.集群日志排查发现OCR不能读导致CRSD资源异常；具体为5/26号12:19分时候asm日志显示OCR仲裁盘有问题，13:23分出现读写错误。
处理：
1.数据库集群因OCR访问问题异常(节点2及其它DATA磁盘组未报错)
2.CRS资源异常
3.命令启动CRS资源(alter diskgroup ocr mount; //crsctl start res ora.crsd -init)，不影响集群其它资源，数据库等均持续正常运行。
4.关闭和重启了监听后，监听出现物理IP 80,srvctl start listener -n TESTdb01
5.已经联系用户协调存储工程师做检查
处理过程：
1.CRS资源异常
[grid@cxhisdb01 ~]$ crsctl check crs
CRS-4638: Oracle High Availability Services is online
CRS-4535: Cannot communicate with Cluster Ready Services
CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is online
2.启动CRS资源
[grid@cxhisdb01 ~]$ crsctl start res ora.crsd -init
CRS-2672: Attempting to start 'ora.crsd' on 'cxhisdb01'
CRS-2676: Start of 'ora.crsd' on 'cxhisdb01' succeeded
3、检查CRS状态
[grid@cxhisdb01 ~]$ crsctl check crs
CRS-4638: Oracle High Availability Services is online
CRS-4537: Cluster Ready Services is online
CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is online
[grid@TESTdb01 ~]$ sqlplus / as sysasm
SQL*Plus: Release 11.2.0.4.0 Production on Thu May 27 12:12:17 2021
Copyright (c) 1982, 2013, Oracle. All rights reserved.
Connected to:
Oracle Database 11g Enterprise Edition Release 11.2.0.4.0 - 64bit Production
With the Real Application Clusters and Automatic Storage Management options
SQL> alter diskgroup ocr mount;
Diskgroup altered.
SQL> exit
Disconnected from Oracle Database 11g Enterprise Edition Release 11.2.0.4.0 - 64bit Production
With the Real Application Clusters and Automatic Storage Management options
[grid@TESTdb01 ~]$ ps -ef|grep crs
grid 96756 95841 0 12:13 pts/32 00:00:00 grep crs
[grid@TESTdb01 ~]$ ps -ef|grep css
grid 96877 95841 0 12:13 pts/32 00:00:00 grep css
root 144383 1 0 2020 ? 09:30:54 /u01/app/11.2.0/grid/bin/cssdmonitor
root 144401 1 0 2020 ? 09:23:19 /u01/app/11.2.0/grid/bin/cssdagent
grid 144412 1 0 2020 ? 2-05:10:35 /u01/app/11.2.0/grid/bin/ocssd.bin
[grid@TESTdb01 ~]$ ps -ef|grep has
root 5926 1 0 2017 ? 00:00:00 /bin/sh /etc/init.d/init.ohasd run
grid 96897 95841 0 12:13 pts/32 00:00:00 grep has
root 144192 1 0 2020 ? 1-14:31:42 /u01/app/11.2.0/grid/bin/ohasd.bin reboot
[grid@TESTdb01 ~]$ crsctl check crs
CRS-4638: Oracle High Availability Services is online
CRS-4535: Cannot communicate with Cluster Ready Services
CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is online
[grid@TESTdb01 ~]$ crsctl start res ora.crsd -init
CRS-2672: Attempting to start 'ora.crsd' on 'TESTdb01'
CRS-2676: Start of 'ora.crsd' on 'TESTdb01' succeeded
[grid@TESTdb01 ~]$ crsctl check crs
CRS-4638: Oracle High Availability Services is online
CRS-4537: Cluster Ready Services is online
CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is online
[grid@TESTdb01 ~]$ crsctl stat res -t
--------------------------------------------------------------------------------
NAME TARGET STATE SERVER STATE_DETAILS
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.DATA.dg
ONLINE ONLINE TESTdb01
ONLINE ONLINE TESTdb02
ora.LISTENER.lsnr
ONLINE ONLINE TESTdb01
ONLINE ONLINE TESTdb02
ora.OCR.dg
ONLINE ONLINE TESTdb01
ONLINE ONLINE TESTdb02
ora.SSD.dg
ONLINE ONLINE TESTdb01
ONLINE ONLINE TESTdb02
ora.asm
ONLINE ONLINE TESTdb01 Started
ONLINE ONLINE TESTdb02 Started
ora.gsd
OFFLINE OFFLINE TESTdb01
OFFLINE OFFLINE TESTdb02
ora.net1.network
ONLINE ONLINE TESTdb01
ONLINE ONLINE TESTdb02
ora.ons
ONLINE ONLINE TESTdb01
ONLINE ONLINE TESTdb02
ora.registry.acfs
ONLINE ONLINE TESTdb01
ONLINE ONLINE TESTdb02
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.LISTENER_SCAN1.lsnr
1 ONLINE ONLINE TESTdb02
ora.cvu
1 ONLINE ONLINE TESTdb01
ora.TESTdb01.vip
1 ONLINE ONLINE TESTdb01
ora.TESTdb02.vip
1 ONLINE ONLINE TESTdb02
ora.hospital.db
1 ONLINE ONLINE TESTdb01 Open
2 ONLINE ONLINE TESTdb02 Open
ora.oc4j
1 ONLINE ONLINE TESTdb01
ora.scan1.vip
1 ONLINE ONLINE TESTdb02
[grid@TESTdb01 ~]$ lsnrctl status
分析过程：
集群日志
2020-12-29 05:06:46.095:
[/u01/app/11.2.0/grid/bin/oraagent.bin(145339)]CRS-5818:Aborted command 'check' for resource 'ora.OCR.dg'. Details at (:CRSAGF00113:) {1:61066:2} in /u01/app/11.2.0/grid/log/TESTdb01/agent/crsd/oraagent_grid/oraagent_grid.log.
2021-05-26 13:23:46.059:
[crsd(145215)]CRS-1006:The OCR location +OCR is inaccessible. Details in /u01/app/11.2.0/grid/log/TESTdb01/crsd/crsd.log.
2021-05-26 13:23:46.068:
[crsd(145215)]CRS-1006:The OCR location +OCR is inaccessible. Details in /u01/app/11.2.0/grid/log/TESTdb01/crsd/crsd.log.
2021-05-26 13:23:56.293:
[/u01/app/11.2.0/grid/bin/oraagent.bin(66885)]CRS-5822:Agent '/u01/app/11.2.0/grid/bin/oraagent_grid' disconnected from server. Details at (:CRSAGF00117:) {0:21:18} in /u01/app/11.2.0/grid/log/TESTdb01/agent/crsd/oraagent_grid/oraagent_grid.log.
2021-05-26 13:23:56.294:
[/u01/app/11.2.0/grid/bin/oraagent.bin(31320)]CRS-5822:Agent '/u01/app/11.2.0/grid/bin/oraagent_oracle' disconnected from server. Details at (:CRSAGF00117:) {0:19:50603} in /u01/app/11.2.0/grid/log/TESTdb01/agent/crsd/oraagent_oracle/oraagent_oracle.log.
2021-05-26 13:23:56.461:
[/u01/app/11.2.0/grid/bin/orarootagent.bin(145347)]CRS-5822:Agent '/u01/app/11.2.0/grid/bin/orarootagent_root' disconnected from server. Details at (:CRSAGF00117:) {0:5:1568} in /u01/app/11.2.0/grid/log/TESTdb01/agent/crsd/orarootagent_root/orarootagent_root.log.
2021-05-26 13:23:56.485:
[/u01/app/11.2.0/grid/bin/scriptagent.bin(145549)]CRS-5822:Agent '/u01/app/11.2.0/grid/bin/scriptagent_grid' disconnected from server. Details at (:CRSAGF00117:) {0:9:68} in /u01/app/11.2.0/grid/log/TESTdb01/agent/crsd/scriptagent_grid/scriptagent_grid.log.
2021-05-26 13:23:56.651:
[ohasd(144192)]CRS-2765:Resource 'ora.crsd' has failed on server 'TESTdb01'.
2021-05-26 13:23:58.540:
[crsd(5795)]CRS-1013:The OCR location in an ASM disk group is inaccessible. Details in /u01/app/11.2.0/grid/log/TESTdb01/crsd/crsd.log.
2021-05-26 13:23:58.548:
[crsd(5795)]CRS-0804:Cluster Ready Service aborted due to Oracle Cluster Registry error [PROC-26: Error while accessing the physical storage
]. Details at (:CRSD00111:) in /u01/app/11.2.0/grid/log/TESTdb01/crsd/crsd.log.
2021-05-26 13:23:58.964:
[ohasd(144192)]CRS-2765:Resource 'ora.crsd' has failed on server 'TESTdb01'.
2021-05-26 13:24:00.374:
[crsd(5834)]CRS-1013:The OCR location in an ASM disk group is inaccessible. Details in /u01/app/11.2.0/grid/log/TESTdb01/crsd/crsd.log.
2021-05-26 13:24:00.382:
[crsd(5834)]CRS-0804:Cluster Ready Service aborted due to Oracle Cluster Registry error [PROC-26: Error while accessing the physical storage
]. Details at (:CRSD00111:) in /u01/app/11.2.0/grid/log/TESTdb01/crsd/crsd.log.
2021-05-26 13:24:01.010:
[ohasd(144192)]CRS-2765:Resource 'ora.crsd' has failed on server 'TESTdb01'.
2021-05-26 13:24:02.447:
[crsd(5886)]CRS-1013:The OCR location in an ASM disk group is inaccessible. Details in /u01/app/11.2.0/grid/log/TESTdb01/crsd/crsd.log.
2021-05-26 13:24:02.455:
[crsd(5886)]CRS-0804:Cluster Ready Service aborted due to Oracle Cluster Registry error [PROC-26: Error while accessing the physical storage
]. Details at (:CRSD00111:) in /u01/app/11.2.0/grid/log/TESTdb01/crsd/crsd.log.
2021-05-26 13:24:03.068:
[ohasd(144192)]CRS-2765:Resource 'ora.crsd' has failed on server 'TESTdb01'.
2021-05-26 13:24:04.457:
[crsd(5909)]CRS-1013:The OCR location in an ASM disk group is inaccessible. Details in /u01/app/11.2.0/grid/log/TESTdb01/crsd/crsd.log.
2021-05-26 13:24:04.465:
[crsd(5909)]CRS-0804:Cluster Ready Service aborted due to Oracle Cluster Registry error [PROC-26: Error while accessing the physical storage
]. Details at (:CRSD00111:) in /u01/app/11.2.0/grid/log/TESTdb01/crsd/crsd.log.
2021-05-26 13:24:05.102:
[ohasd(144192)]CRS-2765:Resource 'ora.crsd' has failed on server 'TESTdb01'.
2021-05-26 13:24:06.492:
[crsd(5937)]CRS-1013:The OCR location in an ASM disk group is inaccessible. Details in /u01/app/11.2.0/grid/log/TESTdb01/crsd/crsd.log.
2021-05-26 13:24:06.501:
[crsd(5937)]CRS-0804:Cluster Ready Service aborted due to Oracle Cluster Registry error [PROC-26: Error while accessing the physical storage
]. Details at (:CRSD00111:) in /u01/app/11.2.0/grid/log/TESTdb01/crsd/crsd.log.
2021-05-26 13:24:07.132:
[ohasd(144192)]CRS-2765:Resource 'ora.crsd' has failed on server 'TESTdb01'.
2021-05-26 13:24:08.517:
[crsd(5986)]CRS-1013:The OCR location in an ASM disk group is inaccessible. Details in /u01/app/11.2.0/grid/log/TESTdb01/crsd/crsd.log.
2021-05-26 13:24:08.525:
[crsd(5986)]CRS-0804:Cluster Ready Service aborted due to Oracle Cluster Registry error [PROC-26: Error while accessing the physical storage
]. Details at (:CRSD00111:) in /u01/app/11.2.0/grid/log/TESTdb01/crsd/crsd.log.
2021-05-26 13:24:09.162:
[ohasd(144192)]CRS-2765:Resource 'ora.crsd' has failed on server 'TESTdb01'.
2021-05-26 13:24:10.544:
[crsd(6015)]CRS-1013:The OCR location in an ASM disk group is inaccessible. Details in /u01/app/11.2.0/grid/log/TESTdb01/crsd/crsd.log.
2021-05-26 13:24:10.552:
[crsd(6015)]CRS-0804:Cluster Ready Service aborted due to Oracle Cluster Registry error [PROC-26: Error while accessing the physical storage
]. Details at (:CRSD00111:) in /u01/app/11.2.0/grid/log/TESTdb01/crsd/crsd.log.
2021-05-26 13:24:11.193:
[ohasd(144192)]CRS-2765:Resource 'ora.crsd' has failed on server 'TESTdb01'.
2021-05-26 13:24:12.581:
[crsd(6051)]CRS-1013:The OCR location in an ASM disk group is inaccessible. Details in /u01/app/11.2.0/grid/log/TESTdb01/crsd/crsd.log.
2021-05-26 13:24:12.589:
[crsd(6051)]CRS-0804:Cluster Ready Service aborted due to Oracle Cluster Registry error [PROC-26: Error while accessing the physical storage
]. Details at (:CRSD00111:) in /u01/app/11.2.0/grid/log/TESTdb01/crsd/crsd.log.
2021-05-26 13:24:13.223:
[ohasd(144192)]CRS-2765:Resource 'ora.crsd' has failed on server 'TESTdb01'.
2021-05-26 13:24:14.614:
[crsd(6070)]CRS-1013:The OCR location in an ASM disk group is inaccessible. Details in /u01/app/11.2.0/grid/log/TESTdb01/crsd/crsd.log.
2021-05-26 13:24:14.622:
[crsd(6070)]CRS-0804:Cluster Ready Service aborted due to Oracle Cluster Registry error [PROC-26: Error while accessing the physical storage
]. Details at (:CRSD00111:) in /u01/app/11.2.0/grid/log/TESTdb01/crsd/crsd.log.
2021-05-26 13:24:15.253:
[ohasd(144192)]CRS-2765:Resource 'ora.crsd' has failed on server 'TESTdb01'.
2021-05-26 13:24:16.643:
[crsd(6090)]CRS-1013:The OCR location in an ASM disk group is inaccessible. Details in /u01/app/11.2.0/grid/log/TESTdb01/crsd/crsd.log.
2021-05-26 13:24:16.650:
[crsd(6090)]CRS-0804:Cluster Ready Service aborted due to Oracle Cluster Registry error [PROC-26: Error while accessing the physical storage
]. Details at (:CRSD00111:) in /u01/app/11.2.0/grid/log/TESTdb01/crsd/crsd.log.
2021-05-26 13:24:17.284:
[ohasd(144192)]CRS-2765:Resource 'ora.crsd' has failed on server 'TESTdb01'.
2021-05-26 13:24:17.284:
[ohasd(144192)]CRS-2771:Maximum restart attempts reached for resource 'ora.crsd'; will not restart.
2021-05-26 13:24:17.315:
[ohasd(144192)]CRS-2769:Unable to failover resource 'ora.crsd'.
2021-05-27 12:18:47.208:
[crsd(99493)]CRS-1012:The OCR service started on node TESTdb01.
2021-05-27 12:18:47.603:
[crsd(99493)]CRS-1201:CRSD started on node TESTdb01.
ASM实例日志
Sun Jan 17 01:12:00 2021
Warning: VKTM detected a time drift.
Time drifts can result in an unexpected behavior such as time-outs. Please check trace file for more details.
Wed May 26 12:19:57 2021
WARNING: Waited 15 secs for write IO to PST disk 0 in group 3.
WARNING: Waited 15 secs for write IO to PST disk 2 in group 3.
WARNING: Waited 15 secs for write IO to PST disk 3 in group 3.
WARNING: Waited 15 secs for write IO to PST disk 4 in group 3.
WARNING: Waited 15 secs for write IO to PST disk 0 in group 3.
WARNING: Waited 15 secs for write IO to PST disk 2 in group 3.
WARNING: Waited 15 secs for write IO to PST disk 3 in group 3.
WARNING: Waited 15 secs for write IO to PST disk 4 in group 3.
Wed May 26 12:19:57 2021
NOTE: process _b000_+asm1 (160488) initiating offline of disk 0.1409468596 (OCR_0000) with mask 0x7e in group 3
NOTE: process _b000_+asm1 (160488) initiating offline of disk 2.1409468594 (OCR_0002) with mask 0x7e in group 3
NOTE: process _b000_+asm1 (160488) initiating offline of disk 3.1409468595 (OCR_0003) with mask 0x7e in group 3
NOTE: process _b000_+asm1 (160488) initiating offline of disk 4.1409468592 (OCR_0004) with mask 0x7e in group 3
NOTE: checking PST: grp = 3
GMON checking disk modes for group 3 at 15 for pid 46, osid 160488
ERROR: no read quorum in group: required 3, found 1 disks
NOTE: checking PST for grp 3 done.
NOTE: initiating PST update: grp = 3, dsk = 0/0x5402c8b4, mask = 0x6a, op = clear
NOTE: initiating PST update: grp = 3, dsk = 2/0x5402c8b2, mask = 0x6a, op = clear
NOTE: initiating PST update: grp = 3, dsk = 3/0x5402c8b3, mask = 0x6a, op = clear
NOTE: initiating PST update: grp = 3, dsk = 4/0x5402c8b0, mask = 0x6a, op = clear
GMON updating disk modes for group 3 at 16 for pid 46, osid 160488
ERROR: no read quorum in group: required 3, found 1 disks
Wed May 26 12:19:57 2021
NOTE: cache dismounting (not clean) group 3/0xA242386E (OCR)
NOTE: messaging CKPT to quiesce pins Unix process pid: 160495, image: oracle@TESTdb01 (B001)
Wed May 26 12:19:57 2021
NOTE: halting all I/Os to diskgroup 3 (OCR)
Wed May 26 12:19:57 2021
NOTE: LGWR doing non-clean dismount of group 3 (OCR)
NOTE: LGWR sync ABA=15.85 last written ABA 15.85
WARNING: Offline for disk OCR_0000 in mode 0x7f failed.
WARNING: Offline for disk OCR_0002 in mode 0x7f failed.
WARNING: Offline for disk OCR_0003 in mode 0x7f failed.
WARNING: Offline for disk OCR_0004 in mode 0x7f failed.
Wed May 26 12:19:58 2021
kjbdomdet send to inst 2
detach from dom 3, sending detach message to inst 2
Wed May 26 12:19:58 2021
NOTE: No asm libraries found in the system
Wed May 26 12:19:58 2021
List of instances:
1 2
Dirty detach reconfiguration started (new ddet inc 1, cluster inc 4)
Global Resource Directory partially frozen for dirty detach
* dirty detach - domain 3 invalid = TRUE
2 GCS resources traversed, 0 cancelled
Dirty Detach Reconfiguration complete
Wed May 26 12:19:58 2021
WARNING: dirty detached from domain 3
NOTE: cache dismounted group 3/0xA242386E (OCR)
SQL> alter diskgroup OCR dismount force /* ASM SERVER:2722248814 */
Wed May 26 12:19:58 2021
NOTE: cache deleting context for group OCR 3/0xa242386e
ASM Health Checker found 1 new failures
GMON dismounting group 3 at 17 for pid 47, osid 160495
NOTE: Disk OCR_0000 in mode 0x7f marked for de-assignment
NOTE: Disk OCR_0001 in mode 0x7f marked for de-assignment
NOTE: Disk OCR_0002 in mode 0x7f marked for de-assignment
NOTE: Disk OCR_0003 in mode 0x7f marked for de-assignment
NOTE: Disk OCR_0004 in mode 0x7f marked for de-assignment
NOTE:Waiting for all pending writes to complete before de-registering: grpnum 3
Wed May 26 12:20:28 2021
SUCCESS: diskgroup OCR was dismounted
SUCCESS: alter diskgroup OCR dismount force /* ASM SERVER:2722248814 */
SUCCESS: ASM-initiated MANDATORY DISMOUNT of group OCR
Wed May 26 12:20:29 2021
NOTE: diskgroup resource ora.OCR.dg is offline
Wed May 26 12:20:29 2021
Errors in file /u01/app/grid/diag/asm/+asm/+ASM1/trace/+ASM1_ora_145227.trc:
ORA-15078: ASM diskgroup was forcibly dismounted
Errors in file /u01/app/grid/diag/asm/+asm/+ASM1/trace/+ASM1_ora_145227.trc:
ORA-15078: ASM diskgroup was forcibly dismounted
Errors in file /u01/app/grid/diag/asm/+asm/+ASM1/trace/+ASM1_ora_145227.trc:
ORA-15078: ASM diskgroup was forcibly dismounted
Errors in file /u01/app/grid/diag/asm/+asm/+ASM1/trace/+ASM1_ora_145227.trc:
ORA-15078: ASM diskgroup was forcibly dismounted
Errors in file /u01/app/grid/diag/asm/+asm/+ASM1/trace/+ASM1_ora_145227.trc:
ORA-15078: ASM diskgroup was forcibly dismounted
Errors in file /u01/app/grid/diag/asm/+asm/+ASM1/trace/+ASM1_ora_145227.trc:
ORA-15078: ASM diskgroup was forcibly dismounted
Errors in file /u01/app/grid/diag/asm/+asm/+ASM1/trace/+ASM1_ora_145227.trc:
ORA-15078: ASM diskgroup was forcibly dismounted
Errors in file /u01/app/grid/diag/asm/+asm/+ASM1/trace/+ASM1_ora_145227.trc:
ORA-15078: ASM diskgroup was forcibly dismounted
Errors in file /u01/app/grid/diag/asm/+asm/+ASM1/trace/+ASM1_ora_145227.trc:
ORA-15078: ASM diskgroup was forcibly dismounted
Wed May 26 13:23:45 2021
Errors in file /u01/app/grid/diag/asm/+asm/+ASM1/trace/+ASM1_ora_145227.trc:
ORA-15078: ASM diskgroup was forcibly dismounted
WARNING: requested mirror side 1 of virtual extent 6 logical extent 0 offset 741376 is not allocated; I/O request failed
WARNING: requested mirror side 2 of virtual extent 6 logical extent 1 offset 741376 is not allocated; I/O request failed
WARNING: requested mirror side 3 of virtual extent 6 logical extent 2 offset 741376 is not allocated; I/O request failed
Errors in file /u01/app/grid/diag/asm/+asm/+ASM1/trace/+ASM1_ora_145227.trc:
ORA-15078: ASM diskgroup was forcibly dismounted
ORA-15078: ASM diskgroup was forcibly dismounted
ORA-15078: ASM diskgroup was forcibly dismounted
Wed May 26 13:23:46 2021
SQL> alter diskgroup OCR check /* proxy */
ORA-15032: not all alterations performed
ORA-15001: diskgroup "OCR" does not exist or is not mounted
ERROR: alter diskgroup OCR check /* proxy */
Wed May 26 13:23:56 2021
NOTE: client exited [145215]
Wed May 26 13:23:58 2021
NOTE: [crsd.bin@TESTdb01 (TNS V1-V3) 5795] opening OCR file
Wed May 26 13:24:00 2021
NOTE: [crsd.bin@TESTdb01 (TNS V1-V3) 5834] opening OCR file
Wed May 26 13:24:02 2021
NOTE: [crsd.bin@TESTdb01 (TNS V1-V3) 5886] opening OCR file
Wed May 26 13:24:04 2021
NOTE: [crsd.bin@TESTdb01 (TNS V1-V3) 5909] opening OCR file
Wed May 26 13:24:06 2021
NOTE: [crsd.bin@TESTdb01 (TNS V1-V3) 5937] opening OCR file
Wed May 26 13:24:08 2021
NOTE: [crsd.bin@TESTdb01 (TNS V1-V3) 5986] opening OCR file
Wed May 26 13:24:10 2021
NOTE: [crsd.bin@TESTdb01 (TNS V1-V3) 6015] opening OCR file
Wed May 26 13:24:12 2021
NOTE: [crsd.bin@TESTdb01 (TNS V1-V3) 6051] opening OCR file
Wed May 26 13:24:14 2021
NOTE: [crsd.bin@TESTdb01 (TNS V1-V3) 6070] opening OCR file
Wed May 26 13:24:16 2021
NOTE: [crsd.bin@TESTdb01 (TNS V1-V3) 6090] opening OCR file
Thu May 27 12:12:25 2021
SQL> alter diskgroup ocr mount
NOTE: cache registered group OCR number=3 incarn=0xa2423bb2
NOTE: cache began mount (not first) of group OCR number=3 incarn=0xa2423bb2
NOTE: Assigning number (3,4) to disk (/dev/asm-ocr5)
NOTE: Assigning number (3,1) to disk (/dev/asm-ocr2)
NOTE: Assigning number (3,2) to disk (/dev/asm-ocr3)
NOTE: Assigning number (3,3) to disk (/dev/asm-ocr4)
NOTE: Assigning number (3,0) to disk (/dev/asm-ocr1)
Thu May 27 12:12:25 2021
GMON querying group 3 at 19 for pid 30, osid 95906
NOTE: cache opening disk 0 of grp 3: OCR_0000 path:/dev/asm-ocr1
NOTE: F1X0 found on disk 0 au 2 fcn 0.0
NOTE: cache opening disk 1 of grp 3: OCR_0001 path:/dev/asm-ocr2
NOTE: F1X0 found on disk 1 au 2 fcn 0.0
NOTE: cache opening disk 2 of grp 3: OCR_0002 path:/dev/asm-ocr3
NOTE: F1X0 found on disk 2 au 2 fcn 0.0
NOTE: cache opening disk 3 of grp 3: OCR_0003 path:/dev/asm-ocr4
NOTE: cache opening disk 4 of grp 3: OCR_0004 path:/dev/asm-ocr5
NOTE: cache mounting (not first) high redundancy group 3/0xA2423BB2 (OCR)
Thu May 27 12:12:26 2021
kjbdomatt send to inst 2
Thu May 27 12:12:26 2021
NOTE: attached to recovery domain 3
NOTE: redo buffer size is 256 blocks (1053184 bytes)
Thu May 27 12:12:26 2021
NOTE: LGWR attempting to mount thread 1 for diskgroup 3 (OCR)
NOTE: LGWR found thread 1 closed at ABA 15.85
NOTE: LGWR mounted thread 1 for diskgroup 3 (OCR)
NOTE: LGWR opening thread 1 at fcn 0.591 ABA 16.86
NOTE: cache mounting group 3/0xA2423BB2 (OCR) succeeded
NOTE: cache ending mount (success) of group OCR number=3 incarn=0xa2423bb2
Thu May 27 12:12:26 2021
NOTE: Instance updated compatible.asm to 11.2.0.0.0 for grp 3
SUCCESS: diskgroup OCR was mounted
SUCCESS: alter diskgroup ocr mount
Thu May 27 12:12:28 2021
WARNING: failed to online diskgroup resource ora.OCR.dg (unable to communicate with CRSD/OHASD)
NOTE: Attempting voting file refresh on diskgroup OCR
NOTE: Refresh completed on diskgroup OCR
. Found 5 voting file(s).
Thu May 27 12:18:41 2021
NOTE: [crsd.bin@TESTdb01 (TNS V1-V3) 99493] opening OCR file