近期，某客户多次出现RAC节点异常重启的问题，对日志进行分析，均为报错IO方面问题；通过了解数据库底层服务器、网络、存储的配置情况，该客户采用双存储（双活）。每个存储放在不通的办公楼，服务器也采用2台搭建RAC，分布在不同大楼。
结合这些信息推测问题可能是存储到服务器的光纤信号波动，后续进行光衰测试，确实存在问题，进行光模块更换和光纤线更换后，后续观察半年左右，系统运行稳定，未再出现IO异常引起的重启问题。
相关日志信息分析如下：
1、数据库日志
ditional information: -1
Additional information: 8192
WARNING: failed to write mirror side 1 of virtual extent 27592 logical extent 0 of file 574 in group 2 on disk 5 allocation unit 857523
Errors in file /u01/app/oracle/diag/rdbms/hisdb/hisdb2/trace/hisdb2_dbw9_29987.trc:
ORA-15080: synchronous I/O operation to a disk failed
ORA-27063: number of bytes read/written is incorrect
HPUX-ia64 Error: 11: Resource temporarily unavailable
Additional information: -1
Additional information: 8192
WARNING: failed to write mirror side 1 of virtual extent 26947 logical extent 0 of file 576 in group 2 on disk 3 allocation unit 46307
KCF: read, write or open error, block=0x34a1a6 online=1
file=85 '+DATA/hisdb/datafile/emr5.576.1092670511'
error=15081 txt: ''
Errors in file /u01/app/oracle/diag/rdbms/hisdb/hisdb2/trace/hisdb2_dbw9_29987.trc:
Errors in file /u01/app/oracle/diag/rdbms/hisdb/hisdb2/trace/hisdb2_dbw9_29987.trc:
ORA-63999: data file suffered media failure
ORA-01114: IO error writing block to file 85 (block # 3449254)
ORA-01110: data file 85: '+DATA/hisdb/datafile/emr5.576.1092670511'
ORA-15081: failed to submit an I/O operation to a disk
ORA-15081: failed to submit an I/O operation to a disk
Thu Jul 03 17:52:43 2025
LGWR waiting for instance termination
DBW9 (ospid: 29987): terminating the instance due to error 63999
Thu Jul 03 17:52:44 2025
opiodr aborting process unknown ospid (16883) as a result of ORA-1092
Thu Jul 03 17:52:44 2025
opiodr aborting process unknown ospid (17135) as a result of ORA-1092
Thu Jul 03 17:52:44 2025
ORA-1092 : opitsk aborting process
Thu Jul 03 17:52:44 2025
2、集群GRID日志
$ tail -200 alertxkhis01.log
2025-03-06 22:57:02.696:
crsd(4872)CRS-2772:Server 'xkhis02' has been assigned to pool 'Generic'.
2025-03-06 22:57:02.697:
crsd(4872)CRS-2772:Server 'xkhis02' has been assigned to pool 'ora.hisdb'.
2025-05-08 17:17:00.496:
/u01/app/11.2.0/grid/bin/oraagent.bin(4930)CRS-5818:Aborted command 'stop' for resource 'ora.LISTENER.lsnr'. Details at (:CRSAGF00113:) {1:46568:8793} in /u01/app/11.2.0/grid/log/xkhis01/agent/crsd/oraagent_grid/oraagent_grid.log.
2025-05-08 17:17:00.546:
/u01/app/11.2.0/grid/bin/oraagent.bin(4930)CRS-5014:Agent "/u01/app/11.2.0/grid/bin/oraagent.bin" timed out starting process "/u01/app/11.2.0/grid/bin/lsnrctl" for action "stop": details at "(:CLSN00009:)" in "/u01/app/11.2.0/grid/log/xkhis01/agent/crsd/oraagent_grid/oraagent_grid.log"
2025-05-08 17:17:04.519:
crsd(4872)CRS-2757:Command 'Stop' timed out waiting for response from the resource 'ora.LISTENER.lsnr'. Details at (:CRSPE00111:) {1:46568:8793} in /u01/app/11.2.0/grid/log/xkhis01/crsd/crsd.log.
2025-05-08 17:17:04.635:
/u01/app/11.2.0/grid/bin/oraagent.bin(4930)CRS-5016:Process "/u01/app/11.2.0/grid/bin/lsnrctl" spawned by agent "/u01/app/11.2.0/grid/bin/oraagent.bin" for action "check" failed: details at "(:CLSN00010:)" in "/u01/app/11.2.0/grid/log/xkhis01/agent/crsd/oraagent_grid/oraagent_grid.log"
2025-05-08 17:17:04.765:
/u01/app/11.2.0/grid/bin/oraagent.bin(4930)CRS-5016:Process "/u01/app/11.2.0/grid/bin/lsnrctl" spawned by agent "/u01/app/11.2.0/grid/bin/oraagent.bin" for action "clean" failed: details at "(:CLSN00010:)" in "/u01/app/11.2.0/grid/log/xkhis01/agent/crsd/oraagent_grid/oraagent_grid.log"
2025-05-08 17:17:04.884:
/u01/app/11.2.0/grid/bin/oraagent.bin(4930)CRS-5016:Process "/u01/app/11.2.0/grid/bin/lsnrctl" spawned by agent "/u01/app/11.2.0/grid/bin/oraagent.bin" for action "check" failed: details at "(:CLSN00010:)" in "/u01/app/11.2.0/grid/log/xkhis01/agent/crsd/oraagent_grid/oraagent_grid.log"
2025-05-08 17:17:29.974:
crsd(4872)CRS-2757:Command 'Stop' timed out waiting for response from the resource 'ora.LISTENER.lsnr'. Details at (:CRSPE00111:) {2:38591:41607} in /u01/app/11.2.0/grid/log/xkhis01/crsd/crsd.log.
2025-05-08 17:57:33.642:
/u01/app/11.2.0/grid/bin/oraagent.bin(4930)CRS-5818:Aborted command 'stop' for resource 'ora.LISTENER.lsnr'. Details at (:CRSAGF00113:) {1:46568:9867} in /u01/app/11.2.0/grid/log/xkhis01/agent/crsd/oraagent_grid/oraagent_grid.log.
2025-05-08 17:57:33.693:
/u01/app/11.2.0/grid/bin/oraagent.bin(4930)CRS-5014:Agent "/u01/app/11.2.0/grid/bin/oraagent.bin" timed out starting process "/u01/app/11.2.0/grid/bin/lsnrctl" for action "stop": details at "(:CLSN00009:)" in "/u01/app/11.2.0/grid/log/xkhis01/agent/crsd/oraagent_grid/oraagent_grid.log"
2025-05-08 17:57:37.663:
crsd(4872)CRS-2757:Command 'Stop' timed out waiting for response from the resource 'ora.LISTENER.lsnr'. Details at (:CRSPE00111:) {1:46568:9867} in /u01/app/11.2.0/grid/log/xkhis01/crsd/crsd.log.
2025-05-08 17:57:37.782:
/u01/app/11.2.0/grid/bin/oraagent.bin(4930)CRS-5016:Process "/u01/app/11.2.0/grid/bin/lsnrctl" spawned by agent "/u01/app/11.2.0/grid/bin/oraagent.bin" for action "check" failed: details at "(:CLSN00010:)" in "/u01/app/11.2.0/grid/log/xkhis01/agent/crsd/oraagent_grid/oraagent_grid.log"
2025-05-08 17:57:37.910:
/u01/app/11.2.0/grid/bin/oraagent.bin(4930)CRS-5016:Process "/u01/app/11.2.0/grid/bin/lsnrctl" spawned by agent "/u01/app/11.2.0/grid/bin/oraagent.bin" for action "clean" failed: details at "(:CLSN00010:)" in "/u01/app/11.2.0/grid/log/xkhis01/agent/crsd/oraagent_grid/oraagent_grid.log"
2025-05-08 17:57:38.032:
/u01/app/11.2.0/grid/bin/oraagent.bin(4930)CRS-5016:Process "/u01/app/11.2.0/grid/bin/lsnrctl" spawned by agent "/u01/app/11.2.0/grid/bin/oraagent.bin" for action "check" failed: details at "(:CLSN00010:)" in "/u01/app/11.2.0/grid/log/xkhis01/agent/crsd/oraagent_grid/oraagent_grid.log"
2025-05-08 17:58:38.410:
crsd(4872)CRS-2757:Command 'Stop' timed out waiting for response from the resource 'ora.LISTENER.lsnr'. Details at (:CRSPE00111:) {2:38591:42605} in /u01/app/11.2.0/grid/log/xkhis01/crsd/crsd.log.
2025-07-03 17:52:44.148:
crsd(4872)CRS-2765:Resource 'ora.hisdb.db' has failed on server 'xkhis02'.
2025-07-03 17:53:09.449:
/u01/app/11.2.0/grid/bin/oraagent.bin(12008)CRS-5011:Check of resource "hisdb" failed: details at "(:CLSN00007:)" in "/u01/app/11.2.0/grid/log/xkhis01/agent/crsd/oraagent_oracle/oraagent_oracle.log"
2025-07-03 17:53:09.456:
crsd(4872)CRS-2765:Resource 'ora.hisdb.db' has failed on server 'xkhis01'.
2025-07-03 17:53:16.114:
cssd(4730)CRS-1649:An I/O error occured for voting file: /dev/rdisk/disk30; details at (:CSSNM00060:) in /u01/app/11.2.0/grid/log/xkhis01/cssd/ocssd.log.
2025-07-03 17:53:16.114:
cssd(4730)CRS-1649:An I/O error occured for voting file: /dev/rdisk/disk32; details at (:CSSNM00060:) in /u01/app/11.2.0/grid/log/xkhis01/cssd/ocssd.log.
2025-07-03 17:53:16.114:
cssd(4730)CRS-1649:An I/O error occured for voting file: /dev/rdisk/disk33; details at (:CSSNM00060:) in /u01/app/11.2.0/grid/log/xkhis01/cssd/ocssd.log.
2025-07-03 17:53:20.514:
cssd(4730)CRS-1649:An I/O error occured for voting file: /dev/rdisk/disk32; details at (:CSSNM00059:) in /u01/app/11.2.0/grid/log/xkhis01/cssd/ocssd.log.
2025-07-03 17:53:46.915:
cssd(4730)CRS-1649:An I/O error occured for voting file: /dev/rdisk/disk30; details at (:CSSNM00059:) in /u01/app/11.2.0/grid/log/xkhis01/cssd/ocssd.log.
2025-07-03 17:53:46.915:
cssd(4730)CRS-1649:An I/O error occured for voting file: /dev/rdisk/disk33; details at (:CSSNM00059:) in /u01/app/11.2.0/grid/log/xkhis01/cssd/ocssd.log.
2025-07-03 17:53:46.919:
cssd(4730)CRS-1605:CSSD voting file is online: /dev/rdisk/disk30; details in /u01/app/11.2.0/grid/log/xkhis01/cssd/ocssd.log.
2025-07-03 17:53:46.921:
cssd(4730)CRS-1605:CSSD voting file is online: /dev/rdisk/disk33; details in /u01/app/11.2.0/grid/log/xkhis01/cssd/ocssd.log.
2025-07-03 17:53:51.315:
cssd(4730)CRS-1649:An I/O error occured for voting file: /dev/rdisk/disk32; details at (:CSSNM00060:) in /u01/app/11.2.0/grid/log/xkhis01/cssd/ocssd.log.
2025-07-03 17:53:51.552:
cssd(4730)CRS-1605:CSSD voting file is online: /dev/rdisk/disk32; details in /u01/app/11.2.0/grid/log/xkhis01/cssd/ocssd.log.
2025-07-03 17:54:05.065:
/u01/app/11.2.0/grid/bin/oraagent.bin(4930)CRS-5822:Agent '/u01/app/11.2.0/grid/bin/oraagent_grid' disconnected from server. Details at (:CRSAGF00117:) {0:3:33} in /u01/app/11.2.0/grid/log/xkhis01/agent/crsd/oraagent_grid/oraagent_grid.log.
2025-07-03 17:54:05.067:
/u01/app/11.2.0/grid/bin/orarootagent.bin(4935)CRS-5822:Agent '/u01/app/11.2.0/grid/bin/orarootagent_root' disconnected from server. Details at (:CRSAGF00117:) {0:5:3458} in /u01/app/11.2.0/grid/log/xkhis01/agent/crsd/orarootagent_root/orarootagent_root.log.
2025-07-03 17:54:05.067:
/u01/app/11.2.0/grid/bin/scriptagent.bin(14469)CRS-5822:Agent '/u01/app/11.2.0/grid/bin/scriptagent_grid' disconnected from server. Details at (:CRSAGF00117:) {0:9:16} in /u01/app/11.2.0/grid/log/xkhis01/agent/crsd/scriptagent_grid/scriptagent_grid.log.
2025-07-03 17:54:05.068:
/u01/app/11.2.0/grid/bin/oraagent.bin(2346)CRS-5822:Agent '/u01/app/11.2.0/grid/bin/oraagent_oracle' disconnected from server. Details at (:CRSAGF00117:) {0:15:3} in /u01/app/11.2.0/grid/log/xkhis01/agent/crsd/oraagent_oracle/oraagent_oracle.log.
2025-07-03 17:54:05.080:
ohasd(4384)CRS-2765:Resource 'ora.crsd' has failed on server 'xkhis01'.
2025-07-03 17:54:06.526:
crsd(2415)CRS-1013:The OCR location in an ASM disk group is inaccessible. Details in /u01/app/11.2.0/grid/log/xkhis01/crsd/crsd.log.
2025-07-03 17:54:06.540:
crsd(2415)CRS-0804:Cluster Ready Service aborted due to Oracle Cluster Registry error [PROC-26: Error while accessing the physical storage
ORA-12549: TNS:operating system resource quota exceeded
]. Details at (:CRSD00111:) in /u01/app/11.2.0/grid/log/xkhis01/crsd/crsd.log.
2025-07-03 17:54:11.610:
ohasd(4384)CRS-2765:Resource 'ora.crsd' has failed on server 'xkhis01'.
2025-07-03 17:54:11.627:
/u01/app/11.2.0/grid/bin/orarootagent.bin(4787)CRS-5013:Agent "/u01/app/11.2.0/grid/bin/orarootagent.bin" failed to start process "/u01/app/11.2.0/grid/bin/crsd" for action "start": details at "(:CLSN00008:)" in "/u01/app/11.2.0/grid/log/xkhis01/agent/ohasd/orarootagent_root/orarootagent_root.log"
2025-07-03 17:54:13.276:
ohasd(4384)CRS-2878:Failed to restart resource 'ora.crsd'
2025-07-03 17:54:13.278:
ohasd(4384)CRS-2769:Unable to failover resource 'ora.crsd'.
2025-07-03 18:18:57.917:
mdnsd(4710)CRS-5602:mDNS service stopping by request.
2025-07-03 18:18:57.919:
ctssd(4791)CRS-2405:The Cluster Time Synchronization Service on host xkhis01 is shutdown by user
2025-07-03 18:28:57.949:
/u01/app/11.2.0/grid/bin/oraagent.bin(4706)CRS-5818:Aborted command 'stop' for resource 'ora.asm'. Details at (:CRSAGF00113:) {0:0:20112} in /u01/app/11.2.0/grid/log/xkhis01/agent/ohasd/oraagent_grid/oraagent_grid.log.
2025-07-03 18:28:59.962:
ohasd(4384)CRS-2757:Command 'Stop' timed out waiting for response from the resource 'ora.asm'. Details at (:CRSPE00111:) {0:0:20112} in /u01/app/11.2.0/grid/log/xkhis01/ohasd/ohasd.log.
2025-07-03 18:29:04.493:
cssd(4730)CRS-1603:CSSD on node xkhis01 shutdown by user.
2025-07-03 18:29:12.642:
ohasd(4384)CRS-2767:Resource state recovery not attempted for 'ora.cssdmonitor' as its target state is OFFLINE
2025-07-03 18:29:12.643:
ohasd(4384)CRS-2769:Unable to failover resource 'ora.cssdmonitor'.
2025-07-03 18:29:12.740:
cssd(4730)CRS-1660:The CSS daemon shutdown has completed
2025-07-03 18:29:15.244:
gpnpd(4714)CRS-2329:GPNPD on node xkhis01 shutdown.
2025-07-03 18:29:17.985:
/u01/app/11.2.0/grid/bin/oraagent.bin(4706)CRS-5822:Agent '/u01/app/11.2.0/grid/bin/oraagent_grid' disconnected from server. Details at (:CRSAGF00117:) {0:9:24} in /u01/app/11.2.0/grid/log/xkhis01/agent/ohasd/oraagent_grid/oraagent_grid.log.
2025-07-03 18:29:17.986:
/u01/app/11.2.0/grid/bin/orarootagent.bin(4787)CRS-5822:Agent '/u01/app/11.2.0/grid/bin/orarootagent_root' disconnected from server. Details at (:CRSAGF00117:) {0:17:23223} in /u01/app/11.2.0/grid/log/xkhis01/agent/ohasd/orarootagent_root/orarootagent_root.log.
2025-07-03 18:30:58.930:
ohasd(7585)CRS-2112:The OLR service started on node xkhis01.
2025-07-03 18:30:58.970:
ohasd(7585)CRS-1301:Oracle High Availability Service started on node xkhis01.
2025-07-03 18:30:58.971:
ohasd(7585)CRS-8017:location: /var/opt/oracle/lastgasp has 2 reboot advisory log files, 0 were announced and 0 errors occurred
2025-07-03 18:31:03.452:
/u01/app/11.2.0/grid/bin/orarootagent.bin(7604)CRS-2302:Cannot get GPnP profile. Error CLSGPNP_NO_DAEMON (GPNPD daemon is not running).
2025-07-03 18:31:09.197:
gpnpd(7686)CRS-2328:GPNPD started on node xkhis01.
2025-07-03 18:31:11.816:
cssd(7702)CRS-1713:CSSD daemon is started in clustered mode
2025-07-03 18:31:13.656:
ohasd(7585)CRS-2767:Resource state recovery not attempted for 'ora.diskmon' as its target state is OFFLINE
2025-07-03 18:31:13.656:
ohasd(7585)CRS-2769:Unable to failover resource 'ora.diskmon'.
2025-07-03 18:31:30.030:
cssd(7702)CRS-1707:Lease acquisition for node xkhis01 number 1 completed
2025-07-03 18:31:31.392:
cssd(7702)CRS-1605:CSSD voting file is online: /dev/rdisk/disk30; details in /u01/app/11.2.0/grid/log/xkhis01/cssd/ocssd.log.
2025-07-03 18:31:31.402:
cssd(7702)CRS-1605:CSSD voting file is online: /dev/rdisk/disk32; details in /u01/app/11.2.0/grid/log/xkhis01/cssd/ocssd.log.
2025-07-03 18:31:31.412:
cssd(7702)CRS-1605:CSSD voting file is online: /dev/rdisk/disk33; details in /u01/app/11.2.0/grid/log/xkhis01/cssd/ocssd.log.
2025-07-03 18:31:33.056:
cssd(7702)CRS-1601:CSSD Reconfiguration complete. Active nodes are xkhis01 xkhis02 .
2025-07-03 18:31:35.124:
ctssd(7800)CRS-2403:The Cluster Time Synchronization Service on host xkhis01 is in observer mode.
2025-07-03 18:31:35.461:
ctssd(7800)CRS-2407:The new Cluster Time Synchronization Service reference node is host xkhis02.
2025-07-03 18:31:35.462:
ctssd(7800)CRS-2401:The Cluster Time Synchronization Service started on host xkhis01.
2025-07-03 18:31:37.118:
ohasd(7585)CRS-2767:Resource state recovery not attempted for 'ora.diskmon' as its target state is OFFLINE
2025-07-03 18:31:37.118:
ohasd(7585)CRS-2769:Unable to failover resource 'ora.diskmon'.
2025-07-03 18:31:47.552:
ctssd(7800)CRS-2409:The clock on host xkhis01 is not synchronous with the mean cluster time. No action has been taken as the Cluster Time Synchronization Service is running in observer mode.
2025-07-03 18:32:01.058:
crsd(7909)CRS-1012:The OCR service started on node xkhis01.
2025-07-03 18:32:01.448:
evmd(7806)CRS-1401:EVMD started on node xkhis01.
2025-07-03 18:32:03.316:
crsd(7909)CRS-1201:CRSD started on node xkhis01.
2025-07-03 18:32:04.242:
/u01/app/11.2.0/grid/bin/oraagent.bin(7974)CRS-5011:Check of resource "hisdb" failed: details at "(:CLSN00007:)" in "/u01/app/11.2.0/grid/log/xkhis01/agent/crsd/oraagent_oracle/oraagent_oracle.log"
2025-07-03 18:32:06.408:
crsd(7909)CRS-2772:Server 'xkhis01' has been assigned to pool 'Generic'.
2025-07-03 18:32:06.409:
crsd(7909)CRS-2772:Server 'xkhis01' has been assigned to pool 'ora.hisdb'.
2025-07-03 18:34:04.412:
ctssd(7800)CRS-2407:The new Cluster Time Synchronization Service reference node is host xkhis01.
2025-07-03 18:35:06.505:
/u01/app/11.2.0/grid/bin/oraagent.bin(7965)CRS-5818:Aborted command 'start' for resource 'ora.LISTENER_SCAN1.lsnr'. Details at (:CRSAGF00113:) {1:30950:2} in /u01/app/11.2.0/grid/log/xkhis01/agent/crsd/oraagent_grid/oraagent_grid.log.
2025-07-03 18:35:06.505:
/u01/app/11.2.0/grid/bin/oraagent.bin(7965)CRS-5818:Aborted command 'start' for resource 'ora.LISTENER.lsnr'. Details at (:CRSAGF00113:) {1:30950:2} in /u01/app/11.2.0/grid/log/xkhis01/agent/crsd/oraagent_grid/oraagent_grid.log.
2025-07-03 18:35:06.584:
/u01/app/11.2.0/grid/bin/oraagent.bin(7965)CRS-5014:Agent "/u01/app/11.2.0/grid/bin/oraagent.bin" timed out starting process "/u01/app/11.2.0/grid/bin/lsnrctl" for action "start": details at "(:CLSN00009:)" in "/u01/app/11.2.0/grid/log/xkhis01/agent/crsd/oraagent_grid/oraagent_grid.log"
2025-07-03 18:35:06.594:
/u01/app/11.2.0/grid/bin/oraagent.bin(7965)CRS-5014:Agent "/u01/app/11.2.0/grid/bin/oraagent.bin" timed out starting process "/u01/app/11.2.0/grid/bin/lsnrctl" for action "start": details at "(:CLSN00009:)" in "/u01/app/11.2.0/grid/log/xkhis01/agent/crsd/oraagent_grid/oraagent_grid.log"
2025-07-03 18:35:10.528:
crsd(7909)CRS-2757:Command 'Start' timed out waiting for response from the resource 'ora.LISTENER_SCAN1.lsnr'. Details at (:CRSPE00111:) {1:30950:2} in /u01/app/11.2.0/grid/log/xkhis01/crsd/crsd.log.
2025-07-03 18:35:10.536:
crsd(7909)CRS-2757:Command 'Start' timed out waiting for response from the resource 'ora.LISTENER.lsnr'. Details at (:CRSPE00111:) {1:30950:2} in /u01/app/11.2.0/grid/log/xkhis01/crsd/crsd.log.
2025-07-03 18:36:21.190:
crsd(7909)CRS-2807:Resource 'ora.xkhis02.vip' failed to start automatically.
2025-07-03 18:43:32.161:
cssd(7702)CRS-1612:Network communication with node xkhis02 (2) missing for 50% of timeout interval. Removal of this node from cluster in 14.772 seconds
2025-07-03 18:43:40.241:
cssd(7702)CRS-1611:Network communication with node xkhis02 (2) missing for 75% of timeout interval. Removal of this node from cluster in 6.692 seconds
2025-07-03 18:43:44.282:
cssd(7702)CRS-1610:Network communication with node xkhis02 (2) missing for 90% of timeout interval. Removal of this node from cluster in 2.651 seconds
2025-07-03 18:43:46.943:
cssd(7702)CRS-1632:Node xkhis02 is being removed from the cluster in cluster incarnation 536359781
2025-07-03 18:43:46.952:
cssd(7702)CRS-1601:CSSD Reconfiguration complete. Active nodes are xkhis01 .
2025-07-03 18:44:32.433:
cssd(7702)CRS-1601:CSSD Reconfiguration complete. Active nodes are xkhis01 xkhis02 .
2025-07-03 18:45:04.846:
crsd(7909)CRS-2772:Server 'xkhis02' has been assigned to pool 'Generic'.
2025-07-03 18:45:04.847:
crsd(7909)CRS-2772:Server 'xkhis02' has been assigned to pool 'ora.hisdb'.
2025-07-03 18:48:08.956:
crsd(7909)CRS-2757:Command 'Start' timed out waiting for response from the resource 'ora.LISTENER.lsnr'. Details at (:CRSPE00111:) {2:21712:2} in /u01/app/11.2.0/grid/log/xkhis01/crsd/crsd.log.
2025-07-08 23:54:22.104:
/u01/app/11.2.0/grid/bin/oraagent.bin(8391)CRS-5011:Check of resource "hisdb" failed: details at "(:CLSN00007:)" in "/u01/app/11.2.0/grid/log/xkhis01/agent/crsd/oraagent_oracle/oraagent_oracle.log"
2025-07-08 23:54:22.113:
crsd(7909)CRS-2765:Resource 'ora.hisdb.db' has failed on server 'xkhis01'.
2025-07-10 02:04:29.186:
/u01/app/11.2.0/grid/bin/oraagent.bin(24799)CRS-5011:Check of resource "hisdb" failed: details at "(:CLSN00007:)" in "/u01/app/11.2.0/grid/log/xkhis01/agent/crsd/oraagent_oracle/oraagent_oracle.log"
2025-07-10 02:04:29.193:
crsd(7909)CRS-2765:Resource 'ora.hisdb.db' has failed on server 'xkhis01'.
2025-07-10 06:34:20.522:
cssd(7702)CRS-1649:An I/O error occured for voting file: /dev/rdisk/disk32; details at (:CSSNM00059:) in /u01/app/11.2.0/grid/log/xkhis01/cssd/ocssd.log.
2025-07-10 06:34:24.666:
cssd(7702)CRS-1605:CSSD voting file is online: /dev/rdisk/disk32; details in /u01/app/11.2.0/grid/log/xkhis01/cssd/ocssd.log.