近期，某客户多次出现RAC节点异常重启的问题，对日志进行分析，均为报错IO方面问题；通过了解数据库底层服务器、网络、存储的配置情况，该客户采用双存储（双活）。每个存储放在不通的办公楼，服务器也采用2台搭建RAC，分布在不同大楼。
结合这些信息推测问题可能是存储到服务器的光纤信号波动，后续进行光衰测试，确实存在问题，进行光模块更换和光纤线更换后，后续观察半年左右，系统运行稳定，未再出现IO异常引起的重启问题。
相关日志信息分析如下：
1、数据库日志
ditional information: -1
Additional information: 8192
WARNING: failed to write mirror side 1 of virtual extent 27592 logical extent 0 of file 574 in group 2 on disk 5 allocation unit 857523
Errors in file /u01/app/oracle/diag/rdbms/hisdb/hisdb2/trace/hisdb2_dbw9_29987.trc:
ORA-15080: synchronous I/O operation to a disk failed
ORA-27063: number of bytes read/written is incorrect
HPUX-ia64 Error: 11: Resource temporarily unavailable
Additional information: -1
Additional information: 8192
WARNING: failed to write mirror side 1 of virtual extent 26947 logical extent 0 of file 576 in group 2 on disk 3 allocation unit 46307
KCF: read, write or open error, block=0x34a1a6 online=1
file=85 '+DATA/hisdb/datafile/emr5.576.1092670511'
error=15081 txt: ''
Errors in file /u01/app/oracle/diag/rdbms/hisdb/hisdb2/trace/hisdb2_dbw9_29987.trc:
Errors in file /u01/app/oracle/diag/rdbms/hisdb/hisdb2/trace/hisdb2_dbw9_29987.trc:
ORA-63999: data file suffered media failure
ORA-01114: IO error writing block to file 85 (block # 3449254)
ORA-01110: data file 85: '+DATA/hisdb/datafile/emr5.576.1092670511'
ORA-15081: failed to submit an I/O operation to a disk
ORA-15081: failed to submit an I/O operation to a disk
Thu Jul 03 17:52:43 2025
LGWR waiting for instance termination
DBW9 (ospid: 29987): terminating the instance due to error 63999
Thu Jul 03 17:52:44 2025
opiodr aborting process unknown ospid (16883) as a result of ORA-1092
Thu Jul 03 17:52:44 2025
opiodr aborting process unknown ospid (17135) as a result of ORA-1092
Thu Jul 03 17:52:44 2025
ORA-1092 : opitsk aborting process
Thu Jul 03 17:52:44 2025
2、集群GRID日志
$ tail -200 alertxkhis01.log
2025-03-06 22:57:02.696:
crsd(4872)\]CRS-2772:Server 'xkhis02' has been assigned to pool 'Generic'. 2025-03-06 22:57:02.697: \[crsd(4872)\]CRS-2772:Server 'xkhis02' has been assigned to pool 'ora.hisdb'. 2025-05-08 17:17:00.496: \[/u01/app/11.2.0/grid/bin/oraagent.bin(4930)\]CRS-5818:Aborted command 'stop' for resource 'ora.LISTENER.lsnr'. Details at (:CRSAGF00113:) {1:46568:8793} in /u01/app/11.2.0/grid/log/xkhis01/agent/crsd/oraagent_grid/oraagent_grid.log. 2025-05-08 17:17:00.546: \[/u01/app/11.2.0/grid/bin/oraagent.bin(4930)\]CRS-5014:Agent "/u01/app/11.2.0/grid/bin/oraagent.bin" timed out starting process "/u01/app/11.2.0/grid/bin/lsnrctl" for action "stop": details at "(:CLSN00009:)" in "/u01/app/11.2.0/grid/log/xkhis01/agent/crsd/oraagent_grid/oraagent_grid.log" 2025-05-08 17:17:04.519: \[crsd(4872)\]CRS-2757:Command 'Stop' timed out waiting for response from the resource 'ora.LISTENER.lsnr'. Details at (:CRSPE00111:) {1:46568:8793} in /u01/app/11.2.0/grid/log/xkhis01/crsd/crsd.log. 2025-05-08 17:17:04.635: \[/u01/app/11.2.0/grid/bin/oraagent.bin(4930)\]CRS-5016:Process "/u01/app/11.2.0/grid/bin/lsnrctl" spawned by agent "/u01/app/11.2.0/grid/bin/oraagent.bin" for action "check" failed: details at "(:CLSN00010:)" in "/u01/app/11.2.0/grid/log/xkhis01/agent/crsd/oraagent_grid/oraagent_grid.log" 2025-05-08 17:17:04.765: \[/u01/app/11.2.0/grid/bin/oraagent.bin(4930)\]CRS-5016:Process "/u01/app/11.2.0/grid/bin/lsnrctl" spawned by agent "/u01/app/11.2.0/grid/bin/oraagent.bin" for action "clean" failed: details at "(:CLSN00010:)" in "/u01/app/11.2.0/grid/log/xkhis01/agent/crsd/oraagent_grid/oraagent_grid.log" 2025-05-08 17:17:04.884: \[/u01/app/11.2.0/grid/bin/oraagent.bin(4930)\]CRS-5016:Process "/u01/app/11.2.0/grid/bin/lsnrctl" spawned by agent "/u01/app/11.2.0/grid/bin/oraagent.bin" for action "check" failed: details at "(:CLSN00010:)" in "/u01/app/11.2.0/grid/log/xkhis01/agent/crsd/oraagent_grid/oraagent_grid.log" 2025-05-08 17:17:29.974: \[crsd(4872)\]CRS-2757:Command 'Stop' timed out waiting for response from the resource 'ora.LISTENER.lsnr'. Details at (:CRSPE00111:) {2:38591:41607} in /u01/app/11.2.0/grid/log/xkhis01/crsd/crsd.log. 2025-05-08 17:57:33.642: \[/u01/app/11.2.0/grid/bin/oraagent.bin(4930)\]CRS-5818:Aborted command 'stop' for resource 'ora.LISTENER.lsnr'. Details at (:CRSAGF00113:) {1:46568:9867} in /u01/app/11.2.0/grid/log/xkhis01/agent/crsd/oraagent_grid/oraagent_grid.log. 2025-05-08 17:57:33.693: \[/u01/app/11.2.0/grid/bin/oraagent.bin(4930)\]CRS-5014:Agent "/u01/app/11.2.0/grid/bin/oraagent.bin" timed out starting process "/u01/app/11.2.0/grid/bin/lsnrctl" for action "stop": details at "(:CLSN00009:)" in "/u01/app/11.2.0/grid/log/xkhis01/agent/crsd/oraagent_grid/oraagent_grid.log" 2025-05-08 17:57:37.663: \[crsd(4872)\]CRS-2757:Command 'Stop' timed out waiting for response from the resource 'ora.LISTENER.lsnr'. Details at (:CRSPE00111:) {1:46568:9867} in /u01/app/11.2.0/grid/log/xkhis01/crsd/crsd.log. 2025-05-08 17:57:37.782: \[/u01/app/11.2.0/grid/bin/oraagent.bin(4930)\]CRS-5016:Process "/u01/app/11.2.0/grid/bin/lsnrctl" spawned by agent "/u01/app/11.2.0/grid/bin/oraagent.bin" for action "check" failed: details at "(:CLSN00010:)" in "/u01/app/11.2.0/grid/log/xkhis01/agent/crsd/oraagent_grid/oraagent_grid.log" 2025-05-08 17:57:37.910: \[/u01/app/11.2.0/grid/bin/oraagent.bin(4930)\]CRS-5016:Process "/u01/app/11.2.0/grid/bin/lsnrctl" spawned by agent "/u01/app/11.2.0/grid/bin/oraagent.bin" for action "clean" failed: details at "(:CLSN00010:)" in "/u01/app/11.2.0/grid/log/xkhis01/agent/crsd/oraagent_grid/oraagent_grid.log" 2025-05-08 17:57:38.032: \[/u01/app/11.2.0/grid/bin/oraagent.bin(4930)\]CRS-5016:Process "/u01/app/11.2.0/grid/bin/lsnrctl" spawned by agent "/u01/app/11.2.0/grid/bin/oraagent.bin" for action "check" failed: details at "(:CLSN00010:)" in "/u01/app/11.2.0/grid/log/xkhis01/agent/crsd/oraagent_grid/oraagent_grid.log" 2025-05-08 17:58:38.410: \[crsd(4872)\]CRS-2757:Command 'Stop' timed out waiting for response from the resource 'ora.LISTENER.lsnr'. Details at (:CRSPE00111:) {2:38591:42605} in /u01/app/11.2.0/grid/log/xkhis01/crsd/crsd.log. 2025-07-03 17:52:44.148: \[crsd(4872)\]CRS-2765:Resource 'ora.hisdb.db' has failed on server 'xkhis02'. 2025-07-03 17:53:09.449: \[/u01/app/11.2.0/grid/bin/oraagent.bin(12008)\]CRS-5011:Check of resource "hisdb" failed: details at "(:CLSN00007:)" in "/u01/app/11.2.0/grid/log/xkhis01/agent/crsd/oraagent_oracle/oraagent_oracle.log" 2025-07-03 17:53:09.456: \[crsd(4872)\]CRS-2765:Resource 'ora.hisdb.db' has failed on server 'xkhis01'. 2025-07-03 17:53:16.114: \[cssd(4730)\]CRS-1649:An I/O error occured for voting file: /dev/rdisk/disk30; details at (:CSSNM00060:) in /u01/app/11.2.0/grid/log/xkhis01/cssd/ocssd.log. 2025-07-03 17:53:16.114: \[cssd(4730)\]CRS-1649:An I/O error occured for voting file: /dev/rdisk/disk32; details at (:CSSNM00060:) in /u01/app/11.2.0/grid/log/xkhis01/cssd/ocssd.log. 2025-07-03 17:53:16.114: \[cssd(4730)\]CRS-1649:An I/O error occured for voting file: /dev/rdisk/disk33; details at (:CSSNM00060:) in /u01/app/11.2.0/grid/log/xkhis01/cssd/ocssd.log. 2025-07-03 17:53:20.514: \[cssd(4730)\]CRS-1649:An I/O error occured for voting file: /dev/rdisk/disk32; details at (:CSSNM00059:) in /u01/app/11.2.0/grid/log/xkhis01/cssd/ocssd.log. 2025-07-03 17:53:46.915: \[cssd(4730)\]CRS-1649:An I/O error occured for voting file: /dev/rdisk/disk30; details at (:CSSNM00059:) in /u01/app/11.2.0/grid/log/xkhis01/cssd/ocssd.log. 2025-07-03 17:53:46.915: \[cssd(4730)\]CRS-1649:An I/O error occured for voting file: /dev/rdisk/disk33; details at (:CSSNM00059:) in /u01/app/11.2.0/grid/log/xkhis01/cssd/ocssd.log. 2025-07-03 17:53:46.919: \[cssd(4730)\]CRS-1605:CSSD voting file is online: /dev/rdisk/disk30; details in /u01/app/11.2.0/grid/log/xkhis01/cssd/ocssd.log. 2025-07-03 17:53:46.921: \[cssd(4730)\]CRS-1605:CSSD voting file is online: /dev/rdisk/disk33; details in /u01/app/11.2.0/grid/log/xkhis01/cssd/ocssd.log. 2025-07-03 17:53:51.315: \[cssd(4730)\]CRS-1649:An I/O error occured for voting file: /dev/rdisk/disk32; details at (:CSSNM00060:) in /u01/app/11.2.0/grid/log/xkhis01/cssd/ocssd.log. 2025-07-03 17:53:51.552: \[cssd(4730)\]CRS-1605:CSSD voting file is online: /dev/rdisk/disk32; details in /u01/app/11.2.0/grid/log/xkhis01/cssd/ocssd.log. 2025-07-03 17:54:05.065: \[/u01/app/11.2.0/grid/bin/oraagent.bin(4930)\]CRS-5822:Agent '/u01/app/11.2.0/grid/bin/oraagent_grid' disconnected from server. Details at (:CRSAGF00117:) {0:3:33} in /u01/app/11.2.0/grid/log/xkhis01/agent/crsd/oraagent_grid/oraagent_grid.log. 2025-07-03 17:54:05.067: \[/u01/app/11.2.0/grid/bin/orarootagent.bin(4935)\]CRS-5822:Agent '/u01/app/11.2.0/grid/bin/orarootagent_root' disconnected from server. Details at (:CRSAGF00117:) {0:5:3458} in /u01/app/11.2.0/grid/log/xkhis01/agent/crsd/orarootagent_root/orarootagent_root.log. 2025-07-03 17:54:05.067: \[/u01/app/11.2.0/grid/bin/scriptagent.bin(14469)\]CRS-5822:Agent '/u01/app/11.2.0/grid/bin/scriptagent_grid' disconnected from server. Details at (:CRSAGF00117:) {0:9:16} in /u01/app/11.2.0/grid/log/xkhis01/agent/crsd/scriptagent_grid/scriptagent_grid.log. 2025-07-03 17:54:05.068: \[/u01/app/11.2.0/grid/bin/oraagent.bin(2346)\]CRS-5822:Agent '/u01/app/11.2.0/grid/bin/oraagent_oracle' disconnected from server. Details at (:CRSAGF00117:) {0:15:3} in /u01/app/11.2.0/grid/log/xkhis01/agent/crsd/oraagent_oracle/oraagent_oracle.log. 2025-07-03 17:54:05.080: \[ohasd(4384)\]CRS-2765:Resource 'ora.crsd' has failed on server 'xkhis01'. 2025-07-03 17:54:06.526: \[crsd(2415)\]CRS-1013:The OCR location in an ASM disk group is inaccessible. Details in /u01/app/11.2.0/grid/log/xkhis01/crsd/crsd.log. 2025-07-03 17:54:06.540: \[crsd(2415)\]CRS-0804:Cluster Ready Service aborted due to Oracle Cluster Registry error \[PROC-26: Error while accessing the physical storage ORA-12549: TNS:operating system resource quota exceeded \]. Details at (:CRSD00111:) in /u01/app/11.2.0/grid/log/xkhis01/crsd/crsd.log. 2025-07-03 17:54:11.610: \[ohasd(4384)\]CRS-2765:Resource 'ora.crsd' has failed on server 'xkhis01'. 2025-07-03 17:54:11.627: \[/u01/app/11.2.0/grid/bin/orarootagent.bin(4787)\]CRS-5013:Agent "/u01/app/11.2.0/grid/bin/orarootagent.bin" failed to start process "/u01/app/11.2.0/grid/bin/crsd" for action "start": details at "(:CLSN00008:)" in "/u01/app/11.2.0/grid/log/xkhis01/agent/ohasd/orarootagent_root/orarootagent_root.log" 2025-07-03 17:54:13.276: \[ohasd(4384)\]CRS-2878:Failed to restart resource 'ora.crsd' 2025-07-03 17:54:13.278: \[ohasd(4384)\]CRS-2769:Unable to failover resource 'ora.crsd'. 2025-07-03 18:18:57.917: \[mdnsd(4710)\]CRS-5602:mDNS service stopping by request. 2025-07-03 18:18:57.919: \[ctssd(4791)\]CRS-2405:The Cluster Time Synchronization Service on host xkhis01 is shutdown by user 2025-07-03 18:28:57.949: \[/u01/app/11.2.0/grid/bin/oraagent.bin(4706)\]CRS-5818:Aborted command 'stop' for resource 'ora.asm'. Details at (:CRSAGF00113:) {0:0:20112} in /u01/app/11.2.0/grid/log/xkhis01/agent/ohasd/oraagent_grid/oraagent_grid.log. 2025-07-03 18:28:59.962: \[ohasd(4384)\]CRS-2757:Command 'Stop' timed out waiting for response from the resource 'ora.asm'. Details at (:CRSPE00111:) {0:0:20112} in /u01/app/11.2.0/grid/log/xkhis01/ohasd/ohasd.log. 2025-07-03 18:29:04.493: \[cssd(4730)\]CRS-1603:CSSD on node xkhis01 shutdown by user. 2025-07-03 18:29:12.642: \[ohasd(4384)\]CRS-2767:Resource state recovery not attempted for 'ora.cssdmonitor' as its target state is OFFLINE 2025-07-03 18:29:12.643: \[ohasd(4384)\]CRS-2769:Unable to failover resource 'ora.cssdmonitor'. 2025-07-03 18:29:12.740: \[cssd(4730)\]CRS-1660:The CSS daemon shutdown has completed 2025-07-03 18:29:15.244: \[gpnpd(4714)\]CRS-2329:GPNPD on node xkhis01 shutdown. 2025-07-03 18:29:17.985: \[/u01/app/11.2.0/grid/bin/oraagent.bin(4706)\]CRS-5822:Agent '/u01/app/11.2.0/grid/bin/oraagent_grid' disconnected from server. Details at (:CRSAGF00117:) {0:9:24} in /u01/app/11.2.0/grid/log/xkhis01/agent/ohasd/oraagent_grid/oraagent_grid.log. 2025-07-03 18:29:17.986: \[/u01/app/11.2.0/grid/bin/orarootagent.bin(4787)\]CRS-5822:Agent '/u01/app/11.2.0/grid/bin/orarootagent_root' disconnected from server. Details at (:CRSAGF00117:) {0:17:23223} in /u01/app/11.2.0/grid/log/xkhis01/agent/ohasd/orarootagent_root/orarootagent_root.log. 2025-07-03 18:30:58.930: \[ohasd(7585)\]CRS-2112:The OLR service started on node xkhis01. 2025-07-03 18:30:58.970: \[ohasd(7585)\]CRS-1301:Oracle High Availability Service started on node xkhis01. 2025-07-03 18:30:58.971: \[ohasd(7585)\]CRS-8017:location: /var/opt/oracle/lastgasp has 2 reboot advisory log files, 0 were announced and 0 errors occurred 2025-07-03 18:31:03.452: \[/u01/app/11.2.0/grid/bin/orarootagent.bin(7604)\]CRS-2302:Cannot get GPnP profile. Error CLSGPNP_NO_DAEMON (GPNPD daemon is not running). 2025-07-03 18:31:09.197: \[gpnpd(7686)\]CRS-2328:GPNPD started on node xkhis01. 2025-07-03 18:31:11.816: \[cssd(7702)\]CRS-1713:CSSD daemon is started in clustered mode 2025-07-03 18:31:13.656: \[ohasd(7585)\]CRS-2767:Resource state recovery not attempted for 'ora.diskmon' as its target state is OFFLINE 2025-07-03 18:31:13.656: \[ohasd(7585)\]CRS-2769:Unable to failover resource 'ora.diskmon'. 2025-07-03 18:31:30.030: \[cssd(7702)\]CRS-1707:Lease acquisition for node xkhis01 number 1 completed 2025-07-03 18:31:31.392: \[cssd(7702)\]CRS-1605:CSSD voting file is online: /dev/rdisk/disk30; details in /u01/app/11.2.0/grid/log/xkhis01/cssd/ocssd.log. 2025-07-03 18:31:31.402: \[cssd(7702)\]CRS-1605:CSSD voting file is online: /dev/rdisk/disk32; details in /u01/app/11.2.0/grid/log/xkhis01/cssd/ocssd.log. 2025-07-03 18:31:31.412: \[cssd(7702)\]CRS-1605:CSSD voting file is online: /dev/rdisk/disk33; details in /u01/app/11.2.0/grid/log/xkhis01/cssd/ocssd.log. 2025-07-03 18:31:33.056: \[cssd(7702)\]CRS-1601:CSSD Reconfiguration complete. Active nodes are xkhis01 xkhis02 . 2025-07-03 18:31:35.124: \[ctssd(7800)\]CRS-2403:The Cluster Time Synchronization Service on host xkhis01 is in observer mode. 2025-07-03 18:31:35.461: \[ctssd(7800)\]CRS-2407:The new Cluster Time Synchronization Service reference node is host xkhis02. 2025-07-03 18:31:35.462: \[ctssd(7800)\]CRS-2401:The Cluster Time Synchronization Service started on host xkhis01. 2025-07-03 18:31:37.118: \[ohasd(7585)\]CRS-2767:Resource state recovery not attempted for 'ora.diskmon' as its target state is OFFLINE 2025-07-03 18:31:37.118: \[ohasd(7585)\]CRS-2769:Unable to failover resource 'ora.diskmon'. 2025-07-03 18:31:47.552: \[ctssd(7800)\]CRS-2409:The clock on host xkhis01 is not synchronous with the mean cluster time. No action has been taken as the Cluster Time Synchronization Service is running in observer mode. 2025-07-03 18:32:01.058: \[crsd(7909)\]CRS-1012:The OCR service started on node xkhis01. 2025-07-03 18:32:01.448: \[evmd(7806)\]CRS-1401:EVMD started on node xkhis01. 2025-07-03 18:32:03.316: \[crsd(7909)\]CRS-1201:CRSD started on node xkhis01. 2025-07-03 18:32:04.242: \[/u01/app/11.2.0/grid/bin/oraagent.bin(7974)\]CRS-5011:Check of resource "hisdb" failed: details at "(:CLSN00007:)" in "/u01/app/11.2.0/grid/log/xkhis01/agent/crsd/oraagent_oracle/oraagent_oracle.log" 2025-07-03 18:32:06.408: \[crsd(7909)\]CRS-2772:Server 'xkhis01' has been assigned to pool 'Generic'. 2025-07-03 18:32:06.409: \[crsd(7909)\]CRS-2772:Server 'xkhis01' has been assigned to pool 'ora.hisdb'. 2025-07-03 18:34:04.412: \[ctssd(7800)\]CRS-2407:The new Cluster Time Synchronization Service reference node is host xkhis01. 2025-07-03 18:35:06.505: \[/u01/app/11.2.0/grid/bin/oraagent.bin(7965)\]CRS-5818:Aborted command 'start' for resource 'ora.LISTENER_SCAN1.lsnr'. Details at (:CRSAGF00113:) {1:30950:2} in /u01/app/11.2.0/grid/log/xkhis01/agent/crsd/oraagent_grid/oraagent_grid.log. 2025-07-03 18:35:06.505: \[/u01/app/11.2.0/grid/bin/oraagent.bin(7965)\]CRS-5818:Aborted command 'start' for resource 'ora.LISTENER.lsnr'. Details at (:CRSAGF00113:) {1:30950:2} in /u01/app/11.2.0/grid/log/xkhis01/agent/crsd/oraagent_grid/oraagent_grid.log. 2025-07-03 18:35:06.584: \[/u01/app/11.2.0/grid/bin/oraagent.bin(7965)\]CRS-5014:Agent "/u01/app/11.2.0/grid/bin/oraagent.bin" timed out starting process "/u01/app/11.2.0/grid/bin/lsnrctl" for action "start": details at "(:CLSN00009:)" in "/u01/app/11.2.0/grid/log/xkhis01/agent/crsd/oraagent_grid/oraagent_grid.log" 2025-07-03 18:35:06.594: \[/u01/app/11.2.0/grid/bin/oraagent.bin(7965)\]CRS-5014:Agent "/u01/app/11.2.0/grid/bin/oraagent.bin" timed out starting process "/u01/app/11.2.0/grid/bin/lsnrctl" for action "start": details at "(:CLSN00009:)" in "/u01/app/11.2.0/grid/log/xkhis01/agent/crsd/oraagent_grid/oraagent_grid.log" 2025-07-03 18:35:10.528: \[crsd(7909)\]CRS-2757:Command 'Start' timed out waiting for response from the resource 'ora.LISTENER_SCAN1.lsnr'. Details at (:CRSPE00111:) {1:30950:2} in /u01/app/11.2.0/grid/log/xkhis01/crsd/crsd.log. 2025-07-03 18:35:10.536: \[crsd(7909)\]CRS-2757:Command 'Start' timed out waiting for response from the resource 'ora.LISTENER.lsnr'. Details at (:CRSPE00111:) {1:30950:2} in /u01/app/11.2.0/grid/log/xkhis01/crsd/crsd.log. 2025-07-03 18:36:21.190: \[crsd(7909)\]CRS-2807:Resource 'ora.xkhis02.vip' failed to start automatically. 2025-07-03 18:43:32.161: \[cssd(7702)\]CRS-1612:Network communication with node xkhis02 (2) missing for 50% of timeout interval. Removal of this node from cluster in 14.772 seconds 2025-07-03 18:43:40.241: \[cssd(7702)\]CRS-1611:Network communication with node xkhis02 (2) missing for 75% of timeout interval. Removal of this node from cluster in 6.692 seconds 2025-07-03 18:43:44.282: \[cssd(7702)\]CRS-1610:Network communication with node xkhis02 (2) missing for 90% of timeout interval. Removal of this node from cluster in 2.651 seconds 2025-07-03 18:43:46.943: \[cssd(7702)\]CRS-1632:Node xkhis02 is being removed from the cluster in cluster incarnation 536359781 2025-07-03 18:43:46.952: \[cssd(7702)\]CRS-1601:CSSD Reconfiguration complete. Active nodes are xkhis01 . 2025-07-03 18:44:32.433: \[cssd(7702)\]CRS-1601:CSSD Reconfiguration complete. Active nodes are xkhis01 xkhis02 . 2025-07-03 18:45:04.846: \[crsd(7909)\]CRS-2772:Server 'xkhis02' has been assigned to pool 'Generic'. 2025-07-03 18:45:04.847: \[crsd(7909)\]CRS-2772:Server 'xkhis02' has been assigned to pool 'ora.hisdb'. 2025-07-03 18:48:08.956: \[crsd(7909)\]CRS-2757:Command 'Start' timed out waiting for response from the resource 'ora.LISTENER.lsnr'. Details at (:CRSPE00111:) {2:21712:2} in /u01/app/11.2.0/grid/log/xkhis01/crsd/crsd.log. 2025-07-08 23:54:22.104: \[/u01/app/11.2.0/grid/bin/oraagent.bin(8391)\]CRS-5011:Check of resource "hisdb" failed: details at "(:CLSN00007:)" in "/u01/app/11.2.0/grid/log/xkhis01/agent/crsd/oraagent_oracle/oraagent_oracle.log" 2025-07-08 23:54:22.113: \[crsd(7909)\]CRS-2765:Resource 'ora.hisdb.db' has failed on server 'xkhis01'. 2025-07-10 02:04:29.186: \[/u01/app/11.2.0/grid/bin/oraagent.bin(24799)\]CRS-5011:Check of resource "hisdb" failed: details at "(:CLSN00007:)" in "/u01/app/11.2.0/grid/log/xkhis01/agent/crsd/oraagent_oracle/oraagent_oracle.log" 2025-07-10 02:04:29.193: \[crsd(7909)\]CRS-2765:Resource 'ora.hisdb.db' has failed on server 'xkhis01'. 2025-07-10 06:34:20.522: \[cssd(7702)\]CRS-1649:An I/O error occured for voting file: /dev/rdisk/disk32; details at (:CSSNM00059:) in /u01/app/11.2.0/grid/log/xkhis01/cssd/ocssd.log. 2025-07-10 06:34:24.666: \[cssd(7702)\]CRS-1605:CSSD voting file is online: /dev/rdisk/disk32; details in /u01/app/11.2.0/grid/log/xkhis01/cssd/ocssd.log.