oracle exadata x9的存储节点重启问题分析

近期遇到一台oracle exadata x9的存储节点重启，由于一体机的存储节点是有冗余的，故障发生后，未对数据库业务产生影响。

通过监控系统发现当时数据库的ASM容量发生了波动变化，发出了告警。进行分析后发现是RAID控制器方面的问题（也可能是BUG），有告警日志megaraid_sas 0000:4b:00.0: Application firmware crash；而重启后存储节点已经自动恢复正常，数据库层面，也存储节点对应的ASM磁盘也自动加回去了，凸显了oracle exadata还是很好用的。

检查ILOM，无告警。

检查MESSAGE日志：

Mar 13 02:50:01 celadm05 systemd: Started Session 64057 of user root.

Mar 13 02:50:01 celadm05 systemd: Started Session 64058 of user root.

Mar 13 03:00:01 celadm05 systemd: Started Session 64059 of user root.

Mar 13 03:00:01 celadm05 systemd: Started Session 64060 of user root.

Mar 13 03:00:01 celadm05 systemd: Started Session 64061 of user root.

Mar 13 03:01:01 celadm05 systemd: Started Session 64062 of user root.

Mar 13 03:03:01 celadm05 systemd: Started Session 64063 of user root.

Mar 13 03:06:06 celadm05 systemd: Stopping LSB: Monitors LSI MegaRAID health....

Mar 13 03:06:06 celadm05 kernel: megaraid_sas 0000:4b:00.0: Application firmware crash dump mode set success

Mar 13 03:06:06 celadm05 mrdiag: Stopping mrdiagd: $OK$

Mar 13 03:06:06 celadm05 systemd: Stopped LSB: Monitors LSI MegaRAID health..

Mar 13 03:06:06 celadm05 systemd: Starting LSB: Monitors LSI MegaRAID health....

Mar 13 03:06:06 celadm05 mrdiag: Starting mrdiagd: $OK$ #015 $OK$

Mar 13 03:06:06 celadm05 systemd: Started LSB: Monitors LSI MegaRAID health..

Mar 13 03:06:06 celadm05 kernel: megaraid_sas 0000:4b:00.0: Application firmware crash dump mode set success

Mar 13 03:06:09 celadm05 kernel: XFS (md25p1): Mounting V4 Filesystem

Mar 13 03:06:09 celadm05 kernel: XFS (md25p1): Ending clean mount

Mar 13 03:06:12 celadm05 kernel: XFS (md25p1): Unmounting Filesystem

Mar 13 03:06:12 celadm05 setup_disks.py: Starting: command line: ['/opt/oracle.cellos/setup_disks.py', '--get-dev

ice', '--label', 'var', '--sys-dev', '/dev/md24p5', '--layout', 'cell_layout_1']

Mar 13 03:06:14 celadm05 setup_disks.py: Starting: command line: ['/opt/oracle.cellos/setup_disks.py', '--get-dev

ice', '--label', 'var', '--sys-dev', '/dev/md24p5', '--inactive', '--layout', 'cell_layout_1']

Mar 13 03:06:18 celadm05 kernel: XFS (md25p1): Mounting V4 Filesystem

Mar 13 03:06:18 celadm05 kernel: XFS (md25p1): Ending clean mount

Mar 13 03:06:21 celadm05 kernel: XFS (md25p1): Unmounting Filesystem

Mar 13 03:06:55 celadm05 kernel: md25: p1 p2

Mar 13 03:06:59 celadm05 kernel: md25: p1 p2

Mar 13 03:07:01 celadm05 systemd: Stopping ExaWatcher...

Mar 13 03:07:02 celadm05 systemd: Stopped ExaWatcher.

Mar 13 03:07:02 celadm05 systemd: Started ExaWatcher.

Mar 13 03:07:02 celadm05 rsyslogd: [origin software="rsyslogd" swVersion="8.24.0-57.0.1.el7_9.2" x-pid="44858" x-

info="http://www.rsyslog.com"] rsyslogd was HUPed

Mar 13 03:10:01 celadm05 systemd: Started Session 64064 of user root.

Mar 13 03:10:01 celadm05 systemd: Started Session 64065 of user root.

Mar 13 03:20:01 celadm05 systemd: Started Session 64066 of user root.

Mar 13 03:20:01 celadm05 systemd: Started Session 64067 of user root.

Mar 13 03:29:39 celadm05 nscd: 6064 monitoring file `/etc/passwd` (1)

Mar 13 03:29:39 celadm05 nscd: 6064 monitoring directory `/etc` (2)

Mar 13 03:29:39 celadm05 nscd: 6064 monitoring file `/etc/group` (3)

Mar 13 03:29:39 celadm05 nscd: 6064 monitoring directory `/etc` (2)

Mar 13 03:29:39 celadm05 nscd: 6064 monitoring file `/etc/hosts` (4)

Mar 13 03:29:39 celadm05 nscd: 6064 monitoring directory `/etc` (2)

Mar 13 03:29:39 celadm05 nscd: 6064 monitoring file `/etc/resolv.conf` (5)

Mar 13 03:29:39 celadm05 nscd: 6064 monitoring directory `/etc` (2)

Mar 13 03:29:39 celadm05 nscd: 6064 monitoring file `/etc/services` (6)

Mar 13 03:29:39 celadm05 nscd: 6064 monitoring directory `/etc` (2)

Mar 13 03:30:01 celadm05 systemd: Started Session 64068 of user root.

Mar 13 03:30:01 celadm05 systemd: Started Session 64069 of user root.

Mar 13 03:33:01 celadm05 systemd: Started Session 64070 of user root.

Mar 13 03:40:01 celadm05 systemd: Started Session 64071 of user root.

Mar 13 03:40:01 celadm05 systemd: Started Session 64072 of user root.

Mar 13 03:50:01 celadm05 systemd: Started Session 64073 of user root.

Mar 13 03:50:01 celadm05 systemd: Started Session 64074 of user root.

Mar 13 04:00:01 celadm05 systemd: Started Session 64075 of user root.

Mar 13 04:00:01 celadm05 systemd: Started Session 64076 of user root.

Mar 13 04:00:01 celadm05 systemd: Started Session 64077 of user root.

Mar 13 04:01:01 celadm05 systemd: Started Session 64078 of user root.

Mar 13 04:03:01 celadm05 systemd: Started Session 64079 of user root.

Mar 13 04:10:01 celadm05 systemd: Started Session 64080 of user root.

Mar 13 04:10:01 celadm05 systemd: Started Session 64081 of user root.

Mar 13 04:20:01 celadm05 systemd: Started Session 64082 of user root.

Mar 13 04:20:01 celadm05 systemd: Started Session 64083 of user root.

Mar 13 04:29:39 celadm05 nscd: 11183 monitoring file `/etc/passwd` (1)