【银河麒麟高级服务器操作系统】服务器外挂存储ioerror分析及处理分享

更多银河麒麟操作系统产品及技术讨论，欢迎加入银河麒麟操作系统官方论坛

forum.kylinos.cn

了解更多银河麒麟操作系统全新产品，请点击访问

麒麟软件产品专区：product.kylinos.cn

开发者专区：developer.kylinos.cn

文档中心：document.kylinos.cn

服务器环境以及配置

|------|--------------|---------------------------------------------------------------|
| 系统环境 | 物理机/虚拟机/云/容器 | 物理机 |
| 网络环境 | 外网/私有网络/无网络 | 私有网络 |
| 硬件环境 | 处理器： | S2500 |
| 硬件环境 | 内存： | 512GB |
| 硬件环境 | 机器型号 | 擎天EF860 |
| 硬件环境 | 整机类型/架构： | arm64 |
| 硬件环境 | BIOS版本： | Great Wall BIOS V3.0 |
| 软件环境 | 具体操作系统版本 | 银河麒麟高级服务器操作系统 Kylin Linux Advanced Server release V10 (Lance) |
| 软件环境 | 内核版本 | 4.19.90-52.30.v2207.ky10.aarch64 |

现象描述

服务器巡检告显示有io error，需要进行分析。

现象分析

查看磁盘存储情况

根据串口日志，报IO Error错误的是dm-4和dm-5磁盘设备，对应的是/dev/mpathxsky02blk01和/dev/mpathxsky02blk02两块多路径盘。

分析串口日志

查看串口日志，系统出现过三次I/O error相关的报错，第一次导致系统发生了hung task，后面两次出现IO error报错后便出现了shutdown相关的日志打印。

分析第一次出现 I/O error

日志中多次出现print_req_error: I/O error, dev sdb和print_req_error: I/O error, dev sdc的错误。这表明设备sdb和sdc发生了I/O错误。另外，日志中还出现了rejecting I/O to offline device的消息（例如：sd 3:0:0:1: rejecting I/O to offline device），这通常意味着设备已经离线，无法再进行I/O操作。这些I/O错误可能是由于硬盘故障、连接问题（例如，SATA线缆故障）或控制器问题引起的。

日志中有多个任务被报告为挂起超过1200秒，如xfsaild/dm-4和containerd。这些任务的挂起是由于无法完成的磁盘I/O请求导致的，因为设备已经离线或不可用。出现了任务挂起（通常和无法访问存储设备相关），导致最后内核触发了恐慌（Kernel panic - not syncing: hung_task: blocked tasks）。

sdb和sdc设备应该对应sd 3:0:0:0和sd 3:0:0:1，查看当前收集的sosreport中的lsscsi命令，由于相隔的时间太过久远，没有3:0:0:0和3:0:0:1相关的设备，变为了5:0:0:0和5:0:0:1。后续也未再出现过rejecting I/O to offline device，问题应该已经修复。

|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| [30551.689416][ 86] print_req_error: I/O error, dev sdb, sector 278120^M [30551.690643][ 86] print_req_error: I/O error, dev sdb, sector 526730880^M [30551.691874][ 86] print_req_error: I/O error, dev sdb, sector 794630784^M [30551.693092][ 86] print_req_error: I/O error, dev sdb, sector 3239552^M [30601.823314][ 24] print_req_error: I/O error, dev sdb, sector 267177832^M [30622.174613][ 23] print_req_error: I/O error, dev sdb, sector 267177832^M [30627.418715][ 86] sd 3:0:0:1: rejecting I/O to offline device^M [30627.420021][ 86] print_req_error: I/O error, dev sdc, sector 209772368^M [30627.421253][ 86] print_req_error: I/O error, dev sdc, sector 209772376^M [30627.438674][ 86] sd 3:0:0:0: rejecting I/O to offline device^M [30627.439942][ 86] print_req_error: I/O error, dev sdb, sector 1745360^M [30627.441180][ 86] print_req_error: I/O error, dev sdb, sector 788207504^M [30627.442345][ 86] print_req_error: I/O error, dev sdb, sector 526731600^M [30627.443517][ 86] print_req_error: I/O error, dev sdb, sector 263186528^M [30627.444683][ 86] print_req_error: I/O error, dev sdb, sector 3239552^M [31446.409648][ 28] print_req_error: I/O error, dev sdb, sector 267177832^M [31467.420538][ 28] print_req_error: I/O error, dev sdb, sector 267177832^M [31476.890512][ 61] print_req_error: I/O error, dev sdc, sector 209772368^M [31476.891710][ 61] print_req_error: I/O error, dev sdc, sector 209772376^M [31482.488033][ 84] sd 3:0:0:1: rejecting I/O to offline device^M [31482.489254][ 84] print_req_error: I/O error, dev sdc, sector 209772392^M [31482.490507][ 84] print_req_error: I/O error, dev sdc, sector 209772384^M [31482.491733][ 84] print_req_error: I/O error, dev sdc, sector 209772400^M [31482.492918][ 84] print_req_error: I/O error, dev sdc, sector 209772408^M [31482.494051][ 84] sd 3:0:0:0: rejecting I/O to offline device^M [31482.495054][ 84] print_req_error: I/O error, dev sdb, sector 270107808^M [31482.496109][ 84] print_req_error: I/O error, dev sdb, sector 270106784^M [31482.497193][ 84] print_req_error: I/O error, dev sdb, sector 788207504^M [31871.417820][ 4] INFO: task xfsaild/dm-4:402347 blocked for more than 1200 seconds.^M [31871.419496][ 4] Tainted: G OE 4.19.90-52.30.v2207.ky10.aarch64 #1^M [31871.421055][ 4] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.^M [31871.422705][ 4] xfsaild/dm-4 D 0 402347 2 0x00000628^M [31871.423726][ 4] Call trace:^M [31871.424436][ 4] __switch_to+0xe8/0x150^M [31871.425293][ 4] __schedule+0x2b0/0x768^M [31871.426152][ 4] schedule+0x30/0xf0^M [31871.426917][ 4] xfs_log_force+0x170/0x358^M [31871.427809][ 4] xfsaild_push+0x5a8/0x6c0^M [31871.428766][ 4] xfsaild+0x11c/0x238^M [31871.429627][ 4] kthread+0x134/0x138^M [31871.430447][ 4] ret_from_fork+0x10/0x18^M [31871.431339][ 4] INFO: task containerd:409512 blocked for more than 1200 seconds.^M [31871.433064][ 4] Tainted: G OE 4.19.90-52.30.v2207.ky10.aarch64 #1^M [31871.434889][ 4] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.^M [31871.436710][ 4] containerd D 0 409512 1 0x00000608^M [31871.437906][ 4] Call trace:^M [31871.438822][ 4] __switch_to+0xe8/0x150^M [31871.439785][ 4] __schedule+0x2b0/0x768^M [31871.440741][ 4] schedule+0x30/0xf0^M [31871.441658][ 4] io_schedule+0x20/0x90^M [31871.442570][ 4] wait_on_page_bit+0x134/0x178^M [31871.443548][ 4] __filemap_fdatawait_range+0xd0/0x120^M [31871.444618][ 4] file_write_and_wait_range+0xb0/0xd8^M [31871.445655][ 4] xfs_file_fsync+0x58/0x1d8^M [31871.446584][ 4] vfs_fsync_range+0x4c/0x90^M [31871.447743][ 4] do_fsync+0x48/0x78^M [31871.448623][ 4] sys_fdatasync+0x24/0x38^M [31871.449503][ 4] __sys_trace_return+0x0/0x4^M [31871.450841][ 4] Kernel panic - not syncing: hung_task: blocked tasks^M [31871.451999][ 4] CPU: 4 PID: 748 Comm: khungtaskd Tainted: G OE 4.19.90-52.30.v2207.ky10.aarch64 #1^M [31871.453937][ 4] Source Version: ccdbfc2c55f0fb8dde14fae29155446fc8a7e941^M [31871.455059][ 4] Hardware name: GreatWall æ\x93\x8eå¤©EF860/GW-748F2A-FTG, BIOS Great Wall BIOS V3.0 2022-11-17^M [31871.457098][ 4] Call trace:^M [31871.457908][ 4] dump_backtrace+0x0/0x1b0^M [31871.458911][ 4] show_stack+0x24/0x30^M [31871.459845][ 4] dump_stack+0xb4/0xf0^M [31871.460748][ 4] panic+0x130/0x310^M [31871.461654][ 4] watchdog+0x2b8/0x468^M [31871.462536][ 4] kthread+0x134/0x138^M [31871.463337][ 4] ret_from_fork+0x10/0x18^M [31871.464354][ 4] SMP: stopping secondary CPUs^M [31871.584509][ 4] Kernel Offset: 0x37db061e0000 from 0xffff000048000000^M [31871.585577][ 4] CPU features: 0x10,a0000008^M [31871.586428][ 4] Memory Limit: none^M [31871.587456][ 4] Rebooting in 10 seconds..^M |

分析第二次出现I/O error

日志中先出现了多个print_req_error: I/O error, dev dm-4/5/6, sector xxxx，说明lvm逻辑卷输入输出出现错误。而后出现了XFS (dm-4): writeback error on sector xxxx，XFS 文件系统报告了writeback error和metadata I/O error，特别是在xlog_iodone操作中，错误发生在不同的扇区，涉及数据和元数据的读写操作。错误代码为error 5，对应于输入/输出错误（EIO），这表示底层存储设备无法完成读写请求。

而后马上出现了shutdown及reboot: Restarting system相关日志，这应是之前执行了关机操作。虽然是先出现了IO error的日志，而后出现的shutdown日志，但是shutdown操作并不会立即写到日志，推测是关机操作使得多路径存储设备下线，文件系统无法写入，而后出现的IO error。

分析第三次出现I/O error

和第二次出现IO error相同，出现IO error后，也立马出现了shutdown相关的日志。

|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| [2001517.911050][102] print_req_error: I/O error, dev dm-6, sector 62984704^M [2001517.911712][100] print_req_error: I/O error, dev dm-5, sector 218187936^M [2001517.912383][102] XFS (dm-6): writeback error on sector 62984832^M [2001517.913028][100] print_req_error: I/O error, dev dm-5, sector 216240008^M [2001517.913138][100] print_req_error: I/O error, dev dm-5, sector 6241168^M [2001517.913144][100] print_req_error: I/O error, dev dm-5, sector 7116376^M [2001517.913166][100] XFS (dm-5): writeback error on sector 6241240^M [2001517.913170][100] XFS (dm-5): writeback error on sector 7116472^M [2001517.913174][100] XFS (dm-5): writeback error on sector 7116568^M [2001517.913229][100] print_req_error: I/O error, dev dm-5, sector 6772344^M [2001517.913254][100] XFS (dm-5): writeback error on sector 6772472^M [2001517.913258][100] XFS (dm-5): writeback error on sector 6772520^M [2001517.913263][100] XFS (dm-5): writeback error on sector 338549088^M [2001517.913392][100] XFS (dm-5): writeback error on sector 110130576^M [2001517.913398][100] XFS (dm-5): writeback error on sector 110130584^M [2001517.923828][101] XFS (dm-5): metadata I/O error in "xlog_iodone" at daddr 0xc82f840 len 64 error 5^M [2001517.925072][101] XFS (dm-5): Log I/O Error Detected. Shutting down filesystem^M [2001517.925237][103] XFS (dm-6): metadata I/O error in "xlog_iodone" at daddr 0x1f44c278 len 64 error 5^M [2001517.925257][101] XFS (dm-6): metadata I/O error in "xlog_iodone" at daddr 0x1f44c2b8 len 64 error 5^M [2001517.925419][103] XFS (dm-6): Log I/O Error Detected. Shutting down filesystem^M [2001517.925421][103] XFS (dm-6): Please umount the filesystem and rectify the problem(s)^M [2001517.925840][103] XFS (dm-5): Please umount the filesystem and rectify the problem(s)^M [2001518.084461][ 59] overlayfs: failed to get xattr trusted.overlay.redirect: err=-5)^M [2001518.134555][ 29] overlayfs: failed to get xattr trusted.overlay.redirect: err=-5)^M [2001518.134679][ 8] overlayfs: failed to get metacopy (-5)^M [2001518.135513][ 29] overlayfs: failed to get xattr trusted.overlay.redirect: err=-5)^M [2001518.137716][ 17] overlayfs: failed to get xattr trusted.overlay.redirect: err=-5)^M [2001518.138605][ 17] overlayfs: failed to get xattr trusted.overlay.redirect: err=-5)^M [2001518.141203][ 71] overlayfs: failed to get metacopy (-5)^M [2001518.143645][ 54] overlayfs: failed to get metacopy (-5)^M [2001518.144968][ 95] overlayfs: failed to get metacopy (-5)^M [2001518.145177][ 46] overlayfs: failed to get xattr trusted.overlay.redirect: err=-5)^M [2001518.146489][ 46] overlayfs: failed to get xattr trusted.overlay.redirect: err=-5)^M [2001518.148168][ 72] overlayfs: failed to get metacopy (-5)^M [2001518.148170][ 34] overlayfs: failed to get metacopy (-5)^M [2001518.148259][ 34] overlayfs: failed to get metacopy (-5)^M [2001518.150749][ 96] overlayfs: failed to get metacopy (-5)^M [2001518.152389][116] overlayfs: failed to get metacopy (-5)^M [2001528.063127][ 97] systemd-shutdown[1]: Waiting for process: local-path-prov^M [2001600.084921][123] kauditd_printk_skb: 23 callbacks suppressed^M [2001613.949707][ 99] shutdown: 7 output lines suppressed due to ratelimiting^M [2001614.020310][100] dracut Warning: Killing all remaining processes^M [2001614.364201][100] dracut Warning: Unmounted /oldroot.^M [2001628.676636][ 0] reboot: Restarting system^M |

查看tuned.log，出现了两次关机操作，应该是对应这两次IO error。

分析结果

第一次出现I/O错误时，日志中显示了 rejecting I/O to offline device的消息。这通常意味着设备已经离线，无法继续进行I/O操作，从而导致了I/O错误。由于这个错误的日志已经存在一段时间，问题应该已经得到解决。

第二次和第三次出现I/O错误时，伴随着关机相关的日志信息。具体来说，关机日志（shutdown 和 reboot）表明系统在I/O错误发生前，应该有执行关机操作。系统关机通常会触发文件系统的卸载和数据的回写，但由于多路径存储设备已经下线，XFS 无法完成日志写回和其他数据的持久化操作。日志中的XFS writeback error和metadata I/O error表明XFS文件系统在尝试进行数据写回时失败。尤其是在元数据操作过程中，XFS 在执行xlog_iodone操作时未能完成I/O请求，导致元数据无法成功写回。这些错误的原因是，XFS尝试将数据写入物理存储设备，但由于设备处于离线状态或无法访问，文件系统无法完成这些操作。