【银河麒麟高级服务器操作系统】服务器外挂存储ioerror分析及处理分享

更多银河麒麟操作系统产品及技术讨论，欢迎加入银河麒麟操作系统官方论坛

forum.kylinos.cn

了解更多银河麒麟操作系统全新产品，请点击访问

麒麟软件产品专区：product.kylinos.cn

开发者专区：developer.kylinos.cn

文档中心：document.kylinos.cn

服务器环境以及配置

|------|--------------|---------------------------------------------------------------|
| 系统环境 | 物理机/虚拟机/云/容器 | 物理机 |
| 网络环境 | 外网/私有网络/无网络 | 私有网络 |
| 硬件环境 | 处理器： | S2500 |
| 硬件环境 | 内存： | 512GB |
| 硬件环境 | 机器型号 | 擎天EF860 |
| 硬件环境 | 整机类型/架构： | arm64 |
| 硬件环境 | BIOS版本： | Great Wall BIOS V3.0 |
| 软件环境 | 具体操作系统版本 | 银河麒麟高级服务器操作系统 Kylin Linux Advanced Server release V10 (Lance) |
| 软件环境 | 内核版本 | 4.19.90-52.30.v2207.ky10.aarch64 |

现象描述

服务器巡检告显示有io error，需要进行分析。

现象分析

查看磁盘存储情况

根据串口日志，报IO Error错误的是dm-4和dm-5磁盘设备，对应的是/dev/mpathxsky02blk01和/dev/mpathxsky02blk02两块多路径盘。

分析串口日志

查看串口日志，系统出现过三次I/O error相关的报错，第一次导致系统发生了hung task，后面两次出现IO error报错后便出现了shutdown相关的日志打印。

分析第一次出现 I/O error

日志中多次出现print_req_error: I/O error, dev sdb和print_req_error: I/O error, dev sdc的错误。这表明设备sdb和sdc发生了I/O错误。另外，日志中还出现了rejecting I/O to offline device的消息（例如：sd 3:0:0:1: rejecting I/O to offline device），这通常意味着设备已经离线，无法再进行I/O操作。这些I/O错误可能是由于硬盘故障、连接问题（例如，SATA线缆故障）或控制器问题引起的。

日志中有多个任务被报告为挂起超过1200秒，如xfsaild/dm-4和containerd。这些任务的挂起是由于无法完成的磁盘I/O请求导致的，因为设备已经离线或不可用。出现了任务挂起（通常和无法访问存储设备相关），导致最后内核触发了恐慌（Kernel panic - not syncing: hung_task: blocked tasks）。

sdb和sdc设备应该对应sd 3:0:0:0和sd 3:0:0:1，查看当前收集的sosreport中的lsscsi命令，由于相隔的时间太过久远，没有3:0:0:0和3:0:0:1相关的设备，变为了5:0:0:0和5:0:0:1。后续也未再出现过rejecting I/O to offline device，问题应该已经修复。

|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| $30551.689416$ $86$ print_req_error: I/O error, dev sdb, sector 278120^M $30551.690643$ $86$ print_req_error: I/O error, dev sdb, sector 526730880^M $30551.691874$ $86$ print_req_error: I/O error, dev sdb, sector 794630784^M $30551.693092$ $86$ print_req_error: I/O error, dev sdb, sector 3239552^M $30601.823314$ $24$ print_req_error: I/O error, dev sdb, sector 267177832^M $30622.174613$ $23$ print_req_error: I/O error, dev sdb, sector 267177832^M $30627.418715$ $86$ sd 3:0:0:1: rejecting I/O to offline device^M $30627.420021$ $86$ print_req_error: I/O error, dev sdc, sector 209772368^M $30627.421253$ $86$ print_req_error: I/O error, dev sdc, sector 209772376^M $30627.438674$ $86$ sd 3:0:0:0: rejecting I/O to offline device^M $30627.439942$ $86$ print_req_error: I/O error, dev sdb, sector 1745360^M $30627.441180$ $86$ print_req_error: I/O error, dev sdb, sector 788207504^M $30627.442345$ $86$ print_req_error: I/O error, dev sdb, sector 526731600^M $30627.443517$ $86$ print_req_error: I/O error, dev sdb, sector 263186528^M $30627.444683$ $86$ print_req_error: I/O error, dev sdb, sector 3239552^M $31446.409648$ $28$ print_req_error: I/O error, dev sdb, sector 267177832^M $31467.420538$ $28$ print_req_error: I/O error, dev sdb, sector 267177832^M $31476.890512$ $61$ print_req_error: I/O error, dev sdc, sector 209772368^M $31476.891710$ $61$ print_req_error: I/O error, dev sdc, sector 209772376^M $31482.488033$ $84$ sd 3:0:0:1: rejecting I/O to offline device^M $31482.489254$ $84$ print_req_error: I/O error, dev sdc, sector 209772392^M $31482.490507$ $84$ print_req_error: I/O error, dev sdc, sector 209772384^M $31482.491733$ $84$ print_req_error: I/O error, dev sdc, sector 209772400^M $31482.492918$ $84$ print_req_error: I/O error, dev sdc, sector 209772408^M $31482.494051$ $84$ sd 3:0:0:0: rejecting I/O to offline device^M $31482.495054$ $84$ print_req_error: I/O error, dev sdb, sector 270107808^M $31482.496109$ $84$ print_req_error: I/O error, dev sdb, sector 270106784^M $31482.497193$ $84$ print_req_error: I/O error, dev sdb, sector 788207504^M $31871.417820$ $4$ INFO: task xfsaild/dm-4:402347 blocked for more than 1200 seconds.^M $31871.419496$ $4$ Tainted: G OE 4.19.90-52.30.v2207.ky10.aarch64 #1^M $31871.421055$ $4$ "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.^M $31871.422705$ $4$ xfsaild/dm-4 D 0 402347 2 0x00000628^M $31871.423726$ $4$ Call trace:^M $31871.424436$ $4$ __switch_to+0xe8/0x150^M $31871.425293$ $4$ __schedule+0x2b0/0x768^M $31871.426152$ $4$ schedule+0x30/0xf0^M $31871.426917$ $4$ xfs_log_force+0x170/0x358^M $31871.427809$ $4$ xfsaild_push+0x5a8/0x6c0^M $31871.428766$ $4$ xfsaild+0x11c/0x238^M $31871.429627$ $4$ kthread+0x134/0x138^M $31871.430447$ $4$ ret_from_fork+0x10/0x18^M $31871.431339$ $4$ INFO: task containerd:409512 blocked for more than 1200 seconds.^M $31871.433064$ $4$ Tainted: G OE 4.19.90-52.30.v2207.ky10.aarch64 #1^M $31871.434889$ $4$ "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.^M $31871.436710$ $4$ containerd D 0 409512 1 0x00000608^M $31871.437906$ $4$ Call trace:^M $31871.438822$ $4$ __switch_to+0xe8/0x150^M $31871.439785$ $4$ __schedule+0x2b0/0x768^M $31871.440741$ $4$ schedule+0x30/0xf0^M $31871.441658$ $4$ io_schedule+0x20/0x90^M $31871.442570$ $4$ wait_on_page_bit+0x134/0x178^M $31871.443548$ $4$ __filemap_fdatawait_range+0xd0/0x120^M $31871.444618$ $4$ file_write_and_wait_range+0xb0/0xd8^M $31871.445655$ $4$ xfs_file_fsync+0x58/0x1d8^M $31871.446584$ $4$ vfs_fsync_range+0x4c/0x90^M $31871.447743$ $4$ do_fsync+0x48/0x78^M $31871.448623$ $4$ sys_fdatasync+0x24/0x38^M $31871.449503$ $4$ __sys_trace_return+0x0/0x4^M $31871.450841$ $4$ Kernel panic - not syncing: hung_task: blocked tasks^M $31871.451999$ $4$ CPU: 4 PID: 748 Comm: khungtaskd Tainted: G OE 4.19.90-52.30.v2207.ky10.aarch64 #1^M $31871.453937$ $4$ Source Version: ccdbfc2c55f0fb8dde14fae29155446fc8a7e941^M $31871.455059$ $4$ Hardware name: GreatWall æ\x93\x8eå¤©EF860/GW-748F2A-FTG, BIOS Great Wall BIOS V3.0 2022-11-17^M $31871.457098$ $4$ Call trace:^M $31871.457908$ $4$ dump_backtrace+0x0/0x1b0^M $31871.458911$ $4$ show_stack+0x24/0x30^M $31871.459845$ $4$ dump_stack+0xb4/0xf0^M $31871.460748$ $4$ panic+0x130/0x310^M $31871.461654$ $4$ watchdog+0x2b8/0x468^M $31871.462536$ $4$ kthread+0x134/0x138^M $31871.463337$ $4$ ret_from_fork+0x10/0x18^M $31871.464354$ $4$ SMP: stopping secondary CPUs^M $31871.584509$ $4$ Kernel Offset: 0x37db061e0000 from 0xffff000048000000^M $31871.585577$ $4$ CPU features: 0x10,a0000008^M $31871.586428$ $4$ Memory Limit: none^M $31871.587456$ $4$ Rebooting in 10 seconds..^M |

分析第二次出现I/O error

日志中先出现了多个print_req_error: I/O error, dev dm-4/5/6, sector xxxx，说明lvm逻辑卷输入输出出现错误。而后出现了XFS (dm-4): writeback error on sector xxxx，XFS 文件系统报告了writeback error和metadata I/O error，特别是在xlog_iodone操作中，错误发生在不同的扇区，涉及数据和元数据的读写操作。错误代码为error 5，对应于输入/输出错误（EIO），这表示底层存储设备无法完成读写请求。

而后马上出现了shutdown及reboot: Restarting system相关日志，这应是之前执行了关机操作。虽然是先出现了IO error的日志，而后出现的shutdown日志，但是shutdown操作并不会立即写到日志，推测是关机操作使得多路径存储设备下线，文件系统无法写入，而后出现的IO error。

分析第三次出现I/O error

和第二次出现IO error相同，出现IO error后，也立马出现了shutdown相关的日志。

|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| $2001517.911050$ $102$ print_req_error: I/O error, dev dm-6, sector 62984704^M $2001517.911712$ $100$ print_req_error: I/O error, dev dm-5, sector 218187936^M $2001517.912383$ $102$ XFS (dm-6): writeback error on sector 62984832^M $2001517.913028$ $100$ print_req_error: I/O error, dev dm-5, sector 216240008^M $2001517.913138$ $100$ print_req_error: I/O error, dev dm-5, sector 6241168^M $2001517.913144$ $100$ print_req_error: I/O error, dev dm-5, sector 7116376^M $2001517.913166$ $100$ XFS (dm-5): writeback error on sector 6241240^M $2001517.913170$ $100$ XFS (dm-5): writeback error on sector 7116472^M $2001517.913174$ $100$ XFS (dm-5): writeback error on sector 7116568^M $2001517.913229$ $100$ print_req_error: I/O error, dev dm-5, sector 6772344^M $2001517.913254$ $100$ XFS (dm-5): writeback error on sector 6772472^M $2001517.913258$ $100$ XFS (dm-5): writeback error on sector 6772520^M $2001517.913263$ $100$ XFS (dm-5): writeback error on sector 338549088^M $2001517.913392$ $100$ XFS (dm-5): writeback error on sector 110130576^M $2001517.913398$ $100$ XFS (dm-5): writeback error on sector 110130584^M $2001517.923828$ $101$ XFS (dm-5): metadata I/O error in "xlog_iodone" at daddr 0xc82f840 len 64 error 5^M $2001517.925072$ $101$ XFS (dm-5): Log I/O Error Detected. Shutting down filesystem^M $2001517.925237$ $103$ XFS (dm-6): metadata I/O error in "xlog_iodone" at daddr 0x1f44c278 len 64 error 5^M $2001517.925257$ $101$ XFS (dm-6): metadata I/O error in "xlog_iodone" at daddr 0x1f44c2b8 len 64 error 5^M $2001517.925419$ $103$ XFS (dm-6): Log I/O Error Detected. Shutting down filesystem^M $2001517.925421$ $103$ XFS (dm-6): Please umount the filesystem and rectify the problem(s)^M $2001517.925840$ $103$ XFS (dm-5): Please umount the filesystem and rectify the problem(s)^M $2001518.084461$ $59$ overlayfs: failed to get xattr trusted.overlay.redirect: err=-5)^M $2001518.134555$ $29$ overlayfs: failed to get xattr trusted.overlay.redirect: err=-5)^M $2001518.134679$ $8$ overlayfs: failed to get metacopy (-5)^M $2001518.135513$ $29$ overlayfs: failed to get xattr trusted.overlay.redirect: err=-5)^M $2001518.137716$ $17$ overlayfs: failed to get xattr trusted.overlay.redirect: err=-5)^M $2001518.138605$ $17$ overlayfs: failed to get xattr trusted.overlay.redirect: err=-5)^M $2001518.141203$ $71$ overlayfs: failed to get metacopy (-5)^M $2001518.143645$ $54$ overlayfs: failed to get metacopy (-5)^M $2001518.144968$ $95$ overlayfs: failed to get metacopy (-5)^M $2001518.145177$ $46$ overlayfs: failed to get xattr trusted.overlay.redirect: err=-5)^M $2001518.146489$ $46$ overlayfs: failed to get xattr trusted.overlay.redirect: err=-5)^M $2001518.148168$ $72$ overlayfs: failed to get metacopy (-5)^M $2001518.148170$ $34$ overlayfs: failed to get metacopy (-5)^M $2001518.148259$ $34$ overlayfs: failed to get metacopy (-5)^M $2001518.150749$ $96$ overlayfs: failed to get metacopy (-5)^M $2001518.152389$ $116$ overlayfs: failed to get metacopy (-5)^M $2001528.063127$ $97$ systemd-shutdown $1$ : Waiting for process: local-path-prov^M $2001600.084921$ $123$ kauditd_printk_skb: 23 callbacks suppressed^M $2001613.949707$ $99$ shutdown: 7 output lines suppressed due to ratelimiting^M $2001614.020310$ $100$ dracut Warning: Killing all remaining processes^M $2001614.364201$ $100$ dracut Warning: Unmounted /oldroot.^M $2001628.676636$ $0$ reboot: Restarting system^M |

查看tuned.log，出现了两次关机操作，应该是对应这两次IO error。

分析结果

第一次出现I/O错误时，日志中显示了 rejecting I/O to offline device的消息。这通常意味着设备已经离线，无法继续进行I/O操作，从而导致了I/O错误。由于这个错误的日志已经存在一段时间，问题应该已经得到解决。

第二次和第三次出现I/O错误时，伴随着关机相关的日志信息。具体来说，关机日志（shutdown 和 reboot）表明系统在I/O错误发生前，应该有执行关机操作。系统关机通常会触发文件系统的卸载和数据的回写，但由于多路径存储设备已经下线，XFS 无法完成日志写回和其他数据的持久化操作。日志中的XFS writeback error和metadata I/O error表明XFS文件系统在尝试进行数据写回时失败。尤其是在元数据操作过程中，XFS 在执行xlog_iodone操作时未能完成I/O请求，导致元数据无法成功写回。这些错误的原因是，XFS尝试将数据写入物理存储设备，但由于设备处于离线状态或无法访问，文件系统无法完成这些操作。