【银河麒麟高级服务器操作系统】服务器异常重启故障分析及处理建议

麒麟软件产品专区：https://product.kylinos.cn

开发者专区：https://developer.kylinos.cn

文档中心：https://document.kylinos.cn

服务器环境以及配置

|------|--------------|-----------------------------------------------------------------|
| 系统环境 | 物理机/虚拟机/云/容器 | 物理机 |
| 网络环境 | 外网/私有网络/无网络 | 私有网络 |
| 硬件环境 | 处理器： | Hygon C86 7390 32-core Processor |
| 硬件环境 | 内存： | 512 GiB |
| 硬件环境 | 机器型号 | New H3C Technologies Co., Ltd. UniServer R4930 G5 H3 PKG/62DB32 |
| 硬件环境 | 整机类型/架构： | x86 |
| 硬件环境 | BIOS版本： | HXYH031027-U01 08/03/2023 |
| 软件环境 | 具体操作系统版本 | 银河麒麟高级服务器操作系统 Kylin Linux Advanced Server release V10 (Sword) |
| 软件环境 | 内核版本 | 4.19.90-25.37.v2101.ky10.x86_64 |
| 软件环境 | 第三方模块 | scap(OE) |

现象描述

系统异常重启

现象分析

分析vmcore-dmesg.txt，"invalid opcode: 0000 $#1$ SMP NOPTI"，表明CPU在执行某条指令时遇到了无效的操作码(opcode)而触发异常,导致系统崩溃。invalid opcode表明CPU执行了无法识别的指令，通常这发生在软件问题或硬件失效上。

RIP(指令指针寄存器)显示异常发生在函数"bpf_prog_9500684e63fa4fd7+0x129d/0x3000"中的0x129d偏移处。结合Code段的反汇编代码"<4c> 09 fb 48 c1",可以看出正是这条指令出错。说明invalid opcode(无效的操作码)来自与函数bpf_prog_9500684e63fa4fd7，该函数来自于自定义bpf程序。

|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| $18594507.173491$ invalid opcode: 0000 $#1$ SMP NOPTI $18594507.174449$ CPU: 77 PID: 718772 Comm: java Kdump: loaded Tainted: G OE 4.19.90-25.37.v2101.ky10.x86_64 #1 $18594507.175720$ Hardware name: New H3C Technologies Co., Ltd. UniServer R4930 G5 H3 PKG/62DB32, BIOS HXYH031027-U01 08/03/2023 $18594507.176994$ RIP: 0010:bpf_prog_9500684e63fa4fd7+0x129d/0x3000 $18594507.177646$ Code: 16 4c 09 f6 48 c1 e6 10 48 09 d6 4d 0f b6 75 19 49 c1 e6 08 49 0f b6 55 18 49 09 d6 4d 0f b6 7d 1a 49 0f b6 5d 1b 48 c1 e3 08 <4c> 09 fb 48 c1 e3 10 4c 09 f3 48 c1 e3 20 48 09 f3 48 89 de 48 81 $18594507.178993$ RSP: 0018:ffffa05fc164fc70 EFLAGS: 00010256 $18594507.179927$ RAX: 000000000000000e RBX: 0000000000000000 RCX: 0000000000000000 $18594507.180621$ RDX: 0000000000000000 RSI: 000000000000001e RDI: 0000000000000000 $18594507.181361$ RBP: ffffa05fc164fe98 R08: 0000000000000000 R09: ffffffffc0d0a0a0 $18594507.182037$ R10: 0000000000000001 R11: 0000000000000000 R12: 0000000000000000 $18594507.182711$ R13: ffff8c91c3665920 R14: 0000000000000000 R15: 0000000000000000 $18594507.183378$ FS: 00007ef74cfff700(0000) GS:ffff8c42ffd40000(0000) knlGS:0000000000000000 $18594507.184049$ CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 $18594507.184717$ CR2: 00007f63c9f1d000 CR3: 0000006e3167c000 CR4: 00000000003406e0 $18594507.185387$ Call Trace: $18594507.186070$ ? tcp_recvmsg+0x33d/0xd30 $18594507.186737$ ? inet_recvmsg+0x5b/0xd0 $18594507.187398$ ? sock_read_iter+0x92/0xf0 $18594507.188051$ ? recalibrate_cpu_khz+0x10/0x10 $18594507.188700$ ? ktime_get_with_offset+0x59/0xc0 $18594507.189351$ ? bpf_trace_run2+0x2d/0x50 $18594507.189993$ ? syscall_trace_enter+0x2ad/0x2e0 $18594507.190627$ ? bpf_trace_run2+0x2d/0x50 $18594507.191324$ ? do_syscall_64+0x153/0x1d0 $18594507.191954$ ? entry_SYSCALL_64_after_hwframe+0x44/0xa9 $18594507.192578$ Modules linked in: nfsv3 nfs_acl rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache cdc_ether usbnet mii iptable_raw xt_CT macvlan scap(OE) ip6table_nat nf_nat_ipv6 ip6_tables xt_set ip_set_hash_ipportnet ip_set_bitmap_port ip_set_hash_ipportip ip_set_hash_ipport ip_set dummy iptable_mangle xt_comment xt_mark xt_conntrack ipt_MASQUERADE nf_conntrack_netlink nfnetlink xt_addrtype iptable_filter iptable_nat nf_nat_ipv4 nf_nat aufs overlay fuse br_netfilter bridge ip_vs_sh ip_vs_wrr ip_vs_rr ip_vs nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 udp_diag tcp_diag inet_diag 8021q garp mrp stp llc bonding rfkill sunrpc amd64_edac_mod edac_mce_amd vfat fat kvm_amd ccp kvm irqbypass ipmi_ssif joydev crct10dif_pclmul ses crc32_pclmul enclosure scsi_transport_sas ghash_clmulni_intel $18594507.197429$ pcspkr sg i2c_piix4 k10temp ipmi_si ipmi_devintf ipmi_msghandler acpi_cpufreq ip_tables xfs libcrc32c sd_mod ast i2c_algo_bit drm_kms_helper mlx5_core syscopyarea sysfillrect sysimgblt fb_sys_fops ttm crc32c_intel ahci libahci drm mlxfw libata devlink megaraid_sas dm_mirror dm_region_hash dm_log dm_mod |

以下是按调用顺序排列的堆栈跟踪分析，堆栈显示异常主要源于与网络相关的系统调用经过BPF程序处理时发生了问题。BPF程序在处理网络套接字数据时，可能因编写错误或与内核版本不匹配而执行了非法指令。

分析结果

堆栈显示异常主要源于与网络相关的系统调用经过BPF程序处理时发生了问题。问题的根源很可能在于BPF规则存在缺陷，生成了非法的指令序列。BPF规则通常由应用程序在用户态定义，然后传入内核执行，不当的BPF规则可能导致内核执行到非法指令。此外CPU硬件故障也可能导致指令码发生跳变。

后续计划与建议

建议针对该BPF程序进行代码审核与测试。