一次由隐藏大页配置引发的数据库OOM故障分析

一次由隐藏大页配置引发的数据库OOM故障分析

一、事故发生

在周日清晨，收到紧急短信告警，数据库实例发生异常重启。首先登录数据库服务器，查看日志记录

2025-12-21T06:54:57.259156+08:00 77 [Note] [MY-010914] [Server] Aborted connection 77 to db: 'unconnected' user: 'root' host: '172.17.139.203' (Got an error reading communication packets).
2025-12-21T06:55:33.224314Z mysqld_safe Number of processes running now: 0
2025-12-21T06:55:33.248143Z mysqld_safe mysqld restarted
2025-12-21T06:55:34.053462+08:00 0 [Warning] [MY-011069] [Server] The syntax '--replica-parallel-type' is deprecated and will be removed in a future release.
2025-12-21T06:55:34.053569+08:00 0 [Warning] [MY-011068] [Server] The syntax '--ssl=off' is deprecated and will be removed in a future release. Please use --tls-version='' instead.

通过该日志内容初步判断重启原因是发生了OOM异常，直接观察系统日志/var/log/messages，确认存在oom异常信息。

[root@gdb-adm ~]#  grep -inr /var/log/messages
5:Dec 21 06:55:33 gdb kernel: [419827.630493] crontab-1 invoked oom-killer: gfp_mask=0x6200ca(GFP_HIGHUSER_MOVABLE), order=0, oom_score_adj=0
11:Dec 21 06:55:33 gdb kernel: [419827.630530]  oom_kill_process+0x24f/0x270
12:Dec 21 06:55:33 gdb kernel: [419827.630532]  ? oom_badness+0x25/0x140
68:Dec 21 06:55:33 gdb kernel: [419827.630752] [  pid  ]   uid  tgid total_vm      rss pgtables_bytes swapents oom_score_adj name
148:Dec 21 06:55:33 gdb kernel: [419827.631062] oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0-1,global_oom,task_memcg=/user.slice/user-2036.slice/session-6188.scope,task=mysqld,pid=2567710,uid=2032

二、问题分析

1、内存设置检查

服务器物理内存376G，而innodb_buffer_pool_size设置为200G，占比为53%，符合预期。

free -h
              total        used        free      shared  buff/cache   available
Mem:          376Gi       267Gi        26Gi       5.0Mi        82Gi        53Gi

2、jemolloc判断

作为GreatSQL数据库或者开源MySQL数据库，出现OOM的情况，很大可能是由于使用默认的glibc内存分配管理，内存使用后释放不完全引起内存泄漏导致，通过命令lsof -p PID| grep jem 观察内存分配管理方式

[root@gdb ~]# lsof -p 25424 | grep jem
mysqld 25424 mysql  mem       REG                8,2    2136088   2355262 /data/svr/greatsql/lib/mysql/libjemalloc.so.1

从返回可以看出配置正常，基本上可以排除此原因。

3、OOM日志详细分析

1）完整OOM日志

Dec 21 06:55:33 gdb kernel: [419827.630493] crontab-1 invoked oom-killer: gfp_mask=0x6200ca(GFP_HIGHUSER_MOVABLE), order=0, oom_score_adj=0
Dec 21 06:55:33 gdb kernel: [419827.630499] CPU: 14 PID: 9458 Comm: crontab-1 Kdump: loaded Not tainted 4.19.90-2107.6.0.0227.28.oe1.bclinux.x86_64 #1
Dec 21 06:55:33 gdb kernel: [419827.630500] Hardware name: FiberHome FitServer/FiberHome Boards, BIOS 3.4.V7 02/01/2023
Dec 21 06:55:33 gdb kernel: [419827.630507] Call Trace:
Dec 21 06:55:33 gdb kernel: [419827.630519]  dump_stack+0x66/0x8b
Dec 21 06:55:33 gdb kernel: [419827.630527]  dump_header+0x4a/0x1fc
Dec 21 06:55:33 gdb kernel: [419827.630530]  oom_kill_process+0x24f/0x270
Dec 21 06:55:33 gdb kernel: [419827.630532]  ? oom_badness+0x25/0x140
Dec 21 06:55:33 gdb kernel: [419827.630533]  out_of_memory+0x11f/0x540
Dec 21 06:55:33 gdb kernel: [419827.630536]  __alloc_pages_slowpath+0x9f5/0xde0
Dec 21 06:55:33 gdb kernel: [419827.630543]  __alloc_pages_nodemask+0x2a8/0x2d0
Dec 21 06:55:33 gdb kernel: [419827.630549]  filemap_fault+0x35e/0x8a0
Dec 21 06:55:33 gdb kernel: [419827.630555]  ? alloc_set_pte+0x244/0x450
Dec 21 06:55:33 gdb kernel: [419827.630558]  ? filemap_map_pages+0x28f/0x480
Dec 21 06:55:33 gdb kernel: [419827.630584]  ext4_filemap_fault+0x2c/0x40 [ext4]
Dec 21 06:55:33 gdb kernel: [419827.630588]  __do_fault+0x33/0x110
Dec 21 06:55:33 gdb kernel: [419827.630592]  do_fault+0x12e/0x490
Dec 21 06:55:33 gdb kernel: [419827.630595]  ? __handle_mm_fault+0x2a/0x690
Dec 21 06:55:33 gdb kernel: [419827.630597]  __handle_mm_fault+0x613/0x690
Dec 21 06:55:33 gdb kernel: [419827.630601]  handle_mm_fault+0xc4/0x200
Dec 21 06:55:33 gdb kernel: [419827.630604]  __do_page_fault+0x2ba/0x4d0
Dec 21 06:55:33 gdb kernel: [419827.630609]  ? __audit_syscall_exit+0x238/0x2c0
Dec 21 06:55:33 gdb kernel: [419827.630611]  do_page_fault+0x31/0x130
Dec 21 06:55:33 gdb kernel: [419827.630616]  ? page_fault+0x8/0x30
Dec 21 06:55:33 gdb kernel: [419827.630620]  page_fault+0x1e/0x30
Dec 21 06:55:33 gdb kernel: [419827.630623] Mem-Info:
Dec 21 06:55:33 gdb kernel: [419827.630635] active_anon:50985791 inactive_anon:354 isolated_anon:0#012 active_file:677 inactive_file:0 isolated_file:0#012 unevictable:0 dirty:105 writeback:123 unstable:0#012 slab_reclaimable:20583 slab_unreclaimable:49628#012 m
apped:319 shmem:1323 pagetables:106803 bounce:0#012 free:5313776 free_pcp:5715 free_cma:0
Dec 21 06:55:33 gdb kernel: [419827.630638] Node 0 active_anon:100766572kB inactive_anon:556kB active_file:1384kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB mapped:76kB dirty:32kB writeback:0kB shmem:2276kB shmem_thp: 0kB shmem_pmdm
apped: 0kB anon_thp: 0kB writeback_tmp:0kB unstable:0kB all_unreclaimable? no
Dec 21 06:55:33 gdb kernel: [419827.630645] Node 1 active_anon:103176592kB inactive_anon:860kB active_file:1324kB inactive_file:80kB unevictable:0kB isolated(anon):0kB isolated(file):0kB mapped:1200kB dirty:388kB writeback:492kB shmem:3016kB shmem_thp: 0kB shme
m_pmdmapped: 0kB anon_thp: 0kB writeback_tmp:0kB unstable:0kB all_unreclaimable? no
Dec 21 06:55:33 gdb kernel: [419827.630650] Node 0 DMA free:15892kB min:824kB low:1028kB high:1232kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB writepending:0kB present:15976kB managed:15892kB mlocked:0kB kernel_stack:0k
B pagetables:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
Dec 21 06:55:33 gdb kernel: [419827.630654] lowmem_reserve[]: 0 1347 191666 191666 191666
Dec 21 06:55:33 gdb kernel: [419827.630661] Node 0 DMA32 free:833940kB min:72972kB low:91212kB high:109452kB active_anon:559420kB inactive_anon:8kB active_file:68kB inactive_file:0kB unevictable:0kB writepending:32kB present:1733384kB managed:1405672kB mlocked:
0kB kernel_stack:52kB pagetables:1084kB bounce:0kB free_pcp:400kB local_pcp:0kB free_cma:0kB
Dec 21 06:55:33 gdb kernel: [419827.630666] lowmem_reserve[]: 0 0 190319 190319 190319
Dec 21 06:55:33 gdb kernel: [419827.630672] Node 0 Normal free:10117540kB min:10117912kB low:12647388kB high:15176864kB active_anon:100207152kB inactive_anon:548kB active_file:808kB inactive_file:0kB unevictable:0kB writepending:0kB present:198180864kB managed:
194894048kB mlocked:0kB kernel_stack:13504kB pagetables:215840kB bounce:0kB free_pcp:536kB local_pcp:0kB free_cma:0kB
Dec 21 06:55:33 gdb kernel: [419827.630679] lowmem_reserve[]: 0 0 0 0 0
Dec 21 06:55:33 gdb kernel: [419827.630683] Node 1 Normal free:10287732kB min:10288284kB low:12860352kB high:15432420kB active_anon:103176592kB inactive_anon:860kB active_file:1324kB inactive_file:80kB unevictable:0kB writepending:880kB present:201326592kB mana
ged:198175752kB mlocked:0kB kernel_stack:11836kB pagetables:210288kB bounce:0kB free_pcp:21924kB local_pcp:332kB free_cma:0kB
Dec 21 06:55:33 gdb kernel: [419827.630686] lowmem_reserve[]: 0 0 0 0 0
Dec 21 06:55:33 gdb kernel: [419827.630688] Node 0 DMA: 1*4kB (U) 0*8kB 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15892kB
Dec 21 06:55:33 gdb kernel: [419827.630694] Node 0 DMA32: 240*4kB (UME) 178*8kB (UME) 140*16kB (UME) 66*32kB (UME) 70*64kB (UME) 53*128kB (UME) 38*256kB (UME) 18*512kB (UE) 3*1024kB (U) 2*2048kB (UE) 193*4096kB (M) = 834640kB
Dec 21 06:55:33 gdb kernel: [419827.630702] Node 0 Normal: 3557*4kB (UE) 1963*8kB (UME) 651*16kB (UME) 1139*32kB (UME) 855*64kB (UME) 572*128kB (UME) 308*256kB (UE) 129*512kB (UME) 50*1024kB (UME) 27*2048kB (UME) 2359*4096kB (UME) = 10118588kB
Dec 21 06:55:33 gdb kernel: [419827.630712] Node 1 Normal: 3636*4kB (UME) 1848*8kB (UME) 2744*16kB (UME) 2139*32kB (UME) 1580*64kB (UME) 1073*128kB (UME) 613*256kB (UME) 280*512kB (UE) 130*1024kB (UE) 81*2048kB (UE) 2273*4096kB (UME) = 10289648kB
Dec 21 06:55:33 gdb kernel: [419827.630731] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB
Dec 21 06:55:33 gdb kernel: [419827.630737] Node 0 hugepages_total=40960 hugepages_free=40960 hugepages_surp=0 hugepages_size=2048kB

Dec 21 06:55:33 gdb kernel: [419827.630738] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB
Dec 21 06:55:33 gdb kernel: [419827.630741] Node 1 hugepages_total=40960 hugepages_free=40960 hugepages_surp=0 hugepages_size=2048kB
Dec 21 06:55:33 gdb kernel: [419827.630742] 3360 total pagecache pages
Dec 21 06:55:33 gdb kernel: [419827.630744] 0 pages in swap cache
Dec 21 06:55:33 gdb kernel: [419827.630746] Swap cache stats: add 0, delete 0, find 0/0
Dec 21 06:55:33 gdb kernel: [419827.630746] Free swap  = 0kB
Dec 21 06:55:33 gdb kernel: [419827.630747] Total swap = 0kB
Dec 21 06:55:33 gdb kernel: [419827.630748] 100314204 pages RAM
Dec 21 06:55:33 gdb kernel: [419827.630749] 0 pages HighMem/MovableOnly
Dec 21 06:55:33 gdb kernel: [419827.630749] 1691363 pages reserved
Dec 21 06:55:33 gdb kernel: [419827.630750] 0 pages hwpoisoned
Dec 21 06:55:33 gdb kernel: [419827.630750] Tasks state (memory values in pages):
Dec 21 06:55:33 gdb kernel: [419827.630752] [  pid  ]   uid  tgid total_vm      rss pgtables_bytes swapents oom_score_adj name
Dec 21 06:55:33 gdb kernel: [419827.630790] [    926]     0   926    72470      811   507904        0          -250 systemd-journal
Dec 21 06:55:33 gdb kernel: [419827.630794] [    960]     0   960     8269     1075    77824        0         -1000 systemd-udevd
Dec 21 06:55:33 gdb kernel: [419827.630798] [   1623]     0  1623      729       28    32768        0             0 mdadm
Dec 21 06:55:33 gdb kernel: [419827.630800] [   1672]     0  1672    23007      217    49152        0         -1000 auditd
Dec 21 06:55:33 gdb kernel: [419827.630803] [   1674]     0  1674     1568       90    36864        0             0 sedispatch
Dec 21 06:55:33 gdb kernel: [419827.630806] [   1712]     0  1712    78709      787    98304        0             0 ModemManager
Dec 21 06:55:33 gdb kernel: [419827.630808] [   1714]     0  1714      571       16    32768        0             0 acpid
Dec 21 06:55:33 gdb kernel: [419827.630811] [   1719]    81  1719     2891      845    49152        0          -900 dbus-daemon
Dec 21 06:55:33 gdb kernel: [419827.630813] [   1727]   992  1727      599       38    32768        0             0 lsmd
Dec 21 06:55:33 gdb kernel: [419827.630815] [   1730]     0  1730      619       33    32768        0             0 mcelog
Dec 21 06:55:33 gdb kernel: [419827.630817] [   1735]   999  1735   743772     1030   229376        0             0 polkitd
Dec 21 06:55:33 gdb kernel: [419827.630820] [   1736]     0  1736    77985      204    90112        0             0 rngd
Dec 21 06:55:33 gdb kernel: [419827.630827] [   1739]     0  1739     2711      421    49152        0             0 smartd
Dec 21 06:55:33 gdb kernel: [419827.630829] [   1741]     0  1741    20070      151    40960        0          -500 irqbalance
Dec 21 06:55:33 gdb kernel: [419827.630831] [   1743]     0  1743     4492      227    61440        0             0 systemd-machine
Dec 21 06:55:33 gdb kernel: [419827.630837] [   1753]     0  1753   114058      472   110592        0             0 abrtd
Dec 21 06:55:33 gdb kernel: [419827.630842] [   1794]     0  1794     4780      468    65536        0             0 systemd-logind
Dec 21 06:55:33 gdb kernel: [419827.630844] [   1830]     0  1830   263593      479   929792        0             0 abrt-dump-journ
Dec 21 06:55:33 gdb kernel: [419827.630846] [   1831]     0  1831   261511      460   925696        0             0 abrt-dump-journ
Dec 21 06:55:33 gdb kernel: [419827.630850] [   2802]     0  2802   199635      606   299008        0             0 esfdaemon
Dec 21 06:55:33 gdb kernel: [419827.630852] [   2803]     0  2803    72799    12101   200704        0             0 bare-agent
Dec 21 06:55:33 gdb kernel: [419827.630855] [   2805]     0  2805    59117      340    86016        0             0 cupsd
Dec 21 06:55:33 gdb kernel: [419827.630856] [   2810]     0  2810   251667      734  1376256        0             0 rsyslogd
Dec 21 06:55:33 gdb kernel: [419827.630863] [   2814]     0  2814     3350      227    53248        0         -1000 sshd
Dec 21 06:55:33 gdb kernel: [419827.630865] [   2815]     0  2815   117707     3324   143360        0             0 tuned
Dec 21 06:55:33 gdb kernel: [419827.630869] [   2828]     0  2828    65710      188    73728        0             0 gssproxy
Dec 21 06:55:33 gdb kernel: [419827.630872] [   2848]     0  2848    53496       92    45056        0             0 init.ohasd
Dec 21 06:55:33 gdb kernel: [419827.630874] [   2890]     0  2890      906       48    32768        0             0 atd
Dec 21 06:55:33 gdb kernel: [419827.630875] [   2896]     0  2896    53748      118    49152        0             0 crond
Dec 21 06:55:33 gdb kernel: [419827.630878] [   3692]     0  3692     3539      148    49152        0             0 xinetd
Dec 21 06:55:33 gdb kernel: [419827.630880] [   3978]     0  3978    10985      242    61440        0             0 master
Dec 21 06:55:33 gdb kernel: [419827.630884] [   4004]    89  4004    11331      527    69632        0             0 qmgr
Dec 21 06:55:33 gdb kernel: [419827.630888] [   4093]     0  4093    43766      216   221184        0             0 sddog
Dec 21 06:55:33 gdb kernel: [419827.630890] [   4112]     0  4112   285705      537   577536        0             0 sdmonitor
Dec 21 06:55:33 gdb kernel: [419827.630891] [   4233]     0  4233   134053      596   466944        0             0 sdcc
Dec 21 06:55:33 gdb kernel: [419827.630895] [   4259]     0  4259   168947     8371   667648        0             0 sdec
Dec 21 06:55:33 gdb kernel: [419827.630897] [   4284]     0  4284   286675     1588   778240        0             0 sdexam
Dec 21 06:55:33 gdb kernel: [419827.630899] [   4310]     0  4310   492216    50216  1331200        0             0 sdsvrd
Dec 21 06:55:33 gdb kernel: [419827.630906] [   4330]     0  4330    29248      278   278528        0             0 udcenter
Dec 21 06:55:33 gdb kernel: [419827.630908] [   8353]     0  8353     2184      321    45056        0             0 dhclient
Dec 21 06:55:33 gdb kernel: [419827.630910] [   9243]  1086  9243     5274      639    73728        0             0 systemd
Dec 21 06:55:33 gdb kernel: [419827.630915] [   9245]  1086  9245     6383     1015    73728        0             0 (sd-pam)
Dec 21 06:55:33 gdb kernel: [419827.630918] [   9348]  1086  9348   470112    50291   761856        0             0 java
Dec 21 06:55:33 gdb kernel: [419827.630920] [   9426]     0  9426     2184      323    45056        0             0 dhclient
Dec 21 06:55:33 gdb kernel: [419827.630922] [   9852]     0  9852    53214       26    36864        0             0 agetty
Dec 21 06:55:33 gdb kernel: [419827.630926] [  11463]  1002 11463     5276      639    73728        0             0 systemd
Dec 21 06:55:33 gdb kernel: [419827.630936] [  11465]  1002 11465     6383     1016    73728        0             0 (sd-pam)
Dec 21 06:55:33 gdb kernel: [419827.630942] [  11611]  1002 11611 14284908     1404   602112        0             0 agent60
Dec 21 06:55:33 gdb kernel: [419827.630945] [ 137615]     0 137615   136163     3215   147456        0             0 lvmdbusd
Dec 21 06:55:33 gdb kernel: [419827.630950] [ 796407]  2036 796407     5301      649    73728        0             0 systemd
Dec 21 06:55:33 gdb kernel: [419827.630952] [ 796409]  2036 796409    43812     1109    94208        0             0 (sd-pam)
Dec 21 06:55:33 gdb kernel: [419827.630954] [ 817343]  2032 817343    53508      130    53248        0             0 mysqld_safe
Dec 21 06:55:33 gdb kernel: [419827.630956] [2270020]  2032 2270020  2778466     1788  1466368        0             0 dbinit
Dec 21 06:55:33 gdb kernel: [419827.630958] [2567710]  2032 2567710 77307141 50817311 424357888        0             0 mysqld
Dec 21 06:55:33 gdb kernel: [419827.630960] [3453494]   998 3453494     1173       50    36864        0             0 chronyd
Dec 21 06:55:33 gdb kernel: [419827.630963] [3621338]    89 3621338    11065      249    65536        0             0 pickup
Dec 21 06:55:33 gdb kernel: [419827.630981] [3662845]     0 3662845     5297      648    73728        0             0 systemd
Dec 21 06:55:33 gdb kernel: [419827.630983] [3662881]     0 3662881    44244     1356    98304        0             0 (sd-pam)
Dec 21 06:55:33 gdb kernel: [419827.630985] [3662906]    89 3662906    11068      242    65536        0             0 trivial-rewrite
Dec 21 06:55:33 gdb kernel: [419827.630987] [3663080]     0 3663080    10991      235    65536        0             0 local
Dec 21 06:55:33 gdb kernel: [419827.630988] [3663097]    89 3663097    11131      254    65536        0             0 smtp
Dec 21 06:55:33 gdb kernel: [419827.630990] [3663098]     0 3663098    10991      235    65536        0             0 local
Dec 21 06:55:33 gdb kernel: [419827.630992] [3663108]    89 3663108    11073      242    65536        0             0 bounce
Dec 21 06:55:33 gdb kernel: [419827.630994] [3663141]     0 3663141    10991      235    65536        0             0 local
Dec 21 06:55:33 gdb kernel: [419827.630997] [3663177]    89 3663177    11066      242    69632        0             0 flush
Dec 21 06:55:33 gdb kernel: [419827.631003] [3663193]    89 3663193    11066      242    69632        0             0 flush
Dec 21 06:55:33 gdb kernel: [419827.631005] [3663201]    89 3663201    11066      242    69632        0             0 flush
Dec 21 06:55:33 gdb kernel: [419827.631007] [3663207]     0 3663207    53463       54    45056        0             0 sh
Dec 21 06:55:33 gdb kernel: [419827.631011] [3663208]     0 3663208   884643     7048   589824        0             0 promtail
Dec 21 06:55:33 gdb kernel: [419827.631019] [3663317]    89 3663317    11131      254    65536        0             0 smtp
Dec 21 06:55:33 gdb kernel: [419827.631023] [3663318]    89 3663318    11131      254    65536        0             0 smtp
Dec 21 06:55:33 gdb kernel: [419827.631025] [3663319]    89 3663319    11131      254    65536        0             0 smtp
Dec 21 06:55:33 gdb kernel: [419827.631026] [3663320]    89 3663320    11131      254    65536        0             0 smtp
Dec 21 06:55:33 gdb kernel: [419827.631028] [3663321]    89 3663321    11064      242    65536        0             0 error
Dec 21 06:55:33 gdb kernel: [419827.631030] [3663322]    89 3663322    11064      242    65536        0             0 error
Dec 21 06:55:33 gdb kernel: [419827.631032] [3663388]     0 3663388    53093       15    40960        0             0 sleep
Dec 21 06:55:33 gdb kernel: [419827.631048] [3663946]     0 3663946     4458       86    61440        0             0 systemd-cgroups
Dec 21 06:55:33 gdb kernel: [419827.631060] [3663947]     0 3663947     4071       84    57344        0             0 systemd-cgroups
Dec 21 06:55:33 gdb kernel: [419827.631062] oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0-1,global_oom,task_memcg=/user.slice/user-2036.slice/session-6188.scope,task=mysqld,pid=2567710,uid=2032
Dec 21 06:55:33 gdb kernel: [419827.631071] Out of memory: Kill process 2567710 (mysqld) score 516 or sacrifice child
Dec 21 06:55:33 gdb kernel: [419827.632542] Killed process 2567710 (mysqld) total-vm:309228564kB, anon-rss:203269244kB, file-rss:0kB, shmem-rss:0kB

2）发生现象

Dec 21 06:55:33 gdb kernel: [419827.630493] crontab-1 invoked oom-killer: gfp_mask=0x6200ca(GFP_HIGHUSER_MOVABLE), order=0, oom_score_adj=0
Dec 21 06:55:33 gdb kernel: [419827.632542] Killed process 2567710 (mysqld) total-vm:309228564kB, anon-rss:203269244kB, file-rss:0kB, shmem-rss:0kB

上述关键信息为进程crontab-1申请新的内存引起oom-killer，而被kill进程为mysqld占用内存大小203269244kB

3) NUMA占用分析

Dec 21 06:55:33 gdb kernel: [419827.630672] Node 0 Normal free:10117540kB min:10117912kB low:12647388kB high:15176864kB active_anon:100207152kB inactive_anon:548kB active_file:808kB inactive_file:0kB unevictable:0kB writepending:0kB present:198180864kB managed:
194894048kB mlocked:0kB kernel_stack:13504kB pagetables:215840kB bounce:0kB free_pcp:536kB local_pcp:0kB free_cma:0kB
Dec 21 06:55:33 gdb kernel: [419827.630679] lowmem_reserve[]: 0 0 0 0 0
Dec 21 06:55:33 gdb kernel: [419827.630683] Node 1 Normal free:10287732kB min:10288284kB low:12860352kB high:15432420kB active_anon:103176592kB inactive_anon:860kB active_file:1324kB inactive_file:80kB unevictable:0kB writepending:880kB present:201326592kB mana
ged:198175752kB mlocked:0kB kernel_stack:11836kB pagetables:210288kB bounce:0kB free_pcp:21924kB local_pcp:332kB free_cma:0kB

从上述日志，可以看出两个numa node的剩余free内存均低于了min的要求内存。

4) 内存占用统计

根据OOM记录的日志信息，内存大概有如下分配(注意，系统日志中rss列的单位为页，默认4k大小)

进程	占用内存
mysqld	193G
其他进程	641M
NUMA剩余	19.5G

上述内存远低于操作系统内存376G，缺失近163G

5) 大页分析

继续查看系统日志

Dec 21 06:55:33 gdb kernel: [419827.630731] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB
Dec 21 06:55:33 gdb kernel: [419827.630737] Node 0 hugepages_total=40960 hugepages_free=40960 hugepages_surp=0 hugepages_size=2048kB
Dec 21 06:55:33 gdb kernel: [419827.630738] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB
Dec 21 06:55:33 gdb kernel: [419827.630741] Node 1 hugepages_total=40960 hugepages_free=40960 hugepages_surp=0 hugepages_size=2048kB

解析为

	页类型	总页数量	空闲页
numanode0	2M	40960	40960
numanode0	1G	0	0
numanode1	2M	40960	40960
numanode1	1G	0	0

可见大页占用了2M x 40960 x 2=160G内存，并且没有被使用，刚好和内存统计相近

4、大页配置查看

1) 检查透明大页配置

cat /sys/kernel/mm/transparent_hugepage/enabled，确认是关闭状态

[root@gdb ~]#  cat /sys/kernel/mm/transparent_hugepage/enabled
always madvise [never] 

2) 检查传统大页配置

sysctl -p | grep vm ，可见并没有相关配置

[root@gdb ~]#  sysctl -p | grep vm
vm.zone_reclaim_mode=0
vm.swappiness=1
vm.min_free_kbytes=20480000

3) 大页特性对比

特性维度	传统大页	透明大页
检查方式	/etc/sysctl.conf 中的 vm.nr_hugepages	/sys/kernel/mm/transparent_hugepage/enabled
管理机制	静态预分配。在系统启动或配置后，内核立即从物理内存中划出指定数量的大页。这部分内存被"锁定"，专用于大页，不能被挪作他用（如进程的普通小页）。	动态分配。内核在运行时根据内存访问模式（如连续的512个4K页被频繁访问），自动将小页合并成一个大页，或者在不再需要时拆分回小页。这是一个"按需"的过程。
配置方式	1. 临时：sysctl -w vm.nr_hugepages=N 2. 永久：在 /etc/sysctl.conf 中添加 vm.nr_hugepages=N，重启或执行 sysctl -p 生效。	1. 临时：echo <mode> > /sys/kernel/mm/transparent_hugepage/enabled 2. 永久：通过内核启动参数 vi /etc/default/grub 在 GRUB_CMDLINE_LINUX变量中添加transparent_hugepage=always，重新生成GRUB配置grub2-mkconfig -o /boot/grub2/grub.cfg
内存使用	专用且独占。分配后即使不使用，也会一直占用物理内存，可能导致内存浪费。	共享池。使用普通的内存页池，只在需要时才转换，内存利用率更高。
性能特点	性能稳定可预测。应用程序（如Oracle DB, Redis）通过mmap()或shmget()显式请求大页时，能100%保证使用大页，无缺页中断或合并操作开销，性能最优、最稳定。	性能有波动风险。虽然大多数情况下能提升性能（减少TLB Miss），但在内存压力大或碎片化时，内核的合并/拆分操作（khugepaged进程）会带来不可预测的延迟尖峰，对延迟敏感型应用不利。

根据故障现象及大页特点，猜测应该是由于配置了传统大页，锁定了160G内存无法被其他进程使用，但是配置文件中并没有该配置，现象很奇怪

4) 深度搜索

使用命令grep -R "nr_hugepages" /etc进行大范围深度搜索，发现了问题所在

[root@gdb ~]#  grep -R "nr_hugepages" /etc
/etc/sysctl.conf.bak-2025-07-13:vm.nr_hugepages=81920

可以看到配置文件在7月13日进行了备份调整，备份前确实是有传统大页配置，并且配置值和目前系统日志中记录值相同。

5) 配置变更测试

通过测试发现，即使配置文件中去传统大页设置，但是依然是存在大页设置的

[root@qdb -]# cat /etc/sysctl.conf | grep h
kernel.shmall = 41943040
kernel.shmmax = 171798691840
kernel.shmmni=4096
#vm.hugetlb_shm_group=54321
#vm.nr_hugepages = 40960
[root@qdb -]# sysctl -p | grep h
kernel.shmall = 41943040
kernel.shmmax = 171798691840
kernel.shmmi=4096
[root@qdb -]# cat /proc/sys/vm/nr_hugepages
40960

调整配置后如果不重启操作系统，需要手动释放该部分内存

[root@gdb ~]# echo 0 > /proc/sys/vm/nr_hugepages
[root@gdb ~]# cat /proc/sys/vm/nr_hugepages
0

三、原因总结改进

1) 根本原因

大量 HugePages 被预留但数据库未实际使用，导致普通内存不足，引发 OOM

2) 不正常的默认大页配置

在操作系统默认情况下，未配置nr_hugepages，因此最初分析时未考虑传统大页方向。后经数据对比，发现传统大页存在内存占用异常现象。经后续核实，由于该服务器为利旧使用，残留了Oracle相关配置，导致该隐藏问题未被及时发现，又是一个国产化过程的小坑。

3) 后续改进

在基于现有服务器初始化步骤中，增加传统大页的检查设置步骤

sed -i '/huge/d' /etc/sysctl.conf
sysctl -p | grep huge
echo 0 > /proc/sys/vm/nr_hugepages