Pod 重启多场景复盘：OOM、日志阻塞、并发 map 读写导致程序崩溃

ezreal_pan2026-09-12 19:02

背景

近期业务Pod多次发生重启现象，经排查发现不同时间点的Pod重启根因完全不同，共计三类典型故障：

  1. 内核OOM Kill：容器内存使用超过cgroup limit，被系统OOM杀手直接杀掉；

  2. 日志通道阻塞：日志同步写阻塞导致HTTP协程无法释放，协程堆积、内存持续上涨，最终应该也有可能导致OOM Kill，只是发现的早，手动重启了；

  3. Go程序panic崩溃：业务存在concurrent map read and map write并发map读写问题，进程主动退出，触发Pod重启。

本次复盘保留完整原始排查命令、日志样本、命令解释、根因分析，并补充标准化故障排查 Checklist，形成完整闭环复盘文档。

一、第一次故障：OOM Kill，Pod被内核杀死

排查命令

bash 复制代码 
dmesg -T | tail -50

命令解释

dmesg 用于读取内核环形缓冲区日志；-T 将内核时间戳转换为可读本地时间；tail -50 展示末尾50行。 主要用于排查主机内核级事件：OOM杀进程、虚拟设备异常、网卡故障、硬件报错等。

日志片段

bash 复制代码 
[Thu Aug 20 12:49:40 2026] RDX: 0000000000000001 RSI: 0000800000000000 RDI: 000000c0ed0e7000
[Thu Aug 20 12:49:40 2026] RBP: 000000c05a5612d8 R08: 0000000000000001 R09: 000000c0ed0e6000
[Thu Aug 20 12:49:40 2026] R10: 0000000000000873 R11: 0000000000000000 R12: 0000000000000001
[Thu Aug 20 12:49:40 2026] R13: 3fffffffffffffff R14: 000000c0883e2d00 R15: 0000000000000000
[Thu Aug 20 12:49:40 2026] memory: usage 4194304kB, limit 4194304kB, failcnt 1274
[Thu Aug 20 12:49:40 2026] memory+swap: usage 0kB, limit 9007199254740988kB, failcnt 0
[Thu Aug 20 12:49:40 2026] kmem: usage 91860kB, limit 9007199254740988kB, failcnt 0
[Thu Aug 20 12:49:40 2026] Memory cgroup stats for /kubepods/burstable/pod6b0a1d95-60de-4804-a0bd-44ba20292ab0:
[Thu Aug 20 12:49:40 2026] anon 4200194048
                           file 192512
                           kernel_stack 221184
                           slab 83238912
                           sock 0
                           shmem 0
                           file_mapped 0
                           file_dirty 0
                           file_writeback 0
                           anon_thp 10485760
                           inactive_anon 0
                           active_anon 4200161280
                           inactive_file 192512
                           active_file 286720
                           unevictable 0
                           slab_reclaimable 55136256
                           slab_unreclaimable 28102656
                           pgfault 1263669
                           pgmajfault 0
                           workingset_refault 0
                           workingset_activate 0
                           workingset_nodereclaim 0
                           pgrefill 12096
                           pgscan 17449
                           pgsteal 4380
                           pgactivate 12111
                           pgdeactivate 12096
                           pglazyfree 0
                           pglazyfreed 0
                           thp_fault_alloc 0
                           thp_collapse_alloc 0
[Thu Aug 20 12:49:40 2026] Tasks state (memory values in pages):
[Thu Aug 20 12:49:40 2026] [  pid  ]   uid  tgid total_vm      rss pgtables_bytes swapents oom_score_adj name
[Thu Aug 20 12:49:40 2026] [ 117418] 65535 117418      241        1    28672        0          -998 pause
[Thu Aug 20 12:49:40 2026] [ 117557]     0 117557  1214228  1036362  8458240        0           985 main
[Thu Aug 20 12:49:40 2026] oom-kill:constraint=CONSTRAINT_MEMCG,nodemask=(null),cpuset=7e85d2511249572630dfcd23f9b6f50d620afb8539e945ddaef1395d32ef9a00,mems_allowed=0,oom_memcg=/kubepods/burstable/pod6b0a1d95-60de-4804-a0bd-44ba20292ab0,task_memcg=/kubepods/burstable/pod6b0a1d95-60de-4804-a0bd-44ba20292ab0/7e85d2511249572630dfcd23f9b6f50d620afb8539e945ddaef1395d32ef9a00,task=main,pid=117557,uid=0
[Thu Aug 20 12:49:40 2026] Memory cgroup out of memory: Killed process 117557 (main) total-vm:4856912kB, anon-rss:4100144kB, file-rss:45308kB, shmem-rss:0kB, UID:0 pgt

现象与根因

日志关键字 oom-kill 明确标识：容器内存用量打满cgroup限制，被内核OOM杀手直接杀死主进程，触发Pod重启。

现象特征：

  • 集群Pod内存异常飙升；

  • 大概率为内存泄漏 或瞬时大任务打爆内存；

  • 历史同类问题为WebSocket出现panic引起连接未释放导致内存持续上涨，也存在批量导出、大数据计算瞬时打满内存的场景。

二、第二次故障：虚拟设备Dead loop、日志同步写阻塞、协程堆积重启

排查命令

bash 复制代码 
dmesg -T | tail -50

命令解释同上。

日志片段

bash 复制代码 
root@scrm-09561:~# dmesg -T | tail -50
[Thu Sep 10 10:51:31 2026] Dead loop on virtual device gw_11cbf51a, fix it urgently!
[Thu Sep 10 10:51:32 2026] Dead loop on virtual device gw_11cbf51a, fix it urgently!
[Thu Sep 10 10:51:32 2026] Dead loop on virtual device gw_11cbf51a, fix it urgently!
[Thu Sep 10 10:51:34 2026] Dead loop on virtual device gw_11cbf51a, fix it urgently!
[Thu Sep 10 10:51:37 2026] Dead loop on virtual device gw_11cbf51a, fix it urgently!
[Thu Sep 10 10:51:38 2026] Dead loop on virtual device gw_11cbf51a, fix it urgently!
[Thu Sep 10 10:51:44 2026] Dead loop on virtual device gw_11cbf51a, fix it urgently!
[Thu Sep 10 10:51:50 2026] Dead loop on virtual device gw_11cbf51a, fix it urgently!
[Thu Sep 10 10:51:58 2026] Dead loop on virtual device gw_11cbf51a, fix it urgently!
[Thu Sep 10 10:52:18 2026] Dead loop on virtual device gw_11cbf51a, fix it urgently!
[Thu Sep 10 10:52:23 2026] Dropped outbound packet type=806
[Thu Sep 10 10:52:24 2026] Dropped outbound packet type=806
[Thu Sep 10 10:52:25 2026] Dropped outbound packet type=806
[Thu Sep 10 11:19:31 2026] [ipvlan]: create ipvlan dev[ipv-85df6672], mode[0]
[Thu Sep 10 11:19:31 2026] eth0: renamed from ipv-85df6672
[Thu Sep 10 11:19:31 2026] [ipvlan] Add IPv4=10.0.0.26 addr
[Thu Sep 10 11:19:34 2026] [ipvlan] Delete IPv4=10.0.0.25 addr
[Thu Sep 10 11:19:34 2026] Dead loop on virtual device gw_11cbf51a, fix it urgently!
[Thu Sep 10 11:19:34 2026] Dead loop on virtual device gw_11cbf51a, fix it urgently!
[Thu Sep 10 11:19:35 2026] Dead loop on virtual device gw_11cbf51a, fix it urgently!
[Thu Sep 10 11:19:35 2026] [ipvlan]: create ipvlan dev[ipv-63e8d174], mode[0]
[Thu Sep 10 11:19:35 2026] eth0: renamed from ipv-63e8d174
[Thu Sep 10 11:19:35 2026] [ipvlan] Add IPv4=10.0.0.27 addr
[Thu Sep 10 11:19:36 2026] Dead loop on virtual device gw_11cbf51a, fix it urgently!
[Thu Sep 10 11:19:37 2026] Dead loop on virtual device gw_11cbf51a, fix it urgently!
[Thu Sep 10 11:19:41 2026] Dead loop on virtual device gw_11cbf51a, fix it urgently!
[Thu Sep 10 11:19:45 2026] [ipvlan] Delete IPv4=10.0.0.24 addr
[Thu Sep 10 11:19:45 2026] Dead loop on virtual device gw_11cbf51a, fix it urgently!
[Thu Sep 10 11:19:46 2026] Dead loop on virtual device gw_11cbf51a, fix it urgently!
[Thu Sep 10 11:19:47 2026] Dead loop on virtual device gw_11cbf51a, fix it urgently!
[Thu Sep 10 11:19:47 2026] Dead loop on virtual device gw_11cbf51a, fix it urgently!
[Thu Sep 10 11:19:49 2026] Dead loop on virtual device gw_11cbf51a, fix it urgently!
[Thu Sep 10 11:20:00 2026] Dead loop on virtual device gw_11cbf51a, fix it urgently!
[Thu Sep 10 11:20:12 2026] Dead loop on virtual device gw_11cbf51a, fix it urgently!
[Thu Sep 10 11:20:27 2026] Dead loop on virtual device gw_11cbf51a, fix it urgently!
[Thu Sep 10 11:20:32 2026] Dropped outbound packet type=806
[Thu Sep 10 11:20:33 2026] Dropped outbound packet type=806
[Thu Sep 10 11:20:34 2026] Dropped outbound packet type=806
[Thu Sep 10 11:20:39 2026] Dead loop on virtual device gw_11cbf51a, fix it urgently!
[Thu Sep 10 11:20:45 2026] Dropped outbound packet type=806
[Thu Sep 10 11:20:46 2026] Dropped outbound packet type=806
[Thu Sep 10 11:20:47 2026] Dropped outbound packet type=806
[Thu Sep 10 13:34:02 2026] [ipvlan]: create ipvlan dev[ipv-4e7dc8c6], mode[0]
[Thu Sep 10 13:34:02 2026] eth0: renamed from ipv-4e7dc8c6
[Thu Sep 10 13:34:02 2026] [ipvlan] Add IPv4=10.0.0.28 addr
[Thu Sep 10 13:34:04 2026] [ipvlan] Delete IPv4=10.0.0.19 addr
[Thu Sep 10 13:56:25 2026] [ipvlan]: create ipvlan dev[ipv-ad32479f], mode[0]
[Thu Sep 10 13:56:25 2026] eth0: renamed from ipv-ad32479f
[Thu Sep 10 13:56:25 2026] [ipvlan] Add IPv4=10.0.0.29 addr
[Thu Sep 10 13:56:28 2026] [ipvlan] Delete IPv4=10.0.0.28 addr

pofile文件

bash 复制代码 
goroutine profile: total 123148
121759 @ 0x4387f6 0x4496cc 0x4496a6 0x4655a5 0x4d54f2 0x4d7af2 0x4d7b03 0x4e0725 0x4e071b 0x91f8a3 0x911315 0x9139f9 0x92ff39 0x949656 0x94960f 0x23ceeb9 0x8dd3a6 0x8dd092 0x8dcd2e 0x6b261b 0x6adcc8 0x4695c1
#        0x4655a4        internal/poll.runtime_Semacquire+0x24                                /usr/local/go/src/runtime/sema.go:61
#        0x4d54f1        internal/poll.(*fdMutex).rwlock+0xd1                                /usr/local/go/src/internal/poll/fd_mutex.go:154
#        0x4d7af1        internal/poll.(*FD).writeLock+0x71                                /usr/local/go/src/internal/poll/fd_mutex.go:239
#        0x4d7b02        internal/poll.(*FD).Write+0x82                                        /usr/local/go/src/internal/poll/fd_unix.go:262
#        0x4e0724        os.(*File).write+0x64                                                /usr/local/go/src/os/file_posix.go:49
#        0x4e071a        os.(*File).Write+0x5a                                                /usr/local/go/src/os/file.go:176
#        0x91f8a2        go.uber.org/zap/zapcore.multiWriteSyncer.Write+0xc2                /go/pkg/mod/go.uber.org/zap@v1.22.0/zapcore/write_syncer.go:105
#        0x911314        go.uber.org/zap/zapcore.(*ioCore).Write+0xb4                        /go/pkg/mod/go.uber.org/zap@v1.22.0/zapcore/core.go:90
#        0x9139f8        go.uber.org/zap/zapcore.(*CheckedEntry).Write+0x1d8                /go/pkg/mod/go.uber.org/zap@v1.22.0/zapcore/entry.go:255
#        0x92ff38        go.uber.org/zap.(*Logger).Info+0x58                                /go/pkg/mod/go.uber.org/zap@v1.22.0/logger.go:213
#        0x949655        whgo_xjd/library/log.doLog+0x95                                        /app/library/log/log.go:285
#        0x94960e        whgo_xjd/library/log.InfoWithCtx+0x4e                                /app/library/log/log.go:211
#        0x23ceeb8        whgo_xjd/service/middleware_service.AddTraceId+0x5d8                /app/service/middleware_service/common_middleware.go:46
#        0x8dd3a5        github.com/gin-gonic/gin.(*Context).Next+0x4e5                        /go/pkg/mod/github.com/gin-gonic/gin@v1.4.0/context.go:124
#        0x8dd091        github.com/gin-gonic/gin.(*Engine).handleHTTPRequest+0x1d1        /go/pkg/mod/github.com/gin-gonic/gin@v1.4.0/gin.go:389
#        0x8dcd2d        github.com/gin-gonic/gin.(*Engine).ServeHTTP+0x16d                /go/pkg/mod/github.com/gin-gonic/gin@v1.4.0/gin.go:351
#        0x6b261a        net/http.serverHandler.ServeHTTP+0x43a                                /usr/local/go/src/net/http/server.go:2879
#        0x6adcc7        net/http.(*conn).serve+0xb07                                        /usr/local/go/src/net/http/server.go:1930

1080 @ 0x4387f6 0x430e73 0x463aa9 0x4d5a72 0x4d6dda 0x4d6dc8 0x5af0a9 0x5c05e5 0x6a7e1f 0x4695c1
#        0x463aa8        internal/poll.runtime_pollWait+0x88                /usr/local/go/src/runtime/netpoll.go:234
#        0x4d5a71        internal/poll.(*pollDesc).wait+0x31                /usr/local/go/src/internal/poll/fd_poll_runtime.go:84
#        0x4d6dd9        internal/poll.(*pollDesc).waitRead+0x259        /usr/local/go/src/internal/poll/fd_poll_runtime.go:89
#        0x4d6dc7        internal/poll.(*FD).Read+0x247                        /usr/local/go/src/internal/poll/fd_unix.go:167
#        0x5af0a8        net.(*netFD).Read+0x28                                /usr/local/go/src/net/fd_posix.go:56
#        0x5c05e4        net.(*conn).Read+0x44                                /usr/local/go/src/net/net.go:183
#        0x6a7e1e        net/http.(*connReader).backgroundRead+0x3e        /usr/local/go/src/net/http/server.go:672

52 @ 0x4387f6 0x4496cc 0x4496a6 0x4655a5 0x4d54f2 0x4d7af2 0x4d7b03 0x4e0725 0x4e071b 0x91f8a3 0x911315 0x9139f9 0x92ff39 0x949656 0x94960f 0x2181d65 0x4695c1
#        0x4655a4        internal/poll.runtime_Semacquire+0x24                        /usr/local/go/src/runtime/sema.go:61
#        0x4d54f1        internal/poll.(*fdMutex).rwlock+0xd1                        /usr/local/go/src/internal/poll/fd_mutex.go:154
#        0x4d7af1        internal/poll.(*FD).writeLock+0x71                        /usr/local/go/src/internal/poll/fd_mutex.go:239
#        0x4d7b02        internal/poll.(*FD).Write+0x82                                /usr/local/go/src/internal/poll/fd_unix.go:262
#        0x4e0724        os.(*File).write+0x64                                        /usr/local/go/src/os/file_posix.go:49
#        0x4e071a        os.(*File).Write+0x5a                                        /usr/local/go/src/os/file.go:176
#        0x91f8a2        go.uber.org/zap/zapcore.multiWriteSyncer.Write+0xc2        /go/pkg/mod/go.uber.org/zap@v1.22.0/zapcore/write_syncer.go:105
#        0x911314        go.uber.org/zap/zapcore.(*ioCore).Write+0xb4                /go/pkg/mod/go.uber.org/zap@v1.22.0/zapcore/core.go:90
#        0x9139f8        go.uber.org/zap/zapcore.(*CheckedEntry).Write+0x1d8        /go/pkg/mod/go.uber.org/zap@v1.22.0/zapcore/entry.go:255
#        0x92ff38        go.uber.org/zap.(*Logger).Info+0x58                        /go/pkg/mod/go.uber.org/zap@v1.22.0/logger.go:213
#        0x949655        whgo_xjd/library/log.doLog+0x95                                /app/library/log/log.go:285
#        0x94960e        whgo_xjd/library/log.InfoWithCtx+0x4e                        /app/library/log/log.go:211
#        0x2181d64        whgo_xjd/library/socket.(*Client).Write+0x464                /app/library/socket/socket.go:504

35 @ 0x4387f6 0x405565 0x40511d 0x217fbc6 0x217fbad 0x4695c1
#        0x217fbc5        whgo_xjd/library/socket.(*Manager).UnRegisterClient+0x565        /app/library/socket/socket.go:675
#        0x217fbac        whgo_xjd/library/socket.(*Manager).heart+0x54c                        /app/library/socket/socket.go:310

35 @ 0x4387f6 0x405565 0x40511d 0x2180dcc 0x2180dae 0x2180b2c 0x4695c1
#        0x2180dcb        whgo_xjd/library/socket.(*Manager).UnRegisterClient+0x18b        /app/library/socket/socket.go:675
#        0x2180dad        whgo_xjd/library/socket.(*Client).Read.func1+0x16d                /app/library/socket/socket.go:326
#        0x2180b2b        whgo_xjd/library/socket.(*Client).Read+0xd2b                        /app/library/socket/socket.go:375

26 @ 0x4387f6 0x4485d2 0x2181a30 0x4695c1
#        0x2181a2f        whgo_xjd/library/socket.(*Client).Write+0x12f        /app/library/socket/socket.go:491

24 @ 0x4387f6 0x4485d2 0x6c906a 0x4695c1
#        0x6c9069        net/http.(*persistConn).readLoop+0xd89        /usr/local/go/src/net/http/transport.go:2207

24 @ 0x4387f6 0x4485d2 0x6ca15b 0x4695c1
#        0x6ca15a        net/http.(*persistConn).writeLoop+0xfa        /usr/local/go/src/net/http/transport.go:2386

23 @ 0x4387f6 0x40640c 0x405e78 0xf100ed 0xf4e1be 0x4695c1
#        0xf100ec        github.com/Shopify/sarama.(*partitionProducer).dispatch+0x14c        /go/pkg/mod/github.com/!shopify/sarama@v1.19.0/async_producer.go:439
#        0xf4e1bd        github.com/Shopify/sarama.withRecover+0x3d                        /go/pkg/mod/github.com/!shopify/sarama@v1.19.0/utils.go:45

10 @ 0x4387f6 0x4496cc 0x4496a6 0x4655a5 0x4d54f2 0x4d7af2 0x4d7b03 0x4e0725 0x4e071b 0x91f8a3 0x911315 0x9139f9 0x92ff39 0x949656 0x94960f 0x23ceeb9 0x8dd48d 0x8dd476 0x8dd250 0x8dcd2e 0x6b261b 0x6adcc8 0x4695c1
#        0x4655a4        internal/poll.runtime_Semacquire+0x24                                /usr/local/go/src/runtime/sema.go:61
#        0x4d54f1        internal/poll.(*fdMutex).rwlock+0xd1                                /usr/local/go/src/internal/poll/fd_mutex.go:154
#        0x4d7af1        internal/poll.(*FD).writeLock+0x71                                /usr/local/go/src/internal/poll/fd_mutex.go:239
#        0x4d7b02        internal/poll.(*FD).Write+0x82                                        /usr/local/go/src/internal/poll/fd_unix.go:262
#        0x4e0724        os.(*File).write+0x64                                                /usr/local/go/src/os/file_posix.go:49
#        0x4e071a        os.(*File).Write+0x5a                                                /usr/local/go/src/os/file.go:176
#        0x91f8a2        go.uber.org/zap/zapcore.multiWriteSyncer.Write+0xc2                /go/pkg/mod/go.uber.org/zap@v1.22.0/zapcore/write_syncer.go:105
#        0x911314        go.uber.org/zap/zapcore.(*ioCore).Write+0xb4                        /go/pkg/mod/go.uber.org/zap@v1.22.0/zapcore/core.go:90
#        0x9139f8        go.uber.org/zap/zapcore.(*CheckedEntry).Write+0x1d8                /go/pkg/mod/go.uber.org/zap@v1.22.0/zapcore/entry.go:255
#        0x92ff38        go.uber.org/zap.(*Logger).Info+0x58                                /go/pkg/mod/go.uber.org/zap@v1.22.0/logger.go:213
#        0x949655        whgo_xjd/library/log.doLog+0x95                                        /app/library/log/log.go:285
#        0x94960e        whgo_xjd/library/log.InfoWithCtx+0x4e                                /app/library/log/log.go:211
#        0x23ceeb8        whgo_xjd/service/middleware_service.AddTraceId+0x5d8                /app/service/middleware_service/common_middleware.go:46
#        0x8dd48c        github.com/gin-gonic/gin.(*Context).Next+0x4c                        /go/pkg/mod/github.com/gin-gonic/gin@v1.4.0/context.go:124
#        0x8dd475        github.com/gin-gonic/gin.serveError+0x35                        /go/pkg/mod/github.com/gin-gonic/gin@v1.4.0/gin.go:425
#        0x8dd24f        github.com/gin-gonic/gin.(*Engine).handleHTTPRequest+0x38f        /go/pkg/mod/github.com/gin-gonic/gin@v1.4.0/gin.go:418
#        0x8dcd2d        github.com/gin-gonic/gin.(*Engine).ServeHTTP+0x16d                /go/pkg/mod/github.com/gin-gonic/gin@v1.4.0/gin.go:351
#        0x6b261a        net/http.serverHandler.ServeHTTP+0x43a                                /usr/local/go/src/net/http/server.go:2879
#        0x6adcc7        net/http.(*conn).serve+0xb07                                        /usr/local/go/src/net/http/server.go:1930

7 @ 0x4387f6 0x40640c 0x405e78 0xf186cb 0xf4e1be 0x4695c1
#        0xf186ca        github.com/Shopify/sarama.(*Broker).responseReceiver+0xaa        /go/pkg/mod/github.com/!shopify/sarama@v1.19.0/broker.go:683
#        0xf4e1bd        github.com/Shopify/sarama.withRecover+0x3d                        /go/pkg/mod/github.com/!shopify/sarama@v1.19.0/utils.go:45

5 @ 0x4387f6 0x4485d2 0x8ffc53 0x4695c1
#        0x8ffc52        database/sql.(*DB).connectionOpener+0x92        /usr/local/go/src/database/sql/sql.go:1196

4 @ 0x4387f6 0x4485d2 0xa72251 0x4695c1
#        0xa72250        github.com/go-redis/redis/v8.(*channel).initHealthCheck.func1+0xf0        /go/pkg/mod/github.com/go-redis/redis/v8@v8.11.5/pubsub.go:542

3 @ 0x4387f6 0x40640c 0x405e78 0xf11345 0xf4e1be 0x4695c1
#        0xf11344        github.com/Shopify/sarama.(*asyncProducer).newBrokerProducer.func1+0x64        /go/pkg/mod/github.com/!shopify/sarama@v1.19.0/async_producer.go:569
#        0xf4e1bd        github.com/Shopify/sarama.withRecover+0x3d                                /go/pkg/mod/github.com/!shopify/sarama@v1.19.0/utils.go:45

3 @ 0x4387f6 0x430e73 0x463aa9 0x4d5a72 0x4d6dda 0x4d6dc8 0x5af0a9 0x5c05e5 0x56b563 0x56c12f 0xa1f870 0xa1f7b9 0xa1fa6a 0xa7198d 0xa71979 0xa23a0e 0xa7189e 0xa7255c 0xa72541 0x4695c1
#        0x463aa8        internal/poll.runtime_pollWait+0x88                                        /usr/local/go/src/runtime/netpoll.go:234
#        0x4d5a71        internal/poll.(*pollDesc).wait+0x31                                        /usr/local/go/src/internal/poll/fd_poll_runtime.go:84
#        0x4d6dd9        internal/poll.(*pollDesc).waitRead+0x259                                /usr/local/go/src/internal/poll/fd_poll_runtime.go:89
#        0x4d6dc7        internal/poll.(*FD).Read+0x247                                                /usr/local/go/src/internal/poll/fd_unix.go:167
#        0x5af0a8        net.(*netFD).Read+0x28                                                        /usr/local/go/src/net/fd_posix.go:56
#        0x5c05e4        net.(*conn).Read+0x44                                                        /usr/local/go/src/net/net.go:183
#        0x56b562        bufio.(*Reader).fill+0x102                                                /usr/local/go/src/bufio/bufio.go:101
#        0x56c12e        bufio.(*Reader).ReadSlice+0x2e                                                /usr/local/go/src/bufio/bufio.go:360
#        0xa1f86f        github.com/go-redis/redis/v8/internal/proto.(*Reader).readLine+0x2f        /go/pkg/mod/github.com/go-redis/redis/v8@v8.11.5/internal/proto/reader.go:73
#        0xa1f7b8        github.com/go-redis/redis/v8/internal/proto.(*Reader).ReadLine+0x18        /go/pkg/mod/github.com/go-redis/redis/v8@v8.11.5/internal/proto/reader.go:59
#        0xa1fa69        github.com/go-redis/redis/v8/internal/proto.(*Reader).ReadReply+0x29        /go/pkg/mod/github.com/go-redis/redis/v8@v8.11.5/internal/proto/reader.go:97
#        0xa7198c        github.com/go-redis/redis/v8.(*Cmd).readReply+0x2c                        /go/pkg/mod/github.com/go-redis/redis/v8@v8.11.5/command.go:467
#        0xa71978        github.com/go-redis/redis/v8.(*PubSub).ReceiveTimeout.func1+0x18        /go/pkg/mod/github.com/go-redis/redis/v8@v8.11.5/pubsub.go:375
#        0xa23a0d        github.com/go-redis/redis/v8/internal/pool.(*Conn).WithReader+0x6d        /go/pkg/mod/github.com/go-redis/redis/v8@v8.11.5/internal/pool/conn.go:70
#        0xa7189d        github.com/go-redis/redis/v8.(*PubSub).ReceiveTimeout+0xfd                /go/pkg/mod/github.com/go-redis/redis/v8@v8.11.5/pubsub.go:374
#        0xa7255b        github.com/go-redis/redis/v8.(*PubSub).Receive+0x9b                        /go/pkg/mod/github.com/go-redis/redis/v8@v8.11.5/pubsub.go:391
#        0xa72540        github.com/go-redis/redis/v8.(*channel).initMsgChan.func1+0x80                /go/pkg/mod/github.com/go-redis/redis/v8@v8.11.5/pubsub.go:571

3 @ 0x4387f6 0x4485d2 0xf11611 0xf4e1be 0x4695c1
#        0xf11610        github.com/Shopify/sarama.(*brokerProducer).run+0x190        /go/pkg/mod/github.com/!shopify/sarama@v1.19.0/async_producer.go:615
#        0xf4e1bd        github.com/Shopify/sarama.withRecover+0x3d                /go/pkg/mod/github.com/!shopify/sarama@v1.19.0/utils.go:45

2 @ 0x4387f6 0x40640c 0x405e38 0xf25bc9 0x1f1f1a5 0x4695c1
#        0xf25bc8        github.com/Shopify/sarama.(*consumerGroup).Consume+0x268        /go/pkg/mod/github.com/!shopify/sarama@v1.19.0/consumer_group.go:180
#        0x1f1f1a4        whgo_xjd/utils/kafka.(*GroupConsumer).Start.func2+0xe4                /app/utils/kafka/consumer.go:94

2 @ 0x4387f6 0x40640c 0x405e78 0x1f1f4c5 0x4695c1
#        0x1f1f4c4        whgo_xjd/utils/kafka.(*GroupConsumer).Start.func1+0x84        /app/utils/kafka/consumer.go:77

2 @ 0x4387f6 0x40640c 0x405e78 0xf0f894 0xf4e1be 0x4695c1
#        0xf0f893        github.com/Shopify/sarama.(*topicProducer).dispatch+0x53        /go/pkg/mod/github.com/!shopify/sarama@v1.19.0/async_producer.go:324
#        0xf4e1bd        github.com/Shopify/sarama.withRecover+0x3d                        /go/pkg/mod/github.com/!shopify/sarama@v1.19.0/utils.go:45

2 @ 0x4387f6 0x4485d2 0xf1dc0e 0xf4e1be 0x4695c1
#        0xf1dc0d        github.com/Shopify/sarama.(*client).backgroundMetadataUpdater+0x12d        /go/pkg/mod/github.com/!shopify/sarama@v1.19.0/client.go:656
#        0xf4e1bd        github.com/Shopify/sarama.withRecover+0x3d                                /go/pkg/mod/github.com/!shopify/sarama@v1.19.0/utils.go:45

2 @ 0x4387f6 0x4485d2 0xf289c5 0x4695c1
#        0xf289c4        github.com/Shopify/sarama.(*consumerGroupSession).heartbeatLoop+0x284        /go/pkg/mod/github.com/!shopify/sarama@v1.19.0/consumer_group.go:674

2 @ 0x4387f6 0x4485d2 0xf3dee7 0xf4e1be 0x4695c1
#        0xf3dee6        github.com/Shopify/sarama.(*offsetManager).mainLoop+0x106        /go/pkg/mod/github.com/!shopify/sarama@v1.19.0/offset_manager.go:217
#        0xf4e1bd        github.com/Shopify/sarama.withRecover+0x3d                        /go/pkg/mod/github.com/!shopify/sarama@v1.19.0/utils.go:45

1 @ 0x40b9f4 0x465d18 0x6dbcb9 0x4695c1
#        0x465d17        os/signal.signal_recv+0x97        /usr/local/go/src/runtime/sigqueue.go:169
#        0x6dbcb8        os/signal.loop+0x18                /usr/local/go/src/os/signal/signal_unix.go:24

1 @ 0x4387f6 0x40640c 0x405e38 0x23cd3de 0x4695c1
#        0x23cd3dd        whgo_xjd/library/metric.(*Metric).sync+0x5d        /app/library/metric/metric.go:35

1 @ 0x4387f6 0x40640c 0x405e38 0x268a305 0x4695c1
#        0x268a304        github.com/fvbock/endless.(*endlessServer).handleSignals+0xc4        /go/pkg/mod/github.com/fvbock/endless@v0.0.0-20170109170031-447134032cb6/endless.go:326

1 @ 0x4387f6 0x40640c 0x405e38 0x268d707 0x438427 0x4695c1
#        0x268d706        main.main+0x426                /app/main.go:121
#        0x438426        runtime.main+0x226        /usr/local/go/src/runtime/proc.go:255

1 @ 0x4387f6 0x40640c 0x405e38 0xefd48a 0x4695c1
#        0xefd489        github.com/rcrowley/go-metrics.(*meterArbiter).tick+0x29        /go/pkg/mod/github.com/rcrowley/go-metrics@v0.0.0-20181016184325-3113b8401b8a/meter.go:239

1 @ 0x4387f6 0x40640c 0x405e38 0xf1327f 0xf4e1be 0x4695c1
#        0xf1327e        github.com/Shopify/sarama.(*asyncProducer).retryHandler+0x19e        /go/pkg/mod/github.com/!shopify/sarama@v1.19.0/async_producer.go:813
#        0xf4e1bd        github.com/Shopify/sarama.withRecover+0x3d                        /go/pkg/mod/github.com/!shopify/sarama@v1.19.0/utils.go:45

1 @ 0x4387f6 0x40640c 0x405e78 0x217e72b 0x4695c1
#        0x217e72a        whgo_xjd/library/socket.sendAllChMessage+0x12a        /app/library/socket/socket.go:189

1 @ 0x4387f6 0x40640c 0x405e78 0x21828bb 0x4695c1
#        0x21828ba        whgo_xjd/library/socket.(*Manager).SendService+0x5a        /app/library/socket/socket.go:521

1 @ 0x4387f6 0x40640c 0x405e78 0x2183072 0x4695c1
#        0x2183071        whgo_xjd/library/socket.(*Manager).SendGroupService+0x51        /app/library/socket/socket.go:564

1 @ 0x4387f6 0x40640c 0x405e78 0x21832b2 0x4695c1
#        0x21832b1        whgo_xjd/library/socket.(*Manager).SendAllService+0x51        /app/library/socket/socket.go:581

1 @ 0x4387f6 0x40640c 0x405e78 0x23a8cf3 0x4695c1
#        0x23a8cf2        whgo_xjd/api/script.SubscriberPrivacy+0x112        /app/api/script/subscriber.go:34

1 @ 0x4387f6 0x40640c 0x405e78 0x23a9013 0x4695c1
#        0x23a9012        whgo_xjd/api/script.SubscriberMallPrivacy+0x112        /app/api/script/subscriber.go:60

1 @ 0x4387f6 0x40640c 0x405e78 0x9408e5 0x4695c1
#        0x9408e4        github.com/natefinch/lumberjack.(*Logger).millRun+0x44        /go/pkg/mod/github.com/natefinch/lumberjack@v2.0.0+incompatible/lumberjack.go:379

1 @ 0x4387f6 0x40640c 0x405e78 0xf0f16c 0xf4e1be 0x4695c1
#        0xf0f16b        github.com/Shopify/sarama.(*asyncProducer).dispatcher+0xcb        /go/pkg/mod/github.com/!shopify/sarama@v1.19.0/async_producer.go:246
#        0xf4e1bd        github.com/Shopify/sarama.withRecover+0x3d                        /go/pkg/mod/github.com/!shopify/sarama@v1.19.0/utils.go:45

1 @ 0x4387f6 0x40640c 0x405e78 0xf2205d 0xf4e1be 0x4695c1
#        0xf2205c        github.com/Shopify/sarama.(*partitionConsumer).dispatcher+0x3c        /go/pkg/mod/github.com/!shopify/sarama@v1.19.0/consumer.go:336
#        0xf4e1bd        github.com/Shopify/sarama.withRecover+0x3d                        /go/pkg/mod/github.com/!shopify/sarama@v1.19.0/utils.go:45

1 @ 0x4387f6 0x40640c 0x405e78 0xf2298f 0xf4e1be 0x4695c1
#        0xf2298e        github.com/Shopify/sarama.(*partitionConsumer).responseFeeder+0x8e        /go/pkg/mod/github.com/!shopify/sarama@v1.19.0/consumer.go:451
#        0xf4e1bd        github.com/Shopify/sarama.withRecover+0x3d                                /go/pkg/mod/github.com/!shopify/sarama@v1.19.0/utils.go:45

1 @ 0x4387f6 0x40640c 0x405e78 0xf27cd4 0x4695c1
#        0xf27cd3        github.com/Shopify/sarama.newConsumerGroupSession.func1+0x93        /go/pkg/mod/github.com/!shopify/sarama@v1.19.0/consumer_group.go:497

1 @ 0x4387f6 0x40640c 0x405e78 0xf2851c 0x4695c1
#        0xf2851b        github.com/Shopify/sarama.(*consumerGroupSession).consume.func1+0x9b        /go/pkg/mod/github.com/!shopify/sarama@v1.19.0/consumer_group.go:579

1 @ 0x4387f6 0x40640c 0x405e78 0xf28f78 0x4695c1
#        0xf28f77        github.com/Shopify/sarama.newConsumerGroupClaim.func1+0x97        /go/pkg/mod/github.com/!shopify/sarama@v1.19.0/consumer_group.go:746

1 @ 0x4387f6 0x40640c 0x405e78 0xf4cb7c 0xf4e1be 0x4695c1
#        0xf4cb7b        github.com/Shopify/sarama.(*syncProducer).handleSuccesses+0x9b        /go/pkg/mod/github.com/!shopify/sarama@v1.19.0/sync_producer.go:131
#        0xf4e1bd        github.com/Shopify/sarama.withRecover+0x3d                        /go/pkg/mod/github.com/!shopify/sarama@v1.19.0/utils.go:45

1 @ 0x4387f6 0x40640c 0x405e78 0xf4cce5 0xf4e1be 0x4695c1
#        0xf4cce4        github.com/Shopify/sarama.(*syncProducer).handleErrors+0xa4        /go/pkg/mod/github.com/!shopify/sarama@v1.19.0/sync_producer.go:139
#        0xf4e1bd        github.com/Shopify/sarama.withRecover+0x3d                        /go/pkg/mod/github.com/!shopify/sarama@v1.19.0/utils.go:45

1 @ 0x4387f6 0x430e73 0x463aa9 0x4d5a72 0x4d6dda 0x4d6dc8 0x5af0a9 0x5c05e5 0x56b563 0x56b6bd 0x215f72c 0x21619db 0x2162a05 0x2162fb9 0x217fef3 0x4695c1
#        0x463aa8        internal/poll.runtime_pollWait+0x88                        /usr/local/go/src/runtime/netpoll.go:234
#        0x4d5a71        internal/poll.(*pollDesc).wait+0x31                        /usr/local/go/src/internal/poll/fd_poll_runtime.go:84
#        0x4d6dd9        internal/poll.(*pollDesc).waitRead+0x259                /usr/local/go/src/internal/poll/fd_poll_runtime.go:89
#        0x4d6dc7        internal/poll.(*FD).Read+0x247                                /usr/local/go/src/internal/poll/fd_unix.go:167
#        0x5af0a8        net.(*netFD).Read+0x28                                        /usr/local/go/src/net/fd_posix.go:56
#        0x5c05e4        net.(*conn).Read+0x44                                        /usr/local/go/src/net/net.go:183
#        0x56b562        bufio.(*Reader).fill+0x102                                /usr/local/go/src/bufio/bufio.go:101
#        0x56b6bc        bufio.(*Reader).Peek+0x5c                                /usr/local/go/src/bufio/bufio.go:139
#        0x215f72b        github.com/gorilla/websocket.(*Conn).read+0x2b                /go/pkg/mod/github.com/gorilla/websocket@v1.5.0/conn.go:371
#        0x21619da        github.com/gorilla/websocket.(*Conn).advanceFrame+0x7a        /go/pkg/mod/github.com/gorilla/websocket@v1.5.0/conn.go:809
#        0x2162a04        github.com/gorilla/websocket.(*Conn).NextReader+0xc4        /go/pkg/mod/github.com/gorilla/websocket@v1.5.0/conn.go:1009
#        0x2162fb8        github.com/gorilla/websocket.(*Conn).ReadMessage+0x18        /go/pkg/mod/github.com/gorilla/websocket@v1.5.0/conn.go:1093
#        0x217fef2        whgo_xjd/library/socket.(*Client).Read+0xf2                /app/library/socket/socket.go:330

1 @ 0x4387f6 0x430e73 0x463aa9 0x4d5a72 0x4d6dda 0x4d6dc8 0x5af0a9 0x5c05e5 0x56bb14 0xf4e565 0x4cdffa 0xf18830 0xf187f7 0xf4e1be 0x4695c1
#        0x463aa8        internal/poll.runtime_pollWait+0x88                                /usr/local/go/src/runtime/netpoll.go:234
#        0x4d5a71        internal/poll.(*pollDesc).wait+0x31                                /usr/local/go/src/internal/poll/fd_poll_runtime.go:84
#        0x4d6dd9        internal/poll.(*pollDesc).waitRead+0x259                        /usr/local/go/src/internal/poll/fd_poll_runtime.go:89
#        0x4d6dc7        internal/poll.(*FD).Read+0x247                                        /usr/local/go/src/internal/poll/fd_unix.go:167
#        0x5af0a8        net.(*netFD).Read+0x28                                                /usr/local/go/src/net/fd_posix.go:56
#        0x5c05e4        net.(*conn).Read+0x44                                                /usr/local/go/src/net/net.go:183
#        0x56bb13        bufio.(*Reader).Read+0x1b3                                        /usr/local/go/src/bufio/bufio.go:227
#        0xf4e564        github.com/Shopify/sarama.(*bufConn).Read+0x24                        /go/pkg/mod/github.com/!shopify/sarama@v1.19.0/utils.go:109
#        0x4cdff9        io.ReadAtLeast+0x99                                                /usr/local/go/src/io/io.go:328
#        0xf1882f        io.ReadFull+0x20f                                                /usr/local/go/src/io/io.go:347
#        0xf187f6        github.com/Shopify/sarama.(*Broker).responseReceiver+0x1d6        /go/pkg/mod/github.com/!shopify/sarama@v1.19.0/broker.go:696
#        0xf4e1bd        github.com/Shopify/sarama.withRecover+0x3d                        /go/pkg/

现象与分析

  1. 主机内核持续打印虚拟设备死循环，伴随ipvlan网卡频繁增删、外网报文丢弃；

  2. 抓取goroutine profile 发现：总协程数高达 12w+ ，几乎全部阻塞在 zap日志同步写 os.File.Write 调用栈；

  3. 业务日志逻辑使用 zapcore.NewMultiWriteSyncer 同步写stdout+文件；

  4. Gin中间件 AddTraceId 在请求前后打印大量链路日志，IO阻塞时请求协程无法退出，形成无限堆积、内存持续上涨。

根因链路（暂未确认是否是这个原因，初步判断是这个原因，这个原因在请云厂商协助排查中）：

虚拟设备死循环 → stdout输出阻塞 → 同步日志写卡死 → 每一条HTTP请求阻塞在日志逻辑 → 协程泄露、内存暴涨 → 服务假死/重启。

短期优化方案

  1. 清理业务非必要冗余日志，减少打印量；

后续优化方案

  1. 将zap同步写日志改造为异步队列日志；

  2. 增加日志降级策略：队列满时丢弃Info/Debug，保留Error/Panic关键日志，保护主服务链路。

相关代码片段

bash 复制代码 
var writeSyncer zapcore.WriteSyncer
if hook != nil {
    writeSyncer = zapcore.NewMultiWriteSyncer(zapcore.AddSync(os.Stdout), zapcore.AddSync(hook))
} else {
    writeSyncer = zapcore.NewMultiWriteSyncer(zapcore.AddSync(os.Stdout))
}

func AddTraceId(c *gin.Context) {
    traceId := uuid.NewV4().String()
    c.Request = c.Request.WithContext(context.WithValue(c.Request.Context(), "trace_id", traceId))
    start := time.Now()
    c.Set("trace_id", traceId)
    metricRequestFrequency.Add()
    log.InfoWithCtx(c, "start", zap.String("path", c.Request.RequestURI), zap.String("time", start.Format("2006-01-02 15:04:05")))
    log.InfoWithCtx(c, "request trace_id")
    c.Next()
    end := time.Now()
    log.InfoWithCtx(c, "end", zap.String("path", c.Request.RequestURI), zap.String("time", time.Now().Format("2006-01-02 15:04:05")), zap.Any("use_time", float64(end.Sub(start).Milliseconds())/1e3))
}

三、第三次故障：Go程序panic------并发map读写导致进程退出（9月11日故障）

排查步骤与命令解释

排查命令

bash 复制代码 
dmesg -T | tail -50

命令解释同上。

日志片段

bash 复制代码 
root@worker-private002:~# dmesg -T | tail -50
                           kernel_stack 847872
                           slab 233197568
                           sock 0
                           shmem 0
                           file_mapped 0
                           file_dirty 0
                           file_writeback 0
                           anon_thp 10485760
                           inactive_anon 0
                           active_anon 4045479936
                           inactive_file 425984
                           active_file 0
                           unevictable 0
                           slab_reclaimable 158601216
                           slab_unreclaimable 74596352
                           pgfault 6741636
                           pgmajfault 0
                           workingset_refault 0
                           workingset_activate 0
                           workingset_nodereclaim 0
                           pgrefill 0
                           pgscan 2944
                           pgsteal 2944
                           pgactivate 0
                           pgdeactivate 0
                           pglazyfree 0
                           pglazyfreed 0
                           thp_fault_alloc 0
                           thp_collapse_alloc 0
[Tue Aug 25 11:58:23 2026] Tasks state (memory values in pages):
[Tue Aug 25 11:58:23 2026] [  pid  ]   uid  tgid total_vm      rss pgtables_bytes swapents oom_score_adj name
[Tue Aug 25 11:58:23 2026] [3870460] 65535 3870460      241        1    28672        0          -998 pause
[Tue Aug 25 11:58:23 2026] [3870572]     0 3870572  1238757   999042  8740864        0           993 main
[Tue Aug 25 11:58:23 2026] oom-kill:constraint=CONSTRAINT_MEMCG,nodemask=(null),cpuset=20d02f6a42575f46f34366abe9445e1144ac8bfe3cfe39ec5c9788ec753eaed4,mems_allowed=0,oom_memcg=/kubepods/burstable/pod5f31f683-355a-4e05-bc43-1747f716c922,task_memcg=/kubepods/burstable/pod5f31f683-355a-4e05-bc43-1747f716c922/20d02f6a42575f46f34366abe9445e1144ac8bfe3cfe39ec5c9788ec753eaed4,task=main,pid=3870572,uid=0
[Tue Aug 25 11:58:23 2026] Memory cgroup out of memory: Killed process 3870572 (main) total-vm:4955028kB, anon-rss:3948204kB, file-rss:48164kB, shmem-rss:0kB, UID:0 pgtables:8536kB oom_score_adj:993
[Tue Aug 25 11:58:23 2026] oom_reaper: reaped process 3870572 (main), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB
[Mon Aug 31 10:51:28 2026] IPv6: ADDRCONF(NETDEV_CHANGE): cali50a13540863: link becomes ready
[Tue Sep  8 15:09:22 2026] IPv6: ADDRCONF(NETDEV_CHANGE): cali67e8aa2ce1c: link becomes ready
[Wed Sep  9 15:13:29 2026] IPv6: ADDRCONF(NETDEV_CHANGE): cali2d76e4efcff: link becomes ready

初步分析

日志中9-11日这一天并没有出现oom-kill，根据这个可以判断，这个pod的重启，和此前pod重启的原因并不一样。

进一步排查分析

1.查看主机运行容器

bash 复制代码 
docker ps

作用：定位故障Pod对应的容器ID、运行时长与状态。

2.排查容器是否OOM重启

bash 复制代码 
docker inspect 容器ID | grep -E 'StartedAt|FinishedAt|ExitCode|OOMKilled|Error'

作用：判断重启是否为内核OOM、是否异常退出。

结果

bash 复制代码 
"OOMKilled": false,
 "ExitCode": 0,
 "Error": "",
 "StartedAt": "2026-09-11T04:29:09.013360035Z",
 "FinishedAt": "0001-01-01T00:00:00Z"

结论：本次重启非OOM，属于程序主动崩溃退出。

3.检索容器panic崩溃日志

bash 复制代码 
LOG=$(docker inspect 容器ID --format '{{.LogPath}}')
grep -n -E 'panic:|fatal error:|runtime error:|SIGSEGV|SIGABRT|signal' $LOG
grep -A 80 'fatal error: concurrent map read and map write' $LOG

日志片段

bash 复制代码 
root@worker-private002:/snap# grep -n -E 'panic:|fatal error:|runtime error:|SIGSEGV|SIGABRT|signal' "$LOG" | head
5748:{"log":"fatal error: concurrent map read and map write\r\n","stream":"stdout","time":"2026-09-11T04:29:07.61654351Z"}
7361:{"log":"os/signal.signal_recv()\r\n","stream":"stdout","time":"2026-09-11T04:29:07.650515781Z"}
7363:{"log":"os/signal.loop()\r\n","stream":"stdout","time":"2026-09-11T04:29:07.650555303Z"}
7364:{"log":"\u0009/usr/local/go/src/os/signal/signal_unix.go:24 +0x19\r\n","stream":"stdout","time":"2026-09-11T04:29:07.650587099Z"}
7365:{"log":"created by os/signal.Notify.func1.1\r\n","stream":"stdout","time":"2026-09-11T04:29:07.650597021Z"}
7366:{"log":"\u0009/usr/local/go/src/os/signal/signal.go:151 +0x2c\r\n","stream":"stdout","time":"2026-09-11T04:29:07.650643195Z"}
root@worker-private002:/snap# 
root@worker-private002:/snap# 
root@worker-private002:/snap# grep -A 80 'fatal error: concurrent map read and map write' "$LOG" | head -100
{"log":"fatal error: concurrent map read and map write\r\n","stream":"stdout","time":"2026-09-11T04:29:07.61654351Z"}
{"log":"\r\n","stream":"stdout","time":"2026-09-11T04:29:07.619783886Z"}
{"log":"goroutine 2902824 [running]:\r\n","stream":"stdout","time":"2026-09-11T04:29:07.61979076Z"}
{"log":"runtime.throw({0x2c31fa0, 0x6c83b773})\r\n","stream":"stdout","time":"2026-09-11T04:29:07.619866701Z"}
{"log":"\u0009/usr/local/go/src/runtime/panic.go:1198 +0x71 fp=0xc004b50658 sp=0xc004b50628 pc=0x435d51\r\n","stream":"stdout","time":"2026-09-11T04:29:07.619883181Z"}
{"log":"runtime.mapaccess2_faststr(0x28c8b60, 0x2bc6eb3, {0x2bc6eb3, 0x8})\r\n","stream":"stdout","time":"2026-09-11T04:29:07.619924299Z"}
{"log":"\u0009/usr/local/go/src/runtime/map_faststr.go:116 +0x3d4 fp=0xc004b506c0 sp=0xc004b50658 pc=0x412e34\r\n","stream":"stdout","time":"2026-09-11T04:29:07.619977094Z"}
{"log":"github.com/gin-gonic/gin.(*Context).Get(...)\r\n","stream":"stdout","time":"2026-09-11T04:29:07.620021036Z"}
{"log":"\u0009/go/pkg/mod/github.com/gin-gonic/gin@v1.4.0/context.go:208\r\n","stream":"stdout","time":"2026-09-11T04:29:07.620027133Z"}
{"log":"github.com/gin-gonic/gin.(*Context).Value(0x34a1bd8, {0x2821c00, 0x34263f0})\r\n","stream":"stdout","time":"2026-09-11T04:29:07.620029891Z"}
{"log":"\u0009/go/pkg/mod/github.com/gin-gonic/gin@v1.4.0/context.go:1019 +0x70 fp=0xc004b506f0 sp=0xc004b506c0 pc=0x8d9430\r\n","stream":"stdout","time":"2026-09-11T04:29:07.620087096Z"}
{"log":"whgo_xjd/library/logfactory/factory.getTraceId({0x34a1bd8, 0xc0038f5970})\r\n","stream":"stdout","time":"2026-09-11T04:29:07.620104287Z"}
{"log":"\u0009/app/library/logfactory/factory/trace.go:31 +0x15c fp=0xc004b507f0 sp=0xc004b506f0 pc=0xc1e67c\r\n","stream":"stdout","time":"2026-09-11T04:29:07.620146398Z"}
{"log":"whgo_xjd/library/logfactory/factory.ExtractTraceID(...)\r\n","stream":"stdout","time":"2026-09-11T04:29:07.620151437Z"}
{"log":"\u0009/app/library/logfactory/factory/trace.go:36\r\n","stream":"stdout","time":"2026-09-11T04:29:07.620181346Z"}
{"log":"whgo_xjd/library/logfactory/factory.(*logfactoryImp).Ctx(0xc0004c9dd0, {0x34a1bd8, 0xc0038f5970})\r\n","stream":"stdout","time":"2026-09-11T04:29:07.620213724Z"}
{"log":"\u0009/app/library/logfactory/factory/imp.go:68 +0x49 fp=0xc004b508a8 sp=0xc004b507f0 pc=0xc1dfc9\r\n","stream":"stdout","time":"2026-09-11T04:29:07.62026408Z"}
{"log":"whgo_xjd/service/customer_service.(*SupplementInfo).OrderCompleteHandle(0x23837f5, {0x34a1bd8, 0xc0038f5970}, 0xc002db2070, {0xc002db2070, 0x10})\r\n","stream":"stdout","time":"2026-09-11T04:29:07.620310363Z"}
{"log":"\u0009/app/service/customer_service/supplement_info.go:369 +0xcaf fp=0xc004b514d8 sp=0xc004b508a8 pc=0x125c50f\r\n","stream":"stdout","time":"2026-09-11T04:29:07.62034913Z"}
{"log":"whgo_xjd/service/tiancai_service.DoPayedAfter({0x34a1bd8, 0xc0038f5970}, {0xc002c34580, 0x10})\r\n","stream":"stdout","time":"2026-09-11T04:29:07.620425298Z"}
{"log":"\u0009/app/service/tiancai_service/order.go:2741 +0x1de fp=0xc004b51fb0 sp=0xc004b514d8 pc=0x20f549e\r\n","stream":"stdout","time":"2026-09-11T04:29:07.620434474Z"}
{"log":"whgo_xjd/service/order_service.CreateOrder·dwrap·3()\r\n","stream":"stdout","time":"2026-09-11T04:29:07.620439095Z"}
{"log":"\u0009/app/service/order_service/order_service.go:1291 +0x32 fp=0xc004b51fe0 sp=0xc004b51fb0 pc=0x213e412\r\n","stream":"stdout","time":"2026-09-11T04:29:07.620519303Z"}
{"log":"runtime.goexit()\r\n","stream":"stdout","time":"2026-09-11T04:29:07.620579941Z"}
{"log":"\u0009/usr/local/go/src/runtime/asm_amd64.s:1581 +0x1 fp=0xc004b51fe8 sp=0xc004b51fe0 pc=0x4695c1\r\n","stream":"stdout","time":"2026-09-11T04:29:07.620591749Z"}
{"log":"created by whgo_xjd/service/order_service.CreateOrder\r\n","stream":"stdout","time":"2026-09-11T04:29:07.620594863Z"}
{"log":"\u0009/app/service/order_service/order_service.go:1291 +0x5187\r\n","stream":"stdout","time":"2026-09-11T04:29:07.620640782Z"}
{"log":"\r\n","stream":"stdout","time":"2026-09-11T04:29:07.620647109Z"}
{"log":"goroutine 1 [chan receive, 1284 minutes]:\r\n","stream":"stdout","time":"2026-09-11T04:29:07.620650262Z"}
{"log":"main.main()\r\n","stream":"stdout","time":"2026-09-11T04:29:07.620675695Z"}
{"log":"\u0009/app/main.go:121 +0x427\r\n","stream":"stdout","time":"2026-09-11T04:29:07.620678917Z"}
{"log":"\r\n","stream":"stdout","time":"2026-09-11T04:29:07.620695873Z"}
{"log":"goroutine 10 [chan receive]:\r\n","stream":"stdout","time":"2026-09-11T04:29:07.620749909Z"}
{"log":"whgo_xjd/library/socket.sendChMessage({0x34a19a8, 0xc000130000})\r\n","stream":"stdout","time":"2026-09-11T04:29:07.620753687Z"}
{"log":"\u0009/app/library/socket/socket.go:140 +0x12b\r\n","stream":"stdout","time":"2026-09-11T04:29:07.620775178Z"}
{"log":"created by whgo_xjd/library/socket.init.2\r\n","stream":"stdout","time":"2026-09-11T04:29:07.620789469Z"}
{"log":"\u0009/app/library/socket/socket.go:126 +0xa6\r\n","stream":"stdout","time":"2026-09-11T04:29:07.620794575Z"}
{"log":"\r\n","stream":"stdout","time":"2026-09-11T04:29:07.620797365Z"}
{"log":"goroutine 11 [chan receive]:\r\n","stream":"stdout","time":"2026-09-11T04:29:07.620799737Z"}
{"log":"whgo_xjd/library/socket.sendGroupChMessage({0x34a19a8, 0xc000130000})\r\n","stream":"stdout","time":"2026-09-11T04:29:07.620842102Z"}
{"log":"\u0009/app/library/socket/socket.go:168 +0x12b\r\n","stream":"stdout","time":"2026-09-11T04:29:07.620845734Z"}
{"log":"created by whgo_xjd/library/socket.init.2\r\n","stream":"stdout","time":"2026-09-11T04:29:07.620848295Z"}
{"log":"\u0009/app/library/socket/socket.go:127 +0xf3\r\n","stream":"stdout","time":"2026-09-11T04:29:07.620911406Z"}
{"log":"\r\n","stream":"stdout","time":"2026-09-11T04:29:07.620915345Z"}
{"log":"goroutine 12 [chan receive, 1284 minutes]:\r\n","stream":"stdout","time":"2026-09-11T04:29:07.620917865Z"}
{"log":"whgo_xjd/library/socket.sendAllChMessage({0x34a19a8, 0xc000130000})\r\n","stream":"stdout","time":"2026-09-11T04:29:07.620920433Z"}
{"log":"\u0009/app/library/socket/socket.go:189 +0x12b\r\n","stream":"stdout","time":"2026-09-11T04:29:07.620923023Z"}
{"log":"created by whgo_xjd/library/socket.init.2\r\n","stream":"stdout","time":"2026-09-11T04:29:07.62093937Z"}
{"log":"\u0009/app/library/socket/socket.go:128 +0x145\r\n","stream":"stdout","time":"2026-09-11T04:29:07.620944069Z"}
{"log":"\r\n","stream":"stdout","time":"2026-09-11T04:29:07.620946656Z"}
{"log":"goroutine 13 [sleep]:\r\n","stream":"stdout","time":"2026-09-11T04:29:07.620949907Z"}
{"log":"time.Sleep(0xb2d05e00)\r\n","stream":"stdout","time":"2026-09-11T04:29:07.620952379Z"}
{"log":"\u0009/usr/local/go/src/runtime/time.go:193 +0x12e\r\n","stream":"stdout","time":"2026-09-11T04:29:07.620995638Z"}
{"log":"whgo_xjd/library/socket.checkSubTask({0x34a19a8, 0xc000130000})\r\n","stream":"stdout","time":"2026-09-11T04:29:07.621002687Z"}
{"log":"\u0009/app/library/socket/socket.go:834 +0x254\r\n","stream":"stdout","time":"2026-09-11T04:29:07.621005365Z"}
{"log":"created by whgo_xjd/library/socket.init.2\r\n","stream":"stdout","time":"2026-09-11T04:29:07.621026114Z"}
{"log":"\u0009/app/library/socket/socket.go:129 +0x192\r\n","stream":"stdout","time":"2026-09-11T04:29:07.621029135Z"}
{"log":"\r\n","stream":"stdout","time":"2026-09-11T04:29:07.621031712Z"}
{"log":"goroutine 14 [chan receive]:\r\n","stream":"stdout","time":"2026-09-11T04:29:07.621034219Z"}
{"log":"whgo_xjd/library/metric.(*Metric).sync(0xc000292780)\r\n","stream":"stdout","time":"2026-09-11T04:29:07.621054039Z"}
{"log":"\u0009/app/library/metric/metric.go:35 +0x5e\r\n","stream":"stdout","time":"2026-09-11T04:29:07.621057052Z"}
{"log":"created by whgo_xjd/library/metric.NewMetric\r\n","stream":"stdout","time":"2026-09-11T04:29:07.621059532Z"}
{"log":"\u0009/app/library/metric/metric.go:26 +0x86\r\n","stream":"stdout","time":"2026-09-11T04:29:07.621085111Z"}
{"log":"\r\n","stream":"stdout","time":"2026-09-11T04:29:07.621097478Z"}
{"log":"goroutine 2898885 [IO wait]:\r\n","stream":"stdout","time":"2026-09-11T04:29:07.621106441Z"}
{"log":"internal/poll.runtime_pollWait(0x7fbf83b2fc28, 0x72)\r\n","stream":"stdout","time":"2026-09-11T04:29:07.621126799Z"}
{"log":"\u0009/usr/local/go/src/runtime/netpoll.go:234 +0x89\r\n","stream":"stdout","time":"2026-09-11T04:29:07.62120609Z"}
{"log":"internal/poll.(*pollDesc).wait(0xc003ea1400, 0xc00476a000, 0x0)\r\n","stream":"stdout","time":"2026-09-11T04:29:07.621213344Z"}
{"log":"\u0009/usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0x32\r\n","stream":"stdout","time":"2026-09-11T04:29:07.621217905Z"}
{"log":"internal/poll.(*pollDesc).waitRead(...)\r\n","stream":"stdout","time":"2026-09-11T04:29:07.621222102Z"}
{"log":"\u0009/usr/local/go/src/internal/poll/fd_poll_runtime.go:89\r\n","stream":"stdout","time":"2026-09-11T04:29:07.621226096Z"}
{"log":"internal/poll.(*FD).Read(0xc003ea1400, {0xc00476a000, 0x1000, 0x1000})\r\n","stream":"stdout","time":"2026-09-11T04:29:07.621230099Z"}
{"log":"\u0009/usr/local/go/src/internal/poll/fd_unix.go:167 +0x25a\r\n","stream":"stdout","time":"2026-09-11T04:29:07.621234765Z"}
{"log":"net.(*netFD).Read(0xc003ea1400, {0xc00476a000, 0x4d5f66, 0x47b07a0})\r\n","stream":"stdout","time":"2026-09-11T04:29:07.621272713Z"}
{"log":"\u0009/usr/local/go/src/net/fd_posix.go:56 +0x29\r\n","stream":"stdout","time":"2026-09-11T04:29:07.621322297Z"}
{"log":"net.(*conn).Read(0xc00d5c63f0, {0xc00476a000, 0x0, 0x0})\r\n","stream":"stdout","time":"2026-09-11T04:29:07.621325779Z"}
{"log":"\u0009/usr/local/go/src/net/net.go:183 +0x45\r\n","stream":"stdout","time":"2026-09-11T04:29:07.621328405Z"}
{"log":"net/http.(*connReader).Read(0xc0055cf440, {0xc00476a000, 0x1000, 0x1000})\r\n","stream":"stdout","time":"2026-09-11T04:29:07.621363449Z"}
{"log":"\u0009/usr/local/go/src/net/http/server.go:780 +0x16d\r\n","stream":"stdout","time":"2026-09-11T04:29:07.621367175Z"}
{"log":"bufio.(*Reader).fill(0xc005f4fd40)\r\n","stream":"stdout","time":"2026-09-11T04:29:07.62138851Z"}
{"log":"\u0009/usr/local/go/src/bufio/bufio.go:101 +0x103\r\n","stream":"stdout","time":"2026-09-11T04:29:07.621391826Z"}
root@worker-private002:/snap#

核心错误日志

bash 复制代码 
{"log":"fatal error: concurrent map read and map write\r\n","stream":"stdout","time":"2026-09-11T04:29:07.61654351Z"}
{"log":"whgo_xjd/service/tiancai_service.DoPayedAfter({0x34a1bd8, 0xc0038f5970}, {0xc002c34580, 0x10})\r\n","stream":"stdout","time":"2026-09-11T04:29:07.620425298Z"}

完整堆栈关键定位

崩溃发生在 gin.Context 链路读写map冲突：

  • 后台goroutine：getTraceId → ctx.Value() 读 c.Keys map

  • 主请求goroutine/连接复用：c.Set() 写 c.Keys map

  • 形成同一gin上下文map并发读写，触发Go runtime直接panic退出进程

根因定位

  1. Gin 的 context.Keys map 非并发安全；

  2. 业务代码直接将原始gin.Context传给异步后台协程；

  3. 主请求与后台协程并发读写同一个ctx的map，触发panic；

  4. Gin官方规范：跨goroutine使用ctx必须执行 ctx.Copy() 生成全新上下文，禁止直接传递原始ctx。

问题代码

bash 复制代码 
go tiancai_service.DoPayedAfter(ctx, orders.OrderNo)

四、整体故障总结

近期Pod重启共计三类完全不同根因：

  1. 内核层：内存超限 → OOM Kill；

  2. 容器IO层：虚拟设备死循环 → 同步日志阻塞 → 协程堆积内存暴涨；

  3. 代码并发层：跨协程误用gin原始ctx → map并发读写panic。

Pod重启 ≠ OOM，必须分层排查：内核日志 → 容器退出信息 → 程序panic堆栈。

五、后续优化计划

紧急修复

  1. 全量修复所有异步任务、后台goroutine，统一使用 ctx.Copy()，杜绝并发map读写panic；

中期优化

  1. 日志架构升级：同步日志全面改为异步队列日志，增加限流、降级、丢弃策略；

  2. 清理全链路冗余日志，降低IO压力；

长期治理

  1. 新增监控：Pod内存增速、goroutine数量、panic次数、日志阻塞指标；

  2. 建立Pod重启标准化排查流程，提前发现内存泄漏、IO阻塞隐患。

六、Pod重启故障标准化排查 Checklist

（一）排查顺序（固定）

内核层 → 容器运行时层 → 业务程序层

（二）第一层：内核排查（必查）

bash 复制代码 
dmesg -T | tail -100

检查关键字：

  • oom-kill → OOM重启

  • Dead loop / Dropped outbound packet → 网络/虚拟设备IO阻塞

（三）第二层：容器退出原因排查

bash 复制代码 
docker ps docker inspect 容器ID | grep -E 'OOMKilled|ExitCode'

判断逻辑：

  • OOMKilled=true → 内存超限

  • OOMKilled=false → 程序崩溃/阻塞重启

（四）第三层：程序崩溃排查

bash 复制代码 
LOG=$(docker inspect 容器ID --format '{{.LogPath}}') grep -E 'panic|fatal error' $LOG grep -A 100 'fatal error' $LOG

（五）三类故障快速判定清单

  1. 瞬间内存打满、Pod重启 → OOM cgroup超限

  2. 服务慢慢卡死、协程数暴涨 → 日志IO阻塞、同步写卡死，最终应该也会导致oom超限。

  3. 瞬时重启、无内存飙升 → Go并发panic崩溃

（六）修复落地检查项

OOM问题

  • 排查内存泄漏（连接、协程、缓存未释放）

  • 调整容器内存配额适配峰值

  • 增加内存增速监控

日志阻塞问题

  • 冗余日志清理完成

  • 同步日志改为异步日志

  • 日志队列+降级策略上线

并发panic问题

  • 所有异步任务ctx均使用Copy()

  • 全局非并发map增加锁保护

  • 线上panic告警接入

七、常态化运维规范

  1. 所有Gin异步任务禁止直接传原始ctx；

  2. 所有服务日志禁止全量同步打印；

  3. 重启故障必须三层次完整排查，不可默认归为OOM。

相关推荐
玉宇夕落2 小时前
Docker + Milvus，数据库永久存放记忆，让对话历史永不丢失
数据库·docker·llm
名字还没想好☜5 小时前
Docker 镜像瘦身进阶:用 distroless/scratch 把 Go 服务打到 10MB,以及没 shell 怎么调试
运维·docker·容器·golang·kubernetes
JavaPub-rodert5 小时前
Docker 深入理解：从容器原理到生产环境最佳实践
运维·docker·容器
暖核7 小时前
Docker 入门｜吃透容器生态与底层原理
运维·docker·容器
滕州市燕猫虎计算机科技工作室个体工商户7 小时前
Docker常用命令
运维·docker·容器
wdfk_prog8 小时前
lely-canopen-rtt：为什么推荐在 RT-Thread 上使用 Lely CANopen 构建主站
java·运维·缓存·docker·容器
智_永无止境8 小时前
一个Docker命令，40万首古诗词API开箱即用
docker·诗泉
33三 三like9 小时前
安装并启动Docker
运维·docker·容器
滕州市燕猫虎计算机科技工作室个体工商户9 小时前
进入Docker容器内的命令行界面
运维·docker·容器
热门推荐
01国内可直接用、免费额度/永久免费的大模型API清单（含 SiliconFlow、火山、阿里、智谱、百度、Kimi、DeepSeek、DMXAPI 等）02蜘蛛磁力 搜索引擎大全，如何使用蜘蛛磁力查找磁力链接03【一等奖版】2026 国赛 A题 药材的烘干问题04GitHub 镜像站点05历年考研数学一、数学二、数学三真题试卷及答案PDF06微信历史版本含下载地址（ Windows PC | 安卓 | MAC ）及设置微信不更新07如何新建文件夹？ 电脑新建文件夹的4种方法082026 国产 AI 大模型横评：DeepSeek、通义千问、Kimi、文心一言、星火、豆包谁更能打？09ZCode 周末送额度活动开启：3 亿 Token 免费领取10Agnes AI 免费 API 接入指南：文本、生图、生视频，一套接口全免费