手动修复 rabbitmq 报错 “Crash dump is being written to“

only火车头2023-11-08 19:49

rabbitmq 报错:

复制代码 
2023-11-07 16:38:52.682 [error] emulator Error in process <0.368.0> on node 'rabbit@rabbitmq-0.rabbitmq-discovery.openstack.svc.cluster.local' with exit value:
{shutdown,[{mnesia_loader,handle_exit,2,[{file,"mnesia_loader.erl"},{line,963}]},{mnesia_loader,tab_receiver,5,[{file,"mnesia_loader.erl"},{line,440}]},{mnesia_loader,spawned_receiver,8,[{file,"mnesia_loader.erl"},{line,343}]}]}
2023-11-07 16:38:52.683 [error] emulator Error in process <0.367.0> on node 'rabbit@rabbitmq-0.rabbitmq-discovery.openstack.svc.cluster.local' with exit value:
{badarg,[{ets,insert,[mnesia_gvar,{last_error,{{shutdown,[{mnesia_loader,handle_exit,2,[{file,"mnesia_loader.erl"},{line,963}]},{mnesia_loader,tab_receiver,5,[{file,"mnesia_loader.erl"},{line,440}]},{mnesia_loader,spawned_receiver,8,[{file,"mnesia_loader.erl"},{line,343}]}]},[{mnesia_loader,wait_on_load_complete,1,[{file,"mnesia_loader.erl"},{line,359}]},{mnesia_tm,apply_fun,3,[{file,"mnesia_tm.erl"},{line,840}]},{mnesia_tm,execute_transaction,5,[{file,"mnesia_tm.erl"},{line,816}]},{mnesia_loader,init_receiver,5,[{file,"mnesia_loader.erl"},{line,285}]},{mnesia_loader,do_get_network_copy,5,[{file,"mnesia_loader.erl"},{line,221}]},{mnesia_controller,'-load_table_fun/1-fun-4-',5,[{file,"mnesia_controller.erl"},{line,2186}]},{mnesia_controller,'-load_and_reply/2-fun-0-',2,[{file,"mnesia_controller.erl"},{line,2133}]}]}}],[]},{mnesia_lib,set,2,[{file,"mnesia_lib.erl"},{line,443}]},{mnesia_lib,fix_error,1,[{file,"mnesia_lib.erl"},{line,906}]},{mnesia_tm,return_abort,3,[{file,"mnesia_tm.erl"},{line,962}]},{mnesia_loader,init_receiver,5,[{file,"mnesia_loader.erl"},{line,285}]},{mnesia_loader,do_get_network_copy,5,[{file,"mnesia_loader.erl"},{line,221}]},{mnesia_controller,'-load_table_fun/1-fun-4-',5,[{file,"mnesia_controller.erl"},{line,2186}]},{mnesia_controller,'-load_and_reply/2-fun-0-',2,[{file,"mnesia_controller.erl"},{line,2133}]}]}
2023-11-07 16:38:52.685 [info] <0.43.0> Application mnesia exited with reason: stopped
2023-11-07 16:38:52.685 [info] <0.43.0> Application tools exited with reason: stopped
2023-11-07 16:38:52.685 [error] <0.8.0> 
Error description:
    init:do_boot/3
    init:start_em/1
    rabbit:start_it/1 line 465
    rabbit:broker_start/1 line 341
    rabbit:start_loaded_apps/2 line 586
    app_utils:manage_applications/6 line 126
    lists:foldl/3 line 1263
    rabbit:'-handle_app_error/1-fun-0-'/3 line 709
throw:{could_not_start,ra,
       {ra,
        {{shutdown,
          {failed_to_start_child,ra_system_sup,
           {shutdown,
            {failed_to_start_child,ra_log_sup,
             {shutdown,
              {failed_to_start_child,ra_log_wal_sup,
               {shutdown,
                {failed_to_start_child,ra_log_wal,
                 {{case_clause,{ok,<<>>}},
                  [{ra_log_wal,open_existing,1,
                    [{file,"src/ra_log_wal.erl"},{line,556}]},
                   {ra_log_wal,'-recover_wal/2-lc$^0/1-0-',1,
                    [{file,"src/ra_log_wal.erl"},{line,240}]},
                   {ra_log_wal,recover_wal,2,
                    [{file,"src/ra_log_wal.erl"},{line,243}]},
                   {ra_log_wal,init,1,
                    [{file,"src/ra_log_wal.erl"},{line,186}]},
                   {gen_batch_server,init_it,6,
                    [{file,"src/gen_batch_server.erl"},{line,125}]},
                   {proc_lib,init_p_do_apply,3,
                    [{file,"proc_lib.erl"},{line,249}]}]}}}}}}}}},
         {ra_app,start,[normal,[]]}}}}
Log file(s) (may contain more information):
   <stdout>

BOOT FAILED
===========

Error description:
    init:do_boot/3
    init:start_em/1
    rabbit:start_it/1 line 465
    rabbit:broker_start/1 line 341
    rabbit:start_loaded_apps/2 line 586
    app_utils:manage_applications/6 line 126
    lists:foldl/3 line 1263
    rabbit:'-handle_app_error/1-fun-0-'/3 line 709
throw:{could_not_start,ra,
       {ra,
        {{shutdown,
          {failed_to_start_child,ra_system_sup,
           {shutdown,
            {failed_to_start_child,ra_log_sup,
             {shutdown,
              {failed_to_start_child,ra_log_wal_sup,
               {shutdown,
                {failed_to_start_child,ra_log_wal,
                 {{case_clause,{ok,<<>>}},
                  [{ra_log_wal,open_existing,1,
                    [{file,"src/ra_log_wal.erl"},{line,556}]},
                   {ra_log_wal,'-recover_wal/2-lc$^0/1-0-',1,
                    [{file,"src/ra_log_wal.erl"},{line,240}]},
                   {ra_log_wal,recover_wal,2,
                    [{file,"src/ra_log_wal.erl"},{line,243}]},
                   {ra_log_wal,init,1,
                    [{file,"src/ra_log_wal.erl"},{line,186}]},
                   {gen_batch_server,init_it,6,
                    [{file,"src/gen_batch_server.erl"},{line,125}]},
                   {proc_lib,init_p_do_apply,3,
                    [{file,"proc_lib.erl"},{line,249}]}]}}}}}}}}},
         {ra_app,start,[normal,[]]}}}}
Log file(s) (may contain more information):
   <stdout>

{"init terminating in do_boot",{could_not_start,ra,{ra,{{shutdown,{failed_to_start_child,ra_system_sup,{shutdown,{failed_to_start_child,ra_log_sup,{shutdown,{failed_to_start_child,ra_log_wal_sup,{shutdown,{failed_to_start_child,ra_log_wal,{{case_clause,{ok,<<>>}},[{ra_log_wal,open_existing,1,[{file,"src/ra_log_wal.erl"},{line,556}]},{ra_log_wal,'-recover_wal/2-lc$^0/1-0-',1,[{file,"src/ra_log_wal.erl"},{line,240}]},{ra_log_wal,recover_wal,2,[{file,"src/ra_log_wal.erl"},{line,243}]},{ra_log_wal,init,1,[{file,"src/ra_log_wal.erl"},{line,186}]},{gen_batch_server,init_it,6,[{file,"src/gen_batch_server.erl"},{line,125}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,249}]}]}}}}}}}}},{ra_app,start,[normal,[]]}}}}}
init terminating in do_boot ({could_not_start,ra,{ra,{{shutdown,{_}},{ra_app,start,[_]}}}})

Crash dump is being written to: /var/log/rabbitmq/erl_crash.dump...done

修复方法:

(1) 找到 rabbitmq 使用的 pv，例如: rabbitmq-0 的 pod:

复制代码 
# kubectl get pv | grep rabbitmq-0
pvc-70ed48bf-bef8-4658-b530-1fd3a6ef5937   200Gi      RWO            Delete           Bound    openstack/rabbitmq-data-rabbitmq-0                                    ceph-ssd                6d17h

(2) 找到 pv 使用的信息:

复制代码 
# kubectl get pv pvc-70ed48bf-bef8-4658-b530-1fd3a6ef5937 -o yaml
apiVersion: v1
kind: PersistentVolume
metadata:
  annotations:
    kubernetes.io/createdby: rbd-dynamic-provisioner
    pv.kubernetes.io/bound-by-controller: "yes"
    pv.kubernetes.io/provisioned-by: kubernetes.io/rbd
  creationTimestamp: "2023-10-31T15:40:59Z"
  finalizers:
  - kubernetes.io/pv-protection
  name: pvc-70ed48bf-bef8-4658-b530-1fd3a6ef5937
  resourceVersion: "7552"
  uid: 6848417a-dd4f-430c-85e5-f3234a1ac6bf
spec:
  accessModes:
  - ReadWriteOnce
  capacity:
    storage: 200Gi
  claimRef:
    apiVersion: v1
    kind: PersistentVolumeClaim
    name: rabbitmq-data-rabbitmq-0
    namespace: openstack
    resourceVersion: "4704"
    uid: 70ed48bf-bef8-4658-b530-1fd3a6ef5937
  persistentVolumeReclaimPolicy: Delete
  rbd:
    image: kubernetes-dynamic-pvc-c8a3585f-dc7b-438c-a22e-cca9d84c341f
    keyring: /etc/ceph/keyring
    monitors:
    - ceph-mon.ceph.svc.cluster.local:6789
    pool: ssdpool
    secretRef:
      name: pvc-ceph-client-key
    user: admin
  storageClassName: ceph-ssd
  volumeMode: Filesystem
status:
  phase: Bound

需要的信息:

复制代码 
    image: kubernetes-dynamic-pvc-c8a3585f-dc7b-438c-a22e-cca9d84c341f

(3) 在 pod 节点上查看对应的物理设备

复制代码 
# ssh node-2 rbd showmapped | grep kubernetes-dynamic-pvc-c8a3585f-dc7b-438c-a22e-cca9d84c341f
0  ssdpool           kubernetes-dynamic-pvc-c8a3585f-dc7b-438c-a22e-cca9d84c341f -    /dev/rbd0

(4) 查看设备挂载目录

复制代码 
# ssh node-2 mount | grep rbd0
/dev/rbd0 on /var/lib/kubelet/plugins/kubernetes.io/rbd/mounts/ssdpool-image-kubernetes-dynamic-pvc-c8a3585f-dc7b-438c-a22e-cca9d84c341f type ext4 (rw,relatime,stripe=1024)
/dev/rbd0 on /var/lib/kubelet/pods/3a37e264-4fd5-4cb8-844b-6b6cd4a6859c/volumes/kubernetes.io~rbd/pvc-70ed48bf-bef8-4658-b530-1fd3a6ef5937 type ext4 (rw,relatime,stripe=1024)

(5) 查找 wal 文件路径，查找的路径来自步骤 (4)

复制代码 
# ssh node-2 find /var/lib/kubelet/pods/3a37e264-4fd5-4cb8-844b-6b6cd4a6859c/volumes/kubernetes.io~rbd/pvc-70ed48bf-bef8-4658-b530-1fd3a6ef5937 -name "*.wal"
/var/lib/kubelet/pods/3a37e264-4fd5-4cb8-844b-6b6cd4a6859c/volumes/kubernetes.io~rbd/pvc-70ed48bf-bef8-4658-b530-1fd3a6ef5937/mnesia/rabbit@rabbitmq-0.rabbitmq-discovery.openstack.svc.cluster.local/quorum/rabbit@rabbitmq-0.rabbitmq-discovery.openstack.svc.cluster.local/00000025.wal

(6) 删除 wal 文件

此步骤请慎重操作，建议将文件备份后再操作。

复制代码 
# ssh node-2 rm -rf /var/lib/kubelet/pods/3a37e264-4fd5-4cb8-844b-6b6cd4a6859c/volumes/kubernetes.io~rbd/pvc-70ed48bf-bef8-4658-b530-1fd3a6ef5937/mnesia/rabbit@rabbitmq-0.rabbitmq-discovery.openstack.svc.cluster.local/quorum/rabbit@rabbitmq-0.rabbitmq-discovery.openstack.svc.cluster.local/00000025.wal
Warning: Permanently added 'node-2' (ED25519) to the list of known hosts.

(7) 删除 pod，重新启动 pod

复制代码 
# kubectl delete pods rabbitmq-0 -n openstack 
pod "rabbitmq-0" deleted

等待 pod 再次启动，过一会重新数据同步恢复。

相关推荐
逐流人15 分钟前
Ceph分布式存储集群配置与池管理：从配置优先级到PG、复本池与纠删码池
运维·分布式·ceph·云原生·云计算·rados
天天喝旺仔3 小时前
gRPC 底层原理：HTTP/2 多路复用、Protobuf 序列化与四种流模式
分布式·http·微服务·rpc
灯澜忆梦6 小时前
【RabbitMQ #5】 | 三大交换机 Fanout ,Direct ,Topic
分布式·rabbitmq·ruby
.冰块.8 小时前
Ceph 分布式存储实战（二）：存储池管理与 cephx 认证授权
分布式·ceph·rados·纠删码·复本池·cephx认证与用户权限
我们从未走散10 小时前
从内存字典到租约内核：分布式上游账号池如何做到不超卖
分布式
.冰块.11 小时前
Ceph 分布式存储实战（一）：架构入门与 cephadm 集群部署
分布式·ceph·架构·cephadm
j7～12 小时前
【Redis】《一文走完服务端高并发架构演进：从单机架构到 K8S 容器编排》
分布式·架构演进·服务端高并发分布式结构演进之路·架构基本概念·架构评价指标
Flynt12 小时前
把 bug tracker 塞进 git 仓库，同步和冲突怎么解决？我把 git-bug 拆开实测了一遍
分布式·git·开源
灯澜忆梦1 天前
【RabbitMQ #3】 | Go 客户端 + SpringAMQP
分布式·golang·rabbitmq
热门推荐
01国内可直接用、免费额度/永久免费的大模型API清单（含 SiliconFlow、火山、阿里、智谱、百度、Kimi、DeepSeek、DMXAPI 等）02历年考研数学一、数学二、数学三真题试卷及答案PDF03GitHub 镜像站点04Telegram APK 下载安装完整指南 — 2026年最新05Windows 10/11 如何退出微软账户并改用本地账户登录（含找不到选项的解决方法）06蜘蛛磁力 搜索引擎大全，如何使用蜘蛛磁力查找磁力链接072026 爱知・名古屋亚运会怎么看？直播平台+赛程安排分享08微信历史版本含下载地址（ Windows PC | 安卓 | MAC ）及设置微信不更新092026 年 9 月国内外主流大模型横向硬核评测：价格、算力、业务场景全维度对比，GPT‑6 Astra横空出世10给AI一张“业务概念地图“：本体如何从哲学走向企业智能