使用breakpad生成和解析minidump

概述

在ubuntu环境下,使用使用breakpad生成和解析minidump来分析C++程序问题。记录下来作为笔记备用。

步骤

1. 下载breakpad

git clone https://github.com/google/breakpad.git

2. 编译

命令行进入breakpad文件夹下:

./configure

make

sudo make install

可能遇到的问题:

推荐使用g++-13来作为g++的版本。

问题1:g++需要支持C++20,电脑原来是g++-9报错,后来升级到g++-13解决。

问题2:make时报错:

cpp 复制代码
In file included from ./src/client/linux/dump_writer_common/thread_info.h:36,
                 from ./src/client/linux/minidump_writer/linux_dumper.h:53,
                 from ./src/client/linux/minidump_writer/minidump_writer.h:41,
                 from src/tools/linux/core2md/core2md.cc:37:
./src/common/memory_allocator.h:51:10: fatal error: third_party/lss/linux_syscall_support.h: 没有那个文件或目录
   51 | #include "third_party/lss/linux_syscall_support.h"
      |          ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
compilation terminated.
make: *** [Makefile:5671:src/tools/linux/core2md/core2md.o] 错误 1

解决方法:

在breakpad/src/third_party/lss/目录下,上网下载文件linux_syscall_support.h放进去,一个可用的地址是:https://gitee.com/keeyou/lss/blob/develop/linux_syscall_support.h

示例程序

//minidump1.cpp

cpp 复制代码
#include <iostream>
#include <breakpad/client/linux/handler/exception_handler.h>
bool DumpCallback(const google_breakpad::MinidumpDescriptor& descriptor, void* context, bool succeeded)
{
        std::cout<<"Dump path:" << descriptor.path() << std::endl;
        return succeeded;
}

int main()
{
        google_breakpad::MinidumpDescriptor descriptor("./");

        google_breakpad::ExceptionHandler eh(descriptor, nullptr, DumpCallback, nullptr, true, -1);

        int *ptr = nullptr;
        *ptr = 1;       //这里有问题
        return 0;
}

编译命令:

g++ -g minidump1.cpp -o minidump1 -I /usr/local/include/breakpad -lbreakpad_client -pthread

-I /usr/local/include/breakpad指定头文件的路径,防止找不到头文件

-g选项表示增加调试信息

-pthread是因为breakpad内部用到

-lbreakpad_client表示引用这个库

运行:

./minidump1

程序报错并生成.dmp文件

3.查看minidump文件信息
3.1,生成符号文件:(dump_syms programName > programName.sym)

dump_syms minidump1 > minidump1.sym

注意,dump_syms命令如果找不到,去下载的breakpad/src/tools/linux/dump_syms目录下找

minidump1.sym这个名字不是随便取的,需要和可执行文件同名

3.2. 设置符号文件的目录:

选一个位置,创建目录文件symbols,在它下面创建文件夹minidump1,注意,这个名字和可执行文件同名,查看上一步文件minidump1.sym的内容,从它的第一行获取UUID

例如:第一行:MODULE Linux x86_64 9F290E00150C9597A510DF9C550D7BCF0 minidump1

得到9F290E00150C9597A510DF9C550D7BCF0

在minidump1目录下创建目录9F290E00150C9597A510DF9C550D7BCF0,并把minidump1.sym放进去

最终目录为:

symbols/minidump1/9F290E00150C9597A510DF9C550D7BCF0/minidump1.sym

3.3. 使用minidump_stackwalk解析文件

如果找不到minidump_stackwalk,去breakpad目录下src/processor文件夹下找。

执行命令: minidump_stackwalk <UUID>.dmp symbols目录 > 栈信息.txt

例如:minidump_stackwalk 0f49290e-008a-42be-b3d19c93-3cbe7f82.dmp symbols/ > symbols/minidump1/9F290E00150C9597A510DF9C550D7BCF0/stack1.txt

前面的dmp文件和symbols文件都用上了

3.4. 查看文件stack1.txt
cpp 复制代码
Operating system: Linux

                  0.0.0 Linux 6.8.0-101-generic #101~22.04.1-Ubuntu SMP PREEMPT_DYNAMIC Wed Feb 11 13:19:54 UTC  x86_64

CPU: amd64

     family 25 model 33 stepping 0

     4 CPUs



GPU: UNKNOWN



Crash reason:  SIGSEGV /SEGV_MAPERR

Crash address: 0x0

Process uptime: not available



Thread 0 (crashed)

 0  minidump1!main [minidump1.cpp : 16 + 0x7]

    rax = 0x0000000000000000   rdx = 0x0000000000000000

    rcx = 0x00005865141a8370   rbx = 0x0000000000000000

    rsi = 0x0000000000000000   rdi = 0x00005865077b97e0

    rbp = 0x00007fff5464df60   rsp = 0x00007fff5464ddb0

     r8 = 0x0000000000000000    r9 = 0x00005865141a8370

    r10 = 0x0000000000000008   r11 = 0x00007f8ebfc1ace0

    r12 = 0x00007fff5464e078   r13 = 0x000058650779f2e2

    r14 = 0x00005865077b89f8   r15 = 0x00007f8ec015c040

    rip = 0x000058650779f3b6

    Found by: given as instruction pointer in context

 1  libc.so.6 + 0x29d90

    rbx = 0x0000000000000000   rbp = 0x0000000000000001

    rsp = 0x00007fff5464df70   r12 = 0x00007fff5464e078

    r13 = 0x000058650779f2e2   r14 = 0x00005865077b89f8

    r15 = 0x00007f8ec015c040   rip = 0x00007f8ebfa29d90

    Found by: call frame info

 2  minidump1!DumpCallback(google_breakpad::MinidumpDescriptor const&, void*, bool) [minidump1.cpp : 7 + 0x6]

    rsp = 0x00007fff5464df80   rip = 0x000058650779f2e2

    Found by: stack scanning

 3  0x100000000

    rbx = 0x0000000000000000   rbp = 0x000058650779f2e2

    rsp = 0x00007fff5464df88   rip = 0x0000000100000000

    Found by: call frame info

 4  minidump1!DumpCallback(google_breakpad::MinidumpDescriptor const&, void*, bool) [minidump1.cpp : 7 + 0x6]

    rsp = 0x00007fff5464dfb0   rip = 0x000058650779f2e2

    Found by: stack scanning

 5  0x5865077b89f8

    rbx = 0x00007fff5464e078   rbp = 0x000058650779f2e2

    rsp = 0x00007fff5464dfb8   rip = 0x00005865077b89f8

    Found by: call frame info

 6  libc.so.6 + 0x29e40

    rsp = 0x00007fff5464e010   rip = 0x00007f8ebfa29e40

    Found by: stack scanning

 7  minidump1!google_breakpad::ConvertUTF8toUTF32(unsigned char const**, unsigned char const*, unsigned long**, unsigned long*, google_breakpad::ConversionFlags) [clone .cold] + 0xe

    rsp = 0x00007fff5464e040   rip = 0x000058650779f190

    Found by: stack scanning

 8  minidump1!_start + 0x25

    rsp = 0x00007fff5464e060   rip = 0x000058650779f1b5

    Found by: stack scanning

 9  0x7fff5464e068

    rsp = 0x00007fff5464e068   rip = 0x00007fff5464e068

    Found by: call frame info



Loaded modules:

0x58650779b000 - 0x5865077b1fff  minidump1  ???  (main)

0x7f8ebfa00000 - 0x7f8ebfbbcfff  libc.so.6  ???  (WARNING: No symbols, libc.so.6, A17B5C09AE4881CA668091F7180470780)

0x7f8ebfd19000 - 0x7f8ebfda2fff  libm.so.6  ???

0x7f8ebfe00000 - 0x7f8ebffe9fff  libstdc++.so.6  ???

0x7f8ec00e1000 - 0x7f8ec0107fff  libgcc_s.so.1  ???

0x7f8ec0122000 - 0x7f8ec014dfff  ld-linux-x86-64.so.2  ???

0x7fff54694000 - 0x7fff54695fff  linux-gate.so  ???

可以看出,minidump1!main [minidump1.cpp : 16 + 0x7] 这里指出了问题位置在16行。

注意要严格遵守上面的步骤,如果不正确创建symbols目录或者sym文件名字不对,则最后解析的文件无法指明错误位置,知识一堆地址。

进一步探索

如果解析文件失败了,没有显示正确位置,用以下方法实现定位:

1.定位地址

错误复现:

比如上面步骤3写成了 minidump_stackwalk 0f49290e-008a-42be-b3d19c93-3cbe7f82.dmp symbols/minidump1/ > symbols/minidump1/9F290E00150C9597A510DF9C550D7BCF0/stack2.txt

查看 stack2.txt文件

cpp 复制代码
Operating system: Linux

                  0.0.0 Linux 6.8.0-101-generic #101~22.04.1-Ubuntu SMP PREEMPT_DYNAMIC Wed Feb 11 13:19:54 UTC  x86_64

CPU: amd64

     family 25 model 33 stepping 0

     4 CPUs



GPU: UNKNOWN



Crash reason:  SIGSEGV /SEGV_MAPERR

Crash address: 0x0

Process uptime: not available



Thread 0 (crashed)

 0  minidump1 + 0x43b6

    rax = 0x0000000000000000   rdx = 0x0000000000000000

    rcx = 0x00005865141a8370   rbx = 0x0000000000000000

    rsi = 0x0000000000000000   rdi = 0x00005865077b97e0

    rbp = 0x00007fff5464df60   rsp = 0x00007fff5464ddb0

     r8 = 0x0000000000000000    r9 = 0x00005865141a8370

    r10 = 0x0000000000000008   r11 = 0x00007f8ebfc1ace0

    r12 = 0x00007fff5464e078   r13 = 0x000058650779f2e2

    r14 = 0x00005865077b89f8   r15 = 0x00007f8ec015c040

    rip = 0x000058650779f3b6

    Found by: given as instruction pointer in context

 1  minidump1 + 0x4279

    rbp = 0x00007fff5464df60   rsp = 0x00007fff5464de70

    rip = 0x000058650779f279

    Found by: stack scanning

 2  libc.so.6 + 0x29d90

    rsp = 0x00007fff5464df70   rip = 0x00007f8ebfa29d90

    Found by: stack scanning

 3  minidump1 + 0x42e2

    rsp = 0x00007fff5464df80   rip = 0x000058650779f2e2

    Found by: stack scanning

 4  minidump1 + 0x42e2

    rsp = 0x00007fff5464dfb0   rip = 0x000058650779f2e2

    Found by: stack scanning

 5  libc.so.6 + 0x29e40

    rsp = 0x00007fff5464e010   rip = 0x00007f8ebfa29e40

    Found by: stack scanning

 6  minidump1 + 0x4190

    rsp = 0x00007fff5464e040   rip = 0x000058650779f190

    Found by: stack scanning

 7  minidump1 + 0x41b5

    rsp = 0x00007fff5464e060   rip = 0x000058650779f1b5

    Found by: stack scanning



Loaded modules:

0x58650779b000 - 0x5865077b1fff  minidump1  ???  (main)  (WARNING: No symbols, minidump1, 9F290E00150C9597A510DF9C550D7BCF0)

0x7f8ebfa00000 - 0x7f8ebfbbcfff  libc.so.6  ???  (WARNING: No symbols, libc.so.6, A17B5C09AE4881CA668091F7180470780)

0x7f8ebfd19000 - 0x7f8ebfda2fff  libm.so.6  ???

0x7f8ebfe00000 - 0x7f8ebffe9fff  libstdc++.so.6  ???

0x7f8ec00e1000 - 0x7f8ec0107fff  libgcc_s.so.1  ???

0x7f8ec0122000 - 0x7f8ec014dfff  ld-linux-x86-64.so.2  ???

0x7fff54694000 - 0x7fff54695fff  linux-gate.so  ???

上面内容可以看出, 0 minidump1 + 0x43b6是停止位置。记录地址是0x43b6

2,解析地址位置

objdump -S minidump1 > objdump1_S.txt

这句指令会把 minidump1 的二进制文件的反汇编代码 + 源代码(如果有调试信息) 导出到 objdump1_S.txt 文本文件中,方便查看和分析。

打开objdump1_S.txt文件,查找43b6,会发现

int *ptr = nullptr;

43a4: 48 c7 85 60 fe ff ff movq $0x0,-0x1a0(%rbp)

43ab: 00 00 00 00

*ptr = 1; //这里有问题

43af: 48 8b 85 60 fe ff ff mov -0x1a0(%rbp),%rax

43b6: c7 00 01 00 00 00 movl $0x1,(%rax)

return 0;

43b6所对应的源码是*ptr = 1;定位出问题位置

相关推荐
茶本无香10 个月前
Arthas:Java诊断利器实战指南
arthas·问题定位
Steven-Russell10 个月前
Clickhouse基于breakpad生成minidump文件,方便问题定位
clickhouse·breakpad
ThisIsClark1 年前
【问题定位记录】哪些情况可能造成403
问题定位
向着百万年薪努力的小赵2 年前
现网/生产/一线问题记录
问题定位
jimbo_lee2 年前
qcom 平台efuse机器抓取dump log的方法
dump·minidump·qcap·qpst·apdp·debugpolicy
jimbo_lee2 年前
qcom 平台SSR (Subsystem Restart)配置
android·ssr·susbsystem·restart_level·minidump
G皮T2 年前
【Flink】Apache Flink 常见问题定位指南
大数据·运维·flink·问题定位