Debian/Ubuntu systemd coredump调试程序Crash

程序是通过systemd监管，当程序出现crash的时候，需要保存crash的日志，也就是coredump日志，按照一般做法设置coredump。而在安装有systemd服务的系统中一般都有systemd-coredump服务。

systemd-coredump 是 systemd 子系统中的一个工具，用于收集、处理和存储 Linux 系统中的 core dump 文件。它是现代 Linux 系统中处理进程崩溃信息的推荐方式，替代了传统的直接生成 core 文件。

systemd-coredump的主要功能

功能	说明
📦 捕获崩溃进程的核心转储	接管核心转储处理，替代传统 core 文件
🧾 保存或丢弃 core 文件	可配置是否保存到磁盘、压缩、限制大小
🗂 统一存储	所有 core 文件保存到 `/var/lib/systemd/coredump/`
🔍 提供调试接口	`coredumpctl` 命令用于列出、查看、调试 core 文件
🧱 安全性	默认以受限权限运行，不会泄露敏感信息

安装systemd-coredump服务

检查是否已安装

shell 复制代码

$ which coredumpctl
/usr/bin/coredumpctl

这里会输出coredumpctl所在的位置，而我最开始的时候，系统是没有安装coredumpctl。

安装systemd-coredump服务

shell 复制代码

$ sudo apt-get install systemd-coredump
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
The following NEW packages will be installed:
  systemd-coredump
0 upgraded, 1 newly installed, 0 to remove and 261 not upgraded.
Need to get 93.0 kB of archives.
After this operation, 374 kB of additional disk space will be used.
Get:1 https://mirrors.aliyun.com/debian-security bookworm-security/main arm64 systemd-coredump arm64 252.38-1~deb12u1 [93.0 kB]
Fetched 93.0 kB in 1s (181 kB/s)
Selecting previously unselected package systemd-coredump.
(Reading database ... 156369 files and directories currently installed.)
Preparing to unpack .../systemd-coredump_252.38-1~deb12u1_arm64.deb ...
Unpacking systemd-coredump (252.38-1~deb12u1) ...
Setting up systemd-coredump (252.38-1~deb12u1) ...

Configuration file '/etc/systemd/coredump.conf'
 ==> File on system created by you or by a script.
 ==> File also in package provided by package maintainer.
   What would you like to do about it ?  Your options are:
    Y or I  : install the package maintainer's version
    N or O  : keep your currently-installed version
      D     : show the differences between the versions
      Z     : start a shell to examine the situation
 The default action is to keep your current version.
*** coredump.conf (Y/I/N/O/D/Z) [default=N] ? Y
Installing new version of config file /etc/systemd/coredump.conf ...
Creating group 'systemd-coredump' with GID 991.
Creating user 'systemd-coredump' (systemd Core Dumper) with UID 991 and GID 991.
Processing triggers for man-db (2.11.2-2) ...

coredump.conf --> 这里输入Y

配置coredump.conf

ini 复制代码

[Coredump]
Storage=external
Compress=yes
ProcessSizeMax=2G
ExternalSizeMax=2G
JournalSizeMax=767M
MaxUse=2G
#KeepFree=

检查core_pattern

在安装systemd-coredump服务时，已经做了对应的配置

shell 复制代码

cat /proc/sys/kernel/core_pattern
|/lib/systemd/systemd-coredump %P %u %g %s %t 9223372036854775808 %h %d

reload systemd

shell 复制代码

sudo systemctl daemon-reexec

测试

测试程序

c++ 复制代码

// crash.cpp
int main() {
    int* p = nullptr;
    *p = 1;
}

编译&触发coredump

shell 复制代码

$ g++ crash.cpp -o crash
ulimit -c unlimited
./crash

检查systemd-coredump的输出

shell 复制代码

$ coredumpctl list
TIME                           PID  UID  GID SIG     COREFILE EXE                            SIZE
Wed 2025-06-11 01:49:07 BST 357407 1000 1000 SIGSEGV present  /home/pi/workspace/core/crash 35.5K

具体coredump信息

shell 复制代码

$ coredumpctl info crash
           PID: 357407 (crash)
           UID: 1000 (pi)
           GID: 1000 (pi)
        Signal: 11 (SEGV)
     Timestamp: Wed 2025-06-11 01:49:06 BST (2min 22s ago)
  Command Line: ./crash
    Executable: /home/pi/workspace/core/crash
 Control Group: /user.slice/user-1000.slice/session-67.scope
          Unit: session-67.scope
         Slice: user-1000.slice
       Session: 67
     Owner UID: 1000 (pi)
       Boot ID: 9f37ddcf177845c5b33baf5ede4df169
    Machine ID: 3193cd42b55048b38d93ded435763681
      Hostname: raspberrypi-CM5
       Storage: /var/lib/systemd/coredump/core.crash.1000.9f37ddcf177845c5b33baf5ede4df169.357407.1749602946000000.zst (present)
  Size on Disk: 35.5K
       Message: Process 357407 (crash) of user 1000 dumped core.

                Stack trace of thread 357407:
                #0  0x000055562d490724 n/a (/home/pi/workspace/core/crash + 0x724)
                #1  0x00007fffb2fc7740 __libc_start_call_main (libc.so.6 + 0x27740)
                #2  0x000055562d490714 n/a (/home/pi/workspace/core/crash + 0x714)
                ELF object binary architecture: AARCH64

coredump debug

shell 复制代码

$ coredumpctl gdb crash
           PID: 357407 (crash)
           UID: 1000 (pi)
           GID: 1000 (pi)
        Signal: 11 (SEGV)
     Timestamp: Wed 2025-06-11 01:49:06 BST (2min 5s ago)
  Command Line: ./crash
    Executable: /home/pi/workspace/core/crash
 Control Group: /user.slice/user-1000.slice/session-67.scope
          Unit: session-67.scope
         Slice: user-1000.slice
       Session: 67
     Owner UID: 1000 (pi)
       Boot ID: 9f37ddcf177845c5b33baf5ede4df169
    Machine ID: 3193cd42b55048b38d93ded435763681
      Hostname: raspberrypi-CM5
       Storage: /var/lib/systemd/coredump/core.crash.1000.9f37ddcf177845c5b33baf5ede4df169.357407.1749602946000000.zst (present)
  Size on Disk: 35.5K
       Message: Process 357407 (crash) of user 1000 dumped core.

                Stack trace of thread 357407:
                #0  0x000055562d490724 n/a (/home/pi/workspace/core/crash + 0x724)
                #1  0x00007fffb2fc7740 __libc_start_call_main (libc.so.6 + 0x27740)
                #2  0x000055562d490714 n/a (/home/pi/workspace/core/crash + 0x714)
                ELF object binary architecture: AARCH64

GNU gdb (Debian 13.1-3) 13.1
Copyright (C) 2023 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "aarch64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<https://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
    <http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /home/pi/workspace/core/crash...
(No debugging symbols found in /home/pi/workspace/core/crash)
[New LWP 357407]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/aarch64-linux-gnu/libthread_db.so.1".
Core was generated by `./crash'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x000055562d490724 in main ()
(gdb) bt
#0  0x000055562d490724 in main ()
(gdb) quit

自定义service配置

systemd 复制代码

[Unit]
Description=application
After=custom.service
Requires=xxxx.service

[Service]
LimitCORE=infinity
LimitSTACK=infinity
Type=simple
ExecStart=/custom/app/custom
Restart=on-failure
TimeoutStopSec=30s
WorkingDirectory=/custom

[Install]
WantedBy=multi-user.target

设置项	说明
`LimitCORE=infinity`	允许 systemd 启动的程序生成 core dump
`WorkingDirectory`	core 文件会默认保存到这里（如果未被 systemd-coredump 截获）
`Restart=on-failure`	保证崩溃时自动重启进程（但不影响 core dump）