文章目录
- 前言
- [一、apt 安装dbgsym vnlinux](#一、apt 安装dbgsym vnlinux)
- [二、使用.ddeb包安装dbgsym vnlinux](#二、使用.ddeb包安装dbgsym vnlinux)
- 三、dbgsym发行版
- 四、crash调试
- 参考资料
前言
最近在适配 ubuntu系统，记录一下其crash的安装。
一、apt 安装dbgsym vnlinux
c
# echo "deb http://ddebs.ubuntu.com $(lsb_release -cs) main restricted universe multiverse
deb http://ddebs.ubuntu.com $(lsb_release -cs)-updates main restricted universe multiverse
deb http://ddebs.ubuntu.com $(lsb_release -cs)-proposed main restricted universe multiverse" | sudo tee -a /etc/apt/sources.list.d/ddebs.list
c
$ sudo apt install ubuntu-dbgsym-keyring
$ sudo apt-get update
c
# apt -y install linux-image-$(uname -r)-dbgsym
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
The following additional packages will be installed:
linux-image-unsigned-5.15.0-122-generic-dbgsym
The following NEW packages will be installed:
linux-image-5.15.0-122-generic-dbgsym linux-image-unsigned-5.15.0-122-generic-dbgsym
0 upgraded, 2 newly installed, 0 to remove and 42 not upgraded.
Need to get 1,061 MB of archives.
After this operation, 6,917 MB of additional disk space will be used.
Get:1 http://ddebs.ubuntu.com jammy-updates/main amd64 linux-image-unsigned-5.15.0-122-generic-dbgsym amd64 5.15.0-122.132 [1,061 MB]二、使用.ddeb包安装dbgsym vnlinux
也可以直接在 http://ddebs.ubuntu.com/pool/main/l/linux/下载对应的.ddeb包：
c
dpkg -i linux-image-unsigned-5.15.0-122-generic-dbgsym_5.15.0-122.132_amd64.ddeb三、dbgsym发行版
在http://ddebs.ubuntu.com/pool/main/l/linux/查看支持的dbgsym vmlinux：
| 发行版 | 内核版本 |
|---|---|
| 14.04 | 3.13.0 |
| 16.04 | 4.4.0 |
| 18.04 | 4.15.0 |
| 20.04 | 5.4.0 |
| 22.04 | 5.15.0 |
| 24.04 | 6.8.0 |
其他内核版本，请参考：http://ddebs.ubuntu.com/pool/main/l/
四、crash调试
cpp
# cat /etc/os-release
PRETTY_NAME="Ubuntu 22.04.1 LTS"
NAME="Ubuntu"
VERSION_ID="22.04"
VERSION="22.04.1 LTS (Jammy Jellyfish)"
VERSION_CODENAME=jammy
ID=ubuntu
ID_LIKE=debian
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
UBUNTU_CODENAME=jammy
# uname -r
6.5.0-18-generic我这里用ubuntu22.04 的6.5.0-18-generic内核调试为例：
c
/*
* 64-bit SYSCALL instruction entry. Up to 6 arguments in registers.
*
* This is the only entry point used for 64-bit system calls. The
* hardware interface is reasonably well designed and the register to
* argument mapping Linux uses fits well with the registers that are
* available when SYSCALL is used.
*
* SYSCALL instructions can be found inlined in libc implementations as
* well as some other programs and libraries. There are also a handful
* of SYSCALL instructions in the vDSO used, for example, as a
* clock_gettimeofday fallback.
*
* 64-bit SYSCALL saves rip to rcx, clears rflags.RF, then saves rflags to r11,
* then loads new ss, cs, and rip from previously programmed MSRs.
* rflags gets masked by a value from another MSR (so CLD and CLAC
* are not needed). SYSCALL does not save anything on the stack
* and does not change rsp.
*
* Registers on entry:
* rax system call number
* rcx return address
* r11 saved rflags (note: r11 is callee-clobbered register in C ABI)
* rdi arg0
* rsi arg1
* rdx arg2
* r10 arg3 (needs to be moved to rcx to conform to C ABI)
* r8 arg4
* r9 arg5
* (note: r12-r15, rbp, rbx are callee-preserved in C ABI)
*
* Only called from user space.
*
* When user can change pt_regs->foo always force IRET. That is because
* it deals with uncanonical addresses better. SYSRET has trouble
* with them due to bugs in both AMD and Intel CPUs.
*/
SYM_CODE_START(entry_SYSCALL_64)
UNWIND_HINT_ENTRY
ENDBR
swapgs
/* tss.sp2 is scratch space. */
movq %rsp, PER_CPU_VAR(cpu_tss_rw + TSS_sp2)
SWITCH_TO_KERNEL_CR3 scratch_reg=%rsp
movq PER_CPU_VAR(pcpu_hot + X86_top_of_stack), %rsp
SYM_INNER_LABEL(entry_SYSCALL_64_safe_stack, SYM_L_GLOBAL)
ANNOTATE_NOENDBR
/* Construct struct pt_regs on stack */
pushq $__USER_DS /* pt_regs->ss */
pushq PER_CPU_VAR(cpu_tss_rw + TSS_sp2) /* pt_regs->sp */
pushq %r11 /* pt_regs->flags */
pushq $__USER_CS /* pt_regs->cs */
pushq %rcx /* pt_regs->ip */
SYM_INNER_LABEL(entry_SYSCALL_64_after_hwframe, SYM_L_GLOBAL)
......
c
SYM_INNER_LABEL(entry_SYSCALL_64_after_hwframe, SYM_L_GLOBAL)
pushq %rax /* pt_regs->orig_ax */
PUSH_AND_CLEAR_REGS rax=$-ENOSYS
/* IRQs are off. */
movq %rsp, %rdi
/* Sign extend the lower 32bit as syscall numbers are treated as int */
movslq %eax, %rsi
/* clobbers %rax, make sure it is after saving the syscall nr */
IBRS_ENTER
UNTRAIN_RET
call do_syscall_64 /* returns with IRQs disabled */
......看其vmlinux：
c
cat /boot/System.map-6.5.0-18-generic | grep entry_SYSCALL_64_after_hwframe
ffffffff82200078 T entry_SYSCALL_64_after_hwframe
c
ffffffff82200078: 50 push %rax
ffffffff82200079: 57 push %rdi
ffffffff8220007a: 56 push %rsi
ffffffff8220007b: 52 push %rdx
ffffffff8220007c: 51 push %rcx
ffffffff8220007d: 6a da push $0xffffffffffffffda
ffffffff8220007f: 41 50 push %r8
ffffffff82200081: 41 51 push %r9
ffffffff82200083: 41 52 push %r10
ffffffff82200085: 41 53 push %r11
ffffffff82200087: 53 push %rbx
ffffffff82200088: 55 push %rbp
ffffffff82200089: 41 54 push %r12
ffffffff8220008b: 41 55 push %r13
ffffffff8220008d: 41 56 push %r14
ffffffff8220008f: 41 57 push %r15
ffffffff82200091: 31 f6 xor %esi,%esi
ffffffff82200093: 31 d2 xor %edx,%edx
......
ffffffff822000d1: 90 nop
ffffffff822000d2: 90 nop
ffffffff822000d3: 90 nop
ffffffff822000d4: 90 nop
ffffffff822000d5: 90 nop
ffffffff822000d6: 90 nop
ffffffff822000d7: 90 nop
ffffffff822000d8: 90 nop
ffffffff822000d9: 90 nop
ffffffff822000da: 90 nop
ffffffff822000db: 90 nop
ffffffff822000dc: 90 nop
ffffffff822000dd: 90 nop
ffffffff822000de: 90 nop
ffffffff822000df: 90 nop
ffffffff822000e0: 90 nop
ffffffff822000e1: e8 9a 4a f1 ff call 0xffffffff82114b80
ffffffff822000e6: 90 nop
ffffffff822000e7: 90 nop
ffffffff822000e8: 90 nop
ffffffff822000e9: 90 nop
ffffffff822000ea: 90 nop到了ffffffff822000d1处，全是 nop指令：
c
# cat /boot/System.map-6.5.0-18-generic | grep '\<do_syscall_64\>'
ffffffff82114b80 T do_syscall_64而我看crash：
c
crash> dis entry_SYSCALL_64_after_hwframe
0xffffffff97400078 <entry_SYSCALL_64_after_hwframe>: push %rax
0xffffffff97400079 <entry_SYSCALL_64_after_hwframe+1>: push %rdi
0xffffffff9740007a <entry_SYSCALL_64_after_hwframe+2>: push %rsi
0xffffffff9740007b <entry_SYSCALL_64_after_hwframe+3>: push %rdx
0xffffffff9740007c <entry_SYSCALL_64_after_hwframe+4>: push %rcx
0xffffffff9740007d <entry_SYSCALL_64_after_hwframe+5>: push $0xffffffffffffffda
0xffffffff9740007f <entry_SYSCALL_64_after_hwframe+7>: push %r8
0xffffffff97400081 <entry_SYSCALL_64_after_hwframe+9>: push %r9
0xffffffff97400083 <entry_SYSCALL_64_after_hwframe+11>: push %r10
0xffffffff97400085 <entry_SYSCALL_64_after_hwframe+13>: push %r11
0xffffffff97400087 <entry_SYSCALL_64_after_hwframe+15>: push %rbx
0xffffffff97400088 <entry_SYSCALL_64_after_hwframe+16>: push %rbp
0xffffffff97400089 <entry_SYSCALL_64_after_hwframe+17>: push %r12
0xffffffff9740008b <entry_SYSCALL_64_after_hwframe+19>: push %r13
0xffffffff9740008d <entry_SYSCALL_64_after_hwframe+21>: push %r14
0xffffffff9740008f <entry_SYSCALL_64_after_hwframe+23>: push %r15
0xffffffff97400091 <entry_SYSCALL_64_after_hwframe+25>: xor %esi,%esi
0xffffffff97400093 <entry_SYSCALL_64_after_hwframe+27>: xor %edx,%edx
0xffffffff97400095 <entry_SYSCALL_64_after_hwframe+29>: xor %ecx,%ecx
0xffffffff97400097 <entry_SYSCALL_64_after_hwframe+31>: xor %r8d,%r8d
0xffffffff9740009a <entry_SYSCALL_64_after_hwframe+34>: xor %r9d,%r9d
0xffffffff9740009d <entry_SYSCALL_64_after_hwframe+37>: xor %r10d,%r10d
0xffffffff974000a0 <entry_SYSCALL_64_after_hwframe+40>: xor %r11d,%r11d
0xffffffff974000a3 <entry_SYSCALL_64_after_hwframe+43>: xor %ebx,%ebx
0xffffffff974000a5 <entry_SYSCALL_64_after_hwframe+45>: xor %ebp,%ebp
0xffffffff974000a7 <entry_SYSCALL_64_after_hwframe+47>: xor %r12d,%r12d
0xffffffff974000aa <entry_SYSCALL_64_after_hwframe+50>: xor %r13d,%r13d
0xffffffff974000ad <entry_SYSCALL_64_after_hwframe+53>: xor %r14d,%r14d
0xffffffff974000b0 <entry_SYSCALL_64_after_hwframe+56>: xor %r15d,%r15d
0xffffffff974000b3 <entry_SYSCALL_64_after_hwframe+59>: mov %rsp,%rdi
0xffffffff974000b6 <entry_SYSCALL_64_after_hwframe+62>: movslq %eax,%rsi
0xffffffff974000b9 <entry_SYSCALL_64_after_hwframe+65>: xchg %ax,%ax
0xffffffff974000bb <entry_SYSCALL_64_after_hwframe+67>: mov $0x48,%ecx
0xffffffff974000c0 <entry_SYSCALL_64_after_hwframe+72>: mov %gs:0x1fbd0,%rdx
0xffffffff974000c9 <entry_SYSCALL_64_after_hwframe+81>: mov %edx,%eax
0xffffffff974000cb <entry_SYSCALL_64_after_hwframe+83>: shr $0x20,%rdx
0xffffffff974000cf <entry_SYSCALL_64_after_hwframe+87>: wrmsr
0xffffffff974000d1 <entry_SYSCALL_64_after_hwframe+89>: jmp 0xffffffff974000e1 <entry_SYSCALL_64_after_hwframe+105>
0xffffffff974000d3 <entry_SYSCALL_64_after_hwframe+91>: int3
0xffffffff974000d4 <entry_SYSCALL_64_after_hwframe+92>: int3
0xffffffff974000d5 <entry_SYSCALL_64_after_hwframe+93>: int3
0xffffffff974000d6 <entry_SYSCALL_64_after_hwframe+94>: int3
0xffffffff974000d7 <entry_SYSCALL_64_after_hwframe+95>: int3
0xffffffff974000d8 <entry_SYSCALL_64_after_hwframe+96>: int3
0xffffffff974000d9 <entry_SYSCALL_64_after_hwframe+97>: int3
0xffffffff974000da <entry_SYSCALL_64_after_hwframe+98>: int3
0xffffffff974000db <entry_SYSCALL_64_after_hwframe+99>: int3
0xffffffff974000dc <entry_SYSCALL_64_after_hwframe+100>: int3
0xffffffff974000dd <entry_SYSCALL_64_after_hwframe+101>: int3
0xffffffff974000de <entry_SYSCALL_64_after_hwframe+102>: int3
0xffffffff974000df <entry_SYSCALL_64_after_hwframe+103>: int3
0xffffffff974000e0 <entry_SYSCALL_64_after_hwframe+104>: int3
0xffffffff974000e1 <entry_SYSCALL_64_after_hwframe+105>: call 0xffffffff97314b80 <do_syscall_64>在0xffffffff974000d1处nop指令变成了jmp：
c
0xffffffff974000d1 <entry_SYSCALL_64_after_hwframe+89>: jmp 0xffffffff974000e1 <entry_SYSCALL_64_after_hwframe+105>
......
0xffffffff974000e1 <entry_SYSCALL_64_after_hwframe+105>: call 0xffffffff97314b80 <do_syscall_64>与UNTRAIN_RET指令有关。
（2）
c
/*
* Save all registers in pt_regs. Return GSBASE related information
* in EBX depending on the availability of the FSGSBASE instructions:
*
* FSGSBASE R/EBX
* N 0 -> SWAPGS on exit
* 1 -> no SWAPGS on exit
*
* Y GSBASE value at entry, must be restored in paranoid_exit
*
* R14 - old CR3
* R15 - old SPEC_CTRL
*/
SYM_CODE_START(paranoid_entry)
ANNOTATE_NOENDBR
UNWIND_HINT_FUNC
PUSH_AND_CLEAR_REGS save_ret=1
ENCODE_FRAME_POINTER 8
/*
* Always stash CR3 in %r14. This value will be restored,
* verbatim, at exit. Needed if paranoid_entry interrupted
* another entry that already switched to the user CR3 value
* but has not yet returned to userspace.
*
* This is also why CS (stashed in the "iret frame" by the
* hardware at entry) can not be used: this may be a return
* to kernel code, but with a user CR3 value.
*
* Switching CR3 does not depend on kernel GSBASE so it can
* be done before switching to the kernel GSBASE. This is
* required for FSGSBASE because the kernel GSBASE has to
* be retrieved from a kernel internal table.
*/
SAVE_AND_SWITCH_TO_KERNEL_CR3 scratch_reg=%rax save_reg=%r14
/*
* Handling GSBASE depends on the availability of FSGSBASE.
*
* Without FSGSBASE the kernel enforces that negative GSBASE
* values indicate kernel GSBASE. With FSGSBASE no assumptions
* can be made about the GSBASE value when entering from user
* space.
*/
ALTERNATIVE "jmp .Lparanoid_entry_checkgs", "", X86_FEATURE_FSGSBASE
/*
* Read the current GSBASE and store it in %rbx unconditionally,
* retrieve and set the current CPUs kernel GSBASE. The stored value
* has to be restored in paranoid_exit unconditionally.
*
* The unconditional write to GS base below ensures that no subsequent
* loads based on a mispredicted GS base can happen, therefore no LFENCE
* is needed here.
*/
SAVE_AND_SET_GSBASE scratch_reg=%rax save_reg=%rbx
jmp .Lparanoid_gsbase_done
.Lparanoid_entry_checkgs:
/* EBX = 1 -> kernel GSBASE active, no restore required */
movl $1, %ebx
/*
* The kernel-enforced convention is a negative GSBASE indicates
* a kernel value. No SWAPGS needed on entry and exit.
*/
movl $MSR_GS_BASE, %ecx
rdmsr
testl %edx, %edx
js .Lparanoid_kernel_gsbase
/* EBX = 0 -> SWAPGS required on exit */
xorl %ebx, %ebx
swapgs
.Lparanoid_kernel_gsbase:
FENCE_SWAPGS_KERNEL_ENTRY
.Lparanoid_gsbase_done:
/*
* Once we have CR3 and %GS setup save and set SPEC_CTRL. Just like
* CR3 above, keep the old value in a callee saved register.
*/
IBRS_ENTER save_reg=%r15
UNTRAIN_RET_FROM_CALL
RET
SYM_CODE_END(paranoid_entry)查看vmlinux：
c
# cat /boot/System.map-6.5.0-18-generic | grep '\<paranoid_entry\>'
ffffffff822013b0 T paranoid_entry
c
ffffffff822013b0: 56 push %rsi
ffffffff822013b1: 48 8b 74 24 08 mov 0x8(%rsp),%rsi
ffffffff822013b6: 48 89 7c 24 08 mov %rdi,0x8(%rsp)
ffffffff822013bb: 52 push %rdx
ffffffff822013bc: 51 push %rcx
ffffffff822013bd: 50 push %rax
ffffffff822013be: 41 50 push %r8
ffffffff822013c0: 41 51 push %r9
ffffffff822013c2: 41 52 push %r10
ffffffff822013c4: 41 53 push %r11
ffffffff822013c6: 53 push %rbx
ffffffff822013c7: 55 push %rbp
ffffffff822013c8: 41 54 push %r12
ffffffff822013ca: 41 55 push %r13
ffffffff822013cc: 41 56 push %r14
ffffffff822013ce: 41 57 push %r15
ffffffff822013d0: 56 push %rsi
ffffffff822013d1: 31 f6 xor %esi,%esi
ffffffff822013d3: 31 d2 xor %edx,%edx
......
ffffffff82201484: 90 nop
ffffffff82201485: 90 nop
ffffffff82201486: 90 nop
ffffffff82201487: 90 nop
ffffffff82201488: 90 nop
ffffffff82201489: 90 nop
ffffffff8220148a: 90 nop
ffffffff8220148b: 90 nop
ffffffff8220148c: 90 nop
ffffffff8220148d: 90 nop
ffffffff8220148e: 90 nop
ffffffff8220148f: 90 nop
ffffffff82201490: 90 nop
ffffffff82201491: 90 nop
ffffffff82201492: 90 nop到了ffffffff82201484处都是nop指令。
crash查看：
c
crash> dis paranoid_entry
0xffffffff974013b0 <paranoid_entry>: push %rsi
0xffffffff974013b1 <paranoid_entry+1>: mov 0x8(%rsp),%rsi
0xffffffff974013b6 <paranoid_entry+6>: mov %rdi,0x8(%rsp)
0xffffffff974013bb <paranoid_entry+11>: push %rdx
0xffffffff974013bc <paranoid_entry+12>: push %rcx
0xffffffff974013bd <paranoid_entry+13>: push %rax
0xffffffff974013be <paranoid_entry+14>: push %r8
0xffffffff974013c0 <paranoid_entry+16>: push %r9
0xffffffff974013c2 <paranoid_entry+18>: push %r10
0xffffffff974013c4 <paranoid_entry+20>: push %r11
0xffffffff974013c6 <paranoid_entry+22>: push %rbx
0xffffffff974013c7 <paranoid_entry+23>: push %rbp
0xffffffff974013c8 <paranoid_entry+24>: push %r12
0xffffffff974013ca <paranoid_entry+26>: push %r13
0xffffffff974013cc <paranoid_entry+28>: push %r14
0xffffffff974013ce <paranoid_entry+30>: push %r15
0xffffffff974013d0 <paranoid_entry+32>: push %rsi
0xffffffff974013d1 <paranoid_entry+33>: xor %esi,%esi
0xffffffff974013d3 <paranoid_entry+35>: xor %edx,%edx
......
0xffffffff97401484 <paranoid_entry+212>: jmp 0xffffffff97401493 <paranoid_entry+227>
0xffffffff97401486 <paranoid_entry+214>: int3
0xffffffff97401487 <paranoid_entry+215>: int3
0xffffffff97401488 <paranoid_entry+216>: int3
0xffffffff97401489 <paranoid_entry+217>: int3
0xffffffff9740148a <paranoid_entry+218>: int3
0xffffffff9740148b <paranoid_entry+219>: int3
0xffffffff9740148c <paranoid_entry+220>: int3
0xffffffff9740148d <paranoid_entry+221>: int3
0xffffffff9740148e <paranoid_entry+222>: int3
0xffffffff9740148f <paranoid_entry+223>: int3
0xffffffff97401490 <paranoid_entry+224>: int3
0xffffffff97401491 <paranoid_entry+225>: int3
0xffffffff97401492 <paranoid_entry+226>: int3
0xffffffff97401493 <paranoid_entry+227>: ret
0xffffffff97401494 <paranoid_entry+228>: int3
0xffffffff97401495 <paranoid_entry+229>: int3
0xffffffff97401496 <paranoid_entry+230>: int3
0xffffffff97401497 <paranoid_entry+231>: int3到了0xffffffff97401484：
c
0xffffffff97401484 <paranoid_entry+212>: jmp 0xffffffff97401493 <paranoid_entry+227>
0xffffffff97401493 <paranoid_entry+227>: ret也是nop指令变成了jmp指令。
这与UNTRAIN_RET_FROM_CALL指令有关。