背景
小米项目的同事,最近遇到了一个crash问题,困扰较久,向我请求帮助。(其实是客户指定要求我来分析这个问题,该项目之前是由我负责的,帮客户解决很多较多crash问题。估计是实力得到了客户认可吧,哈哈,自嗨一波~~~Linux 调试进阶(多场景覆盖)_谢艺华的博客-CSDN博客)
问题描述
经验一: 与同事充分沟通,了解当前问题现状。
描述:客户最近在测试过程中,在文件转移阶段,偶现coredump。
仅仅从该描述而言,并不能有进一步的分析。于是我让同事将寄存器,栈空间的值,堆栈信息打印。如下:
|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| (gdb) (gdb) info register x0 0x0 0 x1 0x7fc3cb2050 548745715792 x2 0x0 0 x3 0x8 8 x4 0x0 0 x5 0xffffffffffffffff -1 x6 0xffffffffffffffff -1 x7 0xffffffffffffffff -1 x8 0x87 135 x9 0x6 6 x10 0xffffffffffffffff -1 x11 0xffffffffffffffff -1 x12 0xffffffffffffffff -1 x13 0xffffffffffffffff -1 x14 0xffffffffffffffff -1 x15 0xffffffffffffffff -1 x16 0xffffffffffffffff -1 x17 0xffffffffffffffff -1 x18 0xffffffffffffffff -1 x19 0xffffffffffffffff -1 x20 0x7f7d01d450 547558118480 x21 0x7f7e4356b8 547579188920 x22 0x556005a670 366683203184 --Type <RET> for more, q to quit, c to continue without paging-- x23 0x55560cb2f8 366515892984 x24 0x55560c8000 366515879936 x25 0x556005a750 366683203408 x26 0x55560c8000 366515879936 x27 0x556005a718 366683203352 x28 0x55560cae00 366515891712 x29 0x7fc3cb2030 548745715760 x30 0x7fc3cb2050 548745715792 sp 0x7fc3cb2030 0x7fc3cb2030 pc 0x7f7e30121c 0x7f7e30121c <__GI_raise+204> cpsr 0x1000 [ EL=0 SSBS ] fpsr 0x10 16 fpcr 0x0 0 (gdb) x /256xg $sp 0x7fc3cb2030: 0x0000007fc3cb2150 0x0000007f7e2ee4f8 0x7fc3cb2040: 0x0000007f7e435000 0x0000000000000000 0x7fc3cb2050: 0x0000000000001000 0x0000000000000000 0x7fc3cb2060: 0x0000000000000000 0x0000000000000000 0x7fc3cb2070: 0x0000000000000000 0x0000000000000000 0x7fc3cb2080: 0x0000000000000000 0x0000000000000000 0x7fc3cb2090: 0x0000000000000000 0x0000000000000000 0x7fc3cb20a0: 0x0000000000000000 0x0000000000000000 0x7fc3cb20b0: 0x0000000000000000 0x0000000000000000 0x7fc3cb20c0: 0x0000000000000000 0x0000000000000000 0x7fc3cb20d0: 0xfffffffe7fffffff 0xffffffffffffffff 0x7fc3cb20e0: 0xffffffffffffffff 0xffffffffffffffff 0x7fc3cb20f0: 0xffffffffffffffff 0xffffffffffffffff 0x7fc3cb2100: 0xffffffffffffffff 0xffffffffffffffff 0x7fc3cb2110: 0xffffffffffffffff 0xffffffffffffffff 0x7fc3cb2120: 0xffffffffffffffff 0xffffffffffffffff 0x7fc3cb2130: 0xffffffffffffffff 0xffffffffffffffff 0x7fc3cb2140: 0xffffffffffffffff 0xffffffffffffffff 0x7fc3cb2150: 0x0000007fc3cb22a0 0x0000007f7e4ff038 0x7fc3cb2160: 0x00000055560c8000 0x0000007fc3cb2368 0x7fc3cb2170: 0x0000007fc3cb2388 0x0000007f7e33ad7c 0x7fc3cb2180: 0x000000000000002d 0x0000000000000020 0x7fc3cb2190: 0x0000007f7e434430 0x000000000000002d --Type <RET> for more, q to quit, c to continue without paging-- 0x7fc3cb21a0: 0x0000007fc3cb21d0 0x0000007f7e33a198 0x7fc3cb21b0: 0x0000007f7e434430 0x0000000000000bd0 0x7fc3cb21c0: 0x0000007f7e4346f8 0x0000007f7e435028 0x7fc3cb21d0: 0x0000007fc3cb2210 0x0000007f7e33b49c 0x7fc3cb21e0: 0x000000000000002d 0x000000000000002d 0x7fc3cb21f0: 0x000000000000002d 0x0000007f7e434430 0x7fc3cb2200: 0x0000007f7e5cf7b8 0x0000000000000001 0x7fc3cb2210: 0x0000007fc3cb2250 0x0000007f7e32fbb0 0x7fc3cb2220: 0x0000007f7e434430 0x000000000000002d 0x7fc3cb2230: 0x0000007f7e435028 0x000000000000002d 0x7fc3cb2240: 0x0000007f7e5cf7b8 0x0000000000000001 0x7fc3cb2250: 0x0000007fc3cb22a0 0x0000007f7e4ff02c 0x7fc3cb2260: 0x00000055560c8000 0x0000007fc3cb2368 0x7fc3cb2270: 0x0000007fc3cb2388 0x000000556005a670 0x7fc3cb2280: 0x00000055560cb2f8 0x00000055560c8000 0x7fc3cb2290: 0x000000556005a750 0x0000007f7e4fef60 0x7fc3cb22a0: 0x0000007fc3cb22e0 0x0000007f7e4fcc0c 0x7fc3cb22b0: 0x00000055560c8000 0x0000007fc3cb2368 0x7fc3cb22c0: 0x0000007fc3cb2388 0x000000556005a670 0x7fc3cb22d0: 0x00000055560cb310 0x0000005560056ff0 0x7fc3cb22e0: 0x0000007fc3cb22f0 0x0000007f7e4fcc70 0x7fc3cb22f0: 0x0000007fc3cb2300 0x000000555600bdbc 0x7fc3cb2300: 0x0000007fc3cb23c0 0x0000005556012184 --Type <RET> for more, q to quit, c to continue without paging-- 0x7fc3cb2310: 0x000000556005a670 0x000000556005ae60 0x7fc3cb2320: 0x0000000000000000 0x0000007f7fb26678 0x7fc3cb2330: 0x000000555607e818 0x0000007f8051ca80 0x7fc3cb2340: 0x000000556005a670 0x000000556006a480 0x7fc3cb2350: 0x0000000000000000 0x0000005556012184 0x7fc3cb2360: 0x000000556005a670 0x00000055560c64c8 0x7fc3cb2370: 0x000000556005a670 0x000000555600b850 0x7fc3cb2380: 0x0000000000000000 0x0000000000000000 0x7fc3cb2390: 0x0000000000000000 0x00000000ffffffff 0x7fc3cb23a0: 0x0000000000000000 0x0000000000000000 0x7fc3cb23b0: 0x0000000000000000 0xe48d6289cdcca000 0x7fc3cb23c0: 0x0000007fc3cb23e0 0x0000005555ff72dc 0x7fc3cb23d0: 0x00000055560c8000 0x0000007f7fb26678 0x7fc3cb23e0: 0x0000007fc3cb2460 0x000000555600a5f8 0x7fc3cb23f0: 0x00000055560cb348 0x00000055560caa28 0x7fc3cb2400: 0x00000055560cb418 0x0000007f7fb26678 0x7fc3cb2410: 0x0000007fc3cb2440 0x0000007f7f8546e0 0x7fc3cb2420: 0x000000556005a670 0x000000556005a750 0x7fc3cb2430: 0x000000555607e818 0x00000055560cae00 0x7fc3cb2440: 0x0000007fc3cb2460 0x000000556005e400 0x7fc3cb2450: 0x00000055560cb348 0x0000007f7e644a20 0x7fc3cb2460: 0x0000007fc3cb24f0 0x0000007f7e3035dc 0x7fc3cb2470: 0x000000556002c270 0x0000007f7e435000 --Type <RET> for more, q to quit, c to continue without paging-- 0x7fc3cb2480: 0x00000000000002b3 0x0000007f7e432000 0x7fc3cb2490: 0x0000007f7e435b20 0x0000000000000000 0x7fc3cb24a0: 0x0000007f7e432000 0x0000007f805a25d8 0x7fc3cb24b0: 0x0000007f7e433648 0x0000000000000006 0x7fc3cb24c0: 0x0000007fc3cb24e0 0x00000055560cad60 0x7fc3cb24d0: 0x000000555606d120 0x0000005556077ca8 0x7fc3cb24e0: 0x000000556002f580 0xe48d6289cdcca000 0x7fc3cb24f0: 0x0000007fc3cb2560 0x0000007f7e30376c 0x7fc3cb2500: 0x000000555606c600 0x0000000000000000 0x7fc3cb2510: 0x0000005555f6ef40 0x0000000000000000 0x7fc3cb2520: 0x0000000000000000 0x0000000000000000 0x7fc3cb2530: 0x0000000000000000 0x0000000000000000 0x7fc3cb2540: 0x0000000000000000 0x0000000000000000 0x7fc3cb2550: 0x0000007f7e435b20 0x0000000100000000 0x7fc3cb2560: 0x0000007fc3cb2570 0x0000007f7e2ee87c 0x7fc3cb2570: 0x0000000000000000 0x0000005555f6ef78 0x7fc3cb2580: 0x0000000000000000 0x0000000000000000 0x7fc3cb2590: 0x0000005555f6ef40 0x0000000000000000 0x7fc3cb25a0: 0x0000000000000000 0x0000000000000000 0x7fc3cb25b0: 0x0000000000000000 0x0000007f7e432000 0x7fc3cb25c0: 0x0000007fc3cb2600 0x0000007fc3cb26c8 0x7fc3cb25d0: 0x00000003c3cb25f0 0x0000005555f6e4e0 0x7fc3cb25e0: 0x0000007f7f1d00b0 0x000000555606c600 (gdb) (gdb) bt #0 __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:49 #1 0x0000007f7e2ee4f8 in __GI_abort () at abort.c:79 #2 0x0000007f7e4ff038 in __gnu_cxx::__verbose_terminate_handler() () from /adaptive/usr/lib/libstdc++.so.6 #3 0x0000007f7e4fcc0c in ?? () from /adaptive/usr/lib/libstdc++.so.6 #4 0x0000007f7e4fcc70 in std::terminate() () from /adaptive/usr/lib/libstdc++.so.6 #5 0x000000555600bdbc in std::thread::~thread (this=<optimized out>, __in_chrg=<optimized out>) at /data/ota_soft/jenkins/workspace/MI-MASTER-CD/jenkins_cross_compile_dir/sysroots/cortexa53-crypto-poky-linux/usr/include/c++/10.2.0/thread:157 #6 abup::ota::ftm::TransferTaskBase::~TransferTaskBase (this=0x556005a670, __in_chrg=<optimized out>) at /data/ota_soft/jenkins/workspace/MI-MASTER-CD/application/ftm/src/task/adm_ftm_task_base.hpp:192 #7 0x0000005556012184 in abup::ota::ftm::TransferTaskSelf::~TransferTaskSelf ( this=0x556005a670, __in_chrg=<optimized out>) at /data/ota_soft/jenkins/workspace/MI-MASTER-CD/application/ftm/src/task/adm_ftm_task_self.cpp:67 #8 0x0000005555ff72dc in std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release (this=0x556005ae60) at /data/ota_soft/jenkins/workspace/MI-MASTER-CD/jenkins_cross_compile_dir/sysroots/cortexa53-crypto-poky-linux/usr/include/c++/10.2.0/ext/atomicity.h:84 --Type <RET> for more, q to quit, c to continue without paging-- #9 std::__shared_count<(__gnu_cxx::_Lock_policy)2>::~__shared_count ( this=0x556006a4b0, __in_chrg=<optimized out>) at /data/ota_soft/jenkins/workspace/MI-MASTER-CD/jenkins_cross_compile_dir/sysroots/cortexa53-crypto-poky-linux/usr/include/c++/10.2.0/bits/shared_ptr_base.h:733 #10 std::__shared_ptr<abup::ota::ftm::TransferTaskBase, (__gnu_cxx::_Lock_policy)2>::~__shared_ptr (this=0x556006a4a8, __in_chrg=<optimized out>) at /data/ota_soft/jenkins/workspace/MI-MASTER-CD/jenkins_cross_compile_dir/sysroots/cortexa53-crypto-poky-linux/usr/include/c++/10.2.0/bits/shared_ptr_base.h:1183 #11 std::shared_ptr<abup::ota::ftm::TransferTaskBase>::~shared_ptr ( this=0x556006a4a8, __in_chrg=<optimized out>) at /data/ota_soft/jenkins/workspace/MI-MASTER-CD/jenkins_cross_compile_dir/sysroots/cortexa53-crypto-poky-linux/usr/include/c++/10.2.0/bits/shared_ptr.h:121 #12 std::pair<abup::ota::ftm::EcuType const, std::shared_ptr<abup::ota::ftm::TransferTaskBase> >::~pair (this=0x556006a4a0, __in_chrg=<optimized out>) at /data/ota_soft/jenkins/workspace/MI-MASTER-CD/jenkins_cross_compile_dir/sysroots/cortexa53-crypto-poky-linux/usr/include/c++/10.2.0/bits/stl_pair.h:211 #13 __gnu_cxx::new_allocator<std::_Rb_tree_node<std::pair<abup::ota::ftm::EcuType const, std::shared_ptr<abup::ota::ftm::TransferTaskBase> > > >::destroy<std::pair<abup::ota::ftm::EcuType const, std::shared_ptr<abup::ota::ftm::TransferTaskBase> > > (__p=0x556006a4a0, this=<optimized out>) --Type <RET> for more, q to quit, c to continue without paging-- at /data/ota_soft/jenkins/workspace/MI-MASTER-CD/jenkins_cross_compile_dir/sysroots/cortexa53-crypto-poky-linux/usr/include/c++/10.2.0/ext/new_allocator.h:156 #14 std::allocator_traits<std::allocator<std::_Rb_tree_node<std::pair<abup::ota::ftm::EcuType const, std::shared_ptr<abup::ota::ftm::TransferTaskBase> > > > >::destroy<std::pair<abup::ota::ftm::EcuType const, std::shared_ptr<abup::ota::ftm::TransferTaskBase> > > (__p=0x556006a4a0, __a=...) at /data/ota_soft/jenkins/workspace/MI-MASTER-CD/jenkins_cross_compile_dir/sysroots/cortexa53-crypto-poky-linux/usr/include/c++/10.2.0/bits/alloc_traits.h:531 #15 std::_Rb_tree<abup::ota::ftm::EcuType, std::pair<abup::ota::ftm::EcuType const, std::shared_ptr<abup::ota::ftm::TransferTaskBase> >, std::_Select1st<std::pair<abup::ota::ftm::EcuType const, std::shared_ptr<abup::ota::ftm::TransferTaskBase> > >, std::less<abup::ota::ftm::EcuType>, std::allocator<std::pair<abup::ota::ftm::EcuType const, std::shared_ptr<abup::ota::ftm::TransferTaskBase> > > >::_M_destroy_node (__p=0x556006a480, this=<optimized out>) at /data/ota_soft/jenkins/workspace/MI-MASTER-CD/jenkins_cross_compile_dir/sysroots/cortexa53-crypto-poky-linux/usr/include/c++/10.2.0/bits/stl_tree.h:646 #16 std::_Rb_tree<abup::ota::ftm::EcuType, std::pair<abup::ota::ftm::EcuType const, std::shared_ptr<abup::ota::ftm::TransferTaskBase> >, std::_Select1st<std::pair<abup::ota::ftm::EcuType const, std::shared_ptr<abup::ota::ftm::TransferTaskBase> > >, std::less<abup::ota::ftm::EcuType>, std::allocator<std::pair<abup::ota::ftm::EcuType const, std::shared_ptr<abup::ota::ftm::TransferTaskBase> > > >::_M--Type <RET> for more, q to quit, c to continue without paging-- _drop_node (this=<optimized out>, __p=0x556006a480) at /data/ota_soft/jenkins/workspace/MI-MASTER-CD/jenkins_cross_compile_dir/sysroots/cortexa53-crypto-poky-linux/usr/include/c++/10.2.0/bits/stl_tree.h:654 #17 std::_Rb_tree<abup::ota::ftm::EcuType, std::pair<abup::ota::ftm::EcuType const, std::shared_ptr<abup::ota::ftm::TransferTaskBase> >, std::_Select1st<std::pair<abup::ota::ftm::EcuType const, std::shared_ptr<abup::ota::ftm::TransferTaskBase> > >, std::less<abup::ota::ftm::EcuType>, std::allocator<std::pair<abup::ota::ftm::EcuType const, std::shared_ptr<abup::ota::ftm::TransferTaskBase> > > >::_M_erase (this=<optimized out>, __x=0x556006a480) at /data/ota_soft/jenkins/workspace/MI-MASTER-CD/jenkins_cross_compile_dir/sysroots/cortexa53-crypto-poky-linux/usr/include/c++/10.2.0/bits/stl_tree.h:1921 #18 std::_Rb_tree<abup::ota::ftm::EcuType, std::pair<abup::ota::ftm::EcuType const, std::shared_ptr<abup::ota::ftm::TransferTaskBase> >, std::_Select1st<std::pair<abup::ota::ftm::EcuType const, std::shared_ptr<abup::ota::ftm::TransferTaskBase> > >, std::less<abup::ota::ftm::EcuType>, std::allocator<std::pair<abup::ota::ftm::EcuType const, std::shared_ptr<abup::ota::ftm::TransferTaskBase> > > >::_M_erase (__x=0x556005e400, this=0x55560cad60 <abup::ota::ftm::TransferManager::instance()::transferManager+824>) at /data/ota_soft/jenkins/workspace/MI-MASTER-CD/jenkins_cross_compile_dir/sysroots/cortexa53-crypto-poky-linux/usr/include/c++/10.2.0/bits/stl_tree.h:1919 #19 0x000000555600a5f8 in std::_Rb_tree<abup::ota::ftm::EcuType, std::pair<abup::ota::ftm::EcuType const, std::shared_ptr<abup::ota::ftm::TransferTaskBase> >, s--Type <RET> for more, q to quit, c to continue without paging-- td::_Select1st<std::pair<abup::ota::ftm::EcuType const, std::shared_ptr<abup::ota::ftm::TransferTaskBase> > >, std::less<abup::ota::ftm::EcuType>, std::allocator<std::pair<abup::ota::ftm::EcuType const, std::shared_ptr<abup::ota::ftm::TransferTaskBase> > > >::~_Rb_tree ( this=0x55560cad60 <abup::ota::ftm::TransferManager::instance()::transferManager+824>, __in_chrg=<optimized out>) at /data/ota_soft/jenkins/workspace/MI-MASTER-CD/jenkins_cross_compile_dir/sysroots/cortexa53-crypto-poky-linux/usr/include/c++/10.2.0/bits/stl_tree.h:990 #20 std::map<abup::ota::ftm::EcuType, std::shared_ptr<abup::ota::ftm::TransferTaskBase>, std::less<abup::ota::ftm::EcuType>, std::allocator<std::pair<abup::ota::ftm::EcuType const, std::shared_ptr<abup::ota::ftm::TransferTaskBase> > > >::~map ( this=0x55560cad60 <abup::ota::ftm::TransferManager::instance()::transferManager+824>, __in_chrg=<optimized out>) at /data/ota_soft/jenkins/workspace/MI-MASTER-CD/jenkins_cross_compile_dir/sysroots/cortexa53-crypto-poky-linux/usr/include/c++/10.2.0/bits/stl_map.h:302 #21 abup::ota::ftm::TransferManager::~TransferManager ( this=0x55560caa28 <abup::ota::ftm::TransferManager::instance()::transferManager>, __in_chrg=<optimized out>) at /data/ota_soft/jenkins/workspace/MI-MASTER-CD/application/ftm/src/adm_ftm_manager.cpp:108 #22 0x0000007f7e3035dc in __run_exit_handlers (status=0, listp=0x7f7e433648 <__exit_funcs>, --Type <RET> for more, q to quit, c to continue without paging-- run_list_atexit=run_list_atexit@entry=true, run_dtors=run_dtors@entry=true) at exit.c:108 #23 0x0000007f7e30376c in __GI_exit (status=<optimized out>) at exit.c:139 #24 0x0000007f7e2ee87c in __libc_start_main (main=0x5555f6e4e0 <main>, argc=3, argv=0x7fc3cb26c8, init=<optimized out>, fini=<optimized out>, rtld_fini=<optimized out>, stack_end=<optimized out>) at ../csu/libc-start.c:348 #25 0x0000005555f6ef78 in _start () at ../sysdeps/aarch64/start.S:93 |
有兴趣的朋友,可以观察一下栈空间中,黄色字体和红色字体的规律,以及和调用栈的相关性。是不是很有趣。
分析
从上面信息中,仔细观察可知,存在两个异常。
异常点一:执行abup::ota::ftm::TransferTaskBase::~TransferTaskBase文件转移实例化对象析构时,出现了 abort 信号,造成了coredump。
我们知道C++中实例化对象,在对象销毁时,会进行析构。但是文件转移实例化对象,属于全局变量,之后在进程退出时,由系统进行回收。但是结合日志文件分析,程序流程,并没有执行main 函数中的Deinit模块(用于释放各模块资源),因此进行析构时,会出现abort 异常。从而引出新的问题点:main 函数没有走到 return ,为什么会进行全局对象的析构?
异常点二:为什么函数调用栈的起始是 _start()
我们知道正常的堆栈信息,调用栈的最上层接口一般是main函数,或子线程的入口接口。但是_start()我们应该是很少遇到的。疑问点:_start() 表示什么?当前的函数调用栈是正常的吗?
程序的加载流程
在《程序员的自我修养》6.5节中,曾这样简述linux加载ELF过程。
- Bash进程会调用fork()系统调用创建一个新的进程,然后新的进程调用execve()系统调用执行指定的ELF文件
- execve()内部会检查ELF文件前128个字节,确认文件格式,选择相应的文件装载处理过程。比如,ELF可执行文件,则选择load_elf_binary();a.out可执行文件就选择load_aout_binary();脚本文件,就选择load_script()。其中load_elf_binary()流程如下。
- 检查ELF可执行文件格式的有效性。比如魔数,程序头表中段的数量。
- 寻找动态连接的".interp"段,设置动态连接器路径。
- 根据ELF可执行文件的程序头表的描述,对ELF文件进行映射,比如代码,数据,只读数据。
- 初始化ELF进程环境,比如进程启动时EDX寄存器的地址应该是DT_FINI的地址。
- 将系统调用的返回地址修改成ELF可执行文件的入口点,这个入口点取决于程序的链接放方式,对于静态链接的ELF可执行文件,这个程序入口就是ELF文件的文件头中e_entry所指的地址;对于动态链接的ELF可执行文件,程序入口点就是动态连接器。
- load_elf_binary()执行完毕后,返回至exexve。此时已经将返回地址改为了被转载的ELF程序的入口地址了,所以继续执行新的程序。
经上述描述,ELF可执行文件的第一个代码段,应该就是入口点。我们结合以下代码验证。
|------------------------------------------------------------------------------------------|
| #include<stdlib.h> #include<stdio.h> int main() { printf("hello world"); return 0; } |
注:若不添加 -static -lc选项,部分符号是无法看到的。
由ELF文件头可知,entry point address 为 0x400a50。test.log 内容如test.log
由上述汇编可知,ELF的入口地址是_start。该接口主要做以下操作。
- 初始化操作,比如栈。
- 将libc_csu_fini,libc_csu_init,main函数的地址分别保存在 r8,rcx,rdi寄存器中。
- 调用libc_start_main接口。
通过观察libc_start_main的汇编实现可知。它内部的调用顺序应该为init,main,exit。exit内部会执行 __run_exit_handlers用于资源的回收,比如析构。实际上调用栈应该为:
|-------------------------------------------------------------------------------------------------|
| _start() --> libc_start_main() --> libc_csu_init --> main --> exit --> __run_exit_handlers |
思路
结合上述内容分析,目前怀疑是main 函数在执行过程中,因为触发了某些条件,导致,main 直接退出。执行了后续的exit --> __run_exit_handlers,在资源析构时,触发了abort信号,导致系统产生了coredump。
按照这个思路分析,怀疑是某个线程,调用了类似exit函数,导致系统默认为是进程正常退出。跳过了main后续操作,直接执行exit 后续内容。
按照这个思路,进行如下验证:
- 在接受任务线程中做了修改。若收到任务,则调用exit()。
- 对文件转移对象析构,进行断点。
经验证,当下发任务后,会按照预期,调用exit(),之后进行资源回收,调用析构函数。执行到断电时,发现函数调用栈信息和上述大体一致。 _start()->__libc_start_main->__GI_exit。
目前结论:某个线程调用了 exit 函数。需要客户和我们一起去排查。可以通过打桩,进行压测排查。