Rust 进程假死问题排查（dragonfly+tokio）

颖月石2026-10-12 14:51

背景：

最近负责Dragonfly Client，一个Rust+tokio框架，用于k8s镜像加速的项目。线上遇到难题，进程在大规模节点（600节点+）拉取大镜像（16G+）时，总是跑几分钟后部分节点（200节点+）的dfdaemon进程进入假死状态。表现形式：1. 进入卡死后不再打印任何日志 2. 在节点拉取镜像失败，containerd不能正常降级。3. 85分钟内4001端口可以连通，kubelet tcpListener liveness探活成功，85分钟后探活开始失败。

原因排查：

  1. dfdemon接收请求的端口是4001端口，该端口请求的tcp全连接队列全满

结合4001处理代码：

复制代码 
    /// run starts the proxy server.
    pub async fn run(&self, grpc_server_started_barrier: Arc<Barrier>) -> ClientResult<()> {
        let listener = TcpListener::bind(self.addr).await?;
        info!("proxy server listening on {}", self.addr);

        loop {
            // Wait for a client connection.
            tokio::select! {
                tcp_accepted = listener.accept() => {
                    // A new client connection has been established.
                    let (tcp, remote_address) = tcp_accepted?;

                    // Spawn a task to handle the connection.
                    let io = TokioIo::new(tcp);
                    debug!("accepted connection from {}", remote_address);

                    let context = context.clone();
                    tokio::task::spawn(async move {
                        if let Err(err) = ServerBuilder::new()
                            .keep_alive(true)
                            .max_buf_size(read_buffer_size)
                            .preserve_header_case(true)
                            .title_case_headers(true)
                            .serve_connection(
                                io,
                                service_fn(move |request|{
                                    let context = context.clone();
                                    async move {
                                        handler(context.config, context.task, request, context.dfdaemon_download_client, context.registry_cert, context.server_ca_cert, remote_address.ip()).await
                                    }
                                } ),
                                )
                            .with_upgrades()
                            .await
                        {
                            collect_proxy_request_failure_metrics();
                            error!("failed to serve connection from {}: {}", remote_address, err);
                        }
                    });
                }
                _ = shutdown.recv() => {
                    // Proxy server shutting down with signals.
                    info!("proxy server shutting down");
                    return Ok(());
                }
            }
        }
    }

　

不断从tcp全连接队列拿取任务，调用异步方法handler处理（handler里面会封装task，GRPC调用/dfdaemon.v2.DfdaemonDownload/DownloadTask 下载task，并不堵塞4001处理接口）。handler处理方法放在 tokio::task::spawn里面，由tokio worker线程池处理（顺便提一下，tokio::task::spawn_blocking才是由tokio Blocking Thread Pool堵塞线程池处理）。

tcp全连接队列全满，说明handler处理未及时拿取任务，但是这个异步方法里面并没有什么可能会被堵塞的处理逻辑，那就只有一种可能，tokio worker线程堵塞了。

  1. 通过gdb获取线程调用栈情况

使用gdb有个小插曲，dfdaemon进程中并未内置gdb工具。尝试针对宿主机查询pod在宿主机对应的pid进行gdb查询（方法如下），得到的线程调用栈涉及到dfdaemon内部代码时都是？？？网上查询说是因为pod里面dfdaemon的二进制文件并未支持打印内部类导致，还是需要重新编译构建二进制。

对宿主机查询pod在宿主机对应的pid进行gdb查询：

2.1 查询dfdaemon的pid：ps -ef | grep dfdaemon | grep -v grep

2.2 gdb查询线程调用栈：gdb -p pid -batch -ex "thread apply all bt" > thread_dump.txt.1

为了使用gdb工具，并支持追溯打印内部类，对dfdaemon做如下代码调整：

2.3、 Cargo.toml (右侧为调整后结果）

2.4 jdock.Dockerfile 加一行

复制代码 
RUN dnf install -y gdb strace ltrace perf iotop util-linux glibc libgcc sysstat

进入dfdaemon容器后，执行gdp指令gdb -p pid -batch -ex "thread apply all bt" > thread_dump.txt.1，展示调用栈信息如下：

复制代码 
Thread 6 (Thread 0x7f3ee3bfe6c0 (LWP 11) "tokio-runtime-w"):
#0  0x00007f3ee5884c5d in syscall () from /lib64/libc.so.6
#1  0x0000564284a767aa in dashmap::lock::RawRwLock::lock_exclusive_slow ()
#2  0x0000564285381d2a in dashmap::DashMap<K,V,S>::entry ()
#3  0x00005642854fc11f in dragonfly_client::resource::parent_selector::ParentSelector::register::{{closure}}::{{closure}} ()
#4  0x00005642855d7dd5 in dragonfly_client::resource::task::Task::download_partial_with_scheduler_from_parent::{{closure}}::{{closure}} ()
#5  0x00005642855cbbe1 in dragonfly_client::resource::task::Task::download_partial_with_scheduler::{{closure}}::{{closure}} ()
#6  0x00005642855c2b47 in dragonfly_client::resource::task::Task::download::{{closure}}::{{closure}} ()
#7  0x00005642855beebd in dragonfly_client::resource::task::Task::download::{{closure}} ()
#8  0x000056428543414d in <dragonfly_client::grpc::dfdaemon_download::DfdaemonDownloadServerHandler as dragonfly_api::dfdaemon::v2::dfdaemon_download_server::DfdaemonDownload>::download_task::{{closure}}::{{closure}}::{{closure}} ()
#9  0x00005642853ae431 in <tracing::instrument::Instrumented<T> as core::future::future::Future>::poll ()
#10 0x00005642853168a9 in tokio::runtime::task::raw::poll ()
#11 0x000056428612abcc in tokio::runtime::scheduler::multi_thread::worker::Context::run_task ()
#12 0x0000564286128efe in tokio::runtime::scheduler::multi_thread::worker::run ()
#13 0x0000564286132dca in tokio::runtime::task::raw::poll ()
#14 0x000056428610f71c in std::sys::backtrace::__rust_begin_short_backtrace ()
#15 0x0000564286112524 in core::ops::function::FnOnce::call_once{{vtable.shim}} ()
#16 0x00005642860c126b in alloc::boxed::{impl#28}::call_once<(), dyn core::ops::function::FnOnce<(), Output=()>, alloc::alloc::Global> () at library/alloc/src/boxed.rs:1993
#17 alloc::boxed::{impl#28}::call_once<(), alloc::boxed::Box<dyn core::ops::function::FnOnce<(), Output=()>, alloc::alloc::Global>, alloc::alloc::Global> () at library/alloc/src/boxed.rs:1993
#18 std::sys::pal::unix::thread::{impl#2}::new::thread_start () at library/std/src/sys/pal/unix/thread.rs:106
#19 0x00007f3ee5807a6e in ?? () from /lib64/libc.so.6
#20 0x00007f3ee5886cbc in ?? () from /lib64/libc.so.6

Thread 5 (Thread 0x7f3ee3dff6c0 (LWP 10) "tokio-runtime-w"):
#0  0x00007f3ee5884c5d in syscall () from /lib64/libc.so.6
#1  0x0000564284a767aa in dashmap::lock::RawRwLock::lock_exclusive_slow ()
#2  0x0000564285381d2a in dashmap::DashMap<K,V,S>::entry ()
#3  0x00005642854fc11f in dragonfly_client::resource::parent_selector::ParentSelector::register::{{closure}}::{{closure}} ()
#4  0x00005642855d7dd5 in dragonfly_client::resource::task::Task::download_partial_with_scheduler_from_parent::{{closure}}::{{closure}} ()
#5  0x00005642855cbbe1 in dragonfly_client::resource::task::Task::download_partial_with_scheduler::{{closure}}::{{closure}} ()
#6  0x00005642855c2b47 in dragonfly_client::resource::task::Task::download::{{closure}}::{{closure}} ()
#7  0x00005642855beebd in dragonfly_client::resource::task::Task::download::{{closure}} ()
#8  0x000056428543414d in <dragonfly_client::grpc::dfdaemon_download::DfdaemonDownloadServerHandler as dragonfly_api::dfdaemon::v2::dfdaemon_download_server::DfdaemonDownload>::download_task::{{closure}}::{{closure}}::{{closure}} ()
#9  0x00005642853ae431 in <tracing::instrument::Instrumented<T> as core::future::future::Future>::poll ()
#10 0x00005642853168a9 in tokio::runtime::task::raw::poll ()
#11 0x000056428612ad4e in tokio::runtime::scheduler::multi_thread::worker::Context::run_task ()
#12 0x0000564286128efe in tokio::runtime::scheduler::multi_thread::worker::run ()
#13 0x0000564286132dca in tokio::runtime::task::raw::poll ()
#14 0x000056428610f71c in std::sys::backtrace::__rust_begin_short_backtrace ()
#15 0x0000564286112524 in core::ops::function::FnOnce::call_once{{vtable.shim}} ()
#16 0x00005642860c126b in alloc::boxed::{impl#28}::call_once<(), dyn core::ops::function::FnOnce<(), Output=()>, alloc::alloc::Global> () at library/alloc/src/boxed.rs:1993
#17 alloc::boxed::{impl#28}::call_once<(), alloc::boxed::Box<dyn core::ops::function::FnOnce<(), Output=()>, alloc::alloc::Global>, alloc::alloc::Global> () at library/alloc/src/boxed.rs:1993
#18 std::sys::pal::unix::thread::{impl#2}::new::thread_start () at library/std/src/sys/pal/unix/thread.rs:106
#19 0x00007f3ee5807a6e in ?? () from /lib64/libc.so.6
#20 0x00007f3ee5886cbc in ?? () from /lib64/libc.so.6

Thread 4 (Thread 0x7f3ee43fd6c0 (LWP 9) "tokio-runtime-w"):
#0  0x00007f3ee5884c5d in syscall () from /lib64/libc.so.6
#1  0x0000564284a767aa in dashmap::lock::RawRwLock::lock_exclusive_slow ()
#2  0x0000564285381d2a in dashmap::DashMap<K,V,S>::entry ()
#3  0x00005642854fc11f in dragonfly_client::resource::parent_selector::ParentSelector::register::{{closure}}::{{closure}} ()
#4  0x00005642855d7dd5 in dragonfly_client::resource::task::Task::download_partial_with_scheduler_from_parent::{{closure}}::{{closure}} ()
#5  0x00005642855cbbe1 in dragonfly_client::resource::task::Task::download_partial_with_scheduler::{{closure}}::{{closure}} ()
#6  0x00005642855c2b47 in dragonfly_client::resource::task::Task::download::{{closure}}::{{closure}} ()
#7  0x00005642855beebd in dragonfly_client::resource::task::Task::download::{{closure}} ()
#8  0x000056428543414d in <dragonfly_client::grpc::dfdaemon_download::DfdaemonDownloadServerHandler as dragonfly_api::dfdaemon::v2::dfdaemon_download_server::DfdaemonDownload>::download_task::{{closure}}::{{closure}}::{{closure}} ()
#9  0x00005642853ae431 in <tracing::instrument::Instrumented<T> as core::future::future::Future>::poll ()
#10 0x00005642853168a9 in tokio::runtime::task::raw::poll ()
#11 0x000056428612ad4e in tokio::runtime::scheduler::multi_thread::worker::Context::run_task ()
#12 0x0000564286128efe in tokio::runtime::scheduler::multi_thread::worker::run ()
#13 0x0000564286132dca in tokio::runtime::task::raw::poll ()
#14 0x000056428610f71c in std::sys::backtrace::__rust_begin_short_backtrace ()
#15 0x0000564286112524 in core::ops::function::FnOnce::call_once{{vtable.shim}} ()
#16 0x00005642860c126b in alloc::boxed::{impl#28}::call_once<(), dyn core::ops::function::FnOnce<(), Output=()>, alloc::alloc::Global> () at library/alloc/src/boxed.rs:1993
#17 alloc::boxed::{impl#28}::call_once<(), alloc::boxed::Box<dyn core::ops::function::FnOnce<(), Output=()>, alloc::alloc::Global>, alloc::alloc::Global> () at library/alloc/src/boxed.rs:1993
#18 std::sys::pal::unix::thread::{impl#2}::new::thread_start () at library/std/src/sys/pal/unix/thread.rs:106
#19 0x00007f3ee5807a6e in ?? () from /lib64/libc.so.6
#20 0x00007f3ee5886cbc in ?? () from /lib64/libc.so.6

Thread 3 (Thread 0x7f3ee45fe6c0 (LWP 8) "tokio-runtime-w"):
#0  0x00007f3ee5884c5d in syscall () from /lib64/libc.so.6
#1  0x0000564284a767aa in dashmap::lock::RawRwLock::lock_exclusive_slow ()
#2  0x0000564285381d2a in dashmap::DashMap<K,V,S>::entry ()
#3  0x00005642854fc11f in dragonfly_client::resource::parent_selector::ParentSelector::register::{{closure}}::{{closure}} ()
#4  0x00005642855d7dd5 in dragonfly_client::resource::task::Task::download_partial_with_scheduler_from_parent::{{closure}}::{{closure}} ()
#5  0x00005642855cbbe1 in dragonfly_client::resource::task::Task::download_partial_with_scheduler::{{closure}}::{{closure}} ()
#6  0x00005642855c2b47 in dragonfly_client::resource::task::Task::download::{{closure}}::{{closure}} ()
#7  0x00005642855beebd in dragonfly_client::resource::task::Task::download::{{closure}} ()
#8  0x000056428543414d in <dragonfly_client::grpc::dfdaemon_download::DfdaemonDownloadServerHandler as dragonfly_api::dfdaemon::v2::dfdaemon_download_server::DfdaemonDownload>::download_task::{{closure}}::{{closure}}::{{closure}} ()
#9  0x00005642853ae431 in <tracing::instrument::Instrumented<T> as core::future::future::Future>::poll ()
#10 0x00005642853168a9 in tokio::runtime::task::raw::poll ()
#11 0x000056428612ad4e in tokio::runtime::scheduler::multi_thread::worker::Context::run_task ()
#12 0x0000564286128efe in tokio::runtime::scheduler::multi_thread::worker::run ()
#13 0x0000564286132dca in tokio::runtime::task::raw::poll ()
#14 0x000056428610f71c in std::sys::backtrace::__rust_begin_short_backtrace ()
#15 0x0000564286112524 in core::ops::function::FnOnce::call_once{{vtable.shim}} ()
#16 0x00005642860c126b in alloc::boxed::{impl#28}::call_once<(), dyn core::ops::function::FnOnce<(), Output=()>, alloc::alloc::Global> () at library/alloc/src/boxed.rs:1993
#17 alloc::boxed::{impl#28}::call_once<(), alloc::boxed::Box<dyn core::ops::function::FnOnce<(), Output=()>, alloc::alloc::Global>, alloc::alloc::Global> () at library/alloc/src/boxed.rs:1993
#18 std::sys::pal::unix::thread::{impl#2}::new::thread_start () at library/std/src/sys/pal/unix/thread.rs:106
#19 0x00007f3ee5807a6e in ?? () from /lib64/libc.so.6
#20 0x00007f3ee5886cbc in ?? () from /lib64/libc.so.6

　

tokio线程池worker线程数=pod.spec.template.spec.requests.limit.Cpu（k8s里面）=4

从👆🏻可以看出，4个tokio-runtime-w（worker线程）都堵塞在 ParentSelector::register方法的DashMap<K,V,S>::entry () 获取读写锁RawRwLock上。当有新的任务出现，tokio调度器因为无worker线程调度，而导致任务被堵塞。

  1. 进一步分析，具体堵塞代码：
复制代码 
        for parent in parents {
            match self.connections.entry(parent_host_id.clone()) {   
# DashMap是按照分片Shard分配RawRwLock读写锁的，分片数=pod.spec.template.spec.requests.limit.Cpu（k8s里面）=4
复制代码 
# entry方法需要拿到DashMap的RawRwLock读写锁，锁粒度是分片维度
复制代码 
# 当某个task下载候选peer足够多，4个tokio worker线程撞击在同一个分片锁里面的概率就很大，继而所有tokio worker线程被锁住导致4001端口handler不处理。
## 这也就解释了，为什么dfdaemon线程hang死现象出现在集群内大面积节点同时拉取某个镜像时，而只有几个节点下载镜像时并未出现过。
复制代码 
                dashmap::mapref::entry::Entry::Occupied(entry) => {
                    entry.get().increment_request();
                    continue;
                }
                dashmap::mapref::entry::Entry::Vacant(entry) => {
                    let dfdaemon_upload_client = DfdaemonUploadClient::new( #跟候选peer建立grpc连接
                        self.config.clone(),
                        format_url(
                            "http",
                            IpAddr::from_str(&parent_host.ip)?,
                            parent_host.port as u16,
                        ),
                        false,
                    )
                    .await?;
# .await会挂起任务，直至new完成后，tokio调度器才会再次调度worker线程池来执行后面的步骤
                    let connection = Connection::new();
                    connection.increment_request();
                    let shutdown = connection.shutdown.clone();
                    entry.insert(connection);
# insert方法里面，针对VacantEntry（也就是空entry）会重新new entry，并重新生成读写锁，分配分片，之前再Entry()方法里面占用的RWLock自然也会被释放掉（rust来说，变量在后续代码
未被用到，自然就会被回收掉，锁自然也消失）。
                }
            }
        }

复盘总结一下导致tokio线程hang住场景：

3.1 一个线程调用DashMap.entry方法，占用了该分片的RWLock同步写锁A，因为是VacantEntry，异步调用了New方法B与候选peer建立grpc连接，同时通过.await将任务挂起，但是RWLock写锁A未释放。

3.2 其他候选peer建连过程，也纷纷调用了DashMap.entry方法。因为候选人足够多，就4个分片锁，导致4个tokio worker线程都在调用该方法时卡住，等待获取RWLock同步写锁A。

3.3 New方法B执行成功，.await唤醒，需要tokio调度器调用新的worker线程处理接下来的代码逻辑，从而将RWLock同步写锁A释放。但是因为4个tokio worker线程全部都hang在等待RWLock同步写锁A上，导致无多余tokio线程可调度。

3.4 大家彻底死锁。

这里面就是犯了一个常见错误：

.await方法不能挟持同步锁跑路！！！

解决方法：

把DfdaemonUploadClient::new().await方法挪出DashMap：：entry()作用域，提前处理了。

复制代码 
        for parent in parents {
            if let Some(connection) = self.connections.get(&parent_host_id) {
                connection.increment_request();
                continue;
            }

            let dfdaemon_upload_client = match DfdaemonUploadClient::new(  ##挪出entry作用域
                self.config.clone(),
                format_url(
                    "http",
                    IpAddr::from_str(&parent_host.ip)?,
                    parent_host.port as u16,
                ),
                false,
            )
                .await
            {
                Ok(client) => client,
                Err(err) => {
                    error!("failed to connect to parent {}: {}", parent_host_id, err);
                    continue;
                }
            };

            match self.connections.entry(parent_host_id.clone()) {
                dashmap::mapref::entry::Entry::Occupied(entry) => {
                    entry.get().increment_request();
                    continue;
                }
                dashmap::mapref::entry::Entry::Vacant(entry) => {
                    let connection = Connection::new();
                    connection.increment_request();
                    let shutdown = connection.shutdown.clone();
                    entry.insert(connection);
              }
            }

　　

上线后观测几个月，再未出现hang死现象，问题得解！

相关推荐
智码看视界3 天前
Day84-Arthas诊断实战：CPU飙高/死锁/内存泄漏排查三板斧
jvm·内存泄漏·arthas·死锁·性能诊断·生产调优
for_ever_love__3 天前
MySQL 锁与死锁讲透：行锁、间隙锁、Next-Key Lock 与排查方法
mysql·lock·行锁·死锁·间隙锁·排查·next-key
~木雨4 天前
Java 线程从生到死：创建三式、六态流转、中断协作与死锁活锁饥饿，一篇讲透
java·java并发·虚拟线程·死锁·线程状态·中断机制
键盘会跳舞1 个月前
【C++多线程】死锁诊断工具实战：从 Windows 到 Linux 的全链路排查
linux·c++·windows·死锁
ly76891 个月前
深入解析 MySQL 间隙锁：从加锁规则到死锁案例
数据库·mysql·死锁·间隙锁·next-key lock·事务隔离
J_bean2 个月前
MySQL InnoDB 如何检测死锁、判定死锁、处理死锁
数据库·mysql·死锁·死锁检测·处理死锁·死锁判定
j7～2 个月前
【Linux】二十七.线程篇四《Linux多线程编程：线程互斥（互斥量的底层到封装）、线程安全和冲入、死锁》---详解
linux·开发语言·c++·线程安全·死锁·线程互斥·重入
同勉共进3 个月前
记一例 vibe coding + gcc bug 导致的线程池死锁问题
c++·线程池·gcc·死锁·vibe coding
是个西兰花3 个月前
Linux：死锁与生产者消费者模型解析
linux·运维·服务器·c++·死锁·生产者消费者模型
热门推荐
01国内可直接用、免费额度/永久免费的大模型API清单（含 SiliconFlow、火山、阿里、智谱、百度、Kimi、DeepSeek、DMXAPI 等）02历年考研数学一、数学二、数学三真题试卷及答案PDF03如何新建文件夹？ 电脑新建文件夹的4种方法04「全语录+省流版」邵艾伦 × 孙宇晨：聊风口、聊个人IP、聊年轻人如何抓住 AI 时代的机会05GitHub 镜像站点06蜘蛛磁力 搜索引擎大全，如何使用蜘蛛磁力查找磁力链接07轻量化AI渗透武器库｜ARTEX自主攻防系统、资产梳理、漏洞挖掘、链路分析全流程智能化落地08Windows 10/11 如何退出微软账户并改用本地账户登录（含找不到选项的解决方法）09Telegram APK 下载安装完整指南 — 2026年最新10微信历史版本含下载地址（ Windows PC | 安卓 | MAC ）及设置微信不更新