BE节点经常挂掉:[IO_ERROR]failed to list /proc/27349/fd/: No such file or directory

最近BE节点经常挂掉

Caused by: java.lang.RuntimeException: Failed to execute internal SQL. org.apache.doris.common.UserException: errCode = 2, detailMessage = There is no scanNode Backend available.[10031: not alive] OriginStatement{originStmt='SELECT * FROM __internal_schema.column_statistics WHERE tbl_id=27273 AND idx_id=-1 AND col_id='CREATE_AID'', idx=0}
        at org.apache.doris.qe.StmtExecutor.executeInternalQuery(StmtExecutor.java:2509)
        at org.apache.doris.statistics.util.StatisticsUtil.execStatisticQuery(StatisticsUtil.java:131)
        at org.apache.doris.statistics.StatisticsRepository.loadColStats(StatisticsRepository.java:439)
        at org.apache.doris.statistics.ColumnStatisticsCacheLoader.loadFromStatsTable(ColumnStatisticsCacheLoader.java:56)
        at org.apache.doris.statistics.ColumnStatisticsCacheLoader.doLoad(ColumnStatisticsCacheLoader.java:38)
        at org.apache.doris.statistics.ColumnStatisticsCacheLoader.doLoad(ColumnStatisticsCacheLoader.java:31)
        at org.apache.doris.statistics.StatisticsCacheLoader.lambda$asyncLoad$0(StatisticsCacheLoader.java:48)
        at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1590)
        ... 3 more
Caused by: org.apache.doris.common.UserException: errCode = 2, detailMessage = There is no scanNode Backend available.[10031: not alive]
        at org.apache.doris.qe.SimpleScheduler.getHost(SimpleScheduler.java:147)
        at org.apache.doris.qe.Coordinator.computeFragmentHosts(Coordinator.java:1806)
        at org.apache.doris.qe.Coordinator.computeFragmentExecParams(Coordinator.java:1267)
        at org.apache.doris.qe.Coordinator.exec(Coordinator.java:573)
        at org.apache.doris.qe.StmtExecutor.executeInternalQuery(StmtExecutor.java:2505)
        ... 10 more

be.out也看不出什么有用日志,查看be.WARNING,发现了如下错误,但还不知道如何解决,先记录一下问题

[IO_ERROR]failed to list /proc/27349/fd/: (2), No such file or directory

W1121 09:36:26.929662 27477 doris_metrics.cpp:379] failed to count fd: [IO_ERROR]failed to list /proc/27349/fd/: (2), No such file or directory
0. /root/src/doris-2.0/be/src/common/stack_trace.cpp:302: StackTrace::tryCapture() @ 0x000000000b9e64c7 in /xxsys/doris-2.0.2/be/lib/doris_be
1. /root/src/doris-2.0/be/src/common/stack_trace.h:0: doris::get_stack_trace[abi:cxx11]() @ 0x000000000b9e4ae5 in /xxsys/doris-2.0.2/be/lib/doris_be
2. /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/unique_ptr.h:173: doris::Status doris::Status::Error<true, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >(int, std::basic_string_view<char, std::char_traits<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&&) @ 0x000000000aecc168 in /xxsys/doris-2.0.2/be/lib/doris_be
3. /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/basic_string.h:187: doris::io::LocalFileSystem::list_impl(std::filesystem::__cxx11::path const&, bool, std::vector<doris::io::FileInfo, std::allocator<doris::io::FileInfo> >*, bool*) @ 0x000000000aec6eac in /xxsys/doris-2.0.2/be/lib/doris_be
4. /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/unique_ptr.h:360: doris::io::LocalFileSystem::iterate_directory_impl(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::function<bool (doris::io::FileInfo const&)> const&) @ 0x000000000aec7fcf in /xxsys/doris-2.0.2/be/lib/doris_be
5. /root/src/doris-2.0/be/src/common/status.h:348: doris::io::LocalFileSystem::iterate_directory(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::function<bool (doris::io::FileInfo const&)> const&) @ 0x000000000aec7e4d in /xxsys/doris-2.0.2/be/lib/doris_be
6. /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/std_function.h:244: doris::DorisMetrics::_update_process_fd_num() @ 0x000000000b97a65a in /xxsys/doris-2.0.2/be/lib/doris_be
7. /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/stl_tree.h:368: doris::MetricRegistry::trigger_all_hooks(bool) const @ 0x000000000b9ba69f in /xxsys/doris-2.0.2/be/lib/doris_be
8. /root/src/doris-2.0/be/src/util/time.h:50: doris::Daemon::calculate_metrics_thread() @ 0x000000000ae9cc0c in /xxsys/doris-2.0.2/be/lib/doris_be
9. /var/local/ldb-toolchain/bin/../usr/include/pthread.h:562: doris::Thread::supervise_thread(void*) @ 0x000000000ba1819a in /xxsys/doris-2.0.2/be/lib/doris_be
10. start_thread @ 0x00007f2f98172aa1 in ?
11. __clone @ 0x00007f2f988f8c4d in ?
相关推荐
墨家巨子@俏如来5 天前
五.海量数据实时分析-FlinkCDC+DorisConnector实现数据的全量增量同步
大数据·doris
SelectDB技术团队6 天前
快手:从 Clickhouse 到 Apache Doris,实现湖仓分离向湖仓一体架构升级
数据仓库·clickhouse·doris·快手·lakehouse
Jet-W7 天前
Doris使用手册以及与Mysql差异整理
大数据·数据库·后端·mysql·doris
最菜的Bird13 天前
Apache Doris 实践
java·kafka·doris
SelectDB技术团队15 天前
Apache Doris 2.1.6 版本正式发布
大数据·数据库·数据仓库·开源·doris
SelectDB技术团队22 天前
Apache Doris Flink Connector 24.0.0 版本正式发布
大数据·flink·doris·flinkcdc·数据同步
墨家巨子@俏如来1 个月前
三.海量数据实时分析-FlinkCDC实现Mysql数据同步到Doris
数据库·mysql·doris
SelectDB技术团队1 个月前
兼容Trino Connector,扩展Apache Doris数据源接入能力|Lakehouse 使用手册(四)
数据库·doris·trino·湖仓一体·lakehouse
cloneme011 个月前
Apache Arrow简介
apache·doris·arrow
墨家巨子@俏如来1 个月前
一.海量数据实时分析-Doris入门和安装
大数据·doris