clickhouse日志疯涨问题

4月2日的时候，磁盘满了，导致服务运行异常，清理出28G的磁盘空间后，仅仅10天磁盘又满了，这次查看大日志情况，发现是clickhouse容器日志占用了29G

bash 复制代码

# 计算log日志大小并排序
du -h /var/lib/docker/containers/*/*.log | sort -rh

# 查看容器id和容器名称
docker ps -a --format "table {{.ID}}\t{{.Names}}"
docker ps -a --format "table {{.ID}}\t{{.Names}}" | grep 6f7be8d

# 查看容器日志
docker logs -tf --tail 150 clickhouse

# 查看日志文件内容
tail -n 150 /var/lib/docker/containers/6f7be8daf0434c51a50e3231c2745c906a3f5cb0b07695c14538a01754931dfe/6f7be8daf0434c51a50e3231c2745c906a3f5cb0b07695c14538a01754931dfe-json.log

# 清空容器json日志文件
truncate -s 0 /var/lib/docker/containers/[p#1 容器id]/[p#1 容器id]-json.log

查看日志就一直显示日志写入失败，查看映射出来的容器日志文件夹，发现日志最后写入时间还停留在4月2日，应该是上次的磁盘满导致的日志文件的损坏

-》停止clickhouse容器

-》删除clickhouse日志文件夹下的所有日志，清空clickhouse的容器json日志文件

-》启动clickhouse容器

-》再次查看clickhouse日志

发现容器开始正常运行，不会再疯涨日志

2026-04-13T01:08:42.000853740Z Cannot add message to the log: MemoryTracking: was 602.44 MiB, peak 3.99 GiB, free memory in arenas 40.50 MiB, will set to 610.33 MiB (RSS), difference: 7.89 MiB

2026-04-13T01:08:42.000878457Z Poco::Exception. Code: 1000, e.code() = 0, File access error: /var/log/clickhouse-server/clickhouse-server.log, Stack trace (when copying this message, always include the lines below):

2026-04-13T01:08:42.000881688Z

2026-04-13T01:08:42.000884434Z 0. Poco::FileException::FileException(String const&, int) @ 0x0000000015c5c709 in /usr/bin/clickhouse

2026-04-13T01:08:42.000887646Z 1. Poco::RotateBySizeStrategy::mustRotate(Poco::LogFile*) @ 0x0000000015c8b31b in /usr/bin/clickhouse

2026-04-13T01:08:42.000890562Z 2. Poco::FileChannel::log(Poco::Message const&) @ 0x0000000015c613eb in /usr/bin/clickhouse

2026-04-13T01:08:42.000893501Z 3. DB::OwnFormattingChannel::logExtended(DB::ExtendedLogMessage const&) @ 0x000000000c922367 in /usr/bin/clickhouse

2026-04-13T01:08:42.000896467Z 4. DB::OwnSplitChannel::tryLogSplit(Poco::Message const&) @ 0x000000000c926ded in /usr/bin/clickhouse

2026-04-13T01:08:42.000899199Z 5. DB::OwnSplitChannel::log(Poco::Message const&) @ 0x000000000c926be9 in /usr/bin/clickhouse

2026-04-13T01:08:42.000902147Z 6. DB::AsynchronousMetrics::update(std::chrono::time_point<std::chrono::system_clock, std::chrono::duration<long long, std::ratio<1l, 1000000l>>>) @ 0x000000000c77aa4f in /usr/bin/clickhouse

2026-04-13T01:08:42.000905466Z 7. void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::AsynchronousMetrics::start():: $_0\>(DB::AsynchronousMetrics::start()::$ _0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x000000000c78bf7f in /usr/bin/clickhouse

2026-04-13T01:08:42.000909557Z 8. void* std::__thread_proxy $abi:v15000$ <std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x000000000c728bc4 in /usr/bin/clickhouse

2026-04-13T01:08:42.000913515Z 9. ? @ 0x00007f18e08cb609 in ?

2026-04-13T01:08:42.000916317Z 10. ? @ 0x00007f18e07f0133 in ?

2026-04-13T01:08:42.000918989Z (version 23.8.6.16 (official build))