StarRocks StreamLoad 持续写入导致 be 内存增长

胖胖胖胖胖虎2026-06-30 8:28

背景：Flink 消费 Kafka 通过 StreamLoad 持续写入 StarRocks，be 节点内存持续增长直至到达 be进程内存资源上限，任务中断

查看 be 内存资源
shell 复制代码 
[root@jys-cce-nodepool-bigdata-qnkr7 curl]# kubectl --kubeconfig /opt/module/jys-uat.yaml --namespace bigdata exec -it curl-d6b588446-67h6d -- curl  -XGET -s http://kube-starrocks-be-service.starrocks.svc.cluster.local:8040/metrics | grep "^starrocks_be_.*_mem_bytes"
starrocks_be_bitmap_index_mem_bytes 2772800
starrocks_be_bloom_filter_index_mem_bytes 2378696
starrocks_be_clone_mem_bytes 0
### 内存占用大头
starrocks_be_column_metadata_mem_bytes 310631160
starrocks_be_column_zonemap_index_mem_bytes 108817696
starrocks_be_compaction_mem_bytes 0
starrocks_be_connector_scan_pool_mem_bytes 0
starrocks_be_consistency_mem_bytes 0
starrocks_be_datacache_mem_bytes 78998594
starrocks_be_jit_cache_mem_bytes 0
starrocks_be_load_mem_bytes 0
### metadata 内存占用大头
starrocks_be_metadata_mem_bytes 525319928
starrocks_be_ordinal_index_mem_bytes 60566288
### be 进程
starrocks_be_process_mem_bytes 1366550648
starrocks_be_query_mem_bytes 0
starrocks_be_rowset_metadata_mem_bytes 25569895
starrocks_be_schema_change_mem_bytes 0
starrocks_be_segment_metadata_mem_bytes 72609652
starrocks_be_segment_zonemap_mem_bytes 63078656
starrocks_be_short_key_index_mem_bytes 2230
starrocks_be_storage_page_cache_mem_bytes 78998594
starrocks_be_tablet_metadata_mem_bytes 116509221
starrocks_be_tablet_schema_mem_bytes 37557
starrocks_be_update_mem_bytes 0
sql 复制代码 
select * from information_schema.be_metrics where name like '%_mem_bytes%';
be 内存资源不足
shell 复制代码 
W20260628 02:45:59.098022 139791567357504 load_channel.cpp:259] tablet writer add chunk, id=a941cc85-a8fd-86e9-7ac2-4f88d21dae8d, index_id=93207, sender_id=0 request_index=0 eos=0 err=kube-starrocks-be-0.kube-starrocks-be-search.starrocks.svc.cluster.local: memory limit exceeded, please reduce load frequency or increase config `load_process_max_memory_hard_limit_ratio` or add more BE nodes
W20260628 02:45:59.098243 139793941890624 tablet_sink_sender.cpp:126] NodeChannel[10001], tablet add chunk failed, load_id=a941cc85-a8fd-86e9-7ac2-4f88d21dae8d, txn_id: 86459, parallel=1, compress_type=2, node=172.17.251.5:8060, errmsg=kube-starrocks-be-0.kube-starrocks-be-search.starrocks.svc.cluster.local: memory limit exceeded, please reduce load frequency or increase config `load_process_max_memory_hard_limit_ratio` or add more BE nodes
StarRocks 内存变化
  • sink.batch.size=5000 通过以下内存变化，较小的批次现象是：metadata_mem_bytes ，column_metadata_mem_bytes 占内存资源大头
  • sink.batch.size=100000 通过以下内存变化，10万批次现象是：update_mem_bytes 占内存资源大头

参考：

衍生问题

be 统计 tabletNum 数 与 statistic 统计不一致
  • be TabletNum: 25498
sql 复制代码 
mysql> show backends;
+-----------+--------------------------------------------------------------------------+---------------+--------+----------+----------+---------------------+---------------------+-------+----------------------+-----------------------+-----------+------------------+---------------+---------------+---------+----------------+--------+----------------+--------------------------------------------------------+-------------------+-------------+----------+----------+-------------------+------------+------------+----------------------------------------------------------+----------+------------+
| BackendId | IP                                                                       | HeartbeatPort | BePort | HttpPort | BrpcPort | LastStartTime       | LastHeartbeat       | Alive | SystemDecommissioned | ClusterDecommissioned | TabletNum | DataUsedCapacity | AvailCapacity | TotalCapacity | UsedPct | MaxDiskUsedPct | ErrMsg | Version        | Status                                                 | DataTotalCapacity | DataUsedPct | CpuCores | MemLimit | NumRunningQueries | MemUsedPct | CpuUsedPct | DataCacheMetrics                                         | Location | StatusCode |
+-----------+--------------------------------------------------------------------------+---------------+--------+----------+----------+---------------------+---------------------+-------+----------------------+-----------------------+-----------+------------------+---------------+---------------+---------+----------------+--------+----------------+--------------------------------------------------------+-------------------+-------------+----------+----------+-------------------+------------+------------+----------------------------------------------------------+----------+------------+
| 10001     | kube-starrocks-be-0.kube-starrocks-be-search.starrocks.svc.cluster.local | 9050          | 9060   | 8040     | 8060     | 2026-06-28 13:15:05 | 2026-06-28 16:48:51 | true  | false                | false                 | 25498     | 1.183 GB         | 875.371 GB    | 1006.850 GB   | 13.06 % | 13.06 %        |        | 4.0.11-9559176 | {"lastSuccessReportTabletsTime":"2026-06-28 16:48:51"} | 876.554 GB        | 0.14 %      | 8        | 6.480GB  | 0                 | 22.48 %    | 0.0 %      | Status: Normal, DiskUsage: 0B/660GB, MemUsage: 2MB/1.2GB |          | OK         |
+-----------+--------------------------------------------------------------------------+---------------+--------+----------+----------+---------------------+---------------------+-------+----------------------+-----------------------+-----------+------------------+---------------+---------------+---------+----------------+--------+----------------+--------------------------------------------------------+-------------------+-------------+----------+----------+-------------------+------------+------------+----------------------------------------------------------+----------+------------+
1 row in set (0.00 sec)
  • fe 记录 170 tabletNum
sql 复制代码 
mysql> SHOW PROC '/statistic';
+-------+--------------------+----------+--------------+----------+-----------+------------+--------------------+-----------------------+------------------+---------------------+
| DbId  | DbName             | TableNum | PartitionNum | IndexNum | TabletNum | ReplicaNum | UnhealthyTabletNum | InconsistentTabletNum | CloningTabletNum | ErrorStateTabletNum |
+-------+--------------------+----------+--------------+----------+-----------+------------+--------------------+-----------------------+------------------+---------------------+
| 10002 | _statistics_       | 12       | 33           | 33       | 170       | 170        | 0                  | 0                     | 0                | 0                   |
| 1     | information_schema | 0        | 0            | 0        | 0         | 0          | 0                  | 0                     | 0                | 0                   |
| 92843 | ods                | 0        | 0            | 0        | 0         | 0          | 0                  | 0                     | 0                | 0                   |
| 100   | sys                | 0        | 0            | 0        | 0         | 0          | 0                  | 0                     | 0                | 0                   |
| Total | 4                  | 12       | 33           | 33       | 170       | 170        | 0                  | 0                     | 0                | 0                   |
+-------+--------------------+----------+--------------+----------+-----------+------------+--------------------+-----------------------+------------------+---------------------+
5 rows in set (0.00 sec)
sql 复制代码 
mysql> SHOW PROC '/backends/10001';
+---------------------------+------------------+-------------------+---------------+---------------+--------------+--------+----------------------+---------------+-----------+-------------------+-------------+
| RootPath                  | DataUsedCapacity | OtherUsedCapacity | AvailCapacity | TotalCapacity | TotalUsedPct | State  | PathHash             | StorageMedium | TabletNum | DataTotalCapacity | DataUsedPct |
+---------------------------+------------------+-------------------+---------------+---------------+--------------+--------+----------------------+---------------+-----------+-------------------+-------------+
| /opt/starrocks/be/storage | 1.183 GB         | 130.260 GB        | 875.407 GB    | 1006.850 GB   | 13.05 %      | ONLINE | -1197932058022076895 | HDD           | 25498     | 876.590 GB        | 0.13 %      |
+---------------------------+------------------+-------------------+---------------+---------------+--------------+--------+----------------------+---------------+-----------+-------------------+-------------+
1 row in set (0.01 sec)
查看 StarRocks 相关配置
  • 查看当前 FE 配置参数
sql 复制代码 
mysql> ADMIN SHOW FRONTEND CONFIG LIKE '%trash%';
+-----------------------------+------------+-------+------+-----------+---------+
| Key                         | AliasNames | Value | Type | IsMutable | Comment |
+-----------------------------+------------+-------+------+-----------+---------+
| catalog_trash_expire_second | []         | 86400 | long | true      |         |
+-----------------------------+------------+-------+------+-----------+---------+
1 row in set (0.00 sec)
be 指标
sql 复制代码 
mysql> select * from information_schema.be_metrics where name like '%_mem_bytes%';
+-------+--------------------------------+--------+------------+
| BE_ID | NAME                           | LABELS | VALUE      |
+-------+--------------------------------+--------+------------+
| 10001 | bitmap_index_mem_bytes         |        |     599200 |
| 10001 | bloom_filter_index_mem_bytes   |        |     444512 |
| 10001 | chunk_allocator_mem_bytes      |        |          0 |
| 10001 | clone_mem_bytes                |        |          0 |
| 10001 | column_metadata_mem_bytes      |        |   75812784 |
| 10001 | column_zonemap_index_mem_bytes |        |   23616768 |
| 10001 | compaction_mem_bytes           |        |          0 |
| 10001 | consistency_mem_bytes          |        |          0 |
| 10001 | datacache_mem_bytes            |        |          0 |
| 10001 | jit_cache_mem_bytes            |        |          0 |
| 10001 | load_mem_bytes                 |        |          0 |
| 10001 | metadata_mem_bytes             |        |  237572764 |
| 10001 | ordinal_index_mem_bytes        |        |   15898064 |
| 10001 | process_mem_bytes              |        | 4519480168 |
| 10001 | query_mem_bytes                |        |          0 |
| 10001 | rowset_metadata_mem_bytes      |        |  132016490 |
| 10001 | schema_change_mem_bytes        |        |          0 |
| 10001 | segment_metadata_mem_bytes     |        |   16589775 |
| 10001 | segment_zonemap_mem_bytes      |        |   13885863 |
| 10001 | short_key_index_mem_bytes      |        |          0 |
| 10001 | storage_page_cache_mem_bytes   |        |     319538 |
| 10001 | tablet_metadata_mem_bytes      |        |   13153715 |
| 10001 | tablet_schema_mem_bytes        |        |      30283 |
| 10001 | update_mem_bytes               |        | 3502819939 |
+-------+--------------------------------+--------+------------+
72 rows in set (0.15 sec)
StreamLoad 导入相关
bash 复制代码 
### a.json
{"trend_side":"BUY","ask_user_id":14006,"buy_type":1,"sell_fee_coin":"USDT","bid_id":0,"mtime":"2026-02-09 13:07:08","ask_id":0,"volume":0.04,"buy_fee_coin":"DOT","buy_fee":0,"sell_type":1,"price":1.297,"ctime":"2026-02-09 13:07:08","id":23288,"bid_user_id":14006,"sell_fee":0,"source_table":"ex_trade_dotusdt","binlogFile":"snapshot","binlogPosition":0,"op":"r","ts":"2026-06-22 04:50:20","dt":"2026-02-09"}

### FE 负载均衡
curl --location-trusted \
   -u root:123456 \
   -H "label: json_load_002" \
   -H "format: json" \
  -H "strip_outer_array: false" \
   -H "strip_outer_array: false" \
   -H "Expect: 100-continue" \
   -T a.json \
   http://hadoop02:8030/api/test/ods_ex_trade_history/_stream_load
  
 ### BE 绕过FE
 curl --location-trusted \
   -u root:123456 \
   -H "label: json_load_003" \
   -H "format: json" \
  -H "strip_outer_array: false" \
   -H "strip_outer_array: false" \
   -H "Expect: 100-continue" \
   -T a.json \
   http://hadoop03:8040/api/test/ods_ex_trade_history/_stream_load
热门推荐
01GitHub 镜像站点022026 年 AI 编程工具终极横评：Cursor vs Claude Code vs Copilot vs Windsurf03AI科技热点日报 | 2026年07月01日04【AI】2026 年具身智能模型和世界模型总结052026 AI 编程工具终极实战指南：Cursor vs Claude Code vs Copilot，开发者该怎么选？062026年6月AI大模型全景报告：GPT-5.6、Claude Opus 4.8、Gemini 3.5，中美AI三足鼎立谁主沉浮？072026 年 AI 大模型 & AI 编程工具实战全总结08Claude Code、Codex、Cursor三分天下：2026年AI编程Agent生态全景剖析09Agnes AI 免费 API 接入指南：文本、生图、生视频，一套接口全免费102026年6月AI行业全景：从百模大战到Agent元年，这30天发生了什么？