Elasticsearch 集群容器化部署：构建 PB 级搜索与分析平台

分布式架构 + 容器化编排，打造生产级高可用数据洞察引擎。

一、为什么需要集群？

1.1 单节点的局限性

单节点往往用于个人学习或者测试环境，集群用于生产环境。单节点的缺点：
#mermaid-svg-3aSWCA9dVNgj9PQ4{font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:16px;fill:#333;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#mermaid-svg-3aSWCA9dVNgj9PQ4 .edge-animation-slow{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 50s linear infinite;stroke-linecap:round;}#mermaid-svg-3aSWCA9dVNgj9PQ4 .edge-animation-fast{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 20s linear infinite;stroke-linecap:round;}#mermaid-svg-3aSWCA9dVNgj9PQ4 .error-icon{fill:#552222;}#mermaid-svg-3aSWCA9dVNgj9PQ4 .error-text{fill:#552222;stroke:#552222;}#mermaid-svg-3aSWCA9dVNgj9PQ4 .edge-thickness-normal{stroke-width:1px;}#mermaid-svg-3aSWCA9dVNgj9PQ4 .edge-thickness-thick{stroke-width:3.5px;}#mermaid-svg-3aSWCA9dVNgj9PQ4 .edge-pattern-solid{stroke-dasharray:0;}#mermaid-svg-3aSWCA9dVNgj9PQ4 .edge-thickness-invisible{stroke-width:0;fill:none;}#mermaid-svg-3aSWCA9dVNgj9PQ4 .edge-pattern-dashed{stroke-dasharray:3;}#mermaid-svg-3aSWCA9dVNgj9PQ4 .edge-pattern-dotted{stroke-dasharray:2;}#mermaid-svg-3aSWCA9dVNgj9PQ4 .marker{fill:#333333;stroke:#333333;}#mermaid-svg-3aSWCA9dVNgj9PQ4 .marker.cross{stroke:#333333;}#mermaid-svg-3aSWCA9dVNgj9PQ4 svg{font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:16px;}#mermaid-svg-3aSWCA9dVNgj9PQ4 p{margin:0;}#mermaid-svg-3aSWCA9dVNgj9PQ4 .label{font-family:"trebuchet ms",verdana,arial,sans-serif;color:#333;}#mermaid-svg-3aSWCA9dVNgj9PQ4 .cluster-label text{fill:#333;}#mermaid-svg-3aSWCA9dVNgj9PQ4 .cluster-label span{color:#333;}#mermaid-svg-3aSWCA9dVNgj9PQ4 .cluster-label span p{background-color:transparent;}#mermaid-svg-3aSWCA9dVNgj9PQ4 .label text,#mermaid-svg-3aSWCA9dVNgj9PQ4 span{fill:#333;color:#333;}#mermaid-svg-3aSWCA9dVNgj9PQ4 .node rect,#mermaid-svg-3aSWCA9dVNgj9PQ4 .node circle,#mermaid-svg-3aSWCA9dVNgj9PQ4 .node ellipse,#mermaid-svg-3aSWCA9dVNgj9PQ4 .node polygon,#mermaid-svg-3aSWCA9dVNgj9PQ4 .node path{fill:#ECECFF;stroke:#9370DB;stroke-width:1px;}#mermaid-svg-3aSWCA9dVNgj9PQ4 .rough-node .label text,#mermaid-svg-3aSWCA9dVNgj9PQ4 .node .label text,#mermaid-svg-3aSWCA9dVNgj9PQ4 .image-shape .label,#mermaid-svg-3aSWCA9dVNgj9PQ4 .icon-shape .label{text-anchor:middle;}#mermaid-svg-3aSWCA9dVNgj9PQ4 .node .katex path{fill:#000;stroke:#000;stroke-width:1px;}#mermaid-svg-3aSWCA9dVNgj9PQ4 .rough-node .label,#mermaid-svg-3aSWCA9dVNgj9PQ4 .node .label,#mermaid-svg-3aSWCA9dVNgj9PQ4 .image-shape .label,#mermaid-svg-3aSWCA9dVNgj9PQ4 .icon-shape .label{text-align:center;}#mermaid-svg-3aSWCA9dVNgj9PQ4 .node.clickable{cursor:pointer;}#mermaid-svg-3aSWCA9dVNgj9PQ4 .root .anchor path{fill:#333333!important;stroke-width:0;stroke:#333333;}#mermaid-svg-3aSWCA9dVNgj9PQ4 .arrowheadPath{fill:#333333;}#mermaid-svg-3aSWCA9dVNgj9PQ4 .edgePath .path{stroke:#333333;stroke-width:2.0px;}#mermaid-svg-3aSWCA9dVNgj9PQ4 .flowchart-link{stroke:#333333;fill:none;}#mermaid-svg-3aSWCA9dVNgj9PQ4 .edgeLabel{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-3aSWCA9dVNgj9PQ4 .edgeLabel p{background-color:rgba(232,232,232, 0.8);}#mermaid-svg-3aSWCA9dVNgj9PQ4 .edgeLabel rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-3aSWCA9dVNgj9PQ4 .labelBkg{background-color:rgba(232, 232, 232, 0.5);}#mermaid-svg-3aSWCA9dVNgj9PQ4 .cluster rect{fill:#ffffde;stroke:#aaaa33;stroke-width:1px;}#mermaid-svg-3aSWCA9dVNgj9PQ4 .cluster text{fill:#333;}#mermaid-svg-3aSWCA9dVNgj9PQ4 .cluster span{color:#333;}#mermaid-svg-3aSWCA9dVNgj9PQ4 div.mermaidTooltip{position:absolute;text-align:center;max-width:200px;padding:2px;font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:12px;background:hsl(80, 100%, 96.2745098039%);border:1px solid #aaaa33;border-radius:2px;pointer-events:none;z-index:100;}#mermaid-svg-3aSWCA9dVNgj9PQ4 .flowchartTitleText{text-anchor:middle;font-size:18px;fill:#333;}#mermaid-svg-3aSWCA9dVNgj9PQ4 rect.text{fill:none;stroke-width:0;}#mermaid-svg-3aSWCA9dVNgj9PQ4 .icon-shape,#mermaid-svg-3aSWCA9dVNgj9PQ4 .image-shape{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-3aSWCA9dVNgj9PQ4 .icon-shape p,#mermaid-svg-3aSWCA9dVNgj9PQ4 .image-shape p{background-color:rgba(232,232,232, 0.8);padding:2px;}#mermaid-svg-3aSWCA9dVNgj9PQ4 .icon-shape .label rect,#mermaid-svg-3aSWCA9dVNgj9PQ4 .image-shape .label rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-3aSWCA9dVNgj9PQ4 .label-icon{display:inline-block;height:1em;overflow:visible;vertical-align:-0.125em;}#mermaid-svg-3aSWCA9dVNgj9PQ4 .node .label-icon path{fill:currentColor;stroke:revert;stroke-width:revert;}#mermaid-svg-3aSWCA9dVNgj9PQ4 :root{--mermaid-font-family:"trebuchet ms",verdana,arial,sans-serif;} 单节点 ES
Elasticsearch

单节点
❌ 单点故障

节点宕机 → 服务不可用
❌ 容量瓶颈

磁盘/内存无法扩展
❌ 性能瓶颈

查询压力集中
❌ 数据安全

无副本 → 数据丢失

单点故障：节点宕机导致服务不可用
容量瓶颈：磁盘和内存无法横向扩展
性能瓶颈：查询压力集中，响应变慢
数据安全：没有副本，数据丢失风险高

1.2 集群的核心价值

集群具有容错性，任意一台或者多台挂了都不会影响业务，所以生产环境都是集群模式部署：
#mermaid-svg-ts4gjIknHmeWo7gn{font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:16px;fill:#333;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#mermaid-svg-ts4gjIknHmeWo7gn .edge-animation-slow{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 50s linear infinite;stroke-linecap:round;}#mermaid-svg-ts4gjIknHmeWo7gn .edge-animation-fast{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 20s linear infinite;stroke-linecap:round;}#mermaid-svg-ts4gjIknHmeWo7gn .error-icon{fill:#552222;}#mermaid-svg-ts4gjIknHmeWo7gn .error-text{fill:#552222;stroke:#552222;}#mermaid-svg-ts4gjIknHmeWo7gn .edge-thickness-normal{stroke-width:1px;}#mermaid-svg-ts4gjIknHmeWo7gn .edge-thickness-thick{stroke-width:3.5px;}#mermaid-svg-ts4gjIknHmeWo7gn .edge-pattern-solid{stroke-dasharray:0;}#mermaid-svg-ts4gjIknHmeWo7gn .edge-thickness-invisible{stroke-width:0;fill:none;}#mermaid-svg-ts4gjIknHmeWo7gn .edge-pattern-dashed{stroke-dasharray:3;}#mermaid-svg-ts4gjIknHmeWo7gn .edge-pattern-dotted{stroke-dasharray:2;}#mermaid-svg-ts4gjIknHmeWo7gn .marker{fill:#333333;stroke:#333333;}#mermaid-svg-ts4gjIknHmeWo7gn .marker.cross{stroke:#333333;}#mermaid-svg-ts4gjIknHmeWo7gn svg{font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:16px;}#mermaid-svg-ts4gjIknHmeWo7gn p{margin:0;}#mermaid-svg-ts4gjIknHmeWo7gn .label{font-family:"trebuchet ms",verdana,arial,sans-serif;color:#333;}#mermaid-svg-ts4gjIknHmeWo7gn .cluster-label text{fill:#333;}#mermaid-svg-ts4gjIknHmeWo7gn .cluster-label span{color:#333;}#mermaid-svg-ts4gjIknHmeWo7gn .cluster-label span p{background-color:transparent;}#mermaid-svg-ts4gjIknHmeWo7gn .label text,#mermaid-svg-ts4gjIknHmeWo7gn span{fill:#333;color:#333;}#mermaid-svg-ts4gjIknHmeWo7gn .node rect,#mermaid-svg-ts4gjIknHmeWo7gn .node circle,#mermaid-svg-ts4gjIknHmeWo7gn .node ellipse,#mermaid-svg-ts4gjIknHmeWo7gn .node polygon,#mermaid-svg-ts4gjIknHmeWo7gn .node path{fill:#ECECFF;stroke:#9370DB;stroke-width:1px;}#mermaid-svg-ts4gjIknHmeWo7gn .rough-node .label text,#mermaid-svg-ts4gjIknHmeWo7gn .node .label text,#mermaid-svg-ts4gjIknHmeWo7gn .image-shape .label,#mermaid-svg-ts4gjIknHmeWo7gn .icon-shape .label{text-anchor:middle;}#mermaid-svg-ts4gjIknHmeWo7gn .node .katex path{fill:#000;stroke:#000;stroke-width:1px;}#mermaid-svg-ts4gjIknHmeWo7gn .rough-node .label,#mermaid-svg-ts4gjIknHmeWo7gn .node .label,#mermaid-svg-ts4gjIknHmeWo7gn .image-shape .label,#mermaid-svg-ts4gjIknHmeWo7gn .icon-shape .label{text-align:center;}#mermaid-svg-ts4gjIknHmeWo7gn .node.clickable{cursor:pointer;}#mermaid-svg-ts4gjIknHmeWo7gn .root .anchor path{fill:#333333!important;stroke-width:0;stroke:#333333;}#mermaid-svg-ts4gjIknHmeWo7gn .arrowheadPath{fill:#333333;}#mermaid-svg-ts4gjIknHmeWo7gn .edgePath .path{stroke:#333333;stroke-width:2.0px;}#mermaid-svg-ts4gjIknHmeWo7gn .flowchart-link{stroke:#333333;fill:none;}#mermaid-svg-ts4gjIknHmeWo7gn .edgeLabel{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-ts4gjIknHmeWo7gn .edgeLabel p{background-color:rgba(232,232,232, 0.8);}#mermaid-svg-ts4gjIknHmeWo7gn .edgeLabel rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-ts4gjIknHmeWo7gn .labelBkg{background-color:rgba(232, 232, 232, 0.5);}#mermaid-svg-ts4gjIknHmeWo7gn .cluster rect{fill:#ffffde;stroke:#aaaa33;stroke-width:1px;}#mermaid-svg-ts4gjIknHmeWo7gn .cluster text{fill:#333;}#mermaid-svg-ts4gjIknHmeWo7gn .cluster span{color:#333;}#mermaid-svg-ts4gjIknHmeWo7gn div.mermaidTooltip{position:absolute;text-align:center;max-width:200px;padding:2px;font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:12px;background:hsl(80, 100%, 96.2745098039%);border:1px solid #aaaa33;border-radius:2px;pointer-events:none;z-index:100;}#mermaid-svg-ts4gjIknHmeWo7gn .flowchartTitleText{text-anchor:middle;font-size:18px;fill:#333;}#mermaid-svg-ts4gjIknHmeWo7gn rect.text{fill:none;stroke-width:0;}#mermaid-svg-ts4gjIknHmeWo7gn .icon-shape,#mermaid-svg-ts4gjIknHmeWo7gn .image-shape{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-ts4gjIknHmeWo7gn .icon-shape p,#mermaid-svg-ts4gjIknHmeWo7gn .image-shape p{background-color:rgba(232,232,232, 0.8);padding:2px;}#mermaid-svg-ts4gjIknHmeWo7gn .icon-shape .label rect,#mermaid-svg-ts4gjIknHmeWo7gn .image-shape .label rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-ts4gjIknHmeWo7gn .label-icon{display:inline-block;height:1em;overflow:visible;vertical-align:-0.125em;}#mermaid-svg-ts4gjIknHmeWo7gn .node .label-icon path{fill:currentColor;stroke:revert;stroke-width:revert;}#mermaid-svg-ts4gjIknHmeWo7gn :root{--mermaid-font-family:"trebuchet ms",verdana,arial,sans-serif;} 集群优势
ES 集群
Master 1
Master 2
Master 3
Data 1

热节点 SSD
Data 2

热节点 SSD
Data 3

温节点 HDD
✅ 高可用

副本分片 → 故障不丢数据
✅ 水平扩展

动态加节点
✅ 负载均衡

查询分散到多节点
✅ 冷热分离

SSD + HDD 分层

高可用：副本分片保证节点故障时数据不丢失
水平扩展：动态添加节点提升存储和计算能力
负载均衡：查询请求分散到多个节点
冷热分离：热数据 SSD 加速，冷数据 HDD 归档

二、集群规划

一般 ES 集群需要的节点数比较多，常规的有 11 台，可以根据业务来进行扩容。

2.1 节点角色设计

#mermaid-svg-nq8BGfVcoYK0PXBB{font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:16px;fill:#333;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#mermaid-svg-nq8BGfVcoYK0PXBB .edge-animation-slow{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 50s linear infinite;stroke-linecap:round;}#mermaid-svg-nq8BGfVcoYK0PXBB .edge-animation-fast{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 20s linear infinite;stroke-linecap:round;}#mermaid-svg-nq8BGfVcoYK0PXBB .error-icon{fill:#552222;}#mermaid-svg-nq8BGfVcoYK0PXBB .error-text{fill:#552222;stroke:#552222;}#mermaid-svg-nq8BGfVcoYK0PXBB .edge-thickness-normal{stroke-width:1px;}#mermaid-svg-nq8BGfVcoYK0PXBB .edge-thickness-thick{stroke-width:3.5px;}#mermaid-svg-nq8BGfVcoYK0PXBB .edge-pattern-solid{stroke-dasharray:0;}#mermaid-svg-nq8BGfVcoYK0PXBB .edge-thickness-invisible{stroke-width:0;fill:none;}#mermaid-svg-nq8BGfVcoYK0PXBB .edge-pattern-dashed{stroke-dasharray:3;}#mermaid-svg-nq8BGfVcoYK0PXBB .edge-pattern-dotted{stroke-dasharray:2;}#mermaid-svg-nq8BGfVcoYK0PXBB .marker{fill:#333333;stroke:#333333;}#mermaid-svg-nq8BGfVcoYK0PXBB .marker.cross{stroke:#333333;}#mermaid-svg-nq8BGfVcoYK0PXBB svg{font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:16px;}#mermaid-svg-nq8BGfVcoYK0PXBB p{margin:0;}#mermaid-svg-nq8BGfVcoYK0PXBB .label{font-family:"trebuchet ms",verdana,arial,sans-serif;color:#333;}#mermaid-svg-nq8BGfVcoYK0PXBB .cluster-label text{fill:#333;}#mermaid-svg-nq8BGfVcoYK0PXBB .cluster-label span{color:#333;}#mermaid-svg-nq8BGfVcoYK0PXBB .cluster-label span p{background-color:transparent;}#mermaid-svg-nq8BGfVcoYK0PXBB .label text,#mermaid-svg-nq8BGfVcoYK0PXBB span{fill:#333;color:#333;}#mermaid-svg-nq8BGfVcoYK0PXBB .node rect,#mermaid-svg-nq8BGfVcoYK0PXBB .node circle,#mermaid-svg-nq8BGfVcoYK0PXBB .node ellipse,#mermaid-svg-nq8BGfVcoYK0PXBB .node polygon,#mermaid-svg-nq8BGfVcoYK0PXBB .node path{fill:#ECECFF;stroke:#9370DB;stroke-width:1px;}#mermaid-svg-nq8BGfVcoYK0PXBB .rough-node .label text,#mermaid-svg-nq8BGfVcoYK0PXBB .node .label text,#mermaid-svg-nq8BGfVcoYK0PXBB .image-shape .label,#mermaid-svg-nq8BGfVcoYK0PXBB .icon-shape .label{text-anchor:middle;}#mermaid-svg-nq8BGfVcoYK0PXBB .node .katex path{fill:#000;stroke:#000;stroke-width:1px;}#mermaid-svg-nq8BGfVcoYK0PXBB .rough-node .label,#mermaid-svg-nq8BGfVcoYK0PXBB .node .label,#mermaid-svg-nq8BGfVcoYK0PXBB .image-shape .label,#mermaid-svg-nq8BGfVcoYK0PXBB .icon-shape .label{text-align:center;}#mermaid-svg-nq8BGfVcoYK0PXBB .node.clickable{cursor:pointer;}#mermaid-svg-nq8BGfVcoYK0PXBB .root .anchor path{fill:#333333!important;stroke-width:0;stroke:#333333;}#mermaid-svg-nq8BGfVcoYK0PXBB .arrowheadPath{fill:#333333;}#mermaid-svg-nq8BGfVcoYK0PXBB .edgePath .path{stroke:#333333;stroke-width:2.0px;}#mermaid-svg-nq8BGfVcoYK0PXBB .flowchart-link{stroke:#333333;fill:none;}#mermaid-svg-nq8BGfVcoYK0PXBB .edgeLabel{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-nq8BGfVcoYK0PXBB .edgeLabel p{background-color:rgba(232,232,232, 0.8);}#mermaid-svg-nq8BGfVcoYK0PXBB .edgeLabel rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-nq8BGfVcoYK0PXBB .labelBkg{background-color:rgba(232, 232, 232, 0.5);}#mermaid-svg-nq8BGfVcoYK0PXBB .cluster rect{fill:#ffffde;stroke:#aaaa33;stroke-width:1px;}#mermaid-svg-nq8BGfVcoYK0PXBB .cluster text{fill:#333;}#mermaid-svg-nq8BGfVcoYK0PXBB .cluster span{color:#333;}#mermaid-svg-nq8BGfVcoYK0PXBB div.mermaidTooltip{position:absolute;text-align:center;max-width:200px;padding:2px;font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:12px;background:hsl(80, 100%, 96.2745098039%);border:1px solid #aaaa33;border-radius:2px;pointer-events:none;z-index:100;}#mermaid-svg-nq8BGfVcoYK0PXBB .flowchartTitleText{text-anchor:middle;font-size:18px;fill:#333;}#mermaid-svg-nq8BGfVcoYK0PXBB rect.text{fill:none;stroke-width:0;}#mermaid-svg-nq8BGfVcoYK0PXBB .icon-shape,#mermaid-svg-nq8BGfVcoYK0PXBB .image-shape{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-nq8BGfVcoYK0PXBB .icon-shape p,#mermaid-svg-nq8BGfVcoYK0PXBB .image-shape p{background-color:rgba(232,232,232, 0.8);padding:2px;}#mermaid-svg-nq8BGfVcoYK0PXBB .icon-shape .label rect,#mermaid-svg-nq8BGfVcoYK0PXBB .image-shape .label rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-nq8BGfVcoYK0PXBB .label-icon{display:inline-block;height:1em;overflow:visible;vertical-align:-0.125em;}#mermaid-svg-nq8BGfVcoYK0PXBB .node .label-icon path{fill:currentColor;stroke:revert;stroke-width:revert;}#mermaid-svg-nq8BGfVcoYK0PXBB :root{--mermaid-font-family:"trebuchet ms",verdana,arial,sans-serif;} 数据层
主节点层
协调层
客户端
集群状态管理
集群状态管理
应用 / Kibana

查询请求
Coordinating 节点

× 2

4核16GB
Master 节点

× 3

4核8GB
Hot Data 节点

× 3

8核32GB + SSD
Warm Data 节点

× 3

8核32GB + HDD

节点名称	角色	配置建议	数量
master	仅 master	4 核 8GB	3
hot-data	data + ingest	8 核 32GB + SSD	3
warm-data	data	8 核 32GB + HDD	3
coordinating	仅 coordinating	4 核 16GB	2

2.2 环境准备

这里我们选择用 Docker Compose 部署，很多组件集成在一起，好部署。Docker Engine 20.10+，Docker Compose 2.0+。

所有宿主机需调整 vm.max_map_count=262144
集群节点间网络互通（建议使用 Docker overlay 网络）

三、3 节点集群部署步骤

#mermaid-svg-Ep2qqgSLdN7pRc0R{font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:16px;fill:#333;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#mermaid-svg-Ep2qqgSLdN7pRc0R .edge-animation-slow{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 50s linear infinite;stroke-linecap:round;}#mermaid-svg-Ep2qqgSLdN7pRc0R .edge-animation-fast{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 20s linear infinite;stroke-linecap:round;}#mermaid-svg-Ep2qqgSLdN7pRc0R .error-icon{fill:#552222;}#mermaid-svg-Ep2qqgSLdN7pRc0R .error-text{fill:#552222;stroke:#552222;}#mermaid-svg-Ep2qqgSLdN7pRc0R .edge-thickness-normal{stroke-width:1px;}#mermaid-svg-Ep2qqgSLdN7pRc0R .edge-thickness-thick{stroke-width:3.5px;}#mermaid-svg-Ep2qqgSLdN7pRc0R .edge-pattern-solid{stroke-dasharray:0;}#mermaid-svg-Ep2qqgSLdN7pRc0R .edge-thickness-invisible{stroke-width:0;fill:none;}#mermaid-svg-Ep2qqgSLdN7pRc0R .edge-pattern-dashed{stroke-dasharray:3;}#mermaid-svg-Ep2qqgSLdN7pRc0R .edge-pattern-dotted{stroke-dasharray:2;}#mermaid-svg-Ep2qqgSLdN7pRc0R .marker{fill:#333333;stroke:#333333;}#mermaid-svg-Ep2qqgSLdN7pRc0R .marker.cross{stroke:#333333;}#mermaid-svg-Ep2qqgSLdN7pRc0R svg{font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:16px;}#mermaid-svg-Ep2qqgSLdN7pRc0R p{margin:0;}#mermaid-svg-Ep2qqgSLdN7pRc0R .label{font-family:"trebuchet ms",verdana,arial,sans-serif;color:#333;}#mermaid-svg-Ep2qqgSLdN7pRc0R .cluster-label text{fill:#333;}#mermaid-svg-Ep2qqgSLdN7pRc0R .cluster-label span{color:#333;}#mermaid-svg-Ep2qqgSLdN7pRc0R .cluster-label span p{background-color:transparent;}#mermaid-svg-Ep2qqgSLdN7pRc0R .label text,#mermaid-svg-Ep2qqgSLdN7pRc0R span{fill:#333;color:#333;}#mermaid-svg-Ep2qqgSLdN7pRc0R .node rect,#mermaid-svg-Ep2qqgSLdN7pRc0R .node circle,#mermaid-svg-Ep2qqgSLdN7pRc0R .node ellipse,#mermaid-svg-Ep2qqgSLdN7pRc0R .node polygon,#mermaid-svg-Ep2qqgSLdN7pRc0R .node path{fill:#ECECFF;stroke:#9370DB;stroke-width:1px;}#mermaid-svg-Ep2qqgSLdN7pRc0R .rough-node .label text,#mermaid-svg-Ep2qqgSLdN7pRc0R .node .label text,#mermaid-svg-Ep2qqgSLdN7pRc0R .image-shape .label,#mermaid-svg-Ep2qqgSLdN7pRc0R .icon-shape .label{text-anchor:middle;}#mermaid-svg-Ep2qqgSLdN7pRc0R .node .katex path{fill:#000;stroke:#000;stroke-width:1px;}#mermaid-svg-Ep2qqgSLdN7pRc0R .rough-node .label,#mermaid-svg-Ep2qqgSLdN7pRc0R .node .label,#mermaid-svg-Ep2qqgSLdN7pRc0R .image-shape .label,#mermaid-svg-Ep2qqgSLdN7pRc0R .icon-shape .label{text-align:center;}#mermaid-svg-Ep2qqgSLdN7pRc0R .node.clickable{cursor:pointer;}#mermaid-svg-Ep2qqgSLdN7pRc0R .root .anchor path{fill:#333333!important;stroke-width:0;stroke:#333333;}#mermaid-svg-Ep2qqgSLdN7pRc0R .arrowheadPath{fill:#333333;}#mermaid-svg-Ep2qqgSLdN7pRc0R .edgePath .path{stroke:#333333;stroke-width:2.0px;}#mermaid-svg-Ep2qqgSLdN7pRc0R .flowchart-link{stroke:#333333;fill:none;}#mermaid-svg-Ep2qqgSLdN7pRc0R .edgeLabel{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-Ep2qqgSLdN7pRc0R .edgeLabel p{background-color:rgba(232,232,232, 0.8);}#mermaid-svg-Ep2qqgSLdN7pRc0R .edgeLabel rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-Ep2qqgSLdN7pRc0R .labelBkg{background-color:rgba(232, 232, 232, 0.5);}#mermaid-svg-Ep2qqgSLdN7pRc0R .cluster rect{fill:#ffffde;stroke:#aaaa33;stroke-width:1px;}#mermaid-svg-Ep2qqgSLdN7pRc0R .cluster text{fill:#333;}#mermaid-svg-Ep2qqgSLdN7pRc0R .cluster span{color:#333;}#mermaid-svg-Ep2qqgSLdN7pRc0R div.mermaidTooltip{position:absolute;text-align:center;max-width:200px;padding:2px;font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:12px;background:hsl(80, 100%, 96.2745098039%);border:1px solid #aaaa33;border-radius:2px;pointer-events:none;z-index:100;}#mermaid-svg-Ep2qqgSLdN7pRc0R .flowchartTitleText{text-anchor:middle;font-size:18px;fill:#333;}#mermaid-svg-Ep2qqgSLdN7pRc0R rect.text{fill:none;stroke-width:0;}#mermaid-svg-Ep2qqgSLdN7pRc0R .icon-shape,#mermaid-svg-Ep2qqgSLdN7pRc0R .image-shape{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-Ep2qqgSLdN7pRc0R .icon-shape p,#mermaid-svg-Ep2qqgSLdN7pRc0R .image-shape p{background-color:rgba(232,232,232, 0.8);padding:2px;}#mermaid-svg-Ep2qqgSLdN7pRc0R .icon-shape .label rect,#mermaid-svg-Ep2qqgSLdN7pRc0R .image-shape .label rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-Ep2qqgSLdN7pRc0R .label-icon{display:inline-block;height:1em;overflow:visible;vertical-align:-0.125em;}#mermaid-svg-Ep2qqgSLdN7pRc0R .node .label-icon path{fill:currentColor;stroke:revert;stroke-width:revert;}#mermaid-svg-Ep2qqgSLdN7pRc0R :root{--mermaid-font-family:"trebuchet ms",verdana,arial,sans-serif;} 内部通信
3 节点 ES 集群
9300
9300
9300
es01

Master + Data

端口 9201
es02

Master + Data

端口 9202
es03

Master + Data

端口 9203
discovery.seed_hosts

节点发现
transport.ssl

TLS 加密通信

3.1 创建目录结构

不同目录存储不同的重要文件，有利于后期管理：

bash 复制代码

mkdir -p /data/es-cluster/{node1,node2,node3}/{data,logs,config,certs}
chmod 755 /data/es-cluster/*/data

3.2 生成集群证书

证书是保障安全的重要机制，集群节点间通信需 TLS 加密，使用官方工具生成证书：

bash 复制代码

# 生成 CA
docker run --rm -v /data/es-cluster/certs:/certs docker.elastic.co/elasticsearch/elasticsearch:8.12.0 \
  bash -c "bin/elasticsearch-certutil ca --pem --out /certs/ca.zip && unzip /certs/ca.zip -d /certs"

# 为每个节点生成证书
for node in node1 node2 node3; do
  docker run --rm -v /data/es-cluster/certs:/certs docker.elastic.co/elasticsearch/elasticsearch:8.12.0 \
    bash -c "bin/elasticsearch-certutil cert --pem --ca-cert /certs/ca/ca.crt --ca-key /certs/ca/ca.key --dns ${node} --ip 127.0.0.1 --out /certs/${node}.zip && unzip /certs/${node}.zip -d /certs/${node}"
done

3.3 编写 docker-compose.yml

yaml 复制代码

version: '3.8'

services:
  es01:
    image: docker.elastic.co/elasticsearch/elasticsearch:8.12.0
    container_name: es01
    environment:
      - node.name=es01
      - cluster.name=prod-cluster
      - discovery.seed_hosts=es02,es03
      - cluster.initial_master_nodes=es01,es02,es03
      - bootstrap.memory_lock=true
      - "ES_JAVA_OPTS=-Xms4g -Xmx4g"
      - xpack.security.enabled=true
      - xpack.security.transport.ssl.enabled=true
      - xpack.security.transport.ssl.verification_mode=certificate
      - xpack.security.transport.ssl.keystore.path=/usr/share/elasticsearch/config/certs/elastic-certificates.p12
      - xpack.security.transport.ssl.truststore.path=/usr/share/elasticsearch/config/certs/elastic-certificates.p12
      - xpack.security.http.ssl.enabled=true
      - xpack.security.http.ssl.keystore.path=/usr/share/elasticsearch/config/certs/http.p12
    ulimits:
      memlock:
        soft: -1
        hard: -1
    volumes:
      - es-data01:/usr/share/elasticsearch/data
      - ./certs/es01:/usr/share/elasticsearch/config/certs
    ports:
      - "9201:9200"
    networks:
      - elastic
    healthcheck:
      test: ["CMD", "curl", "-f", "-k", "https://localhost:9200"]
      interval: 30s

  es02:
    image: docker.elastic.co/elasticsearch/elasticsearch:8.12.0
    container_name: es02
    environment:
      - node.name=es02
      - cluster.name=prod-cluster
      - discovery.seed_hosts=es01,es03
      - cluster.initial_master_nodes=es01,es02,es03
      - bootstrap.memory_lock=true
      - "ES_JAVA_OPTS=-Xms4g -Xmx4g"
      - xpack.security.enabled=true
      - xpack.security.transport.ssl.enabled=true
      - xpack.security.transport.ssl.verification_mode=certificate
      - xpack.security.transport.ssl.keystore.path=/usr/share/elasticsearch/config/certs/elastic-certificates.p12
      - xpack.security.transport.ssl.truststore.path=/usr/share/elasticsearch/config/certs/elastic-certificates.p12
      - xpack.security.http.ssl.enabled=true
      - xpack.security.http.ssl.keystore.path=/usr/share/elasticsearch/config/certs/http.p12
    ulimits:
      memlock:
        soft: -1
        hard: -1
    volumes:
      - es-data02:/usr/share/elasticsearch/data
      - ./certs/es02:/usr/share/elasticsearch/config/certs
    networks:
      - elastic

  es03:
    image: docker.elastic.co/elasticsearch/elasticsearch:8.12.0
    container_name: es03
    environment:
      - node.name=es03
      - cluster.name=prod-cluster
      - discovery.seed_hosts=es01,es02
      - cluster.initial_master_nodes=es01,es02,es03
      - bootstrap.memory_lock=true
      - "ES_JAVA_OPTS=-Xms4g -Xmx4g"
      - xpack.security.enabled=true
      - xpack.security.transport.ssl.enabled=true
      - xpack.security.transport.ssl.verification_mode=certificate
      - xpack.security.transport.ssl.keystore.path=/usr/share/elasticsearch/config/certs/elastic-certificates.p12
      - xpack.security.transport.ssl.truststore.path=/usr/share/elasticsearch/config/certs/elastic-certificates.p12
      - xpack.security.http.ssl.enabled=true
      - xpack.security.http.ssl.keystore.path=/usr/share/elasticsearch/config/certs/http.p12
    ulimits:
      memlock:
        soft: -1
        hard: -1
    volumes:
      - es-data03:/usr/share/elasticsearch/data
      - ./certs/es03:/usr/share/elasticsearch/config/certs
    networks:
      - elastic

volumes:
  es-data01:
    driver: local
  es-data02:
    driver: local
  es-data03:
    driver: local

networks:
  elastic:
    driver: bridge

3.4 启动集群

bash 复制代码

# 设置 elastic 用户密码（可选，可通过环境变量传递）
export ELASTIC_PASSWORD=YourStrongPassword

# 启动
docker-compose up -d

# 查看日志
docker-compose logs -f

3.5 验证集群

bash 复制代码

# 通过任一节点查看集群健康
curl -k -u elastic:YourStrongPassword https://localhost:9201/_cluster/health?pretty

# 期望输出：
{
  "cluster_name" : "prod-cluster",
  "status" : "green",
  "number_of_nodes" : 3,
  "number_of_data_nodes" : 3,
  ...
}

四、集群部署流程图

集群完整的部署流程如下所示：
#mermaid-svg-nLqkkVPht9p8K5UE{font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:16px;fill:#333;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#mermaid-svg-nLqkkVPht9p8K5UE .edge-animation-slow{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 50s linear infinite;stroke-linecap:round;}#mermaid-svg-nLqkkVPht9p8K5UE .edge-animation-fast{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 20s linear infinite;stroke-linecap:round;}#mermaid-svg-nLqkkVPht9p8K5UE .error-icon{fill:#552222;}#mermaid-svg-nLqkkVPht9p8K5UE .error-text{fill:#552222;stroke:#552222;}#mermaid-svg-nLqkkVPht9p8K5UE .edge-thickness-normal{stroke-width:1px;}#mermaid-svg-nLqkkVPht9p8K5UE .edge-thickness-thick{stroke-width:3.5px;}#mermaid-svg-nLqkkVPht9p8K5UE .edge-pattern-solid{stroke-dasharray:0;}#mermaid-svg-nLqkkVPht9p8K5UE .edge-thickness-invisible{stroke-width:0;fill:none;}#mermaid-svg-nLqkkVPht9p8K5UE .edge-pattern-dashed{stroke-dasharray:3;}#mermaid-svg-nLqkkVPht9p8K5UE .edge-pattern-dotted{stroke-dasharray:2;}#mermaid-svg-nLqkkVPht9p8K5UE .marker{fill:#333333;stroke:#333333;}#mermaid-svg-nLqkkVPht9p8K5UE .marker.cross{stroke:#333333;}#mermaid-svg-nLqkkVPht9p8K5UE svg{font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:16px;}#mermaid-svg-nLqkkVPht9p8K5UE p{margin:0;}#mermaid-svg-nLqkkVPht9p8K5UE .label{font-family:"trebuchet ms",verdana,arial,sans-serif;color:#333;}#mermaid-svg-nLqkkVPht9p8K5UE .cluster-label text{fill:#333;}#mermaid-svg-nLqkkVPht9p8K5UE .cluster-label span{color:#333;}#mermaid-svg-nLqkkVPht9p8K5UE .cluster-label span p{background-color:transparent;}#mermaid-svg-nLqkkVPht9p8K5UE .label text,#mermaid-svg-nLqkkVPht9p8K5UE span{fill:#333;color:#333;}#mermaid-svg-nLqkkVPht9p8K5UE .node rect,#mermaid-svg-nLqkkVPht9p8K5UE .node circle,#mermaid-svg-nLqkkVPht9p8K5UE .node ellipse,#mermaid-svg-nLqkkVPht9p8K5UE .node polygon,#mermaid-svg-nLqkkVPht9p8K5UE .node path{fill:#ECECFF;stroke:#9370DB;stroke-width:1px;}#mermaid-svg-nLqkkVPht9p8K5UE .rough-node .label text,#mermaid-svg-nLqkkVPht9p8K5UE .node .label text,#mermaid-svg-nLqkkVPht9p8K5UE .image-shape .label,#mermaid-svg-nLqkkVPht9p8K5UE .icon-shape .label{text-anchor:middle;}#mermaid-svg-nLqkkVPht9p8K5UE .node .katex path{fill:#000;stroke:#000;stroke-width:1px;}#mermaid-svg-nLqkkVPht9p8K5UE .rough-node .label,#mermaid-svg-nLqkkVPht9p8K5UE .node .label,#mermaid-svg-nLqkkVPht9p8K5UE .image-shape .label,#mermaid-svg-nLqkkVPht9p8K5UE .icon-shape .label{text-align:center;}#mermaid-svg-nLqkkVPht9p8K5UE .node.clickable{cursor:pointer;}#mermaid-svg-nLqkkVPht9p8K5UE .root .anchor path{fill:#333333!important;stroke-width:0;stroke:#333333;}#mermaid-svg-nLqkkVPht9p8K5UE .arrowheadPath{fill:#333333;}#mermaid-svg-nLqkkVPht9p8K5UE .edgePath .path{stroke:#333333;stroke-width:2.0px;}#mermaid-svg-nLqkkVPht9p8K5UE .flowchart-link{stroke:#333333;fill:none;}#mermaid-svg-nLqkkVPht9p8K5UE .edgeLabel{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-nLqkkVPht9p8K5UE .edgeLabel p{background-color:rgba(232,232,232, 0.8);}#mermaid-svg-nLqkkVPht9p8K5UE .edgeLabel rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-nLqkkVPht9p8K5UE .labelBkg{background-color:rgba(232, 232, 232, 0.5);}#mermaid-svg-nLqkkVPht9p8K5UE .cluster rect{fill:#ffffde;stroke:#aaaa33;stroke-width:1px;}#mermaid-svg-nLqkkVPht9p8K5UE .cluster text{fill:#333;}#mermaid-svg-nLqkkVPht9p8K5UE .cluster span{color:#333;}#mermaid-svg-nLqkkVPht9p8K5UE div.mermaidTooltip{position:absolute;text-align:center;max-width:200px;padding:2px;font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:12px;background:hsl(80, 100%, 96.2745098039%);border:1px solid #aaaa33;border-radius:2px;pointer-events:none;z-index:100;}#mermaid-svg-nLqkkVPht9p8K5UE .flowchartTitleText{text-anchor:middle;font-size:18px;fill:#333;}#mermaid-svg-nLqkkVPht9p8K5UE rect.text{fill:none;stroke-width:0;}#mermaid-svg-nLqkkVPht9p8K5UE .icon-shape,#mermaid-svg-nLqkkVPht9p8K5UE .image-shape{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-nLqkkVPht9p8K5UE .icon-shape p,#mermaid-svg-nLqkkVPht9p8K5UE .image-shape p{background-color:rgba(232,232,232, 0.8);padding:2px;}#mermaid-svg-nLqkkVPht9p8K5UE .icon-shape .label rect,#mermaid-svg-nLqkkVPht9p8K5UE .image-shape .label rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-nLqkkVPht9p8K5UE .label-icon{display:inline-block;height:1em;overflow:visible;vertical-align:-0.125em;}#mermaid-svg-nLqkkVPht9p8K5UE .node .label-icon path{fill:currentColor;stroke:revert;stroke-width:revert;}#mermaid-svg-nLqkkVPht9p8K5UE :root{--mermaid-font-family:"trebuchet ms",verdana,arial,sans-serif;} 验证
编排部署
证书生成
环境准备
调整 vm.max_map_count

sysctl -w 262144
创建目录结构

/data/es-cluster/{node1,node2,node3}
生成 CA 证书

elasticsearch-certutil ca
为每个节点生成证书

node1/node2/node3
编写 docker-compose.yml

3 节点配置
docker-compose up -d

启动集群
查看集群健康

_cluster/health → green

五、企业实战案例：PB 级日志平台

5.1 背景

某大型电商平台有 600+ 微服务，每天有 20TB 日志，需要构建统一日志平台，支持实时检索、监控告警和长期归档，数据还需要进行热备、冷备等。

5.2 需求

存储 30 天日志，总量约 600TB，要进行定时合理清理
查询响应时间 < 5 秒
支持冷热数据分层，降低存储成本
7×24 小时高可用

5.3 实施方案

5.3.1 集群架构

针对日志平台，需要的节点数量相应多很多，这次用了 15 台节点：

节点角色	数量	配置	职责
主节点	3	4 核 8GB	管理集群状态
热节点	5	8 核 32GB + SSD	实时写入 + 最近 7 天查询
温节点	5	8 核 32GB + HDD	存储 7-30 天历史数据
协调节点	2	4 核 16GB	分担查询压力

5.3.2 索引生命周期管理（ILM）

索引是 ES 的关键，通过索引可以快速查询：
#mermaid-svg-GrBEnwQ09kV0Rwgn{font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:16px;fill:#333;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#mermaid-svg-GrBEnwQ09kV0Rwgn .edge-animation-slow{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 50s linear infinite;stroke-linecap:round;}#mermaid-svg-GrBEnwQ09kV0Rwgn .edge-animation-fast{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 20s linear infinite;stroke-linecap:round;}#mermaid-svg-GrBEnwQ09kV0Rwgn .error-icon{fill:#552222;}#mermaid-svg-GrBEnwQ09kV0Rwgn .error-text{fill:#552222;stroke:#552222;}#mermaid-svg-GrBEnwQ09kV0Rwgn .edge-thickness-normal{stroke-width:1px;}#mermaid-svg-GrBEnwQ09kV0Rwgn .edge-thickness-thick{stroke-width:3.5px;}#mermaid-svg-GrBEnwQ09kV0Rwgn .edge-pattern-solid{stroke-dasharray:0;}#mermaid-svg-GrBEnwQ09kV0Rwgn .edge-thickness-invisible{stroke-width:0;fill:none;}#mermaid-svg-GrBEnwQ09kV0Rwgn .edge-pattern-dashed{stroke-dasharray:3;}#mermaid-svg-GrBEnwQ09kV0Rwgn .edge-pattern-dotted{stroke-dasharray:2;}#mermaid-svg-GrBEnwQ09kV0Rwgn .marker{fill:#333333;stroke:#333333;}#mermaid-svg-GrBEnwQ09kV0Rwgn .marker.cross{stroke:#333333;}#mermaid-svg-GrBEnwQ09kV0Rwgn svg{font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:16px;}#mermaid-svg-GrBEnwQ09kV0Rwgn p{margin:0;}#mermaid-svg-GrBEnwQ09kV0Rwgn .label{font-family:"trebuchet ms",verdana,arial,sans-serif;color:#333;}#mermaid-svg-GrBEnwQ09kV0Rwgn .cluster-label text{fill:#333;}#mermaid-svg-GrBEnwQ09kV0Rwgn .cluster-label span{color:#333;}#mermaid-svg-GrBEnwQ09kV0Rwgn .cluster-label span p{background-color:transparent;}#mermaid-svg-GrBEnwQ09kV0Rwgn .label text,#mermaid-svg-GrBEnwQ09kV0Rwgn span{fill:#333;color:#333;}#mermaid-svg-GrBEnwQ09kV0Rwgn .node rect,#mermaid-svg-GrBEnwQ09kV0Rwgn .node circle,#mermaid-svg-GrBEnwQ09kV0Rwgn .node ellipse,#mermaid-svg-GrBEnwQ09kV0Rwgn .node polygon,#mermaid-svg-GrBEnwQ09kV0Rwgn .node path{fill:#ECECFF;stroke:#9370DB;stroke-width:1px;}#mermaid-svg-GrBEnwQ09kV0Rwgn .rough-node .label text,#mermaid-svg-GrBEnwQ09kV0Rwgn .node .label text,#mermaid-svg-GrBEnwQ09kV0Rwgn .image-shape .label,#mermaid-svg-GrBEnwQ09kV0Rwgn .icon-shape .label{text-anchor:middle;}#mermaid-svg-GrBEnwQ09kV0Rwgn .node .katex path{fill:#000;stroke:#000;stroke-width:1px;}#mermaid-svg-GrBEnwQ09kV0Rwgn .rough-node .label,#mermaid-svg-GrBEnwQ09kV0Rwgn .node .label,#mermaid-svg-GrBEnwQ09kV0Rwgn .image-shape .label,#mermaid-svg-GrBEnwQ09kV0Rwgn .icon-shape .label{text-align:center;}#mermaid-svg-GrBEnwQ09kV0Rwgn .node.clickable{cursor:pointer;}#mermaid-svg-GrBEnwQ09kV0Rwgn .root .anchor path{fill:#333333!important;stroke-width:0;stroke:#333333;}#mermaid-svg-GrBEnwQ09kV0Rwgn .arrowheadPath{fill:#333333;}#mermaid-svg-GrBEnwQ09kV0Rwgn .edgePath .path{stroke:#333333;stroke-width:2.0px;}#mermaid-svg-GrBEnwQ09kV0Rwgn .flowchart-link{stroke:#333333;fill:none;}#mermaid-svg-GrBEnwQ09kV0Rwgn .edgeLabel{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-GrBEnwQ09kV0Rwgn .edgeLabel p{background-color:rgba(232,232,232, 0.8);}#mermaid-svg-GrBEnwQ09kV0Rwgn .edgeLabel rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-GrBEnwQ09kV0Rwgn .labelBkg{background-color:rgba(232, 232, 232, 0.5);}#mermaid-svg-GrBEnwQ09kV0Rwgn .cluster rect{fill:#ffffde;stroke:#aaaa33;stroke-width:1px;}#mermaid-svg-GrBEnwQ09kV0Rwgn .cluster text{fill:#333;}#mermaid-svg-GrBEnwQ09kV0Rwgn .cluster span{color:#333;}#mermaid-svg-GrBEnwQ09kV0Rwgn div.mermaidTooltip{position:absolute;text-align:center;max-width:200px;padding:2px;font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:12px;background:hsl(80, 100%, 96.2745098039%);border:1px solid #aaaa33;border-radius:2px;pointer-events:none;z-index:100;}#mermaid-svg-GrBEnwQ09kV0Rwgn .flowchartTitleText{text-anchor:middle;font-size:18px;fill:#333;}#mermaid-svg-GrBEnwQ09kV0Rwgn rect.text{fill:none;stroke-width:0;}#mermaid-svg-GrBEnwQ09kV0Rwgn .icon-shape,#mermaid-svg-GrBEnwQ09kV0Rwgn .image-shape{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-GrBEnwQ09kV0Rwgn .icon-shape p,#mermaid-svg-GrBEnwQ09kV0Rwgn .image-shape p{background-color:rgba(232,232,232, 0.8);padding:2px;}#mermaid-svg-GrBEnwQ09kV0Rwgn .icon-shape .label rect,#mermaid-svg-GrBEnwQ09kV0Rwgn .image-shape .label rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-GrBEnwQ09kV0Rwgn .label-icon{display:inline-block;height:1em;overflow:visible;vertical-align:-0.125em;}#mermaid-svg-GrBEnwQ09kV0Rwgn .node .label-icon path{fill:currentColor;stroke:revert;stroke-width:revert;}#mermaid-svg-GrBEnwQ09kV0Rwgn :root{--mermaid-font-family:"trebuchet ms",verdana,arial,sans-serif;} Delete 阶段 (超过 30 天)
Warm 阶段 (7-30 天)
Hot 阶段 (0-7 天)
min_age: 7d
min_age: 30d
写入实时日志

SSD 高速存储
rollover

50GB 或 1 天
优先级 100
迁移到 HDD

allocate data_type: warm
设为只读

readonly
优先级 50
删除索引

释放空间

json 复制代码

PUT _ilm/policy/logs_policy
{
  "policy": {
    "phases": {
      "hot": {
        "min_age": "0ms",
        "actions": {
          "rollover": {
            "max_size": "50GB",
            "max_age": "1d"
          },
          "set_priority": {
            "priority": 100
          }
        }
      },
      "warm": {
        "min_age": "7d",
        "actions": {
          "allocate": {
            "require": {
              "data_type": "warm"
            }
          },
          "readonly": {},
          "set_priority": {
            "priority": 50
          }
        }
      },
      "delete": {
        "min_age": "30d",
        "actions": {
          "delete": {}
        }
      }
    }
  }
}

5.3.3 索引模板

json 复制代码

PUT _index_template/logs_template
{
  "index_patterns": ["logs-*"],
  "template": {
    "settings": {
      "number_of_shards": 5,
      "number_of_replicas": 1,
      "index.lifecycle.name": "logs_policy",
      "index.lifecycle.rollover_alias": "logs",
      "routing.allocation.require.data_type": "hot"
    },
    "mappings": {
      "properties": {
        "@timestamp": { "type": "date" },
        "service.name": { "type": "keyword" },
        "host.name": { "type": "keyword" },
        "message": { "type": "text" },
        "level": { "type": "keyword" }
      }
    }
  }
}

5.3.4 数据采集架构

#mermaid-svg-0eyjFOSElYpg8vH5{font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:16px;fill:#333;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#mermaid-svg-0eyjFOSElYpg8vH5 .edge-animation-slow{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 50s linear infinite;stroke-linecap:round;}#mermaid-svg-0eyjFOSElYpg8vH5 .edge-animation-fast{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 20s linear infinite;stroke-linecap:round;}#mermaid-svg-0eyjFOSElYpg8vH5 .error-icon{fill:#552222;}#mermaid-svg-0eyjFOSElYpg8vH5 .error-text{fill:#552222;stroke:#552222;}#mermaid-svg-0eyjFOSElYpg8vH5 .edge-thickness-normal{stroke-width:1px;}#mermaid-svg-0eyjFOSElYpg8vH5 .edge-thickness-thick{stroke-width:3.5px;}#mermaid-svg-0eyjFOSElYpg8vH5 .edge-pattern-solid{stroke-dasharray:0;}#mermaid-svg-0eyjFOSElYpg8vH5 .edge-thickness-invisible{stroke-width:0;fill:none;}#mermaid-svg-0eyjFOSElYpg8vH5 .edge-pattern-dashed{stroke-dasharray:3;}#mermaid-svg-0eyjFOSElYpg8vH5 .edge-pattern-dotted{stroke-dasharray:2;}#mermaid-svg-0eyjFOSElYpg8vH5 .marker{fill:#333333;stroke:#333333;}#mermaid-svg-0eyjFOSElYpg8vH5 .marker.cross{stroke:#333333;}#mermaid-svg-0eyjFOSElYpg8vH5 svg{font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:16px;}#mermaid-svg-0eyjFOSElYpg8vH5 p{margin:0;}#mermaid-svg-0eyjFOSElYpg8vH5 .label{font-family:"trebuchet ms",verdana,arial,sans-serif;color:#333;}#mermaid-svg-0eyjFOSElYpg8vH5 .cluster-label text{fill:#333;}#mermaid-svg-0eyjFOSElYpg8vH5 .cluster-label span{color:#333;}#mermaid-svg-0eyjFOSElYpg8vH5 .cluster-label span p{background-color:transparent;}#mermaid-svg-0eyjFOSElYpg8vH5 .label text,#mermaid-svg-0eyjFOSElYpg8vH5 span{fill:#333;color:#333;}#mermaid-svg-0eyjFOSElYpg8vH5 .node rect,#mermaid-svg-0eyjFOSElYpg8vH5 .node circle,#mermaid-svg-0eyjFOSElYpg8vH5 .node ellipse,#mermaid-svg-0eyjFOSElYpg8vH5 .node polygon,#mermaid-svg-0eyjFOSElYpg8vH5 .node path{fill:#ECECFF;stroke:#9370DB;stroke-width:1px;}#mermaid-svg-0eyjFOSElYpg8vH5 .rough-node .label text,#mermaid-svg-0eyjFOSElYpg8vH5 .node .label text,#mermaid-svg-0eyjFOSElYpg8vH5 .image-shape .label,#mermaid-svg-0eyjFOSElYpg8vH5 .icon-shape .label{text-anchor:middle;}#mermaid-svg-0eyjFOSElYpg8vH5 .node .katex path{fill:#000;stroke:#000;stroke-width:1px;}#mermaid-svg-0eyjFOSElYpg8vH5 .rough-node .label,#mermaid-svg-0eyjFOSElYpg8vH5 .node .label,#mermaid-svg-0eyjFOSElYpg8vH5 .image-shape .label,#mermaid-svg-0eyjFOSElYpg8vH5 .icon-shape .label{text-align:center;}#mermaid-svg-0eyjFOSElYpg8vH5 .node.clickable{cursor:pointer;}#mermaid-svg-0eyjFOSElYpg8vH5 .root .anchor path{fill:#333333!important;stroke-width:0;stroke:#333333;}#mermaid-svg-0eyjFOSElYpg8vH5 .arrowheadPath{fill:#333333;}#mermaid-svg-0eyjFOSElYpg8vH5 .edgePath .path{stroke:#333333;stroke-width:2.0px;}#mermaid-svg-0eyjFOSElYpg8vH5 .flowchart-link{stroke:#333333;fill:none;}#mermaid-svg-0eyjFOSElYpg8vH5 .edgeLabel{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-0eyjFOSElYpg8vH5 .edgeLabel p{background-color:rgba(232,232,232, 0.8);}#mermaid-svg-0eyjFOSElYpg8vH5 .edgeLabel rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-0eyjFOSElYpg8vH5 .labelBkg{background-color:rgba(232, 232, 232, 0.5);}#mermaid-svg-0eyjFOSElYpg8vH5 .cluster rect{fill:#ffffde;stroke:#aaaa33;stroke-width:1px;}#mermaid-svg-0eyjFOSElYpg8vH5 .cluster text{fill:#333;}#mermaid-svg-0eyjFOSElYpg8vH5 .cluster span{color:#333;}#mermaid-svg-0eyjFOSElYpg8vH5 div.mermaidTooltip{position:absolute;text-align:center;max-width:200px;padding:2px;font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:12px;background:hsl(80, 100%, 96.2745098039%);border:1px solid #aaaa33;border-radius:2px;pointer-events:none;z-index:100;}#mermaid-svg-0eyjFOSElYpg8vH5 .flowchartTitleText{text-anchor:middle;font-size:18px;fill:#333;}#mermaid-svg-0eyjFOSElYpg8vH5 rect.text{fill:none;stroke-width:0;}#mermaid-svg-0eyjFOSElYpg8vH5 .icon-shape,#mermaid-svg-0eyjFOSElYpg8vH5 .image-shape{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-0eyjFOSElYpg8vH5 .icon-shape p,#mermaid-svg-0eyjFOSElYpg8vH5 .image-shape p{background-color:rgba(232,232,232, 0.8);padding:2px;}#mermaid-svg-0eyjFOSElYpg8vH5 .icon-shape .label rect,#mermaid-svg-0eyjFOSElYpg8vH5 .image-shape .label rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-0eyjFOSElYpg8vH5 .label-icon{display:inline-block;height:1em;overflow:visible;vertical-align:-0.125em;}#mermaid-svg-0eyjFOSElYpg8vH5 .node .label-icon path{fill:currentColor;stroke:revert;stroke-width:revert;}#mermaid-svg-0eyjFOSElYpg8vH5 :root{--mermaid-font-family:"trebuchet ms",verdana,arial,sans-serif;} 存储层
处理层
采集层
数据源
ILM 7天后
ILM 30天后
600+ 微服务

20TB/天
Filebeat

日志采集
Kafka

消息队列

削峰填谷
Logstash

过滤/转换
ES 热节点

SSD

最近 7 天
ES 温节点

HDD

7-30 天
冷数据归档

5.3.5 性能表现

ES 在日志查询方面基本处于领先地位，是很多企业的首选：

指标	数值
写入吞吐量	8 万条/秒
查询平均响应	320ms
可用性	99.99%
存储成本	降低 60%（冷热分离）

六、集群最佳实践

6.1 节点角色优化

专用主节点：避免数据节点承担 master 职责，防止集群脑裂
协调节点：处理复杂查询，减少数据节点压力
Ingest 节点：如果数据预处理量大，可单独部署 ingest 节点

6.2 分片设计

单个分片大小控制在 20-50GB
分片数 = (数据总量 / 30GB) × (副本数 + 1)
避免过度分片，浪费资源

6.3 内存与磁盘

JVM 堆内存 ≤ 32GB，剩余内存留给文件系统缓存
磁盘使用率达到 85% 时触发告警，95% 时进入只读模式

6.4 监控与告警

使用 Metricbeat 或 Prometheus 监控集群指标
关键指标：集群状态、节点堆内存使用率、搜索延迟、磁盘使用率
设置告警规则（如状态非 green、节点离线）

七、总结

#mermaid-svg-IQU0LFLp0rp2CyWN{font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:16px;fill:#333;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#mermaid-svg-IQU0LFLp0rp2CyWN .edge-animation-slow{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 50s linear infinite;stroke-linecap:round;}#mermaid-svg-IQU0LFLp0rp2CyWN .edge-animation-fast{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 20s linear infinite;stroke-linecap:round;}#mermaid-svg-IQU0LFLp0rp2CyWN .error-icon{fill:#552222;}#mermaid-svg-IQU0LFLp0rp2CyWN .error-text{fill:#552222;stroke:#552222;}#mermaid-svg-IQU0LFLp0rp2CyWN .edge-thickness-normal{stroke-width:1px;}#mermaid-svg-IQU0LFLp0rp2CyWN .edge-thickness-thick{stroke-width:3.5px;}#mermaid-svg-IQU0LFLp0rp2CyWN .edge-pattern-solid{stroke-dasharray:0;}#mermaid-svg-IQU0LFLp0rp2CyWN .edge-thickness-invisible{stroke-width:0;fill:none;}#mermaid-svg-IQU0LFLp0rp2CyWN .edge-pattern-dashed{stroke-dasharray:3;}#mermaid-svg-IQU0LFLp0rp2CyWN .edge-pattern-dotted{stroke-dasharray:2;}#mermaid-svg-IQU0LFLp0rp2CyWN .marker{fill:#333333;stroke:#333333;}#mermaid-svg-IQU0LFLp0rp2CyWN .marker.cross{stroke:#333333;}#mermaid-svg-IQU0LFLp0rp2CyWN svg{font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:16px;}#mermaid-svg-IQU0LFLp0rp2CyWN p{margin:0;}#mermaid-svg-IQU0LFLp0rp2CyWN .label{font-family:"trebuchet ms",verdana,arial,sans-serif;color:#333;}#mermaid-svg-IQU0LFLp0rp2CyWN .cluster-label text{fill:#333;}#mermaid-svg-IQU0LFLp0rp2CyWN .cluster-label span{color:#333;}#mermaid-svg-IQU0LFLp0rp2CyWN .cluster-label span p{background-color:transparent;}#mermaid-svg-IQU0LFLp0rp2CyWN .label text,#mermaid-svg-IQU0LFLp0rp2CyWN span{fill:#333;color:#333;}#mermaid-svg-IQU0LFLp0rp2CyWN .node rect,#mermaid-svg-IQU0LFLp0rp2CyWN .node circle,#mermaid-svg-IQU0LFLp0rp2CyWN .node ellipse,#mermaid-svg-IQU0LFLp0rp2CyWN .node polygon,#mermaid-svg-IQU0LFLp0rp2CyWN .node path{fill:#ECECFF;stroke:#9370DB;stroke-width:1px;}#mermaid-svg-IQU0LFLp0rp2CyWN .rough-node .label text,#mermaid-svg-IQU0LFLp0rp2CyWN .node .label text,#mermaid-svg-IQU0LFLp0rp2CyWN .image-shape .label,#mermaid-svg-IQU0LFLp0rp2CyWN .icon-shape .label{text-anchor:middle;}#mermaid-svg-IQU0LFLp0rp2CyWN .node .katex path{fill:#000;stroke:#000;stroke-width:1px;}#mermaid-svg-IQU0LFLp0rp2CyWN .rough-node .label,#mermaid-svg-IQU0LFLp0rp2CyWN .node .label,#mermaid-svg-IQU0LFLp0rp2CyWN .image-shape .label,#mermaid-svg-IQU0LFLp0rp2CyWN .icon-shape .label{text-align:center;}#mermaid-svg-IQU0LFLp0rp2CyWN .node.clickable{cursor:pointer;}#mermaid-svg-IQU0LFLp0rp2CyWN .root .anchor path{fill:#333333!important;stroke-width:0;stroke:#333333;}#mermaid-svg-IQU0LFLp0rp2CyWN .arrowheadPath{fill:#333333;}#mermaid-svg-IQU0LFLp0rp2CyWN .edgePath .path{stroke:#333333;stroke-width:2.0px;}#mermaid-svg-IQU0LFLp0rp2CyWN .flowchart-link{stroke:#333333;fill:none;}#mermaid-svg-IQU0LFLp0rp2CyWN .edgeLabel{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-IQU0LFLp0rp2CyWN .edgeLabel p{background-color:rgba(232,232,232, 0.8);}#mermaid-svg-IQU0LFLp0rp2CyWN .edgeLabel rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-IQU0LFLp0rp2CyWN .labelBkg{background-color:rgba(232, 232, 232, 0.5);}#mermaid-svg-IQU0LFLp0rp2CyWN .cluster rect{fill:#ffffde;stroke:#aaaa33;stroke-width:1px;}#mermaid-svg-IQU0LFLp0rp2CyWN .cluster text{fill:#333;}#mermaid-svg-IQU0LFLp0rp2CyWN .cluster span{color:#333;}#mermaid-svg-IQU0LFLp0rp2CyWN div.mermaidTooltip{position:absolute;text-align:center;max-width:200px;padding:2px;font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:12px;background:hsl(80, 100%, 96.2745098039%);border:1px solid #aaaa33;border-radius:2px;pointer-events:none;z-index:100;}#mermaid-svg-IQU0LFLp0rp2CyWN .flowchartTitleText{text-anchor:middle;font-size:18px;fill:#333;}#mermaid-svg-IQU0LFLp0rp2CyWN rect.text{fill:none;stroke-width:0;}#mermaid-svg-IQU0LFLp0rp2CyWN .icon-shape,#mermaid-svg-IQU0LFLp0rp2CyWN .image-shape{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-IQU0LFLp0rp2CyWN .icon-shape p,#mermaid-svg-IQU0LFLp0rp2CyWN .image-shape p{background-color:rgba(232,232,232, 0.8);padding:2px;}#mermaid-svg-IQU0LFLp0rp2CyWN .icon-shape .label rect,#mermaid-svg-IQU0LFLp0rp2CyWN .image-shape .label rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-IQU0LFLp0rp2CyWN .label-icon{display:inline-block;height:1em;overflow:visible;vertical-align:-0.125em;}#mermaid-svg-IQU0LFLp0rp2CyWN .node .label-icon path{fill:currentColor;stroke:revert;stroke-width:revert;}#mermaid-svg-IQU0LFLp0rp2CyWN :root{--mermaid-font-family:"trebuchet ms",verdana,arial,sans-serif;} 最终效果
关键能力
单机 -> 集群
容器化 + Docker Compose
节点角色规划
单节点 ES
3 节点集群
15 节点生产集群

主 + 热 + 温 + 协调
索引生命周期管理

Hot → Warm → Delete
冷热分层

SSD + HDD 降本 60%
高可用

99.99% 可用性
PB 级数据存储 ✅
查询 320ms ✅
写入 8 万条/秒 ✅

Elasticsearch 集群容器化部署，让你能够以基础设施即代码的方式管理 PB 级数据平台。

通过合理的节点角色规划、索引生命周期管理和监控体系，可以构建出高可用、高性能、低成本的搜索与分析平台。

从单机到集群，不仅是架构的演进，更是对数据价值的深度挖掘。

希望本文能帮助你顺利搭建生产级 Elasticsearch 集群，为业务创新提供坚实的数据底座。