ES分析系统各个服务日志占用量
系统日志量太大，需要分析出哪里的日志打印导致量大，对于公司es的存储压力比较大。
1、创建带size的索引，reindex到新的索引
如果es中的索引size插件默认开启，则可以省略这一步
这一步可能耗时比较长
创建索引
dsl
PUT /bdo-pm-log-2026.05.29_size
{
"aliases": {},
"mappings": {
"_size": {
"enabled": true
},
"properties": {
"@timestamp": {
"type": "date"
},
"@version": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"APP_NAME": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"BOOT_SERVER": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"HOSTNAME": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"REQUESTID": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"audit": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"caller": {
"properties": {
"class": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"file": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"line": {
"type": "long"
},
"method": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
}
}
},
"event": {
"properties": {
"original": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
}
}
},
"ip": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"level": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"levelVal": {
"type": "long"
},
"logger": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"message": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"stacktrace": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"tags": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"thread": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"topic": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"traceId": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"user": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"userId": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
}
}
},
"settings": {
"index": {
"routing": {
"allocation": {
"include": {
"_tier_preference": "data_content"
}
}
},
"number_of_shards": "1",
"priority": "100",
"number_of_replicas": "1"
}
}
}reindex
dsl
POST _reindex
{
"source": {
"index": "bdo-pm-log-2026.05.29"
},
"dest": {
"index": "bdo-pm-log-2026.05.29_size"
}
}2、开始分析
分析顺序：
1、找出日志量比较大的应用
2、分析日志级别占用（防止错误堆栈，日志级别错误打印问题）
3、分析INFO日志，找出日志输出占比较大的类
找出日志量比较大的应用
dsl
GET /bdo-pm-log-2026.04.21_size/_search
{
"size": 0,
"aggs": {
"group_by_app": {
"terms": {
"field": "APP_NAME.keyword",
"size": 10,
"order": {
"total_disk_size": "desc"
}
},
"aggs": {
"total_log_count": {
"value_count": {
"field": "_seq_no"
}
},
"total_disk_size": {
"sum": {
"field": "_size"
}
},
"total_disk_size_gb": {
"bucket_script": {
"buckets_path": {
"sizeBytes": "total_disk_size"
},
"script": "params.sizeBytes / (1024 * 1024 * 1024)"
}
}
}
}
}
}分析日志级别占用（防止错误堆栈，日志级别错误打印问题）
dsl
GET bdo-pm-log-2026.04.21_size/_search
{
"size": 0,
"query": {
"bool": {
"filter": [
{ "term": { "APP_NAME.keyword": "bdo-mid-base" } }
]
}
},
"aggs": {
"group_by_app": {
"terms": {
"field": "level.keyword"
},
"aggs": {
"total_log_count": {
"value_count": {
"field": "_seq_no"
}
},
"total_disk_size": {
"sum": {
"field": "_size"
}
},
"total_disk_size_gb": {
"bucket_script": {
"buckets_path": {
"sizeBytes": "total_disk_size"
},
"script": "params.sizeBytes / (1024 * 1024 * 1024)"
}
}
}
}
}
}分析INFO日志，找出日志输出占比较大的类
部分服务会将文件内容打印在日志系统里面，从而导致日志文件巨大。
dsl
GET bdo-pm-log-2026.04.21_size/_search
{
"size": 0,
"query": {
"bool": {
"filter": [
{
"term": {
"APP_NAME.keyword": "app name"
}
}
]
}
},
"aggs": {
"group_by_app": {
"terms": {
"field": "logger.keyword",
"size": 1000,
"shard_size": 2000,
"order": {
"total_disk_size": "desc"
}
},
"aggs": {
"total_log_count": {
"value_count": {
"field": "_seq_no"
}
},
"total_disk_size": {
"sum": {
"field": "_size"
}
},
"total_disk_size_gb": {
"bucket_script": {
"buckets_path": {
"sizeBytes": "total_disk_size"
},
"script": "params.sizeBytes / (1024 * 1024 * 1024)"
}
}
}
}
}
}