目录
- [一. 前提](#一. 前提)
-
- [1.1 数据准备](#1.1 数据准备)
- [1.2 数据说明](#1.2 数据说明)
- [二. 查询](#二. 查询)
-
- [2.1 统计每个加盟店搜索的次数](#2.1 统计每个加盟店搜索的次数)
一. 前提
1.1 数据准备
⏹file1.log
140 2024/07/08 12:35:01.547 c1server2 5485 [ERROR] SPLREQUEST seqNo=11459,eventController=PMT.payinfoforprc.test.search,oldest_data_search=2
110 2024/07/08 12:34:56.457 c1server1 5892 [INFO] MYCODE2005 测试方法被调用。(method=selectSvc_Test_Id param cpId=16xx2 status=OK) userid=adminUser
150 2024/07/08 12:35:02.231 c1server3 5634 [INFO] SPLREQUEST seqNo=11460,eventController=PMT.payinfoforprc.test.search,oldest_data_search=1
120 2024/07/08 12:34:57.235 c1server1 5675 [INFO] MYCODE2005 测试方法被调用。(method=selectSvc_Test_Id param cpId=16xx2 status=OK) userid=adminUser
160 2024/07/08 12:36:22.534 c1server3 2564 [WARN] SPLREQUEST seqNo=11461,eventController=PMT.payinfoforprc.test.search,oldest_data_search=1
130 2024/07/08 12:34:58.546 c1server2 6354 [INFO] MYCODE2005 测试方法被调用。(method=selectSvc_Test_Id param cpId=16xx2 status=OK) userid=adminUser
110 2024/07/08 12:34:56.456 c1server1 5892 [INFO] SPLREQUEST seqNo=11456,eventController=PMT.payinfoforprc.test.search,oldest_data_search=1
140 2024/07/08 12:35:01.548 c1server2 5485 [ERROR] MYCODE2005 测试方法被调用。(method=selectSvc_Test_Id param cpId=45xx2 status=OK) userid=adminUser
120 2024/07/08 12:34:57.234 c1server1 5675 [INFO] SPLREQUEST seqNo=11457,eventController=PMT.payinfoforprc.test.search,oldest_data_search=2
150 2024/07/08 12:35:02.232 c1server3 5634 [INFO] MYCODE2005 测试方法被调用。(method=selectSvc_Test_Id param cpId=45xx2 status=OK) userid=adminUser
130 2024/07/08 12:34:58.545 c1server2 6354 [INFO] SPLREQUEST seqNo=11458,eventController=PMT.payinfoforprc.test.search,oldest_data_search=1
160 2024/07/08 12:36:22.535 c1server3 2564 [WARN] MYCODE2005 测试方法被调用。(method=selectSvc_Test_Id param cpId=45xx2 status=OK) userid=adminUser
⏹file2.log
210 2024/07/08 12:34:56.456 c1server1 5892 [INFO] SPLREQUEST seqNo=21456,eventController=PMT.payinfoforprc.test.search,oldest_data_search=1
240 2024/07/08 12:35:01.548 c1server2 5485 [ERROR] MYCODE2005 测试方法被调用。(method=selectSvc_Test_Id param cpId=89xx2 status=OK) userid=adminUser
260 2024/07/08 12:34:57.234 c1server1 5675 [INFO] SPLREQUEST seqNo=21457,eventController=PMT.payinfoforprc.test.search,oldest_data_search=2
210 2024/07/08 12:35:02.232 c1server3 5634 [INFO] MYCODE2005 测试方法被调用。(method=selectSvc_Test_Id param cpId=16xx2 status=OK) userid=adminUser
240 2024/07/08 12:34:58.545 c1server2 6354 [INFO] SPLREQUEST seqNo=21458,eventController=PMT.payinfoforprc.test.search,oldest_data_search=1
260 2024/07/08 12:36:22.535 c1server3 2564 [WARN] MYCODE2005 测试方法被调用。(method=selectSvc_Test_Id param cpId=89xx2 status=OK) userid=adminUser
1.2 数据说明
- 第一列用户存储线程号，一个完整的用户请求要分为很多步骤，每一个步骤的线程号都是相同的
但是由于多个用户同时访问服务器，因此一个请求所产生的各个步骤的日志并不是连续打印，而是错落间隔在日志的各部分中
我们可以通过线程号，来将一个请求的完整内容抽取出来 - 第二列存储时间
- 第三列存储服务器名称，一个系统可能有多个服务器
- eventController：功能的名称
- cpId：加盟店ID
二. 查询
- 查询
PMT.payinfoforprc.test.search功能 - 然后通过
oldest_data_search=1进一步过滤 - 然后通过
awk '{print $1}'获取出第一列(awk默认通过空格分列)./file1.log:150./file1.log:160
- 再通过
awk -F ":" '{print "^"$2 " " $1}'构造查询key^150 ./file1.log^160 ./file1.log
- 再通过
xargs -L 1 grep -a命令，从前面的命令中读取一行输入，并将其作为参数传递给下一个命令grep -a构造查询命令grep -a ^150 ./file1.loggrep -a ^160 ./file1.log
- 最后再从查询的结果中，通过
grep -E "SPLREQUEST|MYCODE2005"进一步筛选
bash
grep -a "PMT.payinfoforprc.test.search" ./file*.log | \
grep -a "oldest_data_search=1" | \
awk '{print $1}' | \
awk -F ":" '{print "^"$2 " " $1}' | \
xargs -L 1 grep -a | \
grep -E "SPLREQUEST|MYCODE2005"2.1 统计每个加盟店搜索的次数
⏹完整命令
cpId=[^ ]*- 查询不为空的cpId
awk '{$1=$1; print $0}'- 清除行中的多余空格。通过重新格式化行
- awk 会用单个空格替换多个连续的空格
bash
grep -a "PMT.payinfoforprc.test.search" ./file*.log | \
grep -a "oldest_data_search=1" | \
awk '{print $1}' | \
awk -F ":" '{print "^"$2 " " $1}' | \
xargs -L 1 grep -a | \
grep -E "MYCODE2005" | \
grep -o -e "param cpId=[^ ]*" | \
uniq -c | \
awk '{$1=$1; print $0}' | \
sort- 查询拆解1
bash
$ grep -a "PMT.payinfoforprc.test.search" ./file*.log | \
> grep -a "oldest_data_search=1" | \
> awk '{print $1}'
./file1.log:150
./file1.log:160
./file1.log:110
./file1.log:130
./file2.log:210
./file2.log:240- 查询拆解2
bash
$ grep -a "PMT.payinfoforprc.test.search" ./file*.log | \
> grep -a "oldest_data_search=1" | \
> awk '{print $1}' | \
> awk -F ":" '{print "^"$2 " " $1}'
^150 ./file1.log
^160 ./file1.log
^110 ./file1.log
^130 ./file1.log
^210 ./file2.log
^240 ./file2.log- 查询拆解3
※为方便打印展示效果，将grep -a替换为echo 'grep -a'
bash
$ grep -a "PMT.payinfoforprc.test.search" ./file*.log | \
> grep -a "oldest_data_search=1" | \
> awk '{print $1}' | \
> awk -F ":" '{print "^"$2 " " $1}' | \
> xargs -L 1 echo 'grep -a'
grep -a ^150 ./file1.log
grep -a ^160 ./file1.log
grep -a ^110 ./file1.log
grep -a ^130 ./file1.log
grep -a ^210 ./file2.log
grep -a ^240 ./file2.log- 查询拆解4
bash
$ grep -a "PMT.payinfoforprc.test.search" ./file*.log | \
> grep -a "oldest_data_search=1" | \
> awk '{print $1}' | \
> awk -F ":" '{print "^"$2 " " $1}' | \
> xargs -L 1 grep -a | \
> grep -E "MYCODE2005"
150 2024/07/08 12:35:02.232 c1server3 5634 [INFO] MYCODE2005 测试方法被调用。(method=selectSvc_Test_Id param cpId=45xx2 status=OK) userid=adminUser
160 2024/07/08 12:36:22.535 c1server3 2564 [WARN] MYCODE2005 测试方法被调用。(method=selectSvc_Test_Id param cpId=45xx2 status=OK) userid=adminUser
110 2024/07/08 12:34:56.457 c1server1 5892 [INFO] MYCODE2005 测试方法被调用。(method=selectSvc_Test_Id param cpId=16xx2 status=OK) userid=adminUser
130 2024/07/08 12:34:58.546 c1server2 6354 [INFO] MYCODE2005 测试方法被调用。(method=selectSvc_Test_Id param cpId=16xx2 status=OK) userid=adminUser
210 2024/07/08 12:35:02.232 c1server3 5634 [INFO] MYCODE2005 测试方法被调用。(method=selectSvc_Test_Id param cpId=16xx2 status=OK) userid=adminUser
240 2024/07/08 12:35:01.548 c1server2 5485 [ERROR] MYCODE2005 测试方法被调用。(method=selectSvc_Test_Id param cpId=89xx2 status=OK) userid=adminUser- 查询拆解5
bash
$ grep -a "PMT.payinfoforprc.test.search" ./file*.log | \
> grep -a "oldest_data_search=1" | \
> awk '{print $1}' | \
> awk -F ":" '{print "^"$2 " " $1}' | \
> xargs -L 1 grep -a | \
> grep -E "MYCODE2005" | \
> grep -o -e "param cpId=[^ ]*"
param cpId=45xx2
param cpId=45xx2
param cpId=16xx2
param cpId=16xx2
param cpId=16xx2
param cpId=89xx2- 查询拆解6
bash
$ grep -a "PMT.payinfoforprc.test.search" ./file*.log | \
> grep -a "oldest_data_search=1" | \
> awk '{print $1}' | \
> awk -F ":" '{print "^"$2 " " $1}' | \
> xargs -L 1 grep -a | \
> grep -E "MYCODE2005" | \
> grep -o -e "param cpId=[^ ]*" | \
> uniq -c
2 param cpId=45xx2
3 param cpId=16xx2
1 param cpId=89xx2- 查询拆解7
bash
$ grep -a "PMT.payinfoforprc.test.search" ./file*.log | \
> grep -a "oldest_data_search=1" | \
> awk '{print $1}' | \
> awk -F ":" '{print "^"$2 " " $1}' | \
> xargs -L 1 grep -a | \
> grep -E "MYCODE2005" | \
> grep -o -e "param cpId=[^ ]*" | \
> uniq -c | \
> awk '{$1=$1; print $0}'
2 param cpId=45xx2
3 param cpId=16xx2
1 param cpId=89xx2