水善利万物而不争,处众人之所恶,故几于道💦
文章目录
-
- [1. 扇入、扇出](#1. 扇入、扇出)
- [2. 示例](#2. 示例)
-
- [1. 扇入(聚合)](#1. 扇入(聚合))
- [2. 扇出(复制)](#2. 扇出(复制))
1. 扇入、扇出
扇入、扇出就是利用多个flume实例进行数据的采集。扇入就是从多个数据源采集数据然后都发送到一个flume里面,像扇子一样进行不同数据源间数据的整合。扇出就是用一个flume将数据同时发往多个flume,将数据落到不同的地方。
这个扇入扇出我也是面试的时候被整不会了,面试官问我了解flume扇入扇出不?我一脸懵逼,听都没听过。然后查了下原来就是多个flume的使用,也叫聚合和复制...🤡
2. 示例
1. 扇入(聚合)
以上图中的组件进行配置,演示扇入。
a1配置:a1_netcat_mem_avro.conf
conf
a1.sources = r1
a1.channels = c1
a1.sinks = k1
a1.sources.r1.type = netcat
a1.sources.r1.bind = hadoop101
a1.sources.r1.port = 6666
a1.channels.c1.type = memory
a1.channels.c1.capacity = 1000
a1.channels.c1.transactionCapacity = 100
a1.sinks.k1.type = avro
a1.sinks.k1.hostname = hadoop103
a1.sinks.k1.port = 4545
a1.sources.r1.channels = c1
a1.sinks.k1.channel = c1
a2配置:a2_taildir_mem_avro.conf
conf
a2.sources = r1
a2.channels = c1
a2.sinks = k1
a2.sources.r1.type = TAILDIR
a2.sources.r1.positionFile = /opt/module/flume-1.9.0/positionFile/taildir_position.json
a2.sources.r1.filegroups = f1
a2.sources.r1.filegroups.f1 = /opt/module/flume-1.9.0/app.*
a2.channels.c1.type = memory
a2.channels.c1.capacity = 1000
a2.channels.c1.transactionCapacity = 100
a2.sinks.k1.type = avro
a2.sinks.k1.hostname = hadoop103
a2.sinks.k1.port = 4545
a2.sources.r1.channels = c1
a2.sinks.k1.channel = c1
a3配置:a3_avro_mem_logger.conf
conf
a3.sources = r1
a3.channels = c1
a3.sinks = k1
a3.sources.r1.type = avro
a3.sources.r1.bind = hadoop103
a3.sources.r1.port = 4545
a3.channels.c1.type = memory
a3.channels.c1.capacity = 1000
a3.channels.c1.transactionCapacity = 100
a3.sinks.k1.type = logger
a3.sources.r1.channels = c1
a3.sinks.k1.channel = c1
分别启动a3、a2、a1
bash
bin/flume-ng agent -n a3 -c conf -f job/Into/a3_avro_mem_logger.conf -Dflume.root.logger=INFO,console
bin/flume-ng agent -n a2 -c conf -f job/Into/a2_taildir_mem_avro.conf -Dflume.root.logger=INFO,console
bin/flume-ng agent -n a1 -c conf -f job/Into/a1_netcat_mem_avro.conf -Dflume.root.logger=INFO,console
然后通过netcat向6666端口发数据观察到a3有日志输出,采集到了数据;向taildir监控的目录下app.log中追加内容,同样也观察到了a3有日志输出,也采集到了数据。
2. 扇出(复制)
以上图中的组件进行配置,演示扇出。
a1配置:a1_taildir_mem_avro.conf
conf
a1.sources = r1
a1.channels = c1 c2
a1.sinks = k1 k2
a1.sources.r1.type = TAILDIR
a1.sources.r1.positionFile = /opt/module/flume-1.9.0/positionFile/taildir_position.json
a1.sources.r1.filegroups = f1
a1.sources.r1.filegroups.f1 = /opt/module/flume-1.9.0/app.*
a1.channels.c1.type = memory
a1.channels.c1.capacity = 1000
a1.channels.c1.transactionCapacity = 100
a1.channels.c2.type = memory
a1.channels.c2.capacity = 1000
a1.channels.c2.transactionCapacity = 100
a1.sinks.k1.type = avro
a1.sinks.k1.hostname = hadoop101
a1.sinks.k1.port = 4545
a1.sinks.k2.type = avro
a1.sinks.k2.hostname = hadoop101
a1.sinks.k2.port = 4546
a1.sources.r1.channels = c1 c2
a1.sinks.k1.channel = c1
a1.sinks.k2.channel = c2
a2配置:a2_avro_mem_hdfs.conf
conf
a2.sources = r1
a2.channels = c1
a2.sinks = k1
a2.sources.r1.type = avro
a2.sources.r1.bind = hadoop101
a2.sources.r1.port = 4545
a2.channels.c1.type = memory
a2.channels.c1.capacity = 1000
a2.channels.c1.transactionCapacity = 100
a2.sinks.k1.type = hdfs
a2.sinks.k1.hdfs.path = hdfs://hadoop101:8020/tmp/flume/applog/%Y-%m-%d/%H%M/%S
a2.sinks.k1.hdfs.filePrefix = applog-
# 是否按照时间滚动文件夹
a2.sinks.k1.hdfs.round = true
# 多少时间建立一个新的文件夹
a2.sinks.k1.hdfs.roundValue = 10
# 时间的单位
a2.sinks.k1.hdfs.roundUnit = second
a2.sinks.k1.hdfs.rollInterval = 60
a2.sinks.k1.hdfs.rollSize = 134217700
a2.sinks.k1.hdfs.rollCount = 0
# 是否使用本地时间戳 默认false使用header里面的时间戳
a2.sinks.k1.hdfs.useLocalTimeStamp = true
# 积攒多少个Event才flush到HDFS一次
a2.sinks.k1.hdfs.batchSize = 100
# 设置文件类型,可支持压缩
a2.sinks.k1.hdfs.fileType = CompressedStream
a2.sinks.k1.hdfs.codeC = gzip
a2.sources.r1.channels = c1
a2.sinks.k1.channel = c1
a3配置:a3_avro_mem_file_roll.conf
conf
a3.sources = r1
a3.channels = c1
a3.sinks = k1
a3.sources.r1.type = avro
a3.sources.r1.bind = hadoop101
a3.sources.r1.port = 4546
a3.channels.c1.type = memory
a3.channels.c1.capacity = 1000
a3.channels.c1.transactionCapacity = 100
a3.sinks.k1.type = file_roll
a3.sinks.k1.sink.directory = /home/atguigu/flume_local_data
a3.sinks.k1.sink.rollInterval = 3600
a3.sources.r1.channels = c1
a3.sinks.k1.channel = c1
分别启动a3、a2、a1
bash
bin/flume-ng agent -n a3 -c conf -f job/a3_avro_mem_file_roll.conf -Dflume.root.logger=INFO,console
bin/flume-ng agent -n a2 -c conf -f job/a2_avro_mem_hdfs.conf -Dflume.root.logger=INFO,console
bin/flume-ng agent -n a1 -c conf -f job/a1_taildir_mem_avro.conf -Dflume.root.logger=INFO,console
向文件中追加内容:
bash
[atguigu@hadoop101 flume-1.9.0]$ echo $(date) >> app.log1
[atguigu@hadoop101 flume-1.9.0]$ echo $(date) >> app.log1
观察到控制台打印出的新的日志,说明已经采集到了,再去hdfs和本地路径中查看,出现了相应的文件。