Flume企业级应用 - 扇入扇出

水善利万物而不争，处众人之所恶，故几于道💦

文章目录

• • [1. 扇入、扇出](#1. 扇入、扇出)
  • [2. 示例](#2. 示例)
  • • [1. 扇入（聚合）](#1. 扇入（聚合）)
    • • a1配置：a1_netcat_mem_avro.conf
      • a2配置：a2_taildir_mem_avro.conf
      • a3配置：a3_avro_mem_logger.conf
    • [2. 扇出（复制）](#2. 扇出（复制）)
    • • a1配置：a1_taildir_mem_avro.conf
      • a2配置：a2_avro_mem_hdfs.conf
      • a3配置：a3_avro_mem_file_roll.conf

1. 扇入、扇出

  扇入、扇出就是利用多个flume实例进行数据的采集。扇入就是从多个数据源采集数据然后都发送到一个flume里面，像扇子一样进行不同数据源间数据的整合。扇出就是用一个flume将数据同时发往多个flume，将数据落到不同的地方。

  这个扇入扇出我也是面试的时候被整不会了，面试官问我了解flume扇入扇出不？我一脸懵逼，听都没听过。然后查了下原来就是多个flume的使用，也叫聚合和复制...🤡

2. 示例

1. 扇入（聚合）

  以上图中的组件进行配置，演示扇入。

a1配置：a1_netcat_mem_avro.conf

a1.sources = r1
a1.channels = c1
a1.sinks = k1


a1.sources.r1.type = netcat
a1.sources.r1.bind = hadoop101
a1.sources.r1.port = 6666


a1.channels.c1.type = memory
a1.channels.c1.capacity = 1000
a1.channels.c1.transactionCapacity = 100


a1.sinks.k1.type = avro
a1.sinks.k1.hostname = hadoop103
a1.sinks.k1.port = 4545


a1.sources.r1.channels = c1
a1.sinks.k1.channel = c1

a2配置：a2_taildir_mem_avro.conf

a2.sources = r1
a2.channels = c1
a2.sinks = k1


a2.sources.r1.type = TAILDIR
a2.sources.r1.positionFile = /opt/module/flume-1.9.0/positionFile/taildir_position.json
a2.sources.r1.filegroups = f1
a2.sources.r1.filegroups.f1 = /opt/module/flume-1.9.0/app.*


a2.channels.c1.type = memory
a2.channels.c1.capacity = 1000
a2.channels.c1.transactionCapacity = 100


a2.sinks.k1.type = avro
a2.sinks.k1.hostname = hadoop103
a2.sinks.k1.port = 4545


a2.sources.r1.channels = c1
a2.sinks.k1.channel = c1

a3配置：a3_avro_mem_logger.conf

a3.sources = r1
a3.channels = c1
a3.sinks = k1


a3.sources.r1.type = avro
a3.sources.r1.bind = hadoop103
a3.sources.r1.port = 4545


a3.channels.c1.type = memory
a3.channels.c1.capacity = 1000
a3.channels.c1.transactionCapacity = 100


a3.sinks.k1.type = logger

a3.sources.r1.channels = c1
a3.sinks.k1.channel = c1

分别启动a3、a2、a1

bin/flume-ng agent -n a3 -c conf -f job/Into/a3_avro_mem_logger.conf -Dflume.root.logger=INFO,console

bin/flume-ng agent -n a2 -c conf -f job/Into/a2_taildir_mem_avro.conf -Dflume.root.logger=INFO,console

bin/flume-ng agent -n a1 -c conf -f job/Into/a1_netcat_mem_avro.conf -Dflume.root.logger=INFO,console

然后通过netcat向6666端口发数据观察到a3有日志输出，采集到了数据；向taildir监控的目录下app.log中追加内容，同样也观察到了a3有日志输出，也采集到了数据。

2. 扇出（复制）

  以上图中的组件进行配置，演示扇出。

a1配置：a1_taildir_mem_avro.conf

a1.sources = r1
a1.channels = c1 c2
a1.sinks = k1 k2


a1.sources.r1.type = TAILDIR
a1.sources.r1.positionFile = /opt/module/flume-1.9.0/positionFile/taildir_position.json
a1.sources.r1.filegroups = f1
a1.sources.r1.filegroups.f1 = /opt/module/flume-1.9.0/app.*


a1.channels.c1.type = memory
a1.channels.c1.capacity = 1000
a1.channels.c1.transactionCapacity = 100
a1.channels.c2.type = memory
a1.channels.c2.capacity = 1000
a1.channels.c2.transactionCapacity = 100

a1.sinks.k1.type = avro
a1.sinks.k1.hostname = hadoop101
a1.sinks.k1.port = 4545

a1.sinks.k2.type = avro
a1.sinks.k2.hostname = hadoop101
a1.sinks.k2.port = 4546


a1.sources.r1.channels = c1 c2
a1.sinks.k1.channel = c1
a1.sinks.k2.channel = c2

a2配置：a2_avro_mem_hdfs.conf

a2.sources = r1
a2.channels = c1
a2.sinks = k1


a2.sources.r1.type = avro
a2.sources.r1.bind = hadoop101
a2.sources.r1.port = 4545


a2.channels.c1.type = memory
a2.channels.c1.capacity = 1000
a2.channels.c1.transactionCapacity = 100


a2.sinks.k1.type = hdfs
a2.sinks.k1.hdfs.path = hdfs://hadoop101:8020/tmp/flume/applog/%Y-%m-%d/%H%M/%S
a2.sinks.k1.hdfs.filePrefix = applog-
# 是否按照时间滚动文件夹
a2.sinks.k1.hdfs.round = true
# 多少时间建立一个新的文件夹
a2.sinks.k1.hdfs.roundValue = 10
# 时间的单位
a2.sinks.k1.hdfs.roundUnit = second

a2.sinks.k1.hdfs.rollInterval = 60
a2.sinks.k1.hdfs.rollSize = 134217700
a2.sinks.k1.hdfs.rollCount = 0

# 是否使用本地时间戳 默认false使用header里面的时间戳
a2.sinks.k1.hdfs.useLocalTimeStamp = true
# 积攒多少个Event才flush到HDFS一次
a2.sinks.k1.hdfs.batchSize = 100
# 设置文件类型，可支持压缩
a2.sinks.k1.hdfs.fileType = CompressedStream
a2.sinks.k1.hdfs.codeC = gzip


a2.sources.r1.channels = c1
a2.sinks.k1.channel = c1

a3配置：a3_avro_mem_file_roll.conf

a3.sources = r1
a3.channels = c1
a3.sinks = k1


a3.sources.r1.type = avro
a3.sources.r1.bind = hadoop101
a3.sources.r1.port = 4546


a3.channels.c1.type = memory
a3.channels.c1.capacity = 1000
a3.channels.c1.transactionCapacity = 100


a3.sinks.k1.type = file_roll
a3.sinks.k1.sink.directory = /home/atguigu/flume_local_data
a3.sinks.k1.sink.rollInterval = 3600

a3.sources.r1.channels = c1
a3.sinks.k1.channel = c1

分别启动a3、a2、a1

bin/flume-ng agent -n a3 -c conf -f job/a3_avro_mem_file_roll.conf -Dflume.root.logger=INFO,console

bin/flume-ng agent -n a2 -c conf -f job/a2_avro_mem_hdfs.conf -Dflume.root.logger=INFO,console

bin/flume-ng agent -n a1 -c conf -f job/a1_taildir_mem_avro.conf -Dflume.root.logger=INFO,console

向文件中追加内容：

[atguigu@hadoop101 flume-1.9.0]$ echo $(date) >> app.log1 
[atguigu@hadoop101 flume-1.9.0]$ echo $(date) >> app.log1

观察到控制台打印出的新的日志，说明已经采集到了，再去hdfs和本地路径中查看，出现了相应的文件。