一、目的
对于Flume的负载均衡和故障转移拓扑结构，进行一个开发测试
二、负载均衡和故障转移
（一）结构含义
Flume支持使用将多个sink逻辑上分到一个sink组
（二）结构特征
sink组配合不同的SinkProcessor可以实现负载均衡和错误恢复的功能
三、需求案例
（一）案例需求
使用 Flume1 监控一个端口，其sink组中的sink分别对接 Flume2 和 Flume3，采用FailoverSinkProcessor，实现故障转移的功能。
（二）需求分析
四、前期准备
（一）安装好Flume工具
（二）在Flume中创建测试任务的文件夹group2
root@hurys23 conf\]# mkdir group2 \[root@hurys23 conf\]# cd ./group2/ \[root@hurys23 group2\]# pwd /usr/local/hurys/dc_env/flume/flume190/conf/group2 ## 五、在group2中创建flume的任务文件 ### （一）创建任务文件 a1 flume-netcat-flume.conf 配置 1 个 netcat source 和 1 个 channel、1 个 sink group（2 个 sink），分别输送给flume-flume-console1 和 flume-flume-console2。 \[root@hurys23 group2\]# vi flume-netcat-flume.conf # Name the components on this agent a1.sources = r1 a1.channels = c1 a1.sinkgroups = g1 a1.sinks = k1 k2 # Describe/configure the source a1.sources.r1.type = netcat a1.sources.r1.bind = localhost a1.sources.r1.port = 44444 a1.sinkgroups.g1.processor.type = failover a1.sinkgroups.g1.processor.priority.k1 = 5 a1.sinkgroups.g1.processor.priority.k2 = 10 a1.sinkgroups.g1.processor.maxpenalty = 10000 # Describe the sink a1.sinks.k1.type = avro a1.sinks.k1.hostname = hurys23 a1.sinks.k1.port = 4141 a1.sinks.k2.type = avro a1.sinks.k2.hostname = hurys23 a1.sinks.k2.port = 4142 # Describe the channel a1.channels.c1.type = memory a1.channels.c1.capacity = 1000 a1.channels.c1.transactionCapacity = 100 # Bind the source and sink to the channel a1.sources.r1.channels = c1 a1.sinkgroups.g1.sinks = k1 k2 a1.sinks.k1.channel = c1 a1.sinks.k2.channel = c1  注意： 1、配置文件中的各项参数需要调式，这里只是为了演示，实现目的、打通路径即可！实际在项目中操作时需要调试参数。 2、a1.sinks.k1.hostname = hurys23 hurys23 为服务器名字 ### （二）创建任务文件 a2 flume-flume-console1.conf 配置上级 Flume 输出的 Source，输出是到本地控制台。 \[root@hurys23 group2\]# vi flume-flume-console1.conf # Name the components on this agent a2.sources = r1 a2.sinks = k1 a2.channels = c1 # Describe/configure the source a2.sources.r1.type = avro a2.sources.r1.bind = hurys23 a2.sources.r1.port = 4141 # Describe the sink a2.sinks.k1.type = logger # Describe the channel a2.channels.c1.type = memory a2.channels.c1.capacity = 1000 a2.channels.c1.transactionCapacity = 100 # Bind the source and sink to the channel a2.sources.r1.channels = c1 a2.sinks.k1.channel = c1 ###  （三）创建任务文件 a3 flume-flume-console2.conf 配置上级 Flume 输出的 Source，输出是到本地控制台。 \[root@hurys23 group2\]# vi flume-flume-console2.conf # Name the components on this agent a3.sources = r1 a3.sinks = k1 a3.channels = c2 # Describe/configure the source a3.sources.r1.type = avro a3.sources.r1.bind = hurys23 a3.sources.r1.port = 4142 # Describe the sink a3.sinks.k1.type = logger # Describe the channel a3.channels.c2.type = memory a3.channels.c2.capacity = 1000 a3.channels.c2.transactionCapacity = 100 # Bind the source and sink to the channel a3.sources.r1.channels = c2 a3.sinks.k1.channel = c2  ## 六、分别启动Flume任务文件 ### （一）首先启动 a3 flume-flume-console2.conf \[root@hurys23 flume190\]# bin/flume-ng agent -n a3 -f /usr/local/hurys/dc_env/flume/flume190/conf/group2/flume-flume-console2.conf  ### （二）其次启动 a2 flume-flume-console1.conf \[root@hurys23 flume190\]# bin/flume-ng agent -n a2 -f /usr/local/hurys/dc_env/flume/flume190/conf/group2/flume-flume-console1.conf  ### （三）最后启动 a1 flume-netcat-flume.conf \[root@hurys23 flume190\]# bin/flume-ng agent -n a1 -f /usr/local/hurys/dc_env/flume/flume190/conf/group2/flume-netcat-flume.conf  ## 七、使用 netcat 工具向本机的 44444 端口发送内容 \[root@hurys23 \~\]# nc localhost 44444 hello world OK hello java OK hello hadoop OK hello flume OK  ## 八、Flume任务运行执行状况 ### （一）a1 a1任务运行截图  ### （二）a2 a2任务运行截图  ### （三）a3 a3任务运行截图 2023-12-12 17:29:38,391 INFO sink.LoggerSink: Event: { headers:{} body: 68 65 6C 6C 6F 20 77 6F 72 6C 64 hello world } 2023-12-12 17:29:43,331 INFO sink.LoggerSink: Event: { headers:{} body: 68 65 6C 6C 6F 20 6A 61 76 61 hello java } 2023-12-12 17:29:49,027 INFO sink.LoggerSink: Event: { headers:{} body: 68 65 6C 6C 6F 20 68 61 64 6F 6F 70 hello hadoop } 2023-12-12 17:29:53,028 INFO sink.LoggerSink: Event: { headers:{} body: 68 65 6C 6C 6F 20 66 6C 75 6D 65 hello flume }  Flume负载均衡和故障转移拓扑结构的测试成功，当然这个案例比较简单。 Flume玩法继续研究中！