Java版Flink使用指南——从RabbitMQ中队列中接入消息流

大纲

《Java版Flink使用指南------安装Flink和使用IntelliJ制作任务包》一文中,我们完成了第一个小型Demo的编写。例子中的数据是代码预先指定的。而现实中,数据往往来源于外部。本文我们将尝试Flink从RabbitMQ中读取数据,然后输出到日志中。

关于RabbitMQ的知识可以参阅《RabbitMQ实践》

创建RabbitMQ队列

我们创建一个Classic队列data.from.rbtmq。注意要选择Durable类型,这是后续用的默认连接器的限制。

具体方法见《RabbitMQ实践------在管理后台测试消息收发功能》

后续我们将在后台通过默认交换器,给这个队列新增消息。

新建工程

我们在IntelliJ中新建一个工程DataFromRabbitMQ。

Archetype填入:org.apache.flink:flink-quickstart-java。

版本填入与Flink的版本:1.19.1

新增依赖

在pom.xml中新增RabbitMQ连接器

xml 复制代码
<dependency>
	<groupId>org.apache.flink</groupId>
	<artifactId>flink-connector-rabbitmq</artifactId>
	<version>3.0.1-1.17</version>
</dependency>

编码

设置数据源配置

java 复制代码
String queueName = "data.from.rbtmq";
String host = "172.21.112.140"; // IP of the rabbitmq server
int port = 5672;
String username = "admin";
String password = "fangliang";
String virtualHost = "/";
int parallelism = 1;

// create a RabbitMQ source
RMQConnectionConfig rmqConnectionConfig = new RMQConnectionConfig.Builder()
		.setHost(host)
		.setPort(port)
		.setUserName(username)
		.setPassword(password)
		.setVirtualHost(virtualHost)
		.build();

RMQSource<String> rmqSource = new RMQSource<>(rmqConnectionConfig, queueName, true, new SimpleStringSchema());

读取、处理数据

下面代码通过addSource添加RabbitMQ数据源。注意,不能使用fromSource方法,是因为RMQSource没有实现SourceFunction方法。

java 复制代码
final DataStream<String> stream = env.addSource(rmqSource).name(username + "'s source from " + queueName).setParallelism(parallelism);

stream.print().name(username + "'s data from " + queueName);

完整代码

java 复制代码
/*
 * Licensed to the Apache Software Foundation (ASF) under one
 * or more contributor license agreements.  See the NOTICE file
 * distributed with this work for additional information
 * regarding copyright ownership.  The ASF licenses this file
 * to you under the Apache License, Version 2.0 (the
 * "License"); you may not use this file except in compliance
 * with the License.  You may obtain a copy of the License at
 *
 *     http://www.apache.org/licenses/LICENSE-2.0
 *
 * Unless required by applicable law or agreed to in writing, software
 * distributed under the License is distributed on an "AS IS" BASIS,
 * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 * See the License for the specific language governing permissions and
 * limitations under the License.
 */

package org.example;

import org.apache.flink.api.common.serialization.SimpleStringSchema;
import org.apache.flink.streaming.api.datastream.DataStream;
import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
import org.apache.flink.streaming.connectors.rabbitmq.RMQSource;
import org.apache.flink.streaming.connectors.rabbitmq.common.RMQConnectionConfig;

/**
 * Skeleton for a Flink DataStream Job.
 *
 * <p>For a tutorial how to write a Flink application, check the
 * tutorials and examples on the <a href="https://flink.apache.org">Flink Website</a>.
 *
 * <p>To package your application into a JAR file for execution, run
 * 'mvn clean package' on the command line.
 *
 * <p>If you change the name of the main class (with the public static void main(String[] args))
 * method, change the respective entry in the POM.xml file (simply search for 'mainClass').
 */
public class DataStreamJob {

	public static void main(String[] args) throws Exception {
		// Sets up the execution environment, which is the main entry point
		// to building Flink applications.
		final StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();

		String queueName = "data.from.rbtmq";
		String host = "172.21.112.140"; // IP of the rabbitmq server
		int port = 5672;
		String username = "admin";
		String password = "fangliang";
		String virtualHost = "/";
		int parallelism = 1;

		// create a RabbitMQ source
		RMQConnectionConfig rmqConnectionConfig = new RMQConnectionConfig.Builder()
				.setHost(host)
				.setPort(port)
				.setUserName(username)
				.setPassword(password)
				.setVirtualHost(virtualHost)
				.build();

		RMQSource<String> rmqSource = new RMQSource<>(rmqConnectionConfig, queueName, true, new SimpleStringSchema());
		final DataStream<String> stream = env.addSource(rmqSource).name(username + "'s source from " + queueName).setParallelism(parallelism);

		stream.print().name(username + "'s data from " + queueName);

		env.execute("Flink Java API Skeleton");
	}
}

打包、上传和运行任务




测试

在RabbitMQ后台的默认交换器中,发布一条消息到data.from.rbtmq

然后使用下面指令可以看到Flink读取到消息并执行了print方法

bash 复制代码
tail log/flink-*-taskexecutor-*.out

==> flink-fangliang-taskexecutor-0-fangliang.out <==

data from http://172.21.112.140:15672/#/exchanges/%2F/amq.default

工程代码

https://github.com/f304646673/FlinkDemo

相关推荐
皮皮林5515 小时前
IDEA 源码阅读利器,你居然还不会?
java·intellij idea
卡尔特斯9 小时前
Android Kotlin 项目代理配置【详细步骤(可选)】
android·java·kotlin
白鲸开源9 小时前
Ubuntu 22 下 DolphinScheduler 3.x 伪集群部署实录
java·ubuntu·开源
ytadpole9 小时前
Java 25 新特性 更简洁、更高效、更现代
java·后端
纪莫9 小时前
A公司一面:类加载的过程是怎么样的? 双亲委派的优点和缺点? 产生fullGC的情况有哪些? spring的动态代理有哪些?区别是什么? 如何排查CPU使用率过高?
java·java面试⑧股
JavaGuide10 小时前
JDK 25(长期支持版) 发布,新特性解读!
java·后端
用户37215742613510 小时前
Java 轻松批量替换 Word 文档文字内容
java
白鲸开源10 小时前
教你数分钟内创建并运行一个 DolphinScheduler Workflow!
java
Java中文社群11 小时前
有点意思!Java8后最有用新特性排行榜!
java·后端·面试
代码匠心11 小时前
从零开始学Flink:数据源
java·大数据·后端·flink