深入探究 Kafka Connect MQTT 连接器：从源码到应用

Edingbrugh.南空2025-06-19 18:55

引言

在现代数据处理和消息传递领域，Apache Kafka 作为一个强大的分布式流处理平台，与 MQTT（Message Queuing Telemetry Transport）这一轻量级的消息传输协议的结合，为我们提供了处理实时数据的有效手段。kafka-connect-mqtt 项目正是实现这一结合的关键，它提供了 MQTT 源连接器（Source Connector）和汇连接器（Sink Connector），使得数据能够在 MQTT 代理和 Kafka 主题之间高效流动。本文将基于该项目的 GitHub 源码（https://github.com/johanvandevenne/kafka-connect-mqtt/tree/master/src/main/java/be/jovacon/kafka/connect），深入探讨其实现原理、构建过程、安装配置以及使用方法。

项目概述

kafka-connect-mqtt 项目包含了 MQTT 源连接器和汇连接器的实现。源连接器用于订阅 MQTT 主题，并将接收到的消息发送到 Kafka 主题；汇连接器则相反，它从 Kafka 主题读取消息，并将其发布到 MQTT 主题。该项目经过了 Kafka 2+ 版本的测试，但目前不支持 SSL 连接，且仅支持单任务模式（即 maxTasks > 1 无效）。

构建连接器

前提条件

在构建连接器之前，需要确保以下软件已安装：

  • Java 8 或更高版本
  • Maven
  • GIT

克隆仓库

使用以下命令克隆项目仓库：

bash 复制代码 
git clone https://github.com/johanvandevenne/kafka-connect-mqtt.git

进入项目目录

bash 复制代码 
cd kafka-connect-mqtt

使用 Maven 构建

bash 复制代码 
mvn clean install

源码分析

版本管理

项目通过 Version 类管理版本信息，该类从 kafka-connect-mqtt-version.properties 文件中加载版本号：

java 复制代码 
package be.jovacon.kafka.connect;

import org.slf4j.Logger;
import org.slf4j.LoggerFactory;

import java.util.Properties;

public class Version {
    private static final Logger log = LoggerFactory.getLogger(Version.class);
    private static String version = "unknown";

    static {
        try {
            Properties props = new Properties();
            props.load(Version.class.getResourceAsStream("/kafka-connect-mqtt-version.properties"));
            version = props.getProperty("version", version).trim();
        } catch (Exception e) {
            log.warn("Error while loading version:", e);
        }
    }

    public static String getVersion() {
        return version;
    }

    public static void main(String[] args) {
        System.out.println(Version.getVersion());
    }
}

源连接器

源连接器的核心类是 MQTTSourceConnector 和 MQTTSourceTask。MQTTSourceConnector 负责初始化配置和任务类，而 MQTTSourceTask 则负责实际的 MQTT 连接、订阅和消息转换：

java 复制代码 
// MQTTSourceConnector.java
package be.jovacon.kafka.connect;

import be.jovacon.kafka.connect.config.MQTTSourceConnectorConfig;
import org.apache.kafka.common.config.ConfigDef;
import org.apache.kafka.connect.connector.Task;
import org.apache.kafka.connect.source.SourceConnector;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;

import java.util.*;

public class MQTTSourceConnector extends SourceConnector {

    private static final Logger log = LoggerFactory.getLogger(MQTTSourceConnector.class);
    private MQTTSourceConnectorConfig mqttSourceConnectorConfig;
    private Map<String, String> configProps;

    public void start(Map<String, String> map) {
        this.mqttSourceConnectorConfig = new MQTTSourceConnectorConfig(map);
        this.configProps = Collections.unmodifiableMap(map);
    }

    public Class<? extends Task> taskClass() {
        return MQTTSourceTask.class;
    }

    public List<Map<String, String>> taskConfigs(int maxTasks) {
        log.debug("Enter taskconfigs");
        if (maxTasks > 1) {
            log.info("maxTasks is " + maxTasks + ". MaxTasks > 1 is not supported in this connector.");
        }
        List<Map<String, String>> taskConfigs = new ArrayList<>(1);
        taskConfigs.add(new HashMap<>(configProps));

        log.debug("Taskconfigs: " + taskConfigs);
        return taskConfigs;
    }

    public void stop() {

    }

    public ConfigDef config() {
        return MQTTSourceConnectorConfig.configDef();
    }

    public String version() {
        return Version.getVersion();
    }
}
java 复制代码 
// MQTTSourceTask.java
package be.jovacon.kafka.connect;

import be.jovacon.kafka.connect.config.MQTTSourceConnectorConfig;
import com.github.jcustenborder.kafka.connect.utils.data.SourceRecordDeque;
import com.github.jcustenborder.kafka.connect.utils.data.SourceRecordDequeBuilder;
import org.apache.kafka.connect.errors.ConnectException;
import org.apache.kafka.connect.source.SourceRecord;
import org.apache.kafka.connect.source.SourceTask;
import org.eclipse.paho.client.mqttv3.*;
import org.eclipse.paho.client.mqttv3.persist.MemoryPersistence;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;

import java.util.List;
import java.util.Map;

public class MQTTSourceTask extends SourceTask implements IMqttMessageListener {

    private Logger log = LoggerFactory.getLogger(MQTTSourceConnector.class);
    private MQTTSourceConnectorConfig config;
    private MQTTSourceConverter mqttSourceConverter;
    private SourceRecordDeque sourceRecordDeque;

    private IMqttClient mqttClient;

    public void start(Map<String, String> props) {
        config = new MQTTSourceConnectorConfig(props);
        mqttSourceConverter = new MQTTSourceConverter(config);
        this.sourceRecordDeque = SourceRecordDequeBuilder.of().batchSize(4096).emptyWaitMs(100).maximumCapacityTimeoutMs(60000).maximumCapacity(50000).build();
        try {
            mqttClient = new MqttClient(config.getString(MQTTSourceConnectorConfig.BROKER), config.getString(MQTTSourceConnectorConfig.CLIENTID), new MemoryPersistence());

            log.info("Connecting to MQTT Broker " + config.getString(MQTTSourceConnectorConfig.BROKER));
            connect(mqttClient);
            log.info("Connected to MQTT Broker");

            String topicSubscription = this.config.getString(MQTTSourceConnectorConfig.MQTT_TOPIC);
            int qosLevel = this.config.getInt(MQTTSourceConnectorConfig.MQTT_QOS);

            log.info("Subscribing to " + topicSubscription + " with QOS " + qosLevel);
            mqttClient.subscribe(topicSubscription, qosLevel, (topic, message) -> {
                log.debug("Message arrived in connector from topic " + topic);
                SourceRecord record = mqttSourceConverter.convert(topic, message);
                log.debug("Converted record: " + record);
                sourceRecordDeque.add(record);
            });
            log.info("Subscribed to " + topicSubscription + " with QOS " + qosLevel);
        }
        catch (MqttException e) {
            throw new ConnectException(e);
        }
    }

    private void connect(IMqttClient mqttClient) throws MqttException{
        MqttConnectOptions connOpts = new MqttConnectOptions();
        connOpts.setCleanSession(config.getBoolean(MQTTSourceConnectorConfig.MQTT_CLEANSESSION));
        connOpts.setKeepAliveInterval(config.getInt(MQTTSourceConnectorConfig.MQTT_KEEPALIVEINTERVAL));
        connOpts.setConnectionTimeout(config.getInt(MQTTSourceConnectorConfig.MQTT_CONNECTIONTIMEOUT));
        connOpts.setAutomaticReconnect(config.getBoolean(MQTTSourceConnectorConfig.MQTT_ARC));

        if (!config.getString(MQTTSourceConnectorConfig.MQTT_USERNAME).equals("") && !config.getPassword(MQTTSourceConnectorConfig.MQTT_PASSWORD).equals("")) {
            connOpts.setUserName(config.getString(MQTTSourceConnectorConfig.MQTT_USERNAME));
            connOpts.setPassword(config.getPassword(MQTTSourceConnectorConfig.MQTT_PASSWORD).value().toCharArray());
        }

        log.info("MQTT Connection properties: " + connOpts);

        mqttClient.connect(connOpts);
    }

    public List<SourceRecord> poll() throws InterruptedException {
        List<SourceRecord> records = sourceRecordDeque.getBatch();
        log.trace("Records returning to poll(): " + records);
        return records;
    }

    public void stop() {
        if (mqttClient.isConnected()) {
            try {
                log.debug("Disconnecting from MQTT Broker " + config.getString(MQTTSourceConnectorConfig.BROKER));
                mqttClient.disconnect();
            } catch (MqttException mqttException) {
                log.error("Exception thrown while disconnecting client.", mqttException);
            }
        }
    }

    public String version() {
        return Version.getVersion();
    }

    @Override
    public void messageArrived(String topic, MqttMessage mqttMessage) throws Exception {
        log.debug("Message arrived in connector from topic " + topic);
        SourceRecord record = mqttSourceConverter.convert(topic, mqttMessage);
        log.debug("Converted record: " + record);
        sourceRecordDeque.add(record);
    }
}

汇连接器

汇连接器的核心类是 MQTTSinkConnector 和 MQTTSinkTask。MQTTSinkConnector 负责初始化配置和任务类，而 MQTTSinkTask 则负责从 Kafka 读取消息并发布到 MQTT 主题：

java 复制代码 
// MQTTSinkConnector.java
package be.jovacon.kafka.connect;

import be.jovacon.kafka.connect.config.MQTTSinkConnectorConfig;
import org.apache.kafka.common.config.ConfigDef;
import org.apache.kafka.connect.connector.Task;
import org.apache.kafka.connect.sink.SinkConnector;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;

import java.util.*;

public class MQTTSinkConnector extends SinkConnector {

    private static final Logger log = LoggerFactory.getLogger(MQTTSinkConnector.class);
    private MQTTSinkConnectorConfig mqttSinkConnectorConfig;
    private Map<String, String> configProps;

    @Override
    public void start(Map<String, String> map) {
        this.mqttSinkConnectorConfig = new MQTTSinkConnectorConfig(map);
        this.configProps = Collections.unmodifiableMap(map);
    }

    @Override
    public Class<? extends Task> taskClass() {
        return MQTTSinkTask.class;
    }

    @Override
    public List<Map<String, String>> taskConfigs(int maxTasks) {
        log.debug("Enter taskconfigs");
        if (maxTasks > 1) {
            log.info("maxTasks is " + maxTasks + ". MaxTasks > 1 is not supported in this connector.");
        }
        List<Map<String, String>> taskConfigs = new ArrayList<>(1);
        taskConfigs.add(new HashMap<>(configProps));

        log.debug("Taskconfigs: " + taskConfigs);
        return taskConfigs;
    }

    @Override
    public void stop() {

    }

    @Override
    public ConfigDef config() {
        return MQTTSinkConnectorConfig.configDef();
    }

    @Override
    public String version() {
        return Version.getVersion();
    }
}
java 复制代码 
// MQTTSinkTask.java
package be.jovacon.kafka.connect;

import be.jovacon.kafka.connect.config.MQTTSinkConnectorConfig;
import be.jovacon.kafka.connect.config.MQTTSourceConnectorConfig;
import org.apache.kafka.connect.errors.ConnectException;
import org.apache.kafka.connect.sink.SinkRecord;
import org.apache.kafka.connect.sink.SinkTask;
import org.eclipse.paho.client.mqttv3.*;
import org.eclipse.paho.client.mqttv3.persist.MemoryPersistence;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;

import java.util.Collection;
import java.util.Iterator;
import java.util.Map;

public class MQTTSinkTask extends SinkTask {

    private Logger log = LoggerFactory.getLogger(MQTTSinkTask.class);
    private MQTTSinkConnectorConfig config;
    private MQTTSinkConverter mqttSinkConverter;

    private IMqttClient mqttClient;

    @Override
    public String version() {
        return Version.getVersion();
    }

    @Override
    public void start(Map<String, String> map) {
        config = new MQTTSinkConnectorConfig(map);
        mqttSinkConverter = new MQTTSinkConverter(config);
        try {
            mqttClient = new MqttClient(config.getString(MQTTSinkConnectorConfig.BROKER), config.getString(MQTTSinkConnectorConfig.CLIENTID), new MemoryPersistence());

            log.info("Connecting to MQTT Broker " + config.getString(MQTTSourceConnectorConfig.BROKER));
            connect(mqttClient);
            log.info("Connected to MQTT Broker. This connector publishes to the " + this.config.getString(MQTTSinkConnectorConfig.MQTT_TOPIC) + " topic");

        }
        catch (MqttException e) {
            throw new ConnectException(e);
        }
    }

    private void connect(IMqttClient mqttClient) throws MqttException{
        MqttConnectOptions connOpts = new MqttConnectOptions();
        connOpts.setCleanSession(config.getBoolean(MQTTSinkConnectorConfig.MQTT_CLEANSESSION));
        connOpts.setKeepAliveInterval(config.getInt(MQTTSinkConnectorConfig.MQTT_KEEPALIVEINTERVAL));
        connOpts.setConnectionTimeout(config.getInt(MQTTSinkConnectorConfig.MQTT_CONNECTIONTIMEOUT));
        connOpts.setAutomaticReconnect(config.getBoolean(MQTTSinkConnectorConfig.MQTT_ARC));

        if (!config.getString(MQTTSinkConnectorConfig.MQTT_USERNAME).equals("") && !config.getPassword(MQTTSinkConnectorConfig.MQTT_PASSWORD).equals("")) {
            connOpts.setUserName(config.getString(MQTTSinkConnectorConfig.MQTT_USERNAME));
            connOpts.setPassword(config.getPassword(MQTTSinkConnectorConfig.MQTT_PASSWORD).value().toCharArray());
        }

        log.debug("MQTT Connection properties: " + connOpts);

        mqttClient.connect(connOpts);
    }

    @Override
    public void put(Collection<SinkRecord> collection) {
        try {
            for (Iterator<SinkRecord> iterator = collection.iterator(); iterator.hasNext(); ) {
                SinkRecord sinkRecord = iterator.next();
                log.debug("Received message with offset " + sinkRecord.kafkaOffset());
                MqttMessage mqttMessage = mqttSinkConverter.convert(sinkRecord);
                if (!mqttClient.isConnected()) mqttClient.connect();
                log.debug("Publishing message to topic " + this.config.getString(MQTTSinkConnectorConfig.MQTT_TOPIC) + " with payload " + new String(mqttMessage.getPayload()));
                mqttClient.publish(this.config.getString(MQTTSinkConnectorConfig.MQTT_TOPIC), mqttMessage);
            }
        } catch (MqttException e) {
            throw new ConnectException(e);
        }

    }

    @Override
    public void stop() {
        if (mqttClient.isConnected()) {
            try {
                log.debug("Disconnecting from MQTT Broker " + config.getString(MQTTSinkConnectorConfig.BROKER));
                mqttClient.disconnect();
            } catch (MqttException mqttException) {
                log.error("Exception thrown while disconnecting client.", mqttException);
            }
        }
    }
}

消息转换

MQTTSourceConverter 类负责将 MQTT 消息转换为 Kafka 消息，而 MQTTSinkConverter 类则负责将 Kafka 消息转换为 MQTT 消息：

java 复制代码 
// MQTTSourceConverter.java
package be.jovacon.kafka.connect;

import be.jovacon.kafka.connect.config.MQTTSourceConnectorConfig;
import org.apache.kafka.connect.data.Schema;
import org.apache.kafka.connect.header.ConnectHeaders;
import org.apache.kafka.connect.source.SourceRecord;
import org.eclipse.paho.client.mqttv3.MqttMessage;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;

import java.util.HashMap;

public class MQTTSourceConverter {

    private MQTTSourceConnectorConfig mqttSourceConnectorConfig;

    private Logger log = LoggerFactory.getLogger(MQTTSourceConverter.class);

    public MQTTSourceConverter(MQTTSourceConnectorConfig mqttSourceConnectorConfig) {
        this.mqttSourceConnectorConfig = mqttSourceConnectorConfig;
    }

    protected SourceRecord convert(String topic, MqttMessage mqttMessage) {
        log.debug("Converting MQTT message: " + mqttMessage);
        ConnectHeaders headers = new ConnectHeaders();
        headers.addInt("mqtt.message.id", mqttMessage.getId());
        headers.addInt("mqtt.message.qos", mqttMessage.getQos());
        headers.addBoolean("mqtt.message.duplicate", mqttMessage.isDuplicate());

        SourceRecord sourceRecord = new SourceRecord(new HashMap<>(),
                new HashMap<>(),
                this.mqttSourceConnectorConfig.getString(MQTTSourceConnectorConfig.KAFKA_TOPIC),
                (Integer) null,
                Schema.STRING_SCHEMA,
                topic,
                Schema.STRING_SCHEMA,
                new String(mqttMessage.getPayload()),
                System.currentTimeMillis(),
                headers);
        log.debug("Converted MQTT Message: " + sourceRecord);
        return sourceRecord;
    }
}
java 复制代码 
// MQTTSinkConverter.java
package be.jovacon.kafka.connect;

import be.jovacon.kafka.connect.config.MQTTSinkConnectorConfig;
import be.jovacon.kafka.connect.config.MQTTSourceConnectorConfig;
import org.apache.kafka.connect.sink.SinkRecord;
import org.eclipse.paho.client.mqttv3.MqttMessage;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;

public class MQTTSinkConverter {

    private MQTTSinkConnectorConfig mqttSinkConnectorConfig;

    private Logger log = LoggerFactory.getLogger(MQTTSinkConverter.class);

    public MQTTSinkConverter(MQTTSinkConnectorConfig mqttSinkConnectorConfig) {
        this.mqttSinkConnectorConfig = mqttSinkConnectorConfig;
    }

    protected MqttMessage convert(SinkRecord sinkRecord) {
        log.trace("Converting Kafka message");

        MqttMessage mqttMessage = new MqttMessage();
        mqttMessage.setPayload(((String)sinkRecord.value()).getBytes());
        mqttMessage.setQos(this.mqttSinkConnectorConfig.getInt(MQTTSourceConnectorConfig.MQTT_QOS));
        log.trace("Result MQTTMessage: " + mqttMessage);
        return mqttMessage;
    }
}

安装连接器

前提条件

必须安装 Kafka 2+ 版本。

安装步骤

  1. 将 /target/kafka-connect-mqtt-1.0-0-package/share/kafka-connect-mqtt 文件夹复制到 Kafka Connect 插件路径。
  2. 重启 Kafka Connect。
  3. 检查连接器是否成功加载：
bash 复制代码 
http://<kafkaconnect>:8083/connector-plugins

如果看到以下条目，则表示连接器已成功安装：

json 复制代码 
{
    "class": "MQTTSinkConnector",
    "type": "sink",
    "version": "1.0.0"
},
{
    "class": "MQTTSourceConnector",
    "type": "source",
    "version": "1.0.0"
}

配置连接器

源连接器配置

源连接器订阅 MQTT 主题，并将消息发送到 Kafka 主题。以下是一个基本的配置示例：

bash 复制代码 
curl -X POST \
  http://<kafkaconnect>:8083/connectors \
  -H 'Content-Type: application/json' \
  -d '{ "name": "mqtt-source-connector",
    "config":
    {
      "connector.class":"be.jovacon.kafka.connect.MQTTSourceConnector",
      "mqtt.topic":"my_mqtt_topic",
      "kafka.topic":"my_kafka_topic",
      "mqtt.clientID":"my_client_id",
      "mqtt.broker":"tcp://127.0.0.1:1883",
      "key.converter":"org.apache.kafka.connect.storage.StringConverter",
      "key.converter.schemas.enable":false,
      "value.converter":"org.apache.kafka.connect.storage.StringConverter",
      "value.converter.schemas.enable":false
    }
}'

源连接器可选配置

  • mqtt.qos（可选）：0 -- 最多一次，1 -- 至少一次，2 -- 恰好一次
  • mqtt.automaticReconnect（可选）（默认：true）：客户端在连接失败时是否自动重新连接
  • mqtt.keepAliveInterval（可选）（默认：60 秒）
  • mqtt.cleanSession（可选）（默认：true）：控制客户端与代理断开连接后的状态
  • mqtt.connectionTimeout（可选）（默认：30 秒）
  • mqtt.username（可选）：连接 MQTT 代理的用户名
  • mqtt.password（可选）：连接 MQTT 代理的密码

汇连接器配置

汇连接器从 Kafka 主题读取消息，并将其发布到 MQTT 主题。以下是一个基本的配置示例：

bash 复制代码 
curl -X POST \
  http://<kafkaconnect>:8083/connectors \
  -H 'Content-Type: application/json' \
  -d '{ "name": "mqtt-sink-connector",
    "config":
    {
      "connector.class":"be.jovacon.kafka.connect.MQTTSinkConnector",
      "mqtt.topic":"my_mqtt_topic",
      "topics":"my_kafka_topic",
      "mqtt.clientID":"my_client_id",
      "mqtt.broker":"tcp://127.0.0.1:1883",
      "key.converter":"org.apache.kafka.connect.storage.StringConverter",
      "key.converter.schemas.enable":false,
      "value.converter":"org.apache.kafka.connect.storage.StringConverter",
      "value.converter.schemas.enable":false
    }
}'

汇连接器可选配置

与源连接器的可选配置相同。

总结

kafka-connect-mqtt 项目为我们提供了一种简单而有效的方式，将 MQTT 消息集成到 Kafka 生态系统中。通过深入分析其源码，我们了解了连接器的实现原理和工作流程。同时，我们也掌握了连接器的构建、安装和配置方法，能够在实际项目中灵活运用。尽管目前该项目不支持 SSL 连接和多任务模式，但它仍然是一个强大的工具，适用于许多实时数据处理场景。

相关推荐
zru_960242 分钟前
Kafka核心概念深入浅出：消费者组（Consumer Group）机制全解析
kafka
学习中的阿陈3 小时前
Hadoop伪分布式环境配置
大数据·hadoop·分布式
CesareCheung4 小时前
JMeter分布式压力测试
分布式·jmeter·压力测试
thginWalker4 小时前
面试鸭Java八股之Kafka
kafka
失散135 小时前
分布式专题——10.5 ShardingSphere的CosID主键生成框架
java·分布式·架构·分库分表·shadingsphere
winfield8216 小时前
Kafka 线上问题排查完整手册
kafka
Cxzzzzzzzzzz9 小时前
RabbitMQ 在实际开发中的应用场景与实现方案
分布式·rabbitmq
在未来等你9 小时前
Kafka面试精讲 Day 16：生产者性能优化策略
大数据·分布式·面试·kafka·消息队列
王大帅の王同学9 小时前
Thinkphp6接入讯飞星火大模型Spark Lite完全免费的API
大数据·分布式·spark
一氧化二氢.h11 小时前
通俗解释redis高级：redis持久化（RDB持久化、AOF持久化）、redis主从、redis哨兵、redis分片集群
redis·分布式·缓存
热门推荐
01KGG转MP3工具|非KGM文件|解密音频02GitHub 镜像站点03conda中设置镜像地址（附所有可换的地址）04UV安装并设置国内源05A股预测还能更准？开源大模型Kronos带你跑通预测+回测全流程06突破百度网盘的下载限速，两种方法教会你【超详细】0746个Nano-banana 精选提示词，持续更新中08UV 工具安装与国内镜像源配置指南09Spec-Kit 使用指南10教你如何认证 Gemini 教育优惠的二次验证，薅个 1年的 Gemini Pro 会员