深入探究 Kafka Connect MQTT 连接器：从源码到应用

Edingbrugh.南空2025-06-19 18:55

引言

在现代数据处理和消息传递领域，Apache Kafka 作为一个强大的分布式流处理平台，与 MQTT（Message Queuing Telemetry Transport）这一轻量级的消息传输协议的结合，为我们提供了处理实时数据的有效手段。kafka-connect-mqtt 项目正是实现这一结合的关键，它提供了 MQTT 源连接器（Source Connector）和汇连接器（Sink Connector），使得数据能够在 MQTT 代理和 Kafka 主题之间高效流动。本文将基于该项目的 GitHub 源码（https://github.com/johanvandevenne/kafka-connect-mqtt/tree/master/src/main/java/be/jovacon/kafka/connect），深入探讨其实现原理、构建过程、安装配置以及使用方法。

项目概述

kafka-connect-mqtt 项目包含了 MQTT 源连接器和汇连接器的实现。源连接器用于订阅 MQTT 主题，并将接收到的消息发送到 Kafka 主题；汇连接器则相反，它从 Kafka 主题读取消息，并将其发布到 MQTT 主题。该项目经过了 Kafka 2+ 版本的测试，但目前不支持 SSL 连接，且仅支持单任务模式（即 maxTasks > 1 无效）。

构建连接器

前提条件

在构建连接器之前，需要确保以下软件已安装：

  • Java 8 或更高版本
  • Maven
  • GIT

克隆仓库

使用以下命令克隆项目仓库：

bash 复制代码 
git clone https://github.com/johanvandevenne/kafka-connect-mqtt.git

进入项目目录

bash 复制代码 
cd kafka-connect-mqtt

使用 Maven 构建

bash 复制代码 
mvn clean install

源码分析

版本管理

项目通过 Version 类管理版本信息，该类从 kafka-connect-mqtt-version.properties 文件中加载版本号：

java 复制代码 
package be.jovacon.kafka.connect;

import org.slf4j.Logger;
import org.slf4j.LoggerFactory;

import java.util.Properties;

public class Version {
    private static final Logger log = LoggerFactory.getLogger(Version.class);
    private static String version = "unknown";

    static {
        try {
            Properties props = new Properties();
            props.load(Version.class.getResourceAsStream("/kafka-connect-mqtt-version.properties"));
            version = props.getProperty("version", version).trim();
        } catch (Exception e) {
            log.warn("Error while loading version:", e);
        }
    }

    public static String getVersion() {
        return version;
    }

    public static void main(String[] args) {
        System.out.println(Version.getVersion());
    }
}

源连接器

源连接器的核心类是 MQTTSourceConnector 和 MQTTSourceTask。MQTTSourceConnector 负责初始化配置和任务类，而 MQTTSourceTask 则负责实际的 MQTT 连接、订阅和消息转换：

java 复制代码 
// MQTTSourceConnector.java
package be.jovacon.kafka.connect;

import be.jovacon.kafka.connect.config.MQTTSourceConnectorConfig;
import org.apache.kafka.common.config.ConfigDef;
import org.apache.kafka.connect.connector.Task;
import org.apache.kafka.connect.source.SourceConnector;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;

import java.util.*;

public class MQTTSourceConnector extends SourceConnector {

    private static final Logger log = LoggerFactory.getLogger(MQTTSourceConnector.class);
    private MQTTSourceConnectorConfig mqttSourceConnectorConfig;
    private Map<String, String> configProps;

    public void start(Map<String, String> map) {
        this.mqttSourceConnectorConfig = new MQTTSourceConnectorConfig(map);
        this.configProps = Collections.unmodifiableMap(map);
    }

    public Class<? extends Task> taskClass() {
        return MQTTSourceTask.class;
    }

    public List<Map<String, String>> taskConfigs(int maxTasks) {
        log.debug("Enter taskconfigs");
        if (maxTasks > 1) {
            log.info("maxTasks is " + maxTasks + ". MaxTasks > 1 is not supported in this connector.");
        }
        List<Map<String, String>> taskConfigs = new ArrayList<>(1);
        taskConfigs.add(new HashMap<>(configProps));

        log.debug("Taskconfigs: " + taskConfigs);
        return taskConfigs;
    }

    public void stop() {

    }

    public ConfigDef config() {
        return MQTTSourceConnectorConfig.configDef();
    }

    public String version() {
        return Version.getVersion();
    }
}
java 复制代码 
// MQTTSourceTask.java
package be.jovacon.kafka.connect;

import be.jovacon.kafka.connect.config.MQTTSourceConnectorConfig;
import com.github.jcustenborder.kafka.connect.utils.data.SourceRecordDeque;
import com.github.jcustenborder.kafka.connect.utils.data.SourceRecordDequeBuilder;
import org.apache.kafka.connect.errors.ConnectException;
import org.apache.kafka.connect.source.SourceRecord;
import org.apache.kafka.connect.source.SourceTask;
import org.eclipse.paho.client.mqttv3.*;
import org.eclipse.paho.client.mqttv3.persist.MemoryPersistence;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;

import java.util.List;
import java.util.Map;

public class MQTTSourceTask extends SourceTask implements IMqttMessageListener {

    private Logger log = LoggerFactory.getLogger(MQTTSourceConnector.class);
    private MQTTSourceConnectorConfig config;
    private MQTTSourceConverter mqttSourceConverter;
    private SourceRecordDeque sourceRecordDeque;

    private IMqttClient mqttClient;

    public void start(Map<String, String> props) {
        config = new MQTTSourceConnectorConfig(props);
        mqttSourceConverter = new MQTTSourceConverter(config);
        this.sourceRecordDeque = SourceRecordDequeBuilder.of().batchSize(4096).emptyWaitMs(100).maximumCapacityTimeoutMs(60000).maximumCapacity(50000).build();
        try {
            mqttClient = new MqttClient(config.getString(MQTTSourceConnectorConfig.BROKER), config.getString(MQTTSourceConnectorConfig.CLIENTID), new MemoryPersistence());

            log.info("Connecting to MQTT Broker " + config.getString(MQTTSourceConnectorConfig.BROKER));
            connect(mqttClient);
            log.info("Connected to MQTT Broker");

            String topicSubscription = this.config.getString(MQTTSourceConnectorConfig.MQTT_TOPIC);
            int qosLevel = this.config.getInt(MQTTSourceConnectorConfig.MQTT_QOS);

            log.info("Subscribing to " + topicSubscription + " with QOS " + qosLevel);
            mqttClient.subscribe(topicSubscription, qosLevel, (topic, message) -> {
                log.debug("Message arrived in connector from topic " + topic);
                SourceRecord record = mqttSourceConverter.convert(topic, message);
                log.debug("Converted record: " + record);
                sourceRecordDeque.add(record);
            });
            log.info("Subscribed to " + topicSubscription + " with QOS " + qosLevel);
        }
        catch (MqttException e) {
            throw new ConnectException(e);
        }
    }

    private void connect(IMqttClient mqttClient) throws MqttException{
        MqttConnectOptions connOpts = new MqttConnectOptions();
        connOpts.setCleanSession(config.getBoolean(MQTTSourceConnectorConfig.MQTT_CLEANSESSION));
        connOpts.setKeepAliveInterval(config.getInt(MQTTSourceConnectorConfig.MQTT_KEEPALIVEINTERVAL));
        connOpts.setConnectionTimeout(config.getInt(MQTTSourceConnectorConfig.MQTT_CONNECTIONTIMEOUT));
        connOpts.setAutomaticReconnect(config.getBoolean(MQTTSourceConnectorConfig.MQTT_ARC));

        if (!config.getString(MQTTSourceConnectorConfig.MQTT_USERNAME).equals("") && !config.getPassword(MQTTSourceConnectorConfig.MQTT_PASSWORD).equals("")) {
            connOpts.setUserName(config.getString(MQTTSourceConnectorConfig.MQTT_USERNAME));
            connOpts.setPassword(config.getPassword(MQTTSourceConnectorConfig.MQTT_PASSWORD).value().toCharArray());
        }

        log.info("MQTT Connection properties: " + connOpts);

        mqttClient.connect(connOpts);
    }

    public List<SourceRecord> poll() throws InterruptedException {
        List<SourceRecord> records = sourceRecordDeque.getBatch();
        log.trace("Records returning to poll(): " + records);
        return records;
    }

    public void stop() {
        if (mqttClient.isConnected()) {
            try {
                log.debug("Disconnecting from MQTT Broker " + config.getString(MQTTSourceConnectorConfig.BROKER));
                mqttClient.disconnect();
            } catch (MqttException mqttException) {
                log.error("Exception thrown while disconnecting client.", mqttException);
            }
        }
    }

    public String version() {
        return Version.getVersion();
    }

    @Override
    public void messageArrived(String topic, MqttMessage mqttMessage) throws Exception {
        log.debug("Message arrived in connector from topic " + topic);
        SourceRecord record = mqttSourceConverter.convert(topic, mqttMessage);
        log.debug("Converted record: " + record);
        sourceRecordDeque.add(record);
    }
}

汇连接器

汇连接器的核心类是 MQTTSinkConnector 和 MQTTSinkTask。MQTTSinkConnector 负责初始化配置和任务类，而 MQTTSinkTask 则负责从 Kafka 读取消息并发布到 MQTT 主题：

java 复制代码 
// MQTTSinkConnector.java
package be.jovacon.kafka.connect;

import be.jovacon.kafka.connect.config.MQTTSinkConnectorConfig;
import org.apache.kafka.common.config.ConfigDef;
import org.apache.kafka.connect.connector.Task;
import org.apache.kafka.connect.sink.SinkConnector;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;

import java.util.*;

public class MQTTSinkConnector extends SinkConnector {

    private static final Logger log = LoggerFactory.getLogger(MQTTSinkConnector.class);
    private MQTTSinkConnectorConfig mqttSinkConnectorConfig;
    private Map<String, String> configProps;

    @Override
    public void start(Map<String, String> map) {
        this.mqttSinkConnectorConfig = new MQTTSinkConnectorConfig(map);
        this.configProps = Collections.unmodifiableMap(map);
    }

    @Override
    public Class<? extends Task> taskClass() {
        return MQTTSinkTask.class;
    }

    @Override
    public List<Map<String, String>> taskConfigs(int maxTasks) {
        log.debug("Enter taskconfigs");
        if (maxTasks > 1) {
            log.info("maxTasks is " + maxTasks + ". MaxTasks > 1 is not supported in this connector.");
        }
        List<Map<String, String>> taskConfigs = new ArrayList<>(1);
        taskConfigs.add(new HashMap<>(configProps));

        log.debug("Taskconfigs: " + taskConfigs);
        return taskConfigs;
    }

    @Override
    public void stop() {

    }

    @Override
    public ConfigDef config() {
        return MQTTSinkConnectorConfig.configDef();
    }

    @Override
    public String version() {
        return Version.getVersion();
    }
}
java 复制代码 
// MQTTSinkTask.java
package be.jovacon.kafka.connect;

import be.jovacon.kafka.connect.config.MQTTSinkConnectorConfig;
import be.jovacon.kafka.connect.config.MQTTSourceConnectorConfig;
import org.apache.kafka.connect.errors.ConnectException;
import org.apache.kafka.connect.sink.SinkRecord;
import org.apache.kafka.connect.sink.SinkTask;
import org.eclipse.paho.client.mqttv3.*;
import org.eclipse.paho.client.mqttv3.persist.MemoryPersistence;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;

import java.util.Collection;
import java.util.Iterator;
import java.util.Map;

public class MQTTSinkTask extends SinkTask {

    private Logger log = LoggerFactory.getLogger(MQTTSinkTask.class);
    private MQTTSinkConnectorConfig config;
    private MQTTSinkConverter mqttSinkConverter;

    private IMqttClient mqttClient;

    @Override
    public String version() {
        return Version.getVersion();
    }

    @Override
    public void start(Map<String, String> map) {
        config = new MQTTSinkConnectorConfig(map);
        mqttSinkConverter = new MQTTSinkConverter(config);
        try {
            mqttClient = new MqttClient(config.getString(MQTTSinkConnectorConfig.BROKER), config.getString(MQTTSinkConnectorConfig.CLIENTID), new MemoryPersistence());

            log.info("Connecting to MQTT Broker " + config.getString(MQTTSourceConnectorConfig.BROKER));
            connect(mqttClient);
            log.info("Connected to MQTT Broker. This connector publishes to the " + this.config.getString(MQTTSinkConnectorConfig.MQTT_TOPIC) + " topic");

        }
        catch (MqttException e) {
            throw new ConnectException(e);
        }
    }

    private void connect(IMqttClient mqttClient) throws MqttException{
        MqttConnectOptions connOpts = new MqttConnectOptions();
        connOpts.setCleanSession(config.getBoolean(MQTTSinkConnectorConfig.MQTT_CLEANSESSION));
        connOpts.setKeepAliveInterval(config.getInt(MQTTSinkConnectorConfig.MQTT_KEEPALIVEINTERVAL));
        connOpts.setConnectionTimeout(config.getInt(MQTTSinkConnectorConfig.MQTT_CONNECTIONTIMEOUT));
        connOpts.setAutomaticReconnect(config.getBoolean(MQTTSinkConnectorConfig.MQTT_ARC));

        if (!config.getString(MQTTSinkConnectorConfig.MQTT_USERNAME).equals("") && !config.getPassword(MQTTSinkConnectorConfig.MQTT_PASSWORD).equals("")) {
            connOpts.setUserName(config.getString(MQTTSinkConnectorConfig.MQTT_USERNAME));
            connOpts.setPassword(config.getPassword(MQTTSinkConnectorConfig.MQTT_PASSWORD).value().toCharArray());
        }

        log.debug("MQTT Connection properties: " + connOpts);

        mqttClient.connect(connOpts);
    }

    @Override
    public void put(Collection<SinkRecord> collection) {
        try {
            for (Iterator<SinkRecord> iterator = collection.iterator(); iterator.hasNext(); ) {
                SinkRecord sinkRecord = iterator.next();
                log.debug("Received message with offset " + sinkRecord.kafkaOffset());
                MqttMessage mqttMessage = mqttSinkConverter.convert(sinkRecord);
                if (!mqttClient.isConnected()) mqttClient.connect();
                log.debug("Publishing message to topic " + this.config.getString(MQTTSinkConnectorConfig.MQTT_TOPIC) + " with payload " + new String(mqttMessage.getPayload()));
                mqttClient.publish(this.config.getString(MQTTSinkConnectorConfig.MQTT_TOPIC), mqttMessage);
            }
        } catch (MqttException e) {
            throw new ConnectException(e);
        }

    }

    @Override
    public void stop() {
        if (mqttClient.isConnected()) {
            try {
                log.debug("Disconnecting from MQTT Broker " + config.getString(MQTTSinkConnectorConfig.BROKER));
                mqttClient.disconnect();
            } catch (MqttException mqttException) {
                log.error("Exception thrown while disconnecting client.", mqttException);
            }
        }
    }
}

消息转换

MQTTSourceConverter 类负责将 MQTT 消息转换为 Kafka 消息，而 MQTTSinkConverter 类则负责将 Kafka 消息转换为 MQTT 消息：

java 复制代码 
// MQTTSourceConverter.java
package be.jovacon.kafka.connect;

import be.jovacon.kafka.connect.config.MQTTSourceConnectorConfig;
import org.apache.kafka.connect.data.Schema;
import org.apache.kafka.connect.header.ConnectHeaders;
import org.apache.kafka.connect.source.SourceRecord;
import org.eclipse.paho.client.mqttv3.MqttMessage;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;

import java.util.HashMap;

public class MQTTSourceConverter {

    private MQTTSourceConnectorConfig mqttSourceConnectorConfig;

    private Logger log = LoggerFactory.getLogger(MQTTSourceConverter.class);

    public MQTTSourceConverter(MQTTSourceConnectorConfig mqttSourceConnectorConfig) {
        this.mqttSourceConnectorConfig = mqttSourceConnectorConfig;
    }

    protected SourceRecord convert(String topic, MqttMessage mqttMessage) {
        log.debug("Converting MQTT message: " + mqttMessage);
        ConnectHeaders headers = new ConnectHeaders();
        headers.addInt("mqtt.message.id", mqttMessage.getId());
        headers.addInt("mqtt.message.qos", mqttMessage.getQos());
        headers.addBoolean("mqtt.message.duplicate", mqttMessage.isDuplicate());

        SourceRecord sourceRecord = new SourceRecord(new HashMap<>(),
                new HashMap<>(),
                this.mqttSourceConnectorConfig.getString(MQTTSourceConnectorConfig.KAFKA_TOPIC),
                (Integer) null,
                Schema.STRING_SCHEMA,
                topic,
                Schema.STRING_SCHEMA,
                new String(mqttMessage.getPayload()),
                System.currentTimeMillis(),
                headers);
        log.debug("Converted MQTT Message: " + sourceRecord);
        return sourceRecord;
    }
}
java 复制代码 
// MQTTSinkConverter.java
package be.jovacon.kafka.connect;

import be.jovacon.kafka.connect.config.MQTTSinkConnectorConfig;
import be.jovacon.kafka.connect.config.MQTTSourceConnectorConfig;
import org.apache.kafka.connect.sink.SinkRecord;
import org.eclipse.paho.client.mqttv3.MqttMessage;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;

public class MQTTSinkConverter {

    private MQTTSinkConnectorConfig mqttSinkConnectorConfig;

    private Logger log = LoggerFactory.getLogger(MQTTSinkConverter.class);

    public MQTTSinkConverter(MQTTSinkConnectorConfig mqttSinkConnectorConfig) {
        this.mqttSinkConnectorConfig = mqttSinkConnectorConfig;
    }

    protected MqttMessage convert(SinkRecord sinkRecord) {
        log.trace("Converting Kafka message");

        MqttMessage mqttMessage = new MqttMessage();
        mqttMessage.setPayload(((String)sinkRecord.value()).getBytes());
        mqttMessage.setQos(this.mqttSinkConnectorConfig.getInt(MQTTSourceConnectorConfig.MQTT_QOS));
        log.trace("Result MQTTMessage: " + mqttMessage);
        return mqttMessage;
    }
}

安装连接器

前提条件

必须安装 Kafka 2+ 版本。

安装步骤

  1. 将 /target/kafka-connect-mqtt-1.0-0-package/share/kafka-connect-mqtt 文件夹复制到 Kafka Connect 插件路径。
  2. 重启 Kafka Connect。
  3. 检查连接器是否成功加载：
bash 复制代码 
http://<kafkaconnect>:8083/connector-plugins

如果看到以下条目，则表示连接器已成功安装：

json 复制代码 
{
    "class": "MQTTSinkConnector",
    "type": "sink",
    "version": "1.0.0"
},
{
    "class": "MQTTSourceConnector",
    "type": "source",
    "version": "1.0.0"
}

配置连接器

源连接器配置

源连接器订阅 MQTT 主题，并将消息发送到 Kafka 主题。以下是一个基本的配置示例：

bash 复制代码 
curl -X POST \
  http://<kafkaconnect>:8083/connectors \
  -H 'Content-Type: application/json' \
  -d '{ "name": "mqtt-source-connector",
    "config":
    {
      "connector.class":"be.jovacon.kafka.connect.MQTTSourceConnector",
      "mqtt.topic":"my_mqtt_topic",
      "kafka.topic":"my_kafka_topic",
      "mqtt.clientID":"my_client_id",
      "mqtt.broker":"tcp://127.0.0.1:1883",
      "key.converter":"org.apache.kafka.connect.storage.StringConverter",
      "key.converter.schemas.enable":false,
      "value.converter":"org.apache.kafka.connect.storage.StringConverter",
      "value.converter.schemas.enable":false
    }
}'

源连接器可选配置

  • mqtt.qos（可选）：0 -- 最多一次，1 -- 至少一次，2 -- 恰好一次
  • mqtt.automaticReconnect（可选）（默认：true）：客户端在连接失败时是否自动重新连接
  • mqtt.keepAliveInterval（可选）（默认：60 秒）
  • mqtt.cleanSession（可选）（默认：true）：控制客户端与代理断开连接后的状态
  • mqtt.connectionTimeout（可选）（默认：30 秒）
  • mqtt.username（可选）：连接 MQTT 代理的用户名
  • mqtt.password（可选）：连接 MQTT 代理的密码

汇连接器配置

汇连接器从 Kafka 主题读取消息，并将其发布到 MQTT 主题。以下是一个基本的配置示例：

bash 复制代码 
curl -X POST \
  http://<kafkaconnect>:8083/connectors \
  -H 'Content-Type: application/json' \
  -d '{ "name": "mqtt-sink-connector",
    "config":
    {
      "connector.class":"be.jovacon.kafka.connect.MQTTSinkConnector",
      "mqtt.topic":"my_mqtt_topic",
      "topics":"my_kafka_topic",
      "mqtt.clientID":"my_client_id",
      "mqtt.broker":"tcp://127.0.0.1:1883",
      "key.converter":"org.apache.kafka.connect.storage.StringConverter",
      "key.converter.schemas.enable":false,
      "value.converter":"org.apache.kafka.connect.storage.StringConverter",
      "value.converter.schemas.enable":false
    }
}'

汇连接器可选配置

与源连接器的可选配置相同。

总结

kafka-connect-mqtt 项目为我们提供了一种简单而有效的方式，将 MQTT 消息集成到 Kafka 生态系统中。通过深入分析其源码，我们了解了连接器的实现原理和工作流程。同时，我们也掌握了连接器的构建、安装和配置方法，能够在实际项目中灵活运用。尽管目前该项目不支持 SSL 连接和多任务模式，但它仍然是一个强大的工具，适用于许多实时数据处理场景。

相关推荐
Edingbrugh.南空1 小时前
多维度剖析Kafka的高性能与高吞吐奥秘
分布式·kafka
Edingbrugh.南空1 小时前
Kafka数据写入流程源码深度剖析（客户端篇）
分布式·kafka
高冷小伙2 小时前
介绍下分布式ID的技术实现及应用场景
分布式
爱吃芝麻汤圆2 小时前
分布式——分布式系统设计策略一
分布式
皮皮林5513 小时前
面试官：kafka 分布式的情况下，如何保证消息的顺序消费？
kafka
计算机毕设定制辅导-无忧学长7 小时前
Kubernetes 部署 Kafka 集群：容器化与高可用方案（二）
kafka·kubernetes·linq
小小工匠7 小时前
Kafka - 并发消费拉取数据过少故障分析
分布式·kafka·并发消费
不倒翁^17 小时前
kafka-生产者-(day-4)
分布式·kafka
程序员小刘7 小时前
HarmonyOS5 分布式测试：断网情况支付场景异常恢复验证
分布式·harmonyos5
热门推荐
01扣子（coze）实战|我用扣子搭建了一个自动分析小红薯笔记内容的AI应用|详细步骤拆解02Coze扣子平台完整体验和实践（附国内和国际版对比）03KGG转MP3工具|非KGM文件|解密音频04零代码入门 | Coze——让大模型接入自己的数据库05DeepSeek各版本说明与优缺点分析06从零安装 LLaMA-Factory 微调 Qwen 大模型成功及所有的坑07【云原生】动态资源分配（DRA）深度洞察报告08AI Agent | Coze 插件使用指南：从功能解析到实操步骤09YOLOv8入门 | 重要性能衡量指标、训练结果评价及分析及影响mAP的因素【发论文关注的指标】10免费可用！最强AI数字人对口型神器：让照片开口说话唱歌，支持多人对口型+全身动作，1分钟学会！（附保姆级教程）