Flink 1.14.* 版本kafkaSink源码

通过Flink 1.14.*版本kafkaSource源码,了解了kafkaSource的源码,这里一起看一下kafkaSink的源码,

一、如何把sink加入到Flink中

首先先找一个类继承FlinkKafkaProducerBase这个抽象类,之后加入到dataStream中就可以了

如下:

java 复制代码
DataStream<Tuple2<String, Integer>> dataStream = source的数据
dataStream..addSink(继承了FlinkKafkaProducerBase的实现类)

二、FlinkKafkaProducerBase的源码

java 复制代码
@Internal
public abstract class FlinkKafkaProducerBase<IN> extends RichSinkFunction<IN> implements CheckpointedFunction {
    public FlinkKafkaProducerBase(String defaultTopicId, KeyedSerializationSchema<IN> serializationSchema, Properties producerConfig, FlinkKafkaPartitioner<IN> customPartitioner) {
        //省略
    }
    //这个是关键,这个是发送的执行逻辑
    public void invoke(IN next, Context context) throws Exception {
        byte[] serializedKey = this.schema.serializeKey(next);
        byte[] serializedValue = this.schema.serializeValue(next);
        String targetTopic = this.schema.getTargetTopic(next);

        if (this.flinkKafkaPartitioner == null) {
            record = new ProducerRecord(targetTopic, serializedKey, serializedValue);
        } else {
            record = new ProducerRecord(targetTopic, this.flinkKafkaPartitioner.partition(next, serializedKey, serializedValue, targetTopic, partitions), serializedKey, serializedValue);
        }

        this.producer.send(record, this.callback);
    }    
}    

其中invoke是父类RichSinkFunction实现的接口SinkFunction中的方法

三、调用SinkFunction的invoke的地方

java 复制代码
@Public
public class DataStream<T> {
    public DataStreamSink<T> addSink(SinkFunction<T> sinkFunction) {
        this.transformation.getOutputType();
        if (sinkFunction instanceof InputTypeConfigurable) {
            ((InputTypeConfigurable)sinkFunction).setInputType(this.getType(), this.getExecutionConfig());
        }
        //这个生成sink算子
        StreamSink<T> sinkOperator = new StreamSink((SinkFunction)this.clean(sinkFunction));
        DataStreamSink<T> sink = new DataStreamSink(this, sinkOperator);
        this.getExecutionEnvironment().addOperator(sink.getLegacyTransformation());
        return sink;
    }

} 

kafkaSource一样,会把sinkFunction赋值给AbstractUdfStreamOperatoruserFunction字段

java 复制代码
public class StreamSink<IN> extends AbstractUdfStreamOperator<Object, SinkFunction<IN>> implements OneInputStreamOperator<IN, Object> {
    public StreamSink(SinkFunction<IN> sinkFunction) {
        super(sinkFunction);
        this.chainingStrategy = ChainingStrategy.ALWAYS;
    }
    public void processElement(StreamRecord<IN> element) throws Exception {
        this.sinkContext.element = element;
        ((SinkFunction)this.userFunction).invoke(element.getValue(), this.sinkContext);
    }
}
java 复制代码
@PublicEvolving
public abstract class AbstractUdfStreamOperator<OUT, F extends Function> extends AbstractStreamOperator<OUT> implements OutputTypeConfigurable<OUT> {
    private static final long serialVersionUID = 1L;
    protected final F userFunction;

    public AbstractUdfStreamOperator(F userFunction) {
        this.userFunction = (Function)Objects.requireNonNull(userFunction);
        this.checkUdfCheckpointingPreconditions();
    }
}

这里StreamSink也是实现了implements OneInputStreamOperator接口,等Flink运行时调用StreamSinkprocessElement方法触发实现类的invoke方法,

下面是FlinkKafkaProducerBase类关系图

相关推荐
W Y15 小时前
【架构-37】Spark和Flink
架构·flink·spark
ycsdn1015 小时前
Caused by: org.apache.flink.api.common.io.ParseException: Row too short:
大数据·flink
晨航2 天前
Flink本地模式安装详解
大数据·flink
sf_www2 天前
flink 内存配置(五):网络缓存调优
大数据·网络·flink
我的K84092 天前
Flink滑动窗口(Sliding)中window和windowAll的区别
flink
inori12562 天前
FlinkSql读取外部Mysql和HBase数据库的方法(scala)
mysql·flink·hbase
小强签名设计2 天前
Flink CDC 同步 Mysql 数据
大数据·mysql·flink
牧竹子2 天前
FLINK单机版安装部署入门-1
大数据·flink
sf_www2 天前
flink 内存配置(四):内存调优和问题处理
大数据·flink
静听山水2 天前
Flink-Kafka-Connector
flink