flink1.16+连接Elasticsearch7官方例子报错解决方案

flink输出至es,连接es7时使用的sink类是Elasticsearch7SinkBuilder,与es6链接方式不同,官网地址:fink连接es官方地址

flink连接es7官方示例

依赖jar:

xml 复制代码
<dependency>
    <groupId>org.apache.flink</groupId>
    <artifactId>flink-connector-elasticsearch7</artifactId>
    <version>3.0.1-1.16</version>
</dependency>

官方连接es7代码

java 复制代码
import org.apache.flink.api.common.functions.MapFunction;
import org.apache.flink.connector.elasticsearch.sink.Elasticsearch7SinkBuilder;
import org.apache.flink.streaming.api.datastream.DataStream;

import org.apache.http.HttpHost;
import org.elasticsearch.action.index.IndexRequest;
import org.elasticsearch.client.Requests;

import java.util.HashMap;
import java.util.Map;

DataStream<String> input = ...;

input.sinkTo(
    new Elasticsearch7SinkBuilder<String>()
        .setBulkFlushMaxActions(1) // Instructs the sink to emit after every element, otherwise they would be buffered
        .setHosts(new HttpHost("127.0.0.1", 9200, "http"))
        .setEmitter(
        (element, context, indexer) ->
        indexer.add(createIndexRequest(element)))
        .build());

private static IndexRequest createIndexRequest(String element) {
    Map<String, Object> json = new HashMap<>();
    json.put("data", element);

    return Requests.indexRequest()
        .index("my-index")
        .id(element)
        .source(json);
}

本地运行报错

java.lang.IllegalStateException: The elasticsearch emitter must be serializable.

错误提示很明确:emitter没有序列化,点击查看ElasticsearchEmitter的源码其实最终是有继承序列化接口的,那就猜测是使用lambda方式不能自己实现序列化,那就自己写个实现类ElasticsearchEmitter接口然后显示序列化一次。

修改后代码

java 复制代码
import org.apache.flink.api.common.functions.MapFunction;
import org.apache.flink.connector.elasticsearch.sink.Elasticsearch7SinkBuilder;
import org.apache.flink.streaming.api.datastream.DataStream;

import org.apache.http.HttpHost;
import org.elasticsearch.action.index.IndexRequest;
import org.elasticsearch.client.Requests;

import java.util.HashMap;
import java.util.Map;

DataStream<String> input = ...;

input.sinkTo(
    new Elasticsearch7SinkBuilder<String>()
        .setBulkFlushMaxActions(1) // Instructs the sink to emit after every element, otherwise they would be buffered
        .setHosts(new HttpHost("127.0.0.1", 9200, "http"))
        .setEmitter(
        (element, context, indexer) ->
        indexer.add(createIndexRequest(element)))
        .build());

private static IndexRequest createIndexRequest(String element) {
    Map<String, Object> json = new HashMap<>();
    json.put("data", element);

    return Requests.indexRequest()
        .index("my-index")
        .id(element)
        .source(json);
}

//自己新增类继承ElasticsearchEmitter,实现序列化接口
    static class MyElasticsearchEmitter implements ElasticsearchEmitter<String>, Serializable {

        @Override
        public void emit(String element, SinkWriter.Context context, RequestIndexer indexer) {
            indexer.add(createIndexRequest(element));
        }

    }

启动后验证成功。

flink连接es7抽出完成工具类

java 复制代码
import com.alibaba.fastjson.JSON;
import lombok.extern.slf4j.Slf4j;
import org.apache.flink.api.connector.sink2.Sink;
import org.apache.flink.api.connector.sink2.SinkWriter;
import org.apache.flink.connector.elasticsearch.sink.*;
import org.apache.http.HttpHost;
import org.elasticsearch.action.index.IndexRequest;
import org.elasticsearch.client.Requests;

import java.io.Serializable;

@Slf4j
public class Es7SinkFactory {

    public static Sink createSink(String host, Integer port, String schema, String user, String pwd, String index) {
        try {

            ElasticsearchSink<String> elasticsearchSink = new Elasticsearch7SinkBuilder<String>()
                    .setHosts(new HttpHost(host, port, schema))
                    .setConnectionUsername(user)
                    .setConnectionPassword(pwd)
                    .setEmitter(new MyElasticsearchEmitter(index))
                    .setBulkFlushMaxSizeMb(5)//最大缓冲大小
                    .setBulkFlushInterval(5000)//刷新间隔
                    .setBulkFlushMaxActions(1000)//批量插入最大条数
                    .setBulkFlushBackoffStrategy(FlushBackoffType.CONSTANT, 5, 100)
                    .build();

            return elasticsearchSink;
        } catch (Exception e) {
            log.error("Es7SinkFactory.createSink error e=", e);
        }

        return null;
    }

    private static IndexRequest createIndexRequest(String element, String index) {
        return Requests.indexRequest()
                .index(index)
                .source(JSON.parseObject(element));
    }


    //匿名内部类/lamda表达式未集成序列化接口,需要实现序列化接口
    static class MyElasticsearchEmitter implements ElasticsearchEmitter<String>, Serializable {

        private String index;

        MyElasticsearchEmitter(String index) {
            this.index = index;
        }

        @Override
        public void emit(String element, SinkWriter.Context context, RequestIndexer indexer) {
            indexer.add(createIndexRequest(element, index));
        }

    }


}
相关推荐
YuTaoShao2 小时前
【LeetCode 热题 100】131. 分割回文串——回溯
java·算法·leetcode·深度优先
源码_V_saaskw2 小时前
JAVA图文短视频交友+自营商城系统源码支持小程序+Android+IOS+H5
java·微信小程序·小程序·uni-app·音视频·交友
超浪的晨2 小时前
Java UDP 通信详解:从基础到实战,彻底掌握无连接网络编程
java·开发语言·后端·学习·个人开发
双力臂4043 小时前
Spring Boot 单元测试进阶:JUnit5 + Mock测试与切片测试实战及覆盖率报告生成
java·spring boot·后端·单元测试
Edingbrugh.南空3 小时前
Aerospike与Redis深度对比:从架构到性能的全方位解析
java·开发语言·spring
QQ_4376643144 小时前
C++11 右值引用 Lambda 表达式
java·开发语言·c++
永卿0014 小时前
设计模式-迭代器模式
java·设计模式·迭代器模式
誰能久伴不乏4 小时前
Linux如何执行系统调用及高效执行系统调用:深入浅出的解析
java·服务器·前端
慕y2744 小时前
Java学习第七十二部分——Zookeeper
java·学习·java-zookeeper
midsummer_woo4 小时前
基于spring boot的医院挂号就诊系统(源码+论文)
java·spring boot·后端