flink如何写入es

提示:文章写完后,目录可以自动生成,如何生成可参考右边的帮助文档

文章目录


前言

Flink sink 流数据写入到es5和es7的简单示例。


一、写入到Elasticsearch5

  • pom maven依赖
xml 复制代码
        <dependency>
            <groupId>org.apache.flink</groupId>
            <artifactId>flink-connector-elasticsearch5_2.11</artifactId>
            <version>${flink.version}</version>
        </dependency>
  • 代码如下(示例):
java 复制代码
public class Es5SinkDemo {
    public static void main(String[] args) throws Exception {
        StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
        Row row=Row.of("张三","001",getTimestamp("2016-10-24 21:59:06"));
        Row row2=Row.of("张三","002",getTimestamp("2016-10-24 21:50:06"));
        Row row3=Row.of("张三","002",getTimestamp("2016-10-24 21:51:06"));
        Row row4=Row.of("李四","003",getTimestamp("2016-10-24 21:50:56"));
        Row row5=Row.of("李四","004",getTimestamp("2016-10-24 00:48:36"));
        Row row6=Row.of("王五","005",getTimestamp("2016-10-24 00:48:36"));
        DataStreamSource<Row> source =env.fromElements(row,row2,row3,row4,row5,row6);

        Map<String, String> config = new HashMap<>();
//        config.put("cluster.name", "my-cluster-name");
//        config.put("bulk.flush.max.actions", "1");

        List<InetSocketAddress> transportAddresses = new ArrayList<>();

        transportAddresses.add(new InetSocketAddress(InetAddress.getByName("10.68.8.60"), 9300));
        //Sink操作
        DataStreamSink<Row> rowDataStreamSink = source.addSink(new ElasticsearchSink<>(config, transportAddresses, new ElasticsearchSinkFunction<Row>() {
            public IndexRequest createIndexRequest(Row element) {
                Map<String, Object> json = new HashMap<>();
                json.put("name22", element.getField(0).toString());
                json.put("no22", element.getField(1));
                json.put("age", 34);
                json.put("create_time", element.getField(2));

                return Requests.indexRequest()
                        .index("cc")
                        .type("mtype")
                        .id(element.getField(1).toString())
                        .source(json);
            }

            @Override
            public void process(Row element, RuntimeContext ctx, RequestIndexer indexer) {
                //利用requestIndexer进行发送请求,写入数据
                indexer.add(createIndexRequest(element));
            }
        }));
        env.execute("es demo");
    }
    private static java.sql.Timestamp getTimestamp(String str) throws Exception {
//		String string = "2016-10-24 21:59:06";
        SimpleDateFormat sdf = new SimpleDateFormat("yyyy-MM-dd HH:mm:ss");
        java.util.Date date=sdf.parse(str);
        java.sql.Timestamp s = new java.sql.Timestamp(date.getTime());
        return s;
    }

二、写入到Elasticsearch7

  • pom maven依赖
xml 复制代码
        <dependency>
            <groupId>org.apache.flink</groupId>
            <artifactId>flink-connector-elasticsearch7_2.11</artifactId>
            <version>${flink.version}</version>
            <scope>provided</scope>
        </dependency>
  • 代码如下(示例):
java 复制代码
import org.apache.flink.api.common.functions.RuntimeContext;
import org.apache.flink.streaming.api.datastream.DataStreamSource;
import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
import org.apache.flink.streaming.connectors.elasticsearch.ElasticsearchSinkFunction;
import org.apache.flink.streaming.connectors.elasticsearch.RequestIndexer;
import org.apache.flink.streaming.connectors.elasticsearch7.ElasticsearchSink;
import org.apache.flink.types.Row;
import org.apache.http.HttpHost;
import org.elasticsearch.action.index.IndexRequest;
import org.elasticsearch.client.Requests;

import java.text.SimpleDateFormat;
import java.util.ArrayList;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
public class EsSinkDemo {
    public static void main(String[] args) throws Exception {
        StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
        Row row=Row.of("张三","001",getTimestamp("2016-10-24 21:59:06"));
        Row row2=Row.of("张三","002",getTimestamp("2016-10-24 21:50:06"));
        Row row3=Row.of("张三","002",getTimestamp("2016-10-24 21:51:06"));
        Row row4=Row.of("李四","003",getTimestamp("2016-10-24 21:50:56"));
        Row row5=Row.of("李四","004",getTimestamp("2016-10-24 00:48:36"));
        Row row6=Row.of("王五","005",getTimestamp("2016-10-24 00:48:36"));
        DataStreamSource<Row> source =env.fromElements(row,row2,row3,row4,row5,row6);

        Map<String, String> config = new HashMap<>();
//        config.put("cluster.name", "my-cluster-name");
// This instructs the sink to emit after every element, otherwise they would be buffered
//        config.put("bulk.flush.max.actions", "1");

        List<HttpHost> hosts = new ArrayList<>();
        hosts.add(new HttpHost("10.68.8.69",9200,"http"));

        ElasticsearchSink.Builder<Row> esSinkBuilder = new ElasticsearchSink.Builder<Row>(hosts,new ElasticsearchSinkFunction<Row>() {
            public IndexRequest createIndexRequest(Row element) {
                Map<String, Object> json = new HashMap<>();
                json.put("name22", element.getField(0).toString());
                json.put("no22", element.getField(1));
                json.put("age", 34);
//                json.put("create_time", element.getField(2));

                return Requests.indexRequest()
                        .index("cc")
                        .id(element.getField(1).toString())
                        .source(json);
            }

            @Override
            public void process(Row element, RuntimeContext ctx, RequestIndexer indexer) {
                //利用requestIndexer进行发送请求,写入数据
                indexer.add(createIndexRequest(element));
            }
        });
        esSinkBuilder.setBulkFlushMaxActions(100);
        //Sink操作
        source.addSink(esSinkBuilder.build());

        env.execute("es demo");
    }

    private static java.sql.Timestamp getTimestamp(String str) throws Exception {
//		String string = "2016-10-24 21:59:06";
        SimpleDateFormat sdf = new SimpleDateFormat("yyyy-MM-dd HH:mm:ss");
        java.util.Date date=sdf.parse(str);
        java.sql.Timestamp s = new java.sql.Timestamp(date.getTime());
        return s;
    }
}

总结

flink写入es5和es7 的区别是引入不同的flink-connector-elasticsearch,es7已没有type的概念故无需再设置type。

相关推荐
数智顾问1 小时前
破解 Shuffle 阻塞:Spark RDD 宽窄依赖在实时特征工程中的实战与未来
大数据·分布式·spark
Giser探索家1 小时前
遥感卫星升轨 / 降轨技术解析:对图像光照、对比度的影响及工程化应用
大数据·人工智能·算法·安全·计算机视觉·分类
lisw052 小时前
数字化科技简化移民流程的 5 种方式
大数据·人工智能·机器学习
NewCarRen2 小时前
基于自动驾驶仿真软件的交通事故档案建模与分析
大数据·汽车功能安全
JAVA学习通2 小时前
OJ竞赛平台----C端题目列表
java·开发语言·jvm·vue.js·elasticsearch
JAVA学习通3 小时前
Replication(下):事务,一致性与共识
大数据·分布式·算法
vivo互联网技术3 小时前
vivo HDFS EC大规模落地实践
大数据·hdfs·大数据计算与存储·erasure coding
api_180079054603 小时前
异步数据采集实践:用 Python/Node.js 构建高并发淘宝商品 API 调用引擎
大数据·开发语言·数据库·数据挖掘·node.js
科研服务器mike_leeso4 小时前
41 年 7 次转型!戴尔从 PC 到 AI 工厂的技术跃迁与组织重构
大数据·人工智能·机器学习
2501_913981784 小时前
2025年智能家居无线数传设备品牌方案精选
大数据·人工智能·智能家居