CDC实战：MySQL实时同步数据到Elasticsearch之数组集合(array)如何处理【CDC实战系列十二】

撑伞接落花看那西风骑瘦马2024-03-25 17:58

需求背景： mysql存储的一个字段，需要同步到elasticsearch，并存储为数组，以便于查询。

如下例子，就是查询预期。

bash 复制代码 
PUT /t_test_1/_doc/1
{
  "name":"苹果",
  "code":[1,2,3]
}

PUT /t_test_1/_doc/2
{
  "name":"香蕉",
  "code":[1,2]
}

PUT /t_test_1/_doc/3
{
  "name":"橙子",
  "code":[1,3,5]
}

查询code包含1或者3中任意一个或者所有coe都包含的数据

bash 复制代码 
POST /t_test_1/_search
{
  "query": {
    "bool": {
      "filter": [
        {
          "terms": {
            "code": [
              "1",
              "3"
            ]
          }
        }
      ]
    }
  }
}

数据同步背景介绍：

1、当前数据同步使用的是kafka + confluent

2、各种数据产品中间件版本：

MySQL5.7.x es7.15.1 kafka3.2.0 Debezium MySQL connector plug-in 1.9.4.Final confluentinc-kafka-connect-elasticsearch：13.1.0

详细内容可参考之前的博文

经过调研，找到两种方法实现：

1、第一种使用elasticsearch自带的ingest pipeline处理。话不多少，直接上code。

（1）首先mysql字段num_array设计成varchar，存储格式为'a,b,c,d,e,f'；

（2）同步到es变成数组，es对应的字段num_array设计成keyword，便于精确查询，提高性能。

bash 复制代码 
# 创建一个pipeline
PUT _ingest/pipeline/string_to_array_pipeline
{
  "description": "Transfer the string which is concat with a separtor  to array.",
  "processors": [
    {
      "split": {
        "field": "num_array",
        "separator": ","
      }
    },
    {
      "set": {
        "field": "update_user",
        "value": "system"
      }
    },
    {
      "set": {
        "field": "name",
        "value": "华山"
      }
    }
  ]
}

##t_mountain 设置字段num_array的mapping为keyword，设置default_pipeline =string_to_array_pipeline
PUT /t_mountain
{
  "settings": {
    "default_pipeline": "string_to_array_pipeline"
  }, 
   "mappings" : {
      "date_detection" : false,
      "properties" : {
        "altitude" : {
          "type" : "float"
        },
        "create_time" : {
          "type" : "date",
          "format" : "yyyy-MM-dd HH:mm:ss || strict_date_optional_time || epoch_millis"
        },
        "create_user" : {
          "type" : "text",
          "fields" : {
            "keyword" : {
              "type" : "keyword",
              "ignore_above" : 256
            }
          }
        },
        "desc" : {
          "type" : "text",
          "fields" : {
            "keyword" : {
              "type" : "keyword",
              "ignore_above" : 256
            }
          }
        },
        "id" : {
          "type" : "long"
        },
        "latitude" : {
          "type" : "text",
          "fields" : {
            "keyword" : {
              "type" : "keyword",
              "ignore_above" : 256
            }
          }
        },
        "location" : {
          "type" : "geo_point"
        },
        "logtitude" : {
          "type" : "text",
          "fields" : {
            "keyword" : {
              "type" : "keyword",
              "ignore_above" : 256
            }
          }
        },
        "name" : {
          "type" : "text",
          "fields" : {
            "keyword" : {
              "type" : "keyword",
              "ignore_above" : 256
            }
          }
        },
        "num_array" : {
          "type" : "keyword"
        },
        "ticket" : {
          "type" : "float"
        },
        "update_time" : {
          "type" : "date",
          "format" : "yyyy-MM-dd HH:mm:ss || strict_date_optional_time || epoch_millis"
        },
        "update_user" : {
          "type" : "text",
          "fields" : {
            "keyword" : {
              "type" : "keyword",
              "ignore_above" : 256
            }
          }
        }
      }
    }
}

#查询
POST t_mountain/_search
{
  "query": {
    "bool": {
      "filter": [
        {
          "terms": {
            "num_array": [
              "a",
              "b"
            ]
          }
        }
      ]
    }
  }
}

2、使用debezium自定义转换实现json字符串转数组

详细内容参考gitee：debezium-custom-converter

或参考github：debezium-custom-converter

简单来说就是mysql存储成varchar格式：比如 'a,b,c,d,e' 或者 '["a","b","c","d"]' 经过自定义converter处理转成list,同步到es就是数组格式，es的所有设置和方法1相同。

------------------------数据同步过程中出现了一个很让人费解的现象---------------------

1、数据同步过程中出现了一个很让人费解的现象：mysql的insert语句可以正常同步的es,且pipeline能正常应用，字段被处理成了数组格式。但是，在mysql中update刚才插入的数据，则数据正常同步到了es,但是也只的default_pipeline没有生效。
2、排查发现是因为同步sink脚本中设置了"write.method":"upsert"导致mysql执行update语句时，目标索引设置的pipeline失效了。

现象令人费解！！！

3、查看官方文档，没有看出来这个解释，怎么就影响到索引设置的pipeline了

Method used for writing data to Elasticsearch, and one of INSERT or UPSERT. The default method is INSERT in which the connector constructs a document from the record value and inserts that document into Elasticsearch, completely replacing any existing document with the same ID; this matches previous behavior. The UPSERT method will create a new document if one with the specified ID does not yet exist, or will update an existing document with the same ID by adding or replacing only those fields present in the record value. The UPSERT method may require additional time and resources of Elasticsearch, so consider increasing the read.timeout.ms and decreasing the batch.size configuration properties.

用于向 Elasticsearch 中写入数据的方法，以及 INSERT 或 UPSERT 中的一种。默认方法是 INSERT，连接器会根据记录值构建一个文档，并将该文档插入 Elasticsearch，同时完全替换具有相同 ID 的现有文档；这与以前的行为一致。如果指定 ID 的文档还不存在，UPSERT 方法将创建一个新文档，或更新具有相同 ID 的现有文档，只添加或替换记录值中存在的字段。UPSERT 方法可能需要额外的时间和 Elasticsearch 资源，因此请考虑增加 read.timeout.ms 和减少 batch.size 配置属性。

4、遂在本地模拟同步数据，复现了上述现象。然后打开es详细日志追踪：
ini 复制代码 
 bin/elasticsearch -E logger.org.elasticsearch.action=trace

先贴出来同步脚本:

source脚本

perl 复制代码 
{
    "name": "goods-connector",
    "config": {
        "connector.class": "io.debezium.connector.mysql.MySqlConnector",
        "database.hostname": "127.0.0.1",
        "database.port": "3306",
        "database.user": "debezium_user",
        "database.password": "@debezium2022",
        "database.server.id": "12358",
        "snapshot.mode": "when_needed",
        "database.server.name": "goods",
        "database.include.list": "goods",
        "table.include.list": "goods.t_mountain,goods.t_sku,goods.t_spu",
        "database.history.kafka.bootstrap.servers": "127.0.0.1:9092",
        "database.history.kafka.topic": "dbhistory.goods",
        "include.schema.changes": "true"
    }
}

sink脚本：

bash 复制代码 
{
  "name": "elasticsearch-sink",
  "config": {
    "connector.class": "io.confluent.connect.elasticsearch.ElasticsearchSinkConnector",
    "tasks.max": "1",
    "topics": "goods.goods.t_sku,goods.goods.t_spu,goods.goods.t_mountain,goods001.goods.t_format_date",
    "key.ignore": "false",
    "connection.url": "http://127.0.0.1:9200",
    "name": "elasticsearch-sink",
    "type.name": "_doc",
    "decimal.handling.mode": "string",
    "transforms": "unwrap,key",
    "transforms.unwrap.type": "io.debezium.transforms.ExtractNewRecordState",
    "transforms.unwrap.drop.tombstones": "true",
    "transforms.unwrap.delete.handling.mode": "drop",
    "transforms.key.type": "org.apache.kafka.connect.transforms.ExtractField$Key",
    "transforms.key.field": "id"
  }
}

可以发现，此时的sink脚本未设置 "write.method":"upsert"，那么该设置的默认值为insert

日志如下：

mysql insert语句，同步到es日志，操作是 index （下面日志中我也做了标记），这是已经经过索引设置的pipeline处理后的数据。

[2024-03-25T01:22:16,232][TRACE][o.e.a.b.TransportShardBulkAction] [esserver001-9200] send action [indices:data/write/bulk[s][p]] to local primary [[goods.goods.t_mountain][0]] for request [BulkShardRequest [[goods.goods.t_mountain][0]] containing [index {[goods.goods.t_mountain][_doc][154], source[{"altitude":1200.0,"num_array":["aaaaaa","b","ccccccc"],"create_time":1710419563000,"ticket":0.0,"latitude":"34.497647","logtitude":"110.073028","update_time":1710420074000,"update_user":"system","name":"华山","location":"34.497647,110.073028","id":154,"create_user":"111111","desc":"少华山在陕西渭南华州区"}]}]] with cluster state version [2312] to [NYz8ptioSBGMQoSB94VGew] [2024-03-25T01:22:16,233][TRACE][o.e.a.b.TransportShardBulkAction] [esserver001-9200] [[goods.goods.t_mountain][0]] op [indices:data/write/bulk[s]] completed on primary for request [BulkShardRequest [[goods.goods.t_mountain][0]] containing [index {[goods.goods.t_mountain][_doc][154], source[{"altitude":1200.0,"num_array":["aaaaaa","b","ccccccc"],"create_time":1710419563000,"ticket":0.0,"latitude":"34.497647","logtitude":"110.073028","update_time":1710420074000,"update_user":"system","name":"华山","location":"34.497647,110.073028","id":154,"create_user":"111111","desc":"少华山在陕西渭南华州区"}]}]] [2024-03-25T01:22:16,235][TRACE][o.e.a.b.TransportShardBulkAction] [esserver001-9200] operation succeeded. action [indices:data/write/bulk[s]],request [BulkShardRequest [[goods.goods.t_mountain][0]] containing [index {[goods.goods.t_mountain][_doc][154], source[{"altitude":1200.0,"num_array":["aaaaaa","b","ccccccc"],"create_time":1710419563000,"ticket":0.0,"latitude":"34.497647","logtitude":"110.073028","update_time":1710420074000,"update_user":"system","name":"华山","location":"34.497647,110.073028","id":154,"create_user":"111111","desc":"少华山在陕西渭南华州区"}]}]]

查看elasticsearch源码，从IndexRequest.java的toString方法可以看出，日志输出的是哪些内容。

从以上日志可以模拟出es的插入DSL，这是已经经过索引设置的pipeline处理后的数据。

json 复制代码 
POST _bulk
{"index":{"_id":"154","_index":"t_mountain"}}
{"altitude":1200.0,"num_array":["aaaaaa","b","ccccccc"],"create_time":1710419563000,"ticket":0.0,"latitude":"34.497647","logtitude":"110.073028","update_time":1710420074000,"update_user":"system","name":"华山","location":"34.497647,110.073028","id":154,"create_user":"111111","desc":"少华山在陕西渭南华州区"}

mysql update语句，同步到es日志，操作是 index （下面日志中我也做了标记），这是已经经过索引设置的pipeline处理后的数据。

[2024-03-25T01:23:18,419][TRACE][o.e.a.b.TransportShardBulkAction] [esserver001-9200] send action [indices:data/write/bulk[s][p]] to local primary [[goods.goods.t_mountain][0]] for request [BulkShardRequest [[goods.goods.t_mountain][0]] containing [index {[goods.goods.t_mountain][_doc][154], source[{"altitude":1200.0,"num_array":["aaaaaa","b","ccccccc"],"create_time":1710419563000,"ticket":0.0,"latitude":"34.497647","logtitude":"110.073028","update_time":1711329798000,"update_user":"system","name":"华山","location":"34.497647,110.073028","id":154,"create_user":"111111","desc":"少华山在陕西渭南华州区"}]}]] with cluster state version [2312] to [NYz8ptioSBGMQoSB94VGew] [2024-03-25T01:23:18,421][TRACE][o.e.a.b.TransportShardBulkAction] [esserver001-9200] [[goods.goods.t_mountain][0]] op [indices:data/write/bulk[s]] completed on primary for request [BulkShardRequest [[goods.goods.t_mountain][0]] containing [index {[goods.goods.t_mountain][_doc][154], source[{"altitude":1200.0,"num_array":["aaaaaa","b","ccccccc"],"create_time":1710419563000,"ticket":0.0,"latitude":"34.497647","logtitude":"110.073028","update_time":1711329798000,"update_user":"system","name":"华山","location":"34.497647,110.073028","id":154,"create_user":"111111","desc":"少华山在陕西渭南华州区"}]}]] [2024-03-25T01:23:18,422][TRACE][o.e.a.b.TransportShardBulkAction] [esserver001-9200] operation succeeded. action [indices:data/write/bulk[s]],request [BulkShardRequest [[goods.goods.t_mountain][0]] containing [index {[goods.goods.t_mountain][_doc][154], source[{"altitude":1200.0,"num_array":["aaaaaa","b","ccccccc"],"create_time":1710419563000,"ticket":0.0,"latitude":"34.497647","logtitude":"110.073028","update_time":1711329798000,"update_user":"system","name":"华山","location":"34.497647,110.073028","id":154,"create_user":"111111","desc":"少华山在陕西渭南华州区"}]}]]

从以上日志可以模拟出es的插入DSL，和上面的类似。都是插入数据。

json 复制代码 
POST _bulk
{"index":{"_id":"154","_index":"t_mountain"}}
{"altitude":1200.0,"num_array":["aaaaaa","b","ccccccc"],"create_time":1710419563000,"ticket":0.0,"latitude":"34.497647","logtitude":"110.073028","update_time":1710420074000,"update_user":"system","name":"华山","location":"34.497647,110.073028","id":154,"create_user":"111111","desc":"少华山在陕西渭南华州区"}
5、修改sink脚本，添加一行配置 "write.method":"upsert"，重启kafka-connect，继续数据同步

日志如下：

5.1、mysql insert语句，同步到es日志，操作是update（下面日志中我也做了标记），可以看到数据分别存储在doc对象（未经索引设置的pipeline处理的原始数据）和upsert对象（经过索引设置的pipeline处理的原始数据）中：

[2024-03-22T05:44:42,500][TRACE][o.e.a.b.TransportShardBulkAction] [esserver001-9200] send action [indices:data/write/bulk[s][p]] to local primary [[goods.goods.t_mountain][0]] for request [BulkShardRequest [[goods.goods.t_mountain][0]] containing [update {[goods.goods.t_mountain][_doc][151], doc_as_upsert[false], doc[index {[null][_doc][null], source[{"id":151,"name":"少华山","location":"34.497647,110.073028","latitude":"34.497647","logtitude":"110.073028","altitude":1200.0,"create_user":"111111","create_time":1710419563000,"update_user":null,"update_time":1710420074000,"ticket":0.0,"desc":"少华山在陕西渭南华州区","num_array":"aaaaaa,b,ccccccc"}]}], upsert[index {[null][_doc][null], source[{"altitude":1200.0,"num_array":["aaaaaa","b","ccccccc"],"create_time":1710419563000,"ticket":0.0,"latitude":"34.497647","logtitude":"110.073028","update_time":1710420074000,"update_user":"system","name":"华山","location":"34.497647,110.073028","id":151,"create_user":"111111","desc":"少华山在陕西渭南华州区"}]}], scripted_upsert[false], detect_noop[true]}]] with cluster state version [1858] to [NYz8ptioSBGMQoSB94VGew] [2024-03-22T05:44:42,498][TRACE][o.e.a.b.TransportShardBulkAction] [esserver001-9200] [[goods.goods.t_mountain][0]] op [indices:data/write/bulk[s]] completed on primary for request [BulkShardRequest [[goods.goods.t_mountain][0]] containing [index {[goods.goods.t_mountain][_doc][151], source[{"altitude":1200.0,"num_array":["aaaaaa","b","ccccccc"],"create_time":1710419563000,"ticket":0.0,"latitude":"34.497647","logtitude":"110.073028","update_time":1710420074000,"update_user":"system","name":"华山","location":"34.497647,110.073028","id":151,"create_user":"111111","desc":"少华山在陕西渭南华州区"}]}]] [2024-03-22T05:44:42,500][TRACE][o.e.a.b.TransportShardBulkAction] [esserver001-9200] operation succeeded. action [indices:data/write/bulk[s]],request [BulkShardRequest [[goods.goods.t_mountain][0]] containing [index {[goods.goods.t_mountain][_doc][151], source[{"altitude":1200.0,"num_array":["aaaaaa","b","ccccccc"],"create_time":1710419563000,"ticket":0.0,"latitude":"34.497647","logtitude":"110.073028","update_time":1710420074000,"update_user":"system","name":"华山","location":"34.497647,110.073028","id":151,"create_user":"111111","desc":"少华山在陕西渭南华州区"}]}]]

从elasticsearch源码的 UpdateRequest.java类的toString方法中，可以看出日志输出的是哪些内容：

从以上日志可以模拟出es的插入DSL，可以正常同步数据，且经过索引设置的pipeline处理过的数据，正确同步到了es中。

----下面upsert中内容正常保存到了es中，符合预期。

json 复制代码 
POST _bulk
{"update":{"_id":"151","_index":"t_mountain"}}
{"doc":{"id":151,"name":"少华山","location":"34.497647,110.073028","latitude":"34.497647","logtitude":"110.073028","altitude":1200.0,"create_user":"111111","create_time":1710419563000,"update_user":null,"update_time":1710420074000,"ticket":0.0,"desc":"少华山在陕西渭南华州区","num_array":"aaaaaa,b,ccccccc"},"upsert":{"altitude":1200.0,"num_array":["aaaaaa","b","ccccccc"],"create_time":1710419563000,"ticket":0.0,"latitude":"34.497647","logtitude":"110.073028","update_time":1710420074000,"update_user":"system","name":"华山","location":"34.497647,110.073028","id":151,"create_user":"111111","desc":"少华山在陕西渭南华州区"},"doc_as_upsert":false,"scripted_upsert":false, "detect_noop":true}
5.2、mysql update语句，同步到es日志，操作是update（下面日志中我也做了标记），可以看到数据分别存储在doc对象（未经索引设置的pipeline处理的原始数据）和upsert对象（经过索引设置的pipeline处理的原始数据）中：

[2024-03-22T05:49:56,606][TRACE][o.e.a.b.TransportShardBulkAction] [esserver001-9200] send action [indices:data/write/bulk[s][p]] to local primary [[goods.goods.t_mountain][0]] for request [BulkShardRequest [[goods.goods.t_mountain][0]] containing [update {[goods.goods.t_mountain][_doc][151], doc_as_upsert[false], doc[index {[null][_doc][null], source[{"id":151,"name":"泰山","location":"34.497647,110.073028","latitude":"34.497647","logtitude":"110.073028","altitude":1200.0,"create_user":"111111","create_time":1710419563000,"update_user":"666666","update_time":1711086596000,"ticket":0.0,"desc":"少华山在陕西渭南华州区","num_array":"a,b,c,d,e,f,g,h"}]}], upsert[index {[null][_doc][null], source[{"altitude":1200.0,"num_array":["a","b","c","d","e","f","g","h"],"create_time":1710419563000,"ticket":0.0,"latitude":"34.497647","logtitude":"110.073028","update_time":1711086596000,"update_user":"system","name":"华山","location":"34.497647,110.073028","id":151,"create_user":"111111","desc":"少华山在陕西渭南华州区"}]}], scripted_upsert[false], detect_noop[true]}]] with cluster state version [1927] to [NYz8ptioSBGMQoSB94VGew] [2024-03-22T05:49:56,610][TRACE][o.e.a.b.TransportShardBulkAction] [esserver001-9200] [[goods.goods.t_mountain][0]] op [indices:data/write/bulk[s]] completed on primary for request [BulkShardRequest [[goods.goods.t_mountain][0]] containing [index {[goods.goods.t_mountain][_doc][151], source[{"altitude":1200.0,"num_array":"a,b,c,d,e,f,g,h","create_time":1710419563000,"ticket":0.0,"latitude":"34.497647","logtitude":"110.073028","update_time":1711086596000,"update_user":"666666","name":"泰山","location":"34.497647,110.073028","id":151,"create_user":"111111","desc":"少华山在陕西渭南华州区"}]}]] [2024-03-22T05:49:56,613][TRACE][o.e.a.b.TransportShardBulkAction] [esserver001-9200] operation succeeded. action [indices:data/write/bulk[s]],request [BulkShardRequest [[goods.goods.t_mountain][0]] containing [index {[goods.goods.t_mountain][_doc][151], source[{"altitude":1200.0,"num_array":"a,b,c,d,e,f,g,h","create_time":1710419563000,"ticket":0.0,"latitude":"34.497647","logtitude":"110.073028","update_time":1711086596000,"update_user":"666666","name":"泰山","location":"34.497647,110.073028","id":151,"create_user":"111111","desc":"少华山在陕西渭南华州区"}]}]]

从以上日志可以模拟出es的插入DSL:

----下面doc中内容正常保存到了es中，upsert中的内容未更新到es中，问题就出在这里！！！！！！

json 复制代码 
POST _bulk
{"update":{"_id":"151","_index":"t_mountain"}}
{"doc":{"id":151,"name":"泰山","location":"34.497647,110.073028","latitude":"34.497647","logtitude":"110.073028","altitude":1200.0,"create_user":"111111","create_time":1710419563000,"update_user":"666666","update_time":1711086596000,"ticket":0.0,"desc":"少华山在陕西渭南华州区","num_array":"a,b,c,d,e,f,g,h"},"upsert":{"altitude":1200.0,"num_array":["a","b","c","d","e","f","g","h"],"create_time":1710419563000,"ticket":0.0,"latitude":"34.497647","logtitude":"110.073028","update_time":1711086596000,"update_user":"system","name":"华山","location":"34.497647,110.073028","id":151,"create_user":"111111","desc":"少华山在陕西渭南华州区"},"doc_as_upsert":false,"scripted_upsert":false, "detect_noop":true}

elasticsearch更新，doc里面的内容是部分更新，设置了几个字段就更新几个字段，不会覆盖doc未设置的字段。doc和upsert都存在的情况下，如果指定更新的文档不存在，则"If the document does not already exist, the contents of the upsert element are inserted as a new document."，所以本例中5.1同步数据符合预期。如果指定的文档存在，doc和upsert都存在的情况下，则doc里面的内容更新到es中，所以本例中5.2同步数据不符合预期，upsert中经过索引设置的pipeline处理过的数据为正确同步更新到es中。这就是bug出现的原因。

总结：

1、合理设置参数，"write.method":"upsert" 、"write.method":"insert"

设置了参数"write.method":"upsert"。mysql的update语句执行到es就是走的bulk部分更新，就会导致索引设置的pipeline失效，官方文档没看到特殊说明，这一块描述不是很清晰，但仔细查看，能略显端倪。

confluent这个设置看github上说的是为了避免数据被覆盖。所以部分更新的情况就设置为upsert。如果同步的时候不管是新增还是更新，source读取的数据都是包含完整字段的，就不存在数据被覆盖的问题，就可以设置成insert。

因为是通过debezium+kafka cnnect+confluent同步数据，无法设置elasticsearch批量更新的参数，因此在设置上述参数时要慎重，通常mysql同步数据到es设置成insert就可以了，不存在更新数据时旧数据被覆盖的情况。

如果不能修改sink脚本配置，那么可以创建一个单独的sink脚本只处理这一张设置了指定pipeline的索引，同时配合es索引设置的alias，亦可轻松解决问题。

2、如果是直接操作elasticsearch的更新，可以直接设置doc或者，upsert等对象。结合"doc_as_upsert"参数也可以完成数据的正确更新。

bash 复制代码 
# 创建一个pipeline
PUT _ingest/pipeline/string_to_array_pipeline
{
  "description": "Transfer the string which is concat with a separtor  to array.",
  "processors": [
    {
      "split": {
        "field": "num_array",
        "separator": ","
      }
    },
    {
      "set": {
        "field": "update_user",
        "value": "system"
      }
    },
    {
      "set": {
        "field": "name",
        "value": "华山"
      }
    }
  ]
}


##t_mountain
PUT /t_mountain
{
  "settings": {
    "default_pipeline": "string_to_array_pipeline"
  }, 
   "mappings" : {
      "date_detection" : false,
      "properties" : {
        "altitude" : {
          "type" : "float"
        },
        "create_time" : {
          "type" : "date",
          "format" : "yyyy-MM-dd HH:mm:ss || strict_date_optional_time || epoch_millis"
        },
        "create_user" : {
          "type" : "text",
          "fields" : {
            "keyword" : {
              "type" : "keyword",
              "ignore_above" : 256
            }
          }
        },
        "desc" : {
          "type" : "text",
          "fields" : {
            "keyword" : {
              "type" : "keyword",
              "ignore_above" : 256
            }
          }
        },
        "id" : {
          "type" : "long"
        },
        "latitude" : {
          "type" : "text",
          "fields" : {
            "keyword" : {
              "type" : "keyword",
              "ignore_above" : 256
            }
          }
        },
        "location" : {
          "type" : "geo_point"
        },
        "logtitude" : {
          "type" : "text",
          "fields" : {
            "keyword" : {
              "type" : "keyword",
              "ignore_above" : 256
            }
          }
        },
        "name" : {
          "type" : "text",
          "fields" : {
            "keyword" : {
              "type" : "keyword",
              "ignore_above" : 256
            }
          }
        },
        "num_array" : {
          "type" : "keyword"
        },
        "ticket" : {
          "type" : "float"
        },
        "update_time" : {
          "type" : "date",
          "format" : "yyyy-MM-dd HH:mm:ss || strict_date_optional_time || epoch_millis"
        },
        "update_user" : {
          "type" : "text",
          "fields" : {
            "keyword" : {
              "type" : "keyword",
              "ignore_above" : 256
            }
          }
        }
      }
    }
}
#  #A 部分buck更新t_mountain
POST _bulk
{"update":{"_id":"155","_index":"t_mountain"}}
{"doc":{"id":155,"name":"泰山4","location":"34.497647,110.073028","latitude":"34.497647","logtitude":"110.073028","altitude":1200,"create_user":"111111","create_time":1710419563000,"update_user":"777777","update_time":1711086596000,"ticket":0,"desc":"少华山在陕西渭南华州区","num_array":"a,b,c,d"}}
#  #B  buck插入新数据t_mountain
POST _bulk
{"index":{"_id":"155","_index":"t_mountain"}}
{"id":155,"name":"泰山3","location":"34.497647,110.073028","latitude":"34.497647","logtitude":"110.073028","altitude":1200,"create_user":"111111","create_time":1710419563000,"update_user":"777777","update_time":1711086596000,"ticket":0,"desc":"少华山在陕西渭南华州区","num_array":"a,b,c,d"}


GET t_mountain/_search?version=true
{
  "query": {
    "term": {
      "id": {
        "value": "155"
      }
    }
  }
}

--上面例子说明：

1、先执行#B，正常写入，索引设置的pipeline生效，再执行#A，正常更新，但是索引设置的pipeline没有生效。（完美复现上面问题）
2、先执行#A，此时报错： "reason" : "[_doc][155]: document missing"，因为默认的"doc_as_upsert":false设置是false，将这个修改为true，正常写入数据。且索引设置的pipeline正常执行。

3、先执行#B，正常写入，索引设置的pipeline生效，再执行#A，此时设置"doc_as_upsert":true，正常更新，索引设置的pipeline正常执行。

----在这个请求中，"doc_as_upsert":true指示Elasticsearch，如果指定的文档不存在，就将doc部分的内容作为新文档插入。这样，你就不会因为文档不存在而收到错误，而是会创建一个新的文档。

好了，"doc_as_upsert":true这个参数就可以帮助我们灵活处理elasticsearch更新相关的需求.

参考文档： elasticsearch官方文档update elasticsearch官方文档bulk kafka-connect-elasticsearch confluent文档

相关推荐
未 顾7 小时前
day12:版本控制器
大数据·elasticsearch·搜索引擎
Elastic 中国社区官方博客13 小时前
Lucene 和 Elasticsearch 中更好的二进制量化 (BBQ)
大数据·人工智能·elasticsearch·搜索引擎·ai·全文检索·lucene
报名搜谷安1 天前
CCSK：面试云计算岗的高频问题
elasticsearch·flume·memcached
CoderJia程序员甲2 天前
重学SpringBoot3-整合 Elasticsearch 8.x (三)使用Repository
java·大数据·spring boot·elasticsearch
东方巴黎～Sunsiny2 天前
如何优化Elasticsearch的查询性能？
大数据·elasticsearch·搜索引擎
NoneCoder2 天前
命令行工具进阶指南
大数据·elasticsearch·搜索引擎
许苑向上2 天前
【Elasticsearch】Elasticsearch集成Spring Boot
spring boot·elasticsearch·jenkins
东方巴黎～Sunsiny2 天前
Elasticsearch中什么是倒排索引？
大数据·elasticsearch·jenkins
qq_356408662 天前
es 数据清理delete_by_query
elasticsearch
Elastic 中国社区官方博客2 天前
AutoOps 使每个 Elasticsearch 部署都更易于管理
大数据·人工智能·elasticsearch·搜索引擎·全文检索·devops
热门推荐
01软件工程从理论到实践客观题汇总(头歌第一章至第八章)02【HarmonyOS】HUAWEI DevEco Studio 下载地址汇总03组基轨迹建模 GBTM的介绍与实现（Stata 或 R）04VSCode插件 —— Cody AI （免费AI助手！）05全面解析：构建基于深度学习的安全帽检测系统（UI界面+YOLO代码+数据集）06Windows10安装PCL1.14.0及点云配准07Ubuntu 20.04使用Livox mid 360 测试 FAST_LIO08文件或文件夹名称中有空格如何批量去除09基于YOLOv10深度学习的CT扫描图像肾结石智能检测系统【python源码+Pyqt5界面+数据集+训练代码】深度学习实战、目标检测10【QuikGraph】C#调用第三方库实现迪杰斯特拉(Dijkstra)算法功能