elasticsearch备份恢复

备份恢复

备份恢复的总体流程

  1. elasticsearch.yaml中配置的备份路径,拷贝到新的机器上。

    bash 复制代码
    # 备份路径
    path.repo: ["/data/elasticsearch/snapshots/snapshot01","/data/elasticsearch/snapshots/snapshot02"]
  2. 将备份路径通过nfs挂载到新的集群实例的每个节点上

  3. 创建snapshot repo

    • 创建snapshot repo

      bash 复制代码
      esname=elastic
      espassword=xxxxxx
      curl -X PUT -H 'Content-Type: application/json' --user ${esname}:${espassword} -d @snapshot-repo.json  "http://localhost:9200/_snapshot/recover_repo" 
    • snapshot-repo.json

      如下所示, location为备份路径

      bash 复制代码
      {
          "type": "fs",
          "settings": {
              "compress": true,
              "location": "/data/els_backup_new"
          }
      }
  4. 从snapshot repo恢复数据

    bash 复制代码
    esname=elastic
    espassword=xxxxxx
    
    # close duplicated index
    # curl -X POST -H 'Content-Type: application/json' --user ${esname}:${espassword}  "http://localhost:9200/.security-7/_close"
    
    # snapshot的名字
    snapshot=snapshot_2024-11-05_01-00
    
    curl -X POST -H 'Content-Type: application/json' --user ${esname}:${espassword} -d @recover-settings.json  "http://localhost:9200/_snapshot/recover_repo/$snapshot/_restore?wait_for_completion=true" | python -m json.tool

    其中recover-settings.json:

    bash 复制代码
    {
    "indices": "*,-.*"
    }

    snapshot的名称可以通过以下接口获取:

    bash 复制代码
    esname=elastic
    espassword=xxxxx
    curl -X GET -H 'Content-Type: application/json' --user ${esname}:${espassword}  "http://localhost:9200/_snapshot/recover_repo" 
  5. 查看迁移进度

    bash 复制代码
    esname=elastic
    espassword=xxxx
    
    curl -X GET --user ${esname}:${espassword}   "http://localhost:9200/_recovery/" 

    响应输出格式为json,可以通过grep -v '100.0%'来查看未恢复完成的索引。

备份恢复成功输出

bash 复制代码
[portal@es-node1 tmp]$ bash recover-es-from-snapshot.sh 
{"acknowledged":true,"shards_acknowledged":false,"indices":{}}  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 27914  100 27914    0     0     61      0  0:07:37  0:07:30  0:00:07  7721
{
    "snapshot": {
        "indices": [
            // 恢复的索引
        ],
        "shards": {
            "failed": 0,
            "successful": 1714,
            "total": 1714
        },
        "snapshot": "snapshot_2024-11-05_01-00"
    }
}

备份恢复的常见问题:

索引冲突

bash 复制代码
[portal@es-node1 tmp]$ bash recover-es-from-snapshot.sh 
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   743  100   743    0     0    354      0  0:00:02  0:00:02 --:--:--   354
{
    "error": {
        "reason": "[recover_repo:snapshot_2024-11-05_01-00/l_5Y6dxoSvGAhAHH5zjwGQ] cannot restore index [.security-7] because an open index with same name already exists in the cluster. Either close or delete the existing index or restore the index under a different name by providing a rename pattern and replacement name",
        "root_cause": [
            {
                "reason": "[recover_repo:snapshot_2024-11-05_01-00/l_5Y6dxoSvGAhAHH5zjwGQ] cannot restore index [.security-7] because an open index with same name already exists in the cluster. Either close or delete the existing index or restore the index under a different name by providing a rename pattern and replacement name",
                "type": "snapshot_restore_exception"
            }
        ],
        "type": "snapshot_restore_exception"
    },
    "status": 500
}

出现这种情况,需要把索引关闭:

bash 复制代码
esname=elastic
espassword=xxx
# 关闭索引
curl -X POST -H 'Content-Type: application/json' --user ${esname}:${espassword}  "http://localhost:9200/.security-7/_close"

其中.security-7是索引的名称

bash 复制代码
4.11.06-000001] failed on step [{"phase":"hot","action":"rollover","name":"check-rollover-ready"}]. Moving to ERROR step
java.lang.IllegalArgumentException: setting [index.lifecycle.rollover_alias] for index [.ds-.logs-deprecation.elasticsearch-default-2024.11.06-000001] is empty or not defined
	at org.elasticsearch.xpack.core.ilm.WaitForRolloverReadyStep.evaluateCondition(WaitForRolloverReadyStep.java:89) [x-pack-core-7.17.6.jar:7.17.6]
	at org.elasticsearch.xpack.ilm.IndexLifecycleRunner.runPeriodicStep(IndexLifecycleRunner.java:226) [x-pack-ilm-7.17.6.jar:7.17.6]
	at org.elasticsearch.xpack.ilm.IndexLifecycleService.triggerPolicies(IndexLifecycleService.java:418) [x-pack-ilm-7.17.6.jar:7.17.6]
	at org.elasticsearch.xpack.ilm.IndexLifecycleService.triggered(IndexLifecycleService.java:349) [x-pack-ilm-7.17.6.jar:7.17.6]
	at org.elasticsearch.xpack.core.scheduler.SchedulerEngine.notifyListeners(SchedulerEngine.java:186) [x-pack-core-7.17.6.jar:7.17.6]
	at org.elasticsearch.xpack.core.scheduler.SchedulerEngine$ActiveSchedule.run(SchedulerEngine.java:220) [x-pack-core-7.17.6.jar:7.17.6]
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539) [?:?]
	at java.util.concurrent.FutureTask.run(FutureTask.java:264) [?:?]
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) [?:?]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) [?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) [?:?]
	at java.lang.Thread.run(Thread.java:833) [?:?]
[2024-11-06T17:00:44,102][ERROR][o.e.x.i.IndexLifecycleRunner] [node-1] policy [ilm-history-ilm-policy] for index [.ds-ilm-history-5-2024.11.06-000001] failed on step [{"phase":"hot","action":"rollover","name":"check-rollover-ready"}]. Moving to ERROR step
java.lang.IllegalArgumentException: setting [index.lifecycle.rollover_alias] for index [.ds-ilm-history-5-2024.11.06-000001] is empty or not defined
	at org.elasticsearch.xpack.core.ilm.WaitForRolloverReadyStep.evaluateCondition(WaitForRolloverReadyStep.java:89) [x-pack-core-7.17.6.jar:7.17.6]
	at org.elasticsearch.xpack.ilm.IndexLifecycleRunner.runPeriodicStep(IndexLifecycleRunner.java:226) [x-pack-ilm-7.17.6.jar:7.17.6]
	at org.elasticsearch.xpack.ilm.IndexLifecycleService.triggerPolicies(IndexLifecycleService.java:418) [x-pack-ilm-7.17.6.jar:7.17.6]
	at org.elasticsearch.xpack.ilm.IndexLifecycleService.triggered(IndexLifecycleService.java:349) [x-pack-ilm-7.17.6.jar:7.17.6]
	at org.elasticsearch.xpack.core.scheduler.SchedulerEngine.notifyListeners(SchedulerEngine.java:186) [x-pack-core-7.17.6.jar:7.17.6]
	at org.elasticsearch.xpack.core.scheduler.SchedulerEngine$ActiveSchedule.run(SchedulerEngine.java:220) [x-pack-core-7.17.6.jar:7.17.6]
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539) [?:?]
	at java.util.concurrent.FutureTask.run(FutureTask.java:264) [?:?]
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) [?:?]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) [?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) [?:?]
	at java.lang.Thread.run(Thread.java:833) [?:?]
相关推荐
诗旸的技术记录与分享19 分钟前
Flink-1.19.0源码详解6-JobGraph生成-后篇
大数据·flink
IvanCodes2 小时前
Oracle 视图
大数据·数据库·sql·oracle
cal_2 小时前
数据分析中的拉链表解析
大数据·数据挖掘·数据分析
亿信华辰软件2 小时前
亿信华辰赋能大学教务数据分析实战记
大数据·数据分析
qyresearch_3 小时前
全球机械工业设计服务市场:技术驱动下的创新与增长
大数据·人工智能
前端开发与ui设计的老司机4 小时前
大数据时代UI前端的智能化升级:基于机器学习的用户意图预测
大数据·ui
rui锐rui5 小时前
大数据学习6:Sqoop数据迁移工具
大数据·学习·sqoop
小牛头#9 小时前
clickhouse 各个引擎适用的场景
大数据·clickhouse·机器学习