ElasticSearch 高可用加密方案笔记

响应银行客户的内部安全要求:

  1. 所有数据库或基础服务,不得采用匿名方式访问。
  2. 所有数据库或基础服务,必须采用高可用方案。

于是有了本文。源代码

单节点实例

Shell 复制代码
docker run -d \
  --name es \
  --restart=always \
  --env "node.name=es" \
  --env "ELASTIC_PASSWORD=example" \
  --env discovery.type=single-node \
  --env "xpack.security.enabled=true" \
  --env "xpack.security.transport.ssl.enabled=true" \
  --env "xpack.license.self_generated.type=basic" \
  --publish 9200:9200 \
  --volume /hdd/es/data:/usr/share/elasticsearch/data \
  --volume /hdd/es/plugins:/usr/share/elasticsearch/plugins \
  --network "bank" \
  elasticsearch:7.17.6

默认的用户名为elastic,调用方式:

Shell 复制代码
 curl -X GET --user elastic:example 'http://localhost:9200/_cat/indices?v'
 curl -X GET "localhost:9200/_cat/nodes?v=true&pretty"

上述命令,环境变量为最小配置。需要开启用户名和密码的验证xpack.security.enabled=true,开启安全后,transport.ssl.enabled必须也一并开启。xpack开启后,则需要指定license,basic为社区版本,trial为体验版,只有30天有效期。[1]

多节点实例

根据[2]可以直接实现集群设置,但是设置密码验证就复杂,主要:

  1. 需要密码验证,xpack的安全功能xpack.security.enabled=true
  2. 安全功能,必须开启传输SSLxpack.security.transport.ssl.enabled=true。到这里,单节点还好,因为没有节点之间的通讯,如果是集群,那么节点之间产生的通讯就会因此产生错误(如下),导致集群内的节点无法通信,所以必须开启xpack.security.http.ssl.enabled=true
vbnet 复制代码
ERROR: [1] bootstrap checks failed
[1]: Transport SSL must be enabled if security is enabled on a [basic] license. Please set [xpack.security.transport.ssl.enabled] to [true] or disable security by setting [xpack.security.enabled] to [false]
ERROR: Elasticsearch did not exit normally - check the logs at /usr/share/elasticsearch/logs/es-cluster.log
  1. 开启了(2),没有证书是不可以的,必须要为集群配置证书

正确的配置步骤应该是

  1. elasticsearch-certutil工具生成证书
  2. 配置所有节点引用生成的证书
  3. 配置健康验证命令

生成证书

工具可以批量生成节点证书(包括kibana节点),提前做好:

  1. 实例配置
  2. 环境变量配置(方便修改版本和密码等)
yaml 复制代码
# instance.yml
instances:
  - name: es01
    dns:
      - es01
      - localhost
    ip:
      - 127.0.0.1

  - name: es02
    dns:
      - es02
      - localhost
    ip:
      - 127.0.0.1

  - name: es03
    dns:
      - es03
      - localhost
    ip:
      - 127.0.0.1
# 根据需要添加
  - name: "kib01"
    dns:
      - kib01
      - localhost

.env 环境变量文件

dotenv 复制代码
COMPOSE_PROJECT_NAME=es-cluster
CERTS_DIR=/usr/share/elasticsearch/config/certificates
VERSION=7.9.3
ELASTIC_PASSWORD=example

先用docker命令启动一个es容器,用来生成证书之用,命令(注意与后面的集群在同一个目录中):

Shell 复制代码
docker run -it --rm \
  -v ${PWD}:/usr/share/elasticsearch/config/certificates \
  -v ${PWD}/certs:/certs \
  --workdir /usr/share/elasticsearch \
  --name es-cert elasticsearch:7.9.3 \
  /bin/bash -c " \
    bin/elasticsearch-certutil cert --silent --pem --in config/certificates/instances.yml --out /certs/bundle.zip; \
    unzip /certs/bundle.zip -d /certs;
    rm /certs/bundle.zip;" -

这条命令的意图是启动es容器,执行es的证书工具,以上述instances.yml文件为集群配置,省去了交互过程。根据上面的配置生成证书集bundle.zip,再将其解压到certs文件夹中,完成上述动作后销毁容器。

启动集群

docker-compose.yml文件如下

yaml 复制代码
x-elastic-search: &elastic-search
  image: docker.elastic.co/elasticsearch/elasticsearch:7.9.3
  restart: always
  healthcheck:
    test: curl -k -u elastic:$ELASTIC_PASSWORD -silent --fail "https://localhost:9200/_cluster/health" || exit 1
  networks:
    - es-network
  ulimits:
    memlock:
      soft: -1
      hard: -1

services:
  es01:
    <<: *elastic-search
    container_name: es01
    hostname: es01
    environment:
      - node.name=es01
      - cluster.name=es-cluster
      - bootstrap.memory_lock=true
      - discovery.seed_hosts=es02,es03
      - cluster.initial_master_nodes=es01,es02,es03
      - "ES_JAVA_OPTS=-Xms1g -Xmx1g"
      - ELASTIC_PASSWORD=$ELASTIC_PASSWORD
      - xpack.license.self_generated.type=basic
      - xpack.security.enabled=true
      - xpack.security.http.ssl.enabled=true
      - xpack.security.http.ssl.key=$CERTS_DIR/es01/es01.key
      - xpack.security.http.ssl.certificate_authorities=$CERTS_DIR/ca/ca.crt
      - xpack.security.http.ssl.certificate=$CERTS_DIR/es01/es01.crt
      - xpack.security.transport.ssl.enabled=true
      - xpack.security.transport.ssl.verification_mode=certificate
      - xpack.security.transport.ssl.certificate_authorities=$CERTS_DIR/ca/ca.crt
      - xpack.security.transport.ssl.certificate=$CERTS_DIR/es01/es01.crt
      - xpack.security.transport.ssl.key=$CERTS_DIR/es01/es01.key
    ports:
      - 9200:9200
    volumes:
      - ${PWD}/es-db1/:/usr/share/elasticsearch/data
      - ${PWD}/certs:$CERTS_DIR
  es02:
    <<: *elastic-search
    container_name: es02
    hostname: es02
    environment:
      - node.name=es02
      - cluster.name=es-cluster
      - bootstrap.memory_lock=true
      - discovery.seed_hosts=es01,es03
      - cluster.initial_master_nodes=es01,es02,es03
      - "ES_JAVA_OPTS=-Xms1g -Xmx1g"
      - ELASTIC_PASSWORD=$ELASTIC_PASSWORD
      - xpack.license.self_generated.type=basic
      - xpack.security.enabled=true
      - xpack.security.http.ssl.enabled=true
      - xpack.security.http.ssl.key=$CERTS_DIR/es02/es02.key
      - xpack.security.http.ssl.certificate_authorities=$CERTS_DIR/ca/ca.crt
      - xpack.security.http.ssl.certificate=$CERTS_DIR/es02/es02.crt
      - xpack.security.transport.ssl.enabled=true
      - xpack.security.transport.ssl.verification_mode=certificate
      - xpack.security.transport.ssl.certificate_authorities=$CERTS_DIR/ca/ca.crt
      - xpack.security.transport.ssl.certificate=$CERTS_DIR/es02/es02.crt
      - xpack.security.transport.ssl.key=$CERTS_DIR/es02/es02.key
    volumes:
      - ${PWD}/es-db2/:/usr/share/elasticsearch/data
      - ${PWD}/certs:$CERTS_DIR
  es03:
    <<: *elastic-search
    container_name: es03
    hostname: es03
    environment:
      - node.name=es03
      - cluster.name=es-cluster
      - bootstrap.memory_lock=true
      - discovery.seed_hosts=es01,es02
      - cluster.initial_master_nodes=es01,es02,es03
      - "ES_JAVA_OPTS=-Xms1g -Xmx1g"
      - ELASTIC_PASSWORD=$ELASTIC_PASSWORD
      - xpack.license.self_generated.type=basic
      - xpack.security.enabled=true
      - xpack.security.http.ssl.enabled=true
      - xpack.security.http.ssl.key=$CERTS_DIR/es03/es03.key
      - xpack.security.http.ssl.certificate_authorities=$CERTS_DIR/ca/ca.crt
      - xpack.security.http.ssl.certificate=$CERTS_DIR/es03/es03.crt
      - xpack.security.transport.ssl.enabled=true
      - xpack.security.transport.ssl.verification_mode=certificate
      - xpack.security.transport.ssl.certificate_authorities=$CERTS_DIR/ca/ca.crt
      - xpack.security.transport.ssl.certificate=$CERTS_DIR/es03/es03.crt
      - xpack.security.transport.ssl.key=$CERTS_DIR/es03/es03.key
    volumes:
      - ${PWD}/es-db3/:/usr/share/elasticsearch/data
      - ${PWD}/certs:$CERTS_DIR

networks:
  es-network:
    name: es-network
    driver: bridge

以上http.ssltransport.ssl分别是客户端访问ssl和节点之间通信的ssl配置。每个节点有各自的证书配置路径(命令创建的)。如果上述的命令和过程没有出现错误状况,那么到这已经实现通过用户名和密码访问集群的功能。但是有些许缺憾:由于是自己生成的证书,不被认可;无论用curl还是浏览器直接访问,必须忽略验证问题。

添加证书

打开先前生成证书的目录:

  1. crt结尾的文件,会打开系统证书管理工具;
  2. 把ca、es01、es02....所访需要反问的证书添加到系统内;
  3. 将证书全部设置为信任;

再次打开网页后,就不会有证书问题。

反向代理

以上的方案虽然可以解决证书问题,但如果客户端情况比较复杂不好控制,或者有很多个客户端,没有企业证书,就会显得很麻烦。那么就在集群之前设置一个反向代理来解决这个问题,顺便把https转换为http。配置nginx.conf如下:

nginx.conf 复制代码
    upstream es {
      server es01:9200;
      server es02:9200;
      server es03:9200;
    }

    server {
        listen       80;
        listen  [::]:80;
        server_name  localhost;

        ignore_invalid_headers off;

        client_max_body_size 0;

        proxy_buffering off;
        proxy_request_buffering off;

        location / {
            # root   /usr/share/nginx/html;
            # index  index.html index.htm;
            proxy_pass https://es;
            proxy_set_header Host $http_host;
            proxy_set_header X-Real-IP $remote_addr;
            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
            proxy_set_header X-Forwarded-Proto $scheme;
            proxy_set_header Authorization $http_authorization;
            # proxy_set_header Authorization "elastic example";

            # proxy_ssl_trusted_certificate /etc/nginx/certs/es01/es01.crt;
            proxy_ssl_verify off;
        }

        #error_page  404              /404.html;

        # redirect server error pages to the static page /50x.html
        #
        error_page   500 502 503 504  /50x.html;
        location = /50x.html {
            root   /usr/share/nginx/html;
        }
    }

将nginx和elasticsearch放在一个网络,即可解决证书问题的同时,顺手把负载均衡也一并做好了。参考[5],第一次启动将create-cert取消注释,自动生成证书,后续再用可以将其注释。

参考

1\] [Install Elasticsearch with Docker \| Elasticsearch Guide \[7.17\] \| Elastic](https://link.juejin.cn?target=https%3A%2F%2Fwww.elastic.co%2Fguide%2Fen%2Felasticsearch%2Freference%2F7.17%2Fdocker.html "https://www.elastic.co/guide/en/elasticsearch/reference/7.17/docker.html") \[2\] [Running the Elastic Stack on Docker \| Getting Started \[7.17\] \| Elastic](https://link.juejin.cn?target=https%3A%2F%2Fwww.elastic.co%2Fguide%2Fen%2Felastic-stack-get-started%2F7.17%2Fget-started-docker.html "https://www.elastic.co/guide/en/elastic-stack-get-started/7.17/get-started-docker.html") \[3\] [elasticsearch-certutil \| Elasticsearch Guide \[7.17\] \| Elastic](https://link.juejin.cn?target=https%3A%2F%2Fwww.elastic.co%2Fguide%2Fen%2Felasticsearch%2Freference%2F7.17%2Fcertutil.html "https://www.elastic.co/guide/en/elasticsearch/reference/7.17/certutil.html") \[4\] [License settings \| Elasticsearch Guide \[8.13\] \| Elastic](https://link.juejin.cn?target=https%3A%2F%2Fwww.elastic.co%2Fguide%2Fen%2Felasticsearch%2Freference%2Fcurrent%2Flicense-settings.html "https://www.elastic.co/guide/en/elasticsearch/reference/current/license-settings.html") \[5\] [docker-compose案例](https://link.juejin.cn?target=https%3A%2F%2Fgitee.com%2Fanuoxiang%2Fresearch-of-db-cluster%2Fblob%2Fmain%2Fes-cluster%2Fdocker-compose-security.yaml "https://gitee.com/anuoxiang/research-of-db-cluster/blob/main/es-cluster/docker-compose-security.yaml")

相关推荐
hrrrrb1 小时前
【Spring Security】Spring Security 概念
java·数据库·spring
心止水j1 小时前
spark
javascript·数据库·spark
xujiangyan_2 小时前
Redis详解
数据库·redis·缓存
摇滚侠3 小时前
Spring Boot 3零基础教程,IOC容器中组件的注册,笔记08
spring boot·笔记·后端
Y编程小白5 小时前
PostgreSQL在Linux中的部署和安装教程
数据库·postgresql
程序员小凯5 小时前
Spring Boot测试框架详解
java·spring boot·后端
你的人类朋友6 小时前
什么是断言?
前端·后端·安全
程序员小凯7 小时前
Spring Boot缓存机制详解
spring boot·后端·缓存
TiAmo zhang7 小时前
SQL Server 2019实验 │ 数据库和表的创建、修改与删除
数据库·oracle
yannan201903137 小时前
Docker容器
运维·docker·容器