安全通信设置：使用 OpenSSL 为 Logstash 和 Filebeat 提供 SSL 证书

在为 Elasticsearch 采集数据时，我们经常使用到 Filebeat 及 Logstash。在我们之前的很多教程中，我们通常不为 Filebeat 和 Logstash 之前的通信做安全配置。

如何为 Filebeat 及 Logstash 直接建立安全的链接？这个在很多的情况下是非常有用的。在我之前的文章 "Elasticsearch：为日志分析设置安全的 Elasticsearch 管道"，我有涉及到这个问题。最主要的是，它是针对 Elastic Stack 8.x 而设计的。在今天的文章中，我将针对最普通的情况来进行展示。

本指南提供了有关使用 OpenSSL 生成和配置 SSL 证书的详细说明，以增强 Logstash 和 Filebeat 之间通信的安全性。概述的步骤包括为证书颁发机构 (CA) 生成私钥、为 CA 创建自签名 X509 证书以及为 Logstash 和 Filebeat 生成密钥和证书。还包括使用这些证书的 Filebeat 和 Logstash 的配置步骤。

为 CA 生成私钥

ruby 复制代码

1.  # Generate a private key for the Certificate Authority (CA)
2.  openssl genrsa 2048 > /path/to/your/ca-key.pem

bash 复制代码

1.  $ pwd
2.  /Users/liuxg/demos/filebeat_logstash
3.  $ openssl genrsa 2048 > ca-key.pem
4.  $ ls
5.  ca-key.pem

为 CA 生成 X509 证书

bash 复制代码

1.  # Generate a self-signed X509 certificate for the CA
2.  openssl req -new -x509 -nodes -days 365000 \
3.      -key /path/to/your/ca-key.pem \
4.      -out /path/to/your/ca-cert.pem

vbnet 复制代码

openssl req -new -x509 -nodes -days 365000 -key ca-key.pem -out ca-cert.pem

sql 复制代码

1.  $ pwd
2.  /Users/liuxg/demos/filebeat_logstash
3.  $ ls
4.  ca-key.pem
5.  $ openssl req -new -x509 -nodes -days 365000 -key ca-key.pem -out ca-cert.pem
6.  You are about to be asked to enter information that will be incorporated
7.  into your certificate request.
8.  What you are about to enter is what is called a Distinguished Name or a DN.
9.  There are quite a few fields but you can leave some blank
10.  For some fields there will be a default value,
11.  If you enter '.', the field will be left blank.
12.  -----
13.  Country Name (2 letter code) [AU]:CN
14.  State or Province Name (full name) [Some-State]:Beijing
15.  Locality Name (eg, city) []:Beijing
16.  Organization Name (eg, company) [Internet Widgits Pty Ltd]:Elastic
17.  Organizational Unit Name (eg, section) []:Community
18.  Common Name (e.g. server FQDN or YOUR name) []:Xiaoguo
19.  Email Address []:x@y.com
20.  $ ls
21.  ca-cert.pem ca-key.pem

为 Logstash 和 Filebeat 创建证书和密钥

vbnet 复制代码

1.  # Generate a private key and certificate for Logstash and Filebeat using the CA
2.  openssl req -x509 -nodes -days 365 -newkey rsa:2048 \
3.      -keyout /path/to/your/server.key \
4.      -out /path/to/your/server.crt \
5.      -CA /path/to/your/ca-cert.pem \
6.      -CAkey /path/to/your/ca-key.pem

markdown 复制代码

1.  openssl req -x509 -nodes -days 365 -newkey rsa:2048 \
2.      -keyout ./server.key \
3.      -out ./server.crt \
4.      -CA ./ca-cert.pem \
5.      -CAkey ./ca-key.pem

sql 复制代码

1.  $ pwd
2.  /Users/liuxg/demos/filebeat_logstash
3.  $ ls
4.  ca-cert.pem ca-key.pem
5.  $ openssl req -x509 -nodes -days 365 -newkey rsa:2048 \
6.  >     -keyout ./server.key \
7.  >     -out ./server.crt \
8.  >     -CA ./ca-cert.pem \
9.  >     -CAkey ./ca-key.pem
10.  ........+...+.....+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*...........+..+.+...+.........+..+.............+...+..+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*..........+..........+.....+...+.........+.+..+......+.......+...............+..+...+.......+...+...+.....+......+......+.......+..+....+..+..........+..+......+......+.+..+.+..............+...+.......+..+...+.......+...+........+...+......+....+..+.......+.....+...............+......+................+......+...+......+.....+.+..+...+..........+........+................+...............+......+..+.+..+.........+...+.......+......+...+.....+.+...............+...+........+...+.........+...+.........+....+.....+.+........+...+.........+..........+......+..+.............+.....+....+...+..+.............+......+......+.........+..+...+....+...+............+...+..+.+......+.....+.+..+...+..................+.....................+....+..............+.+..+..........+..+...+....+...+......+.................+...+....+........+...+.+...........+..........+.....+...............+....+......+..............+...+......+......+...+.......+...+.....+...+.............+..+.......+.....+.+..+....+...+......+.....+..........+..+.........+.........+.+........+.+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
11.  .+.+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*.........+......+...+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*........+.........+.........+.....+.+.....+.+.........+..+....+........+...+.......+........+...+......+...+...+............+..........+...+..+..........+.....+.............+..+...+...+....+...+..............+....+..+......................+..+...+...+...+.+.........+......+...+..+.+...+.....+.........+...+...+.+...+..+.+........+............+..........+..+.+.....................+......+........+.+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
12.  -----
13.  You are about to be asked to enter information that will be incorporated
14.  into your certificate request.
15.  What you are about to enter is what is called a Distinguished Name or a DN.
16.  There are quite a few fields but you can leave some blank
17.  For some fields there will be a default value,
18.  If you enter '.', the field will be left blank.
19.  -----
20.  Country Name (2 letter code) [AU]:CN
21.  State or Province Name (full name) [Some-State]:Beijing
22.  Locality Name (eg, city) []:Beijing
23.  Organization Name (eg, company) [Internet Widgits Pty Ltd]:Elastic
24.  Organizational Unit Name (eg, section) []:Community 
25.  Common Name (e.g. server FQDN or YOUR name) []:localhost
26.  Email Address []:x@y.com

按照同样的方法，我们生成相应的 client 证书：

markdown 复制代码

1.  openssl req -x509 -nodes -days 365 -newkey rsa:2048 \
2.      -keyout client.key \
3.      -out client.crt \
4.      -CA ./ca-cert.pem \
5.      -CAkey ./ca-key.pem

markdown 复制代码

1.  $ openssl req -x509 -nodes -days 365 -newkey rsa:2048 \
2.  >     -keyout ./server.key \
3.  >     -out ./server.crt \
4.  >     -CA ./ca-cert.pem \
5.  >     -CAkey ./ca-key.pem
6.  ........+...+.....+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*...........+..+.+...+.........+..+.............+...+..+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*..........+..........+.....+...+.........+.+..+......+.......+...............+..+...+.......+...+...+.....+......+......+.......+..+....+..+..........+..+......+......+.+..+.+..............+...+.......+..+...+.......+...+........+...+......+....+..+.......+.....+...............+......+................+......+...+......+.....+.+..+...+..........+........+................+...............+......+..+.+..+.........+...+.......+......+...+.....+.+...............+...+........+...+.........+...+.........+....+.....+.+........+...+.........+..........+......+..+.............+.....+....+...+..+.............+......+......+.........+..+...+....+...+............+...+..+.+......+.....+.+..+...+..................+.....................+....+..............+.+..+..........+..+...+....+...+......+.................+...+....+........+...+.+...........+..........+.....+...............+....+......+..............+...+......+......+...+.......+...+.....+...+.............+..+.......+.....+.+..+....+...+......+.....+..........+..+.........+.........+.+........+.+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
7.  .+.+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*.........+......+...+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*........+.........+.........+.....+.+.....+.+.........+..+....+........+...+.......+........+...+......+...+...+............+..........+...+..+..........+.....+.............+..+...+...+....+...+..............+....+..+......................+..+...+...+...+.+.........+......+...+..+.+...+.....+.........+...+...+.+...+..+.+........+............+..........+..+.+.....................+......+........+.+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
8.  -----
9.  You are about to be asked to enter information that will be incorporated
10.  into your certificate request.
11.  What you are about to enter is what is called a Distinguished Name or a DN.
12.  There are quite a few fields but you can leave some blank
13.  For some fields there will be a default value,
14.  If you enter '.', the field will be left blank.
15.  -----
16.  Country Name (2 letter code) [AU]:CN
17.  State or Province Name (full name) [Some-State]:Beijing
18.  Locality Name (eg, city) []:Beijing
19.  Organization Name (eg, company) [Internet Widgits Pty Ltd]:Elastic
20.  Organizational Unit Name (eg, section) []:Community 
21.  Common Name (e.g. server FQDN or YOUR name) []:localhost
22.  Email Address []:x@y.com
23.  $ openssl req -x509 -nodes -days 365 -newkey rsa:2048 \
24.  >     -keyout client.key \
25.  >     -out client.crt \
26.  >     -CA ./ca-cert.pem \
27.  >     -CAkey ./ca-key.pem
28.  ....+...+.+.....+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*.+.........+........+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*......+....+.....+....+...+...+..+..................+.+..............+.+.....................+...+..+.............+..+.......+..+......+............+......+.+......+.........+.........+......+...+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
29.  ..+.....+..................+.+..+............+.+..+....+......+...+.....+.......+...+.....+......+...+.............+........+.+.....+....+..+.......+......+..+....+...+..+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*....+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*.+.....+......+.+...+..+....+.....+.............+........+.......+...+............+.....+..................+.......+..+.+.........+...............+.....+.+..+...+...+.+......+.....+.......+.....+...+.........+......+....+...+...........+....+...........+......+.........+....+.....+......+......+.+.....+..........+.....+............+.+......+...+...+...+..+.......+..+...+......+.+......+.....+....+...+..+...............+....+...............+..+.........+...+.+.....+................+.....+.+...+......+.....+......+.......+..+...+......+.+......+...+.....+.........+......................+...+.....+...+...+......+.........+.+.....+....+.....+...............+....+...+.....+............+.+..+.......+..+...+...+............+..........+..+................+..+...+...+.+...+........+...+.+...........+.........+......+..................+....+.....+............+....+.....+......+....+..+....+.....+.+.....+...+...............+....+.....+..........+..+................+..+............+.........+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
30.  -----
31.  You are about to be asked to enter information that will be incorporated
32.  into your certificate request.
33.  What you are about to enter is what is called a Distinguished Name or a DN.
34.  There are quite a few fields but you can leave some blank
35.  For some fields there will be a default value,
36.  If you enter '.', the field will be left blank.
37.  -----
38.  Country Name (2 letter code) [AU]:CN
39.  State or Province Name (full name) [Some-State]:Beijing
40.  Locality Name (eg, city) []:Beijing
41.  Organization Name (eg, company) [Internet Widgits Pty Ltd]:Elastic
42.  Organizational Unit Name (eg, section) []:Community
43.  Common Name (e.g. server FQDN or YOUR name) []:localhost
44.  Email Address []:x@y.com
45.  $ ls
46.  ca-cert.pem                           filebeat-8.12.0-darwin-aarch64.tar.gz
47.  ca-key.pem                            logstash-8.12.0
48.  client.crt                            logstash-8.12.0-darwin-aarch64.tar.gz
49.  client.key                            server.crt
50.  filebeat-8.12.0-darwin-aarch64        server.key

配置 Filebeat（或任何 Beat）

yaml 复制代码

1.  output.logstash:
2.    hosts: ["localhost:5044"]

4.    # Specify the path to the CA certificate for verification
5.    ssl.certificate_authorities: ["/path/to/your/ca-cert.pem"]

7.    # Specify the path to the client certificate
8.    ssl.certificate: "/path/to/your/server.crt"

10.    # Specify the path to the client private key
11.    ssl.key: "/path/to/your/server.key"

配置 Logstash

ini 复制代码

1.  input {
2.    beats {
3.      port => 5044
4.      ssl => true

6.      # Specify the path to the CA certificate for verification
7.      ssl_certificate_authorities => ["/path/to/your/ca-cert.pem"]

9.      # Specify the path to the server certificate
10.      ssl_certificate => "/path/to/your/server.crt"

12.      # Specify the path to the server private key
13.      ssl_key => "/path/to/your/server.key"

15.      # Set peer verification mode to force_peer
16.      ssl_verify_mode => "force_peer"
17.    }
18.  }

确保将 /path/to/your/ 替换为密钥和证书的实际所需路径。

测试

我们在当前的目录下安装 Filebeat 及 Logstash：

filebeat_ssl.yml

bash 复制代码

1.  $ pwd
2.  /Users/liuxg/demos/filebeat_logstash
3.  $ ls 
4.  ca-cert.pem                           logstash-8.12.0
5.  ca-key.pem                            logstash-8.12.0-darwin-aarch64.tar.gz
6.  filebeat-8.12.0-darwin-aarch64        server.crt
7.  filebeat-8.12.0-darwin-aarch64.tar.gz server.key
8.  $ cd filebeat-8.12.0-darwin-aarch64
9.  $ ls filebeat_ssl.yml 
10.  filebeat_ssl.yml

yaml 复制代码

1.  filebeat.inputs:
2.  - type: log
3.    enabled: true
4.    paths:
5.      - ./logfile.log
6.  output.logstash:
7.    hosts: ["localhost:5044"]
8.    ssl.certificate_authorities: ["../ca-cert.pem"]
9.    ssl.certificate: "../server.crt"
10.    ssl.key: "../server.key"
11.  logging:
12.    level: info
13.    to_files: true
14.    files:
15.    path: .
16.    name: filebeat.log
17.    keepfiles: 7
18.    permissions: 0644

我们在 filebeat 安装目录下创建一个叫做 logfile.log 的文件。它的内容如下：

logfile.log

bash 复制代码

This is a test log

logstash.conf

ini 复制代码

1.  input {
2.    beats {
3.      port => 5044
4.      ssl => true
5.      ssl_certificate_authorities => "../ca-cert.pem"
6.      ssl_certificate => "../server.crt"
7.      ssl_key => "../server.key"
8.      ssl_verify_mode => "force_peer"
9.    }
10.  }

12.  output {
13.    stdout { codec => rubydebug }
14.    file {
15.      path => "../outputfile.log"
16.    }
17.  }

我们首先运行 logstash：

bash 复制代码

./bin/logstash -f logstash.conf

我们再接下来运行 Filebeat：

我们回到 Logstash 运行的 terminal 中进行查看：

从上面的输出中，我们可以看到日志被正确地采集到 Logstash。我们可以更进一步输出到 Elasticsearch。在这里，我们就不再累述了。