在为 Elasticsearch 采集数据时,我们经常使用到 Filebeat 及 Logstash。在我们之前的很多教程中,我们通常不为 Filebeat 和 Logstash 之前的通信做安全配置。
如何为 Filebeat 及 Logstash 直接建立安全的链接?这个在很多的情况下是非常有用的。在我之前的文章 "Elasticsearch:为日志分析设置安全的 Elasticsearch 管道",我有涉及到这个问题。最主要的是,它是针对 Elastic Stack 8.x 而设计的。在今天的文章中,我将针对最普通的情况来进行展示。
本指南提供了有关使用 OpenSSL 生成和配置 SSL 证书的详细说明,以增强 Logstash 和 Filebeat 之间通信的安全性。 概述的步骤包括为证书颁发机构 (CA) 生成私钥、为 CA 创建自签名 X509 证书以及为 Logstash 和 Filebeat 生成密钥和证书。 还包括使用这些证书的 Filebeat 和 Logstash 的配置步骤。
为 CA 生成私钥
ruby
1. # Generate a private key for the Certificate Authority (CA)
2. openssl genrsa 2048 > /path/to/your/ca-key.pem
bash
1. $ pwd
2. /Users/liuxg/demos/filebeat_logstash
3. $ openssl genrsa 2048 > ca-key.pem
4. $ ls
5. ca-key.pem
为 CA 生成 X509 证书
bash
1. # Generate a self-signed X509 certificate for the CA
2. openssl req -new -x509 -nodes -days 365000 \
3. -key /path/to/your/ca-key.pem \
4. -out /path/to/your/ca-cert.pem
vbnet
openssl req -new -x509 -nodes -days 365000 -key ca-key.pem -out ca-cert.pem
sql
1. $ pwd
2. /Users/liuxg/demos/filebeat_logstash
3. $ ls
4. ca-key.pem
5. $ openssl req -new -x509 -nodes -days 365000 -key ca-key.pem -out ca-cert.pem
6. You are about to be asked to enter information that will be incorporated
7. into your certificate request.
8. What you are about to enter is what is called a Distinguished Name or a DN.
9. There are quite a few fields but you can leave some blank
10. For some fields there will be a default value,
11. If you enter '.', the field will be left blank.
12. -----
13. Country Name (2 letter code) [AU]:CN
14. State or Province Name (full name) [Some-State]:Beijing
15. Locality Name (eg, city) []:Beijing
16. Organization Name (eg, company) [Internet Widgits Pty Ltd]:Elastic
17. Organizational Unit Name (eg, section) []:Community
18. Common Name (e.g. server FQDN or YOUR name) []:Xiaoguo
19. Email Address []:x@y.com
20. $ ls
21. ca-cert.pem ca-key.pem
为 Logstash 和 Filebeat 创建证书和密钥
vbnet
1. # Generate a private key and certificate for Logstash and Filebeat using the CA
2. openssl req -x509 -nodes -days 365 -newkey rsa:2048 \
3. -keyout /path/to/your/server.key \
4. -out /path/to/your/server.crt \
5. -CA /path/to/your/ca-cert.pem \
6. -CAkey /path/to/your/ca-key.pem
markdown
1. openssl req -x509 -nodes -days 365 -newkey rsa:2048 \
2. -keyout ./server.key \
3. -out ./server.crt \
4. -CA ./ca-cert.pem \
5. -CAkey ./ca-key.pem
sql
1. $ pwd
2. /Users/liuxg/demos/filebeat_logstash
3. $ ls
4. ca-cert.pem ca-key.pem
5. $ openssl req -x509 -nodes -days 365 -newkey rsa:2048 \
6. > -keyout ./server.key \
7. > -out ./server.crt \
8. > -CA ./ca-cert.pem \
9. > -CAkey ./ca-key.pem
10. ........+...+.....+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*...........+..+.+...+.........+..+.............+...+..+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*..........+..........+.....+...+.........+.+..+......+.......+...............+..+...+.......+...+...+.....+......+......+.......+..+....+..+..........+..+......+......+.+..+.+..............+...+.......+..+...+.......+...+........+...+......+....+..+.......+.....+...............+......+................+......+...+......+.....+.+..+...+..........+........+................+...............+......+..+.+..+.........+...+.......+......+...+.....+.+...............+...+........+...+.........+...+.........+....+.....+.+........+...+.........+..........+......+..+.............+.....+....+...+..+.............+......+......+.........+..+...+....+...+............+...+..+.+......+.....+.+..+...+..................+.....................+....+..............+.+..+..........+..+...+....+...+......+.................+...+....+........+...+.+...........+..........+.....+...............+....+......+..............+...+......+......+...+.......+...+.....+...+.............+..+.......+.....+.+..+....+...+......+.....+..........+..+.........+.........+.+........+.+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
11. .+.+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*.........+......+...+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*........+.........+.........+.....+.+.....+.+.........+..+....+........+...+.......+........+...+......+...+...+............+..........+...+..+..........+.....+.............+..+...+...+....+...+..............+....+..+......................+..+...+...+...+.+.........+......+...+..+.+...+.....+.........+...+...+.+...+..+.+........+............+..........+..+.+.....................+......+........+.+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
12. -----
13. You are about to be asked to enter information that will be incorporated
14. into your certificate request.
15. What you are about to enter is what is called a Distinguished Name or a DN.
16. There are quite a few fields but you can leave some blank
17. For some fields there will be a default value,
18. If you enter '.', the field will be left blank.
19. -----
20. Country Name (2 letter code) [AU]:CN
21. State or Province Name (full name) [Some-State]:Beijing
22. Locality Name (eg, city) []:Beijing
23. Organization Name (eg, company) [Internet Widgits Pty Ltd]:Elastic
24. Organizational Unit Name (eg, section) []:Community
25. Common Name (e.g. server FQDN or YOUR name) []:localhost
26. Email Address []:x@y.com
按照同样的方法,我们生成相应的 client 证书:
markdown
1. openssl req -x509 -nodes -days 365 -newkey rsa:2048 \
2. -keyout client.key \
3. -out client.crt \
4. -CA ./ca-cert.pem \
5. -CAkey ./ca-key.pem
markdown
1. $ openssl req -x509 -nodes -days 365 -newkey rsa:2048 \
2. > -keyout ./server.key \
3. > -out ./server.crt \
4. > -CA ./ca-cert.pem \
5. > -CAkey ./ca-key.pem
6. ........+...+.....+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*...........+..+.+...+.........+..+.............+...+..+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*..........+..........+.....+...+.........+.+..+......+.......+...............+..+...+.......+...+...+.....+......+......+.......+..+....+..+..........+..+......+......+.+..+.+..............+...+.......+..+...+.......+...+........+...+......+....+..+.......+.....+...............+......+................+......+...+......+.....+.+..+...+..........+........+................+...............+......+..+.+..+.........+...+.......+......+...+.....+.+...............+...+........+...+.........+...+.........+....+.....+.+........+...+.........+..........+......+..+.............+.....+....+...+..+.............+......+......+.........+..+...+....+...+............+...+..+.+......+.....+.+..+...+..................+.....................+....+..............+.+..+..........+..+...+....+...+......+.................+...+....+........+...+.+...........+..........+.....+...............+....+......+..............+...+......+......+...+.......+...+.....+...+.............+..+.......+.....+.+..+....+...+......+.....+..........+..+.........+.........+.+........+.+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
7. .+.+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*.........+......+...+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*........+.........+.........+.....+.+.....+.+.........+..+....+........+...+.......+........+...+......+...+...+............+..........+...+..+..........+.....+.............+..+...+...+....+...+..............+....+..+......................+..+...+...+...+.+.........+......+...+..+.+...+.....+.........+...+...+.+...+..+.+........+............+..........+..+.+.....................+......+........+.+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
8. -----
9. You are about to be asked to enter information that will be incorporated
10. into your certificate request.
11. What you are about to enter is what is called a Distinguished Name or a DN.
12. There are quite a few fields but you can leave some blank
13. For some fields there will be a default value,
14. If you enter '.', the field will be left blank.
15. -----
16. Country Name (2 letter code) [AU]:CN
17. State or Province Name (full name) [Some-State]:Beijing
18. Locality Name (eg, city) []:Beijing
19. Organization Name (eg, company) [Internet Widgits Pty Ltd]:Elastic
20. Organizational Unit Name (eg, section) []:Community
21. Common Name (e.g. server FQDN or YOUR name) []:localhost
22. Email Address []:x@y.com
23. $ openssl req -x509 -nodes -days 365 -newkey rsa:2048 \
24. > -keyout client.key \
25. > -out client.crt \
26. > -CA ./ca-cert.pem \
27. > -CAkey ./ca-key.pem
28. ....+...+.+.....+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*.+.........+........+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*......+....+.....+....+...+...+..+..................+.+..............+.+.....................+...+..+.............+..+.......+..+......+............+......+.+......+.........+.........+......+...+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
29. ..+.....+..................+.+..+............+.+..+....+......+...+.....+.......+...+.....+......+...+.............+........+.+.....+....+..+.......+......+..+....+...+..+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*....+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*.+.....+......+.+...+..+....+.....+.............+........+.......+...+............+.....+..................+.......+..+.+.........+...............+.....+.+..+...+...+.+......+.....+.......+.....+...+.........+......+....+...+...........+....+...........+......+.........+....+.....+......+......+.+.....+..........+.....+............+.+......+...+...+...+..+.......+..+...+......+.+......+.....+....+...+..+...............+....+...............+..+.........+...+.+.....+................+.....+.+...+......+.....+......+.......+..+...+......+.+......+...+.....+.........+......................+...+.....+...+...+......+.........+.+.....+....+.....+...............+....+...+.....+............+.+..+.......+..+...+...+............+..........+..+................+..+...+...+.+...+........+...+.+...........+.........+......+..................+....+.....+............+....+.....+......+....+..+....+.....+.+.....+...+...............+....+.....+..........+..+................+..+............+.........+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
30. -----
31. You are about to be asked to enter information that will be incorporated
32. into your certificate request.
33. What you are about to enter is what is called a Distinguished Name or a DN.
34. There are quite a few fields but you can leave some blank
35. For some fields there will be a default value,
36. If you enter '.', the field will be left blank.
37. -----
38. Country Name (2 letter code) [AU]:CN
39. State or Province Name (full name) [Some-State]:Beijing
40. Locality Name (eg, city) []:Beijing
41. Organization Name (eg, company) [Internet Widgits Pty Ltd]:Elastic
42. Organizational Unit Name (eg, section) []:Community
43. Common Name (e.g. server FQDN or YOUR name) []:localhost
44. Email Address []:x@y.com
45. $ ls
46. ca-cert.pem filebeat-8.12.0-darwin-aarch64.tar.gz
47. ca-key.pem logstash-8.12.0
48. client.crt logstash-8.12.0-darwin-aarch64.tar.gz
49. client.key server.crt
50. filebeat-8.12.0-darwin-aarch64 server.key
配置 Filebeat(或任何 Beat)
yaml
1. output.logstash:
2. hosts: ["localhost:5044"]
4. # Specify the path to the CA certificate for verification
5. ssl.certificate_authorities: ["/path/to/your/ca-cert.pem"]
7. # Specify the path to the client certificate
8. ssl.certificate: "/path/to/your/server.crt"
10. # Specify the path to the client private key
11. ssl.key: "/path/to/your/server.key"
配置 Logstash
ini
1. input {
2. beats {
3. port => 5044
4. ssl => true
6. # Specify the path to the CA certificate for verification
7. ssl_certificate_authorities => ["/path/to/your/ca-cert.pem"]
9. # Specify the path to the server certificate
10. ssl_certificate => "/path/to/your/server.crt"
12. # Specify the path to the server private key
13. ssl_key => "/path/to/your/server.key"
15. # Set peer verification mode to force_peer
16. ssl_verify_mode => "force_peer"
17. }
18. }
确保将 /path/to/your/ 替换为密钥和证书的实际所需路径。
测试
我们在当前的目录下安装 Filebeat 及 Logstash:
filebeat_ssl.yml
bash
1. $ pwd
2. /Users/liuxg/demos/filebeat_logstash
3. $ ls
4. ca-cert.pem logstash-8.12.0
5. ca-key.pem logstash-8.12.0-darwin-aarch64.tar.gz
6. filebeat-8.12.0-darwin-aarch64 server.crt
7. filebeat-8.12.0-darwin-aarch64.tar.gz server.key
8. $ cd filebeat-8.12.0-darwin-aarch64
9. $ ls filebeat_ssl.yml
10. filebeat_ssl.yml
yaml
1. filebeat.inputs:
2. - type: log
3. enabled: true
4. paths:
5. - ./logfile.log
6. output.logstash:
7. hosts: ["localhost:5044"]
8. ssl.certificate_authorities: ["../ca-cert.pem"]
9. ssl.certificate: "../server.crt"
10. ssl.key: "../server.key"
11. logging:
12. level: info
13. to_files: true
14. files:
15. path: .
16. name: filebeat.log
17. keepfiles: 7
18. permissions: 0644
我们在 filebeat 安装目录下创建一个叫做 logfile.log 的文件。它的内容如下:
logfile.log
bash
This is a test log
logstash.conf
ini
1. input {
2. beats {
3. port => 5044
4. ssl => true
5. ssl_certificate_authorities => "../ca-cert.pem"
6. ssl_certificate => "../server.crt"
7. ssl_key => "../server.key"
8. ssl_verify_mode => "force_peer"
9. }
10. }
12. output {
13. stdout { codec => rubydebug }
14. file {
15. path => "../outputfile.log"
16. }
17. }
我们首先运行 logstash:
bash
./bin/logstash -f logstash.conf
我们再接下来运行 Filebeat:
我们回到 Logstash 运行的 terminal 中进行查看:
从上面的输出中,我们可以看到日志被正确地采集到 Logstash。我们可以更进一步输出到 Elasticsearch。在这里,我们就不再累述了。