Minio 集群搭建
文章目录
- [1. 环境前准备工作](#1. 环境前准备工作)
-
- [1.1 磁盘分区](#1.1 磁盘分区)
- [1.2 磁盘格式化](#1.2 磁盘格式化)
- [1.3 磁盘挂载](#1.3 磁盘挂载)
- [1.4 关闭防火墙](#1.4 关闭防火墙)
- [1.5 创建用户](#1.5 创建用户)
- [2. 安装启动Minio](#2. 安装启动Minio)
-
- [2.1 下载Minio](#2.1 下载Minio)
- [2.2 配置文件编写](#2.2 配置文件编写)
-
- [2.2.1 主机一](#2.2.1 主机一)
- [2.2.2 主机二](#2.2.2 主机二)
- [2.2.3 主机三](#2.2.3 主机三)
- [2.3 编写启动服务 （rpm 安装可不操作）](#2.3 编写启动服务 （rpm 安装可不操作）)
- [2.4 启动Minio](#2.4 启动Minio)
- [3. 安装nginx](#3. 安装nginx)
-
- [3.1 安装前提(可选)](#3.1 安装前提(可选))
- [3.2 安装nginx](#3.2 安装nginx)
- [3.3 修改nginx配置文件](#3.3 修改nginx配置文件)
- [4. Minio 负载](#4. Minio 负载)
-
- [4.1 编写nginx 配置文件](#4.1 编写nginx 配置文件)
- [4.2 启动nginx](#4.2 启动nginx)
环境介绍
| 操作系统 | Kylin Linux Advanced Server V10 (Lance) | Kylin Linux Advanced Server V10 (Lance) | Kylin Linux Advanced Server V10 (Lance) |
|---|---|---|---|
| 内核版本 | Linux 4.19.90-52.22.v2207.ky10.aarch64 | Linux 4.19.90-52.22.v2207.ky10.aarch64 | Linux 4.19.90-52.22.v2207.ky10.aarch64 |
| IP | **192.168.31.82 ** | 192.168.31.83 | 192.168.31.84 |
| Minio | 20220802235916.0.0.aarch64 | 20220802235916.0.0.aarch64 | 20220802235916.0.0.aarch64 |
1. 环境前准备工作
三个节点同时挂载200G的硬盘 集群模式必须挂载新磁盘，与系统共用磁盘会导致启动失败
三台节点各添加200G硬盘
1.1 磁盘分区
sh
# fdisk -l 查看当前硬盘的分区情况
# 硬盘分区
fdisk /dev/vdb
输入n 创建新的分区
输入回车
默认分区输入1，回车，或者 指定存储大小 默认也行
输入w回车保存
# 查看所以块设备(硬盘驱动器、固态硬盘、USB 驱动器)
lsblk1.2 磁盘格式化
sh
mkfs -t ext4 /dev/vdb11.3 磁盘挂载
sh
# 创建挂载目录
mkdir -p /data/minio
# 挂载
mount /dev/vdb1 /data/minio
# 查看挂载是否成功
df -h
#/dev/vdb1 196G 61M 186G 1% /data/minio
# 设置分区在系统重启后自动挂载
vi /etc/fstab
/dev/vdb1 /data/minio ext4 defaults 0 0
# 重新加载配置
mount -a1.4 关闭防火墙
sh
systemctl stop firewalld
systemctl disable firewalld1.5 创建用户
sh
groupadd -r minio-user
useradd -M -r -g minio-user minio-user
chown minio-user:minio-user /data/minio/2. 安装启动Minio
2.1 下载Minio
sh
mkdir -p /opt/software/minio && cd /opt/software/minio
# 下载rpm包
wget https://dl.min.io/server/minio/release/linux-arm64/archive/minio-20220802235916.0.0.aarch64.rpm
# 查询本机是否存在minio rpm包
rpm -e | grep minio
# 移除依赖
# rpm -e --nodeps minio-20220802235916.0.0.aarch64.rpm
# 安装rpm
rpm -ivh minio-20220802235916.0.0.aarch64.rpm
# 查看minio 是否安装成功
[root@192.168.31.82 minio]# minio -version
minio version RELEASE.2022-08-02T23-59-16Z (commit-id=76f950c6632fe67b17e9f0cc00905d7093ef837d)
Runtime: go1.18.5 linux/arm64
License: GNU AGPLv3 <https://www.gnu.org/licenses/agpl-3.0.html>
Copyright: 2015-2022 MinIO, Inc.2.2 配置文件编写
注意如果系统存在多个ip 请在MINIO_OPTS --address 和 --console-address 指定指定ip
不然后面账号密码即使正确也登陆不了 ！！！
2.2.1 主机一
sh
vi /etc/default/minio
cat > /etc/default/minio <<EOF
# 文件磁盘的位置 因为我们是集群节点是82-84 挂载文件夹是/data/minio,生产环境不要使用默认密码
MINIO_VOLUMES="http://192.168.31.8{2...4}:9000/data/minio"
# minio-console的地址 就是web界面控制台
MINIO_OPTS="--address 192.168.31.82:9000 --console-address 192.168.31.82:9001"
# console的登陆账号
MINIO_ROOT_USER=yfkj
# # console的登陆密码
MINIO_ROOT_PASSWORD=yfkj_2024
EOF2.2.2 主机二
sh
cat > /etc/default/minio <<EOF
MINIO_VOLUMES="http://192.168.31.8{2...4}:9000/data/minio"
# minio-console的地址 就是web界面控制台
MINIO_OPTS="--address 192.168.31.83:9000 --console-address 192.168.31.83:9001"
# console的登陆账号
MINIO_ROOT_USER=yfkj
# # console的登陆密码
MINIO_ROOT_PASSWORD=yfkj_2024
EOF2.2.3 主机三
sh
cat > /etc/default/minio <<EOF
MINIO_VOLUMES="http://192.168.31.8{2...4}:9000/data/minio"
# minio-console的地址 就是web界面控制台
MINIO_OPTS="--address 192.168.31.84:9000 --console-address 192.168.31.84:9001"
# console的登陆账号
MINIO_ROOT_USER=yfkj
# # console的登陆密码
MINIO_ROOT_PASSWORD=yfkj_2024
EOF
MINIO_VOLUMES="http://192.168.31.82:9000/data/minio http://192.168.31.83:9000/data/minio http://192.168.31.84:9000/data/minio"2.3 编写启动服务 （rpm 安装可不操作）
sh
vi /etc/systemd/system/minio.service
[Unit]
Description=MinIO
Documentation=https://min.io/docs/minio/linux/index.html
Wants=network-online.target
After=network-online.target
AssertFileIsExecutable=/usr/local/bin/minio
[Service]
WorkingDirectory=/usr/local
User=minio-user
Group=minio-user
ProtectProc=invisible
EnvironmentFile=-/etc/default/minio
ExecStartPre=/bin/bash -c "if [ -z \"${MINIO_VOLUMES}\" ]; then echo \"Variable MINIO_VOLUMES not set in /etc/default/minio\"; exit 1; fi"
ExecStart=/usr/local/bin/minio server $MINIO_OPTS $MINIO_VOLUMES
# MinIO RELEASE.2023-05-04T21-44-30Z adds support for Type=notify (https://www.freedesktop.org/software/systemd/man/systemd.service.html#Type=)
# This may improve systemctl setups where other services use `After=minio.server`
# Uncomment the line to enable the functionality
# Type=notify
# Let systemd restart this service always
Restart=always
# Specifies the maximum file descriptor number that can be opened by this process
LimitNOFILE=65536
# Specifies the maximum number of threads this process can create
TasksMax=infinity
# Disable timeout logic and wait until process is stopped
TimeoutStopSec=infinity
SendSIGKILL=no
[Install]
WantedBy=multi-user.target
# Built for ${project.name}-${project.version} (${project.name})2.4 启动Minio
sh
# 刷新配置 配置文件修改记得执行
# systemctl daemon-reload
# 启动
systemctl start minio.service
# 查看状态
systemctl status minio.service
# 停止
systemctl stop minio.service
# 重新启动
systemctl restart minio.service
# 设置开机自启动
systemctl enable minio.service
# 查看minio 日志
journalctl -f -u minio.service
# 然后查看日志 发现有 ble to initialize backend: Storage resources are insufficient for the write
# 请删除 /data/minio/.minio.sys 文件重新执行 systemctl restart minio.service3. 安装nginx
3.1 安装前提(可选)
sh
# 检查电脑是否安装了 pcre-config openssl 如果没有则安装
# 下载pcre(其他)
wget http://downloads.sourceforge.net/project/pcre/pcre/8.37/pcre-8.37.tar.gz
# 解压
tar -zxvf pcre-8.37.tar.gz
# 进入目录
cd pcre-8.37
# 编译安装
./configure
make install
# 检测是否安装成功
pcre-config --version
# 下载 openssl(其他)
https://github.com/openssl/openssl/releases/download/openssl-3.2.1/openssl-3.2.1.tar.gz
# 解压
tar -zxvf openssl-3.2.1.tar.gz
# 进入目录
cd openssl-3.2.1
# 编译安装
./config
make install
# 安装zlib
yum -y install zlib zlib-devel libtool3.2 安装nginx
sh
# 官方地址
https://nginx.org/en/download.html
# 下载
mkdir -p /opt/software/nginx && cd /opt/software/nginx
wget https://nginx.org/download/nginx-1.26.2.tar.gz
# 解压
tar -zxvf nginx-1.26.2.tar.gz
# 进入到nginx目录编译安装
cd nginx-1.26.2
# 编译安装
./configure --prefix=/usr/local/nginx-1.26.2 \
--user=root \
--group=root \
--with-http_ssl_module \
--with-http_v2_module \
--with-http_realip_module \
--with-http_stub_status_module \
--with-http_gzip_static_module \
--with-pcre \
--with-stream \
--with-stream_ssl_module \
--with-stream_realip_module \
--with-mail \
--with-mail_ssl_module
# 安装
make && make install
# 创建软连接文件 方便后面升级
ln -s /usr/local/nginx-1.26.2 /usr/local/nginx
# 配置环境变量
vi /etc/profile.d/nginx.sh
#!/bin/bash
export NGINX_HOME=/usr/local/nginx
export PATH="${NGINX_HOME}/sbin:$PATH"
# 生效
source /etc/profile.d/nginx.sh
# 编写nginx服务
vi /usr/lib/systemd/system/nginx.service
[Unit]
Description=The nginx HTTP and reverse proxy server
After=network-online.target remote-fs.target nss-lookup.target
Wants=network-online.target
[Service]
Type=forking
PIDFile=/usr/local/nginx/logs/nginx.pid
ExecStartPre=/usr/bin/rm -f /usr/local/nginx/logs/nginx.pid
ExecStartPre=/usr/local/nginx/sbin/nginx -t
ExecStart=/usr/local/nginx/sbin/nginx
ExecReload=/usr/local/nginx/sbin/nginx -s reload
KillSignal=SIGQUIT
TimeoutStopSec=5
KillMode=process
PrivateTmp=true
[Install]
WantedBy=multi-user.target
# 增加可执行权限
chmod a+x /usr/lib/systemd/system/nginx.service
# 重新加载
systemctl daemon-reload
# 查看nginx版本
[root@192.168.31.82 ~]# nginx -v
nginx version: nginx/1.26.23.3 修改nginx配置文件
sh
# 清空nginx.conf
echo > /usr/local/nginx/conf/nginx.conf
# 创建配置文件存放的目录
mkdir -p /usr/local/nginx/conf.d
# 导入配置文件信息
cat >> /usr/local/nginx/conf/nginx.conf << EOF
#user nobody;
worker_processes 1;
#error_log logs/error.log;
#error_log logs/error.log notice;
#error_log logs/error.log info;
#pid logs/nginx.pid;
events {
worker_connections 1024;
}
http {
include mime.types;
default_type application/octet-stream;
#log_format main '$remote_addr - $remote_user [$time_local] "$request" '
# '$status $body_bytes_sent "$http_referer" '
# '"$http_user_agent" "$http_x_forwarded_for"';
#access_log logs/access.log main;
sendfile on;
#tcp_nopush on;
#keepalive_timeout 0;
keepalive_timeout 65;
#gzip on;
include /usr/local/nginx/conf.d/*.conf;
}
EOF
# 检查配置文件是否错误
# nginx -t
[root@192.168.31.82 ~]# nginx -t
nginx: the configuration file /usr/local/nginx-1.26.2/conf/nginx.conf syntax is ok
nginx: configuration file /usr/local/nginx-1.26.2/conf/nginx.conf test is successful4. Minio 负载
4.1 编写nginx 配置文件
sh
# 提前创建日志存储文件
touch /usr/local/nginx/logs/minio_server.log
# 写入配置信息
vi /usr/local/nginx/conf.d/minio_load.conf
# 日志格式
log_format minio_log '$remote_addr - $remote_user [$time_local] "$request" '
'$status $body_bytes_sent "$http_referer" '
'"$http_user_agent" "$http_x_forwarded_for"';
# 七层负载
upstream minio_server {
server 192.168.31.82:9000;
server 192.168.31.83:9000;
server 192.168.31.84:9000;
# 使用轮询方式请求
least_conn;
}
server {
listen 19000 ;
server_name 192.168.31.82;
access_log /usr/local/nginx/logs/minio_server.log minio_log;
location / {
client_body_buffer_size 10M;
client_max_body_size 10G;
proxy_buffers 1024 4k;
proxy_pass http://minio_server;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
}
}
# 检查配置文件是否存在错误
[root@192.168.31.82 ~]# nginx -t
nginx: the configuration file /usr/local/nginx-1.26.2/conf/nginx.conf syntax is ok
nginx: configuration file /usr/local/nginx-1.26.2/conf/nginx.conf test is successful4.2 启动nginx
sh
systemctl start nginx