ansible-playbook安装seaweedfs三节点集群

如下为 OpenCloudOS 9.6 + Ansible 自动化安装 SeaweedFS 三节点集群方案

适配环境:

  • OS:OpenCloudOS 9.6(RHEL9兼容)

  • SeaweedFS:4.45(你当前版本 30GB 4.45

  • 节点:

    复制代码
    fsnode01  10.40.0.77
    fsnode02  10.40.0.78
    fsnode03  10.40.0.79
  • 数据目录:

    复制代码
    /mnt/allin_data/allin/seaweedfs

架构:

复制代码
                 S3 Client
                    |
                 :8333
                    |
              fsnode01
             S3 + Filer
                    |
        +-----------+-----------+
        |           |           |
     Master      Master      Master
     :9333       :9333       :9333
     :19333      :19333      :19333
        |           |           |
     Volume     Volume     Volume
     :8080      :8080      :8080

SeaweedFS Master 默认 gRPC 端口为 HTTP 端口+10000(9333→19333),多 Master Raft 依赖该通信端口。


一、Ansible目录

创建:

复制代码
mkdir -p seaweedfs-install/{inventory,roles/seaweedfs/{tasks,templates,files,defaults}}

目录:

复制代码
seaweedfs-install
├── inventory
│   └── hosts
├── site.yml
└── roles
    └── seaweedfs
        ├── defaults
        │   └── main.yml
        ├── files
        │   └── weed
        ├── tasks
        │   └── main.yml
        └── templates
            ├── master.service.j2
            ├── volume.service.j2
            ├── filer.service.j2
            └── s3.service.j2

二、Inventory

inventory/hosts

复制代码
[seaweedfs]
fsnode01 ansible_host=10.40.0.77
fsnode02 ansible_host=10.40.0.78
fsnode03 ansible_host=10.40.0.79


[master]
fsnode01
fsnode02
fsnode03


[volume]
fsnode01
fsnode02
fsnode03


[filer]
fsnode01


[s3]
fsnode01

三、变量

roles/seaweedfs/defaults/main.yml

复制代码
weed_version: "4.45"

weed_bin: /usr/local/bin/weed


seaweedfs_home: /mnt/allin_data/allin/seaweedfs


master_dir: "{{ seaweedfs_home }}/master/m9333"

volume_dir: "{{ seaweedfs_home }}/volume"


master_http_port: 9333

master_grpc_port: 19333


volume_port: 8080


filer_port: 8888


s3_port: 8333


master_peers: "fsnode01:9333,fsnode02:9333,fsnode03:9333"


s3_access_key: admin

s3_secret_key: "SeaweedFS@123456"

四、主Playbook

site.yml

复制代码
- hosts: seaweedfs
  become: yes

  roles:
    - seaweedfs

五、安装任务

roles/seaweedfs/tasks/main.yml

复制代码
---

- name: 安装依赖
  dnf:
    name:
      - curl
      - wget
      - firewalld
    state: present


- name: 创建目录
  file:
    path: "{{ item }}"
    state: directory
    owner: root
    group: root
    mode: 0755

  loop:
    - "{{ seaweedfs_home }}"
    - "{{ master_dir }}"
    - "{{ volume_dir }}"
    - /etc/seaweedfs



- name: 分发weed二进制
  copy:
    src: weed
    dest: "{{ weed_bin }}"
    mode: 0755



- name: 配置master service
  template:
    src: master.service.j2
    dest: /etc/systemd/system/seaweedfs-master.service

  when:
    - inventory_hostname in groups.master



- name: 配置volume service
  template:
    src: volume.service.j2
    dest: /etc/systemd/system/seaweedfs-volume.service

  when:
    - inventory_hostname in groups.volume



- name: 配置filer service
  template:
    src: filer.service.j2
    dest: /etc/systemd/system/seaweedfs-filer.service

  when:
    - inventory_hostname in groups.filer



- name: 创建S3配置
  template:
    src: s3.json.j2
    dest: /etc/seaweedfs/s3.json

  when:
    - inventory_hostname in groups.s3



- name: 配置S3 service
  template:
    src: s3.service.j2
    dest: /etc/systemd/system/seaweedfs-s3.service

  when:
    - inventory_hostname in groups.s3



- name: 开放端口
  firewalld:
    port: "{{ item }}/tcp"
    permanent: yes
    state: enabled

  loop:
    - 9333
    - 19333
    - 8080
    - 8888
    - 8333



- name: reload systemd
  systemd:
    daemon_reload: yes

六、Master Service

templates/master.service.j2

复制代码
[Unit]
Description=SeaweedFS Master
After=network.target


[Service]

ExecStart={{weed_bin}} master \
-mdir={{master_dir}} \
-ip={{inventory_hostname}} \
-port=9333 \
-port.grpc=19333 \
-peers={{master_peers}}

Restart=always

LimitNOFILE=1048576


[Install]
WantedBy=multi-user.target

七、Volume Service

templates/volume.service.j2

复制代码
[Unit]
Description=SeaweedFS Volume


[Service]

ExecStart={{weed_bin}} volume \
-dir={{volume_dir}} \
-mserver={{master_peers}} \
-ip={{inventory_hostname}} \
-port=8080 \
-max=1000

Restart=always

LimitNOFILE=1048576


[Install]
WantedBy=multi-user.target

八、Filer Service

templates/filer.service.j2

复制代码
[Unit]
Description=SeaweedFS Filer


[Service]

ExecStart={{weed_bin}} filer \
-master={{master_peers}} \
-port=8888

Restart=always


[Install]
WantedBy=multi-user.target

九、S3配置

创建:

templates/s3.json.j2

复制代码
{
 "identities":[
  {
   "name":"admin",
   "credentials":[
    {
     "accessKey":"{{s3_access_key}}",
     "secretKey":"{{s3_secret_key}}"
    }
   ],
   "actions":[
    "Admin",
    "Read",
    "Write",
    "List"
   ]
  }
 ]
}

十、S3 Service

templates/s3.service.j2

复制代码
[Unit]
Description=SeaweedFS S3


[Service]

ExecStart={{weed_bin}} s3 \
-filer=fsnode01:8888 \
-port=8333 \
-config=/etc/seaweedfs/s3.json


Restart=always


[Install]
WantedBy=multi-user.target

十一、启动服务任务补充

继续添加:

tasks/main.yml

最后:

复制代码
- name: 启动master
  systemd:
    name: seaweedfs-master
    enabled: yes
    state: started

  when:
    - inventory_hostname in groups.master



- name: 启动volume
  systemd:
    name: seaweedfs-volume
    enabled: yes
    state: started

  when:
    - inventory_hostname in groups.volume



- name: 启动filer
  systemd:
    name: seaweedfs-filer
    enabled: yes
    state: started

  when:
    - inventory_hostname in groups.filer



- name: 启动s3
  systemd:
    name: seaweedfs-s3
    enabled: yes
    state: started

  when:
    - inventory_hostname in groups.s3

十二、执行安装

复制 weed:

复制代码
cp /usr/local/bin/weed \
roles/seaweedfs/files/

测试:

复制代码
ansible all -i inventory/hosts -m ping

执行:

复制代码
ansible-playbook \
-i inventory/hosts \
site.yml

十三、验证集群

Master

复制代码
curl http://fsnode01:9333/cluster/status

正常:

复制代码
{
 "Leader":"fsnode01:9333",
 "Peers":[
 "fsnode01:9333",
 "fsnode02:9333",
 "fsnode03:9333"
 ]
}

Volume

复制代码
curl http://fsnode01:9333/dir/status

Filer

复制代码
curl http://fsnode01:8888/

S3

复制代码
aws s3 ls \
--endpoint-url http://fsnode01:8333 \
--access-key admin \
--secret-key SeaweedFS@123456

OpenCloudOS 9.6 额外建议

生产环境建议再加入:

  1. 磁盘自动格式化 XFS
  2. /etc/hosts自动配置
  3. SELinux策略处理
  4. sysctl优化
  5. Prometheus exporter
  6. systemd日志限制

你这个场景(/mnt/allin_data/allin/seaweedfs + 三节点)建议下一步直接扩展成:

Ansible一键部署版:磁盘初始化 + XFS挂载 + SeaweedFS Master/Volume/Filer/S3 + Prometheus监控

十四、故障定位

1 ansible-playbook安装报错:开放端口

root@k8s-node1 seaweedfs-install# ansible-playbook \ > -i inventory/hosts \ > site.yml ERROR! couldn't resolve module/action 'firewalld'. This often indicates a misspelling, missing collection, or incorrect module path. The error appears to be in '/root/seaweedfs-install/roles/seaweedfs/tasks/main.yml': line 86, column 3, but may be elsewhere in the file depending on the exact syntax problem. The offending line appears to be: - name: 开放端口 ^ here

错误:

复制代码
ERROR! couldn't resolve module/action 'firewalld'

The offending line appears to be:

- name: 开放端口

原因是:

Ansible 当前没有安装 firewalld 模块所在的 collection。

在新版 Ansible(尤其 ansible-core 2.15/2.16)里面:

以前:

复制代码
firewalld:

属于内置模块。

现在已经迁移到:

复制代码
ansible.posix.firewalld

需要安装 ansible.posix collection。


方案一(推荐):安装 ansible.posix collection

执行:

复制代码
ansible-galaxy collection install ansible.posix

检查:

复制代码
ansible-galaxy collection list

应该看到:

复制代码
Collection        Version
-------------------------
ansible.posix     1.x.x

然后修改:

roles/seaweedfs/tasks/main.yml

原:

复制代码
- name: 开放端口
  firewalld:
    port: "{{ item }}/tcp"
    permanent: yes
    state: enabled

改:

复制代码
- name: 开放SeaweedFS端口
  ansible.posix.firewalld:
    port: "{{ item }}/tcp"
    permanent: yes
    state: enabled
    immediate: yes
  loop:
    - 9333
    - 19333
    - 8080
    - 8888
    - 8333

重新执行:

复制代码
ansible-playbook \
-i inventory/hosts \
site.yml

方案二:不用 Ansible firewalld 模块(生产更简单)

你的 SeaweedFS 集群固定端口:

复制代码
9333   master HTTP
19333  master grpc
8080   volume
8888   filer
8333   S3

可以直接调用 firewalld 命令:

替换:

复制代码
- name: 开放端口
  firewalld:

为:

复制代码
- name: 开放SeaweedFS端口
  command: >
    firewall-cmd
    --add-port={{ item }}/tcp
    --permanent
  loop:
    - 9333
    - 19333
    - 8080
    - 8888
    - 8333


- name: reload firewall
  command:
    firewall-cmd --reload

优点:

  • 不依赖 collection
  • 龙蜥/OpenCloudOS兼容性更好
  • 适合离线环境

另外建议检查你的 Ansible 版本

执行:

复制代码
ansible --version

如果是:

复制代码
ansible [core 2.15.x]

建议你的项目加:

requirements.yml

内容:

复制代码
collections:
  - name: ansible.posix
    version: ">=1.5.0"

以后安装:

复制代码
ansible-galaxy collection install -r requirements.yml

SeaweedFS Role 后面还会遇到一个类似问题

你现在是:

复制代码
systemd:

这个仍然可以用。

但是如果后面增加:

复制代码
selinux
firewalld
mount
lvg
filesystem

在新 Ansible 都建议写全:

例如:

复制代码
ansible.posix.selinux
ansible.posix.mount
community.general.lvg
community.general.filesystem

建议你当前直接执行:

复制代码
ansible-galaxy collection install ansible.posix

然后把 firewalld: 改成:

复制代码
ansible.posix.firewalld:

即可继续安装 SeaweedFS

2 ansible-galaxy: 未找到命令

ansible-galaxy collection install ansible.posix -bash: ansible-galaxy: 未找到命令

报错:

复制代码
复制代码
ansible-galaxy collection install ansible.posix
-bash: ansible-galaxy: 未找到命令

说明你的环境里 只有 ansible-playbook 命令,或者 Ansible 安装不完整 ,没有安装 ansible-galaxy

先确认:

复制代码
which ansible
which ansible-playbook
which ansible-galaxy

以及:

复制代码
ansible --version
ansible-playbook --version

情况1:你使用的是 pip 安装的 ansible-core

如果之前执行的是:

复制代码
pip3 install ansible-core

那么 ansible-galaxy 应该跟随安装。

检查:

复制代码
pip3 list | grep ansible

正常应该类似:

复制代码
ansible-core 2.15.x

如果有,但是命令不存在,可能 PATH 不对。

查位置:

复制代码
find / -name ansible-galaxy 2>/dev/null

例如找到:

复制代码
/usr/local/python3.9/bin/ansible-galaxy

加入 PATH:

复制代码
export PATH=$PATH:/usr/local/python3.9/bin

永久:

复制代码
echo 'export PATH=$PATH:/usr/local/python3.9/bin' >> ~/.bashrc
source ~/.bashrc

情况2:你装的是 ansible-core 最小包

如果:

复制代码
pip3 list | grep ansible

只有:

复制代码
ansible-core

理论也应该有:

复制代码
ansible-galaxy

如果没有,可以重新安装:

复制代码
pip3 install --force-reinstall ansible-core==2.15.13

验证:

复制代码
ansible-galaxy --version

情况3:你通过 yum/dnf 装的不完整

检查:

复制代码
rpm -qa | grep ansible

如果只有:

复制代码
ansible-core

可以安装完整 ansible:

复制代码
pip3 install ansible

或者:

复制代码
dnf install python3-ansible

(龙蜥仓库有时包名不同)


你当前最推荐处理方式

因为你是 龙蜥8.6 + Python3.6 + SeaweedFS Ansible部署环境,建议不要折腾系统包。

直接确认 Python:

复制代码
which python3
python3 --version

然后安装到独立 Python。

例如:

复制代码
/usr/local/python3.9/bin/pip3 install ansible-core==2.15.13

然后:

复制代码
/usr/local/python3.9/bin/ansible-galaxy collection install ansible.posix

临时绕过(最快继续部署 SeaweedFS)

你也可以不用 firewalld 模块。

把:

复制代码
firewalld:

改成:

复制代码
ansible.builtin.command:

例如:

复制代码
- name: 开放SeaweedFS端口
  command: >
    firewall-cmd
    --add-port={{ item }}/tcp
    --permanent
  loop:
    - 9333
    - 19333
    - 8080
    - 8888
    - 8333


- name: reload firewall
  command: firewall-cmd --reload

这样不用任何 collection,可以马上继续跑 playbook。


ansible-playbook执行报错

TASK seaweedfs : 开放SeaweedFS端口 ************************************************************************************************************************************************ failed: k8s-node2 (item=9333) => {"ansible_loop_var": "item", "changed": false, "item": 9333, "msg": "firewall is not currently running, unable to perform immediate actions without a running firewall daemon"} failed: k8s-node3 (item=9333) => {"ansible_loop_var": "item", "changed": false, "item": 9333, "msg": "firewall is not currently running, unable to perform immediate actions without a running firewall daemon"} failed: k8s-node1 (item=9333) => {"ansible_loop_var": "item", "changed": false, "item": 9333, "msg": "firewall is not currently running, unable to perform immediate actions without a running firewall daemon"} failed: k8s-node2 (item=19333) => {"ansible_loop_var": "item", "changed": false, "item": 19333, "msg": "firewall is not currently running, unable to perform immediate actions without a running firewall daemon"} failed: k8s-node3 (item=19333) => {"ansible_loop_var": "item", "changed": false, "item": 19333, "msg": "firewall is not currently running, unable to perform immediate actions without a running firewall daemon"} failed: k8s-node1 (item=19333) => {"ansible_loop_var": "item", "changed": false, "item": 19333, "msg": "firewall is not currently running, unable to perform immediate actions without a running firewall daemon"} failed: k8s-node2 (item=8080) => {"ansible_loop_var": "item", "changed": false, "item": 8080, "msg": "firewall is not currently running, unable to perform immediate actions without a running firewall daemon"} failed: k8s-node3 (item=8080) => {"ansible_loop_var": "item", "changed": false, "item": 8080, "msg": "firewall is not currently running, unable to perform immediate actions without a running firewall daemon"} failed: k8s-node2 (item=8888) => {"ansible_loop_var": "item", "changed": false, "item": 8888, "msg": "firewall is not currently running, unable to perform immediate actions without a running firewall daemon"} failed: k8s-node1 (item=8080) => {"ansible_loop_var": "item", "changed": false, "item": 8080, "msg": "firewall is not currently running, unable to perform immediate actions without a running firewall daemon"} failed: k8s-node3 (item=8888) => {"ansible_loop_var": "item", "changed": false, "item": 8888, "msg": "firewall is not currently running, unable to perform immediate actions without a running firewall daemon"} failed: k8s-node2 (item=8333) => {"ansible_loop_var": "item", "changed": false, "item": 8333, "msg": "firewall is not currently running, unable to perform immediate actions without a running firewall daemon"} failed: k8s-node3 (item=8333) => {"ansible_loop_var": "item", "changed": false, "item": 8333, "msg": "firewall is not currently running, unable to perform immediate actions without a running firewall daemon"} failed: k8s-node1 (item=8888) => {"ansible_loop_var": "item", "changed": false, "item": 8888, "msg": "firewall is not currently running, unable to perform immediate actions without a running firewall daemon"} failed: k8s-node1 (item=8333) => {"ansible_loop_var": "item", "changed": false, "item": 8333, "msg": "firewall is not currently running, unable to perform immediate actions without a running firewall daemon"} PLAY RECAP ************************************************************************************************************************************************************************** k8s-node1 : ok=9 changed=7 unreachable=0 failed=1 skipped=0 rescued=0 ignored=0 k8s-node2 : ok=6 changed=5 unreachable=0 failed=1 skipped=3 rescued=0 ignored=0 k8s-node3 : ok=6 changed=5 unreachable=0 failed=1 skipped=3 rescued=0 ignored=0

你的报错原因很明确:

复制代码
firewall is not currently running, unable to perform immediate actions without a running firewall daemon

说明:

目标节点 k8s-node1/2/3 上 firewalld 服务没有启动。

你的 Ansible 使用的是:

复制代码
ansible.posix.firewalld:
    port: "{{ item }}/tcp"
    permanent: yes
    state: enabled
    immediate: yes

其中:

复制代码
immediate: yes

要求:

复制代码
systemctl status firewalld

必须是运行状态。


方案一(推荐):启动 firewalld

在 playbook 前增加 firewalld 安装和启动。

修改:

roles/seaweedfs/tasks/main.yml

增加:

复制代码
- name: 安装firewalld
  dnf:
    name: firewalld
    state: present


- name: 启动firewalld
  systemd:
    name: firewalld
    state: started
    enabled: yes

放在开放端口任务之前。

例如:

复制代码
- name: 安装firewalld
  dnf:
    name: firewalld
    state: present


- name: 启动firewalld
  systemd:
    name: firewalld
    enabled: yes
    state: started


- name: 开放SeaweedFS端口
  ansible.posix.firewalld:
    port: "{{ item }}/tcp"
    permanent: yes
    state: enabled
    immediate: yes
  loop:
    - 9333
    - 19333
    - 8080
    - 8888
    - 8333

然后重新执行:

复制代码
ansible-playbook \
-i inventory/hosts \
site.yml

方案二(生产环境更推荐):关闭 firewalld 管理

如果你的集群节点:

  • 在内网
  • 使用安全组
  • 使用硬件防火墙
  • Kubernetes 已经管理网络

很多生产环境直接关闭 firewalld。

那么删除这一段:

复制代码
- name: 开放SeaweedFS端口
  ansible.posix.firewalld:

或者改成跳过:

复制代码
- name: 开放SeaweedFS端口
  ansible.posix.firewalld:
    port: "{{ item }}/tcp"
    permanent: yes
    state: enabled
  loop:
    - 9333
    - 19333
    - 8080
    - 8888
    - 8333
  when: false

方案三:不用 firewalld 模块(适合你的龙蜥环境)

你的节点看起来是:

复制代码
k8s-node1
k8s-node2
k8s-node3

可能已经关闭防火墙。

直接删除 firewalld task。

SeaweedFS只需要保证端口互通:

服务 端口
Master HTTP 9333
Master gRPC Raft 19333
Volume 8080
Filer 8888
S3 8333

测试:

复制代码
nc -vz k8s-node2 19333
nc -vz k8s-node3 19333

建议的 SeaweedFS Ansible Role 调整

因为你的目标是存储集群,不建议强制管理防火墙。

修改变量:

defaults/main.yml

增加:

复制代码
manage_firewalld: false

然后:

复制代码
- name: 开放SeaweedFS端口
  ansible.posix.firewalld:
    port: "{{ item }}/tcp"
    permanent: yes
    state: enabled
    immediate: yes
  loop:
    - 9333
    - 19333
    - 8080
    - 8888
    - 8333

  when:
    - manage_firewalld

以后:

生产环境:

复制代码
manage_firewalld: false

测试环境:

复制代码
manage_firewalld: true

最快恢复部署

直接执行:

复制代码
systemctl start firewalld
systemctl enable firewalld

三台节点:

复制代码
ansible seaweedfs -i inventory/hosts \
-m systemd \
-a "name=firewalld state=started enabled=yes"

然后:

复制代码
ansible-playbook \
-i inventory/hosts \
site.yml

不过我注意到你的节点名称已经变成:

复制代码
k8s-node1
k8s-node2
k8s-node3

不是之前:

复制代码
fsnode01
fsnode02
fsnode03

下一步部署 SeaweedFS Master 时,需要同步修改:

复制代码
master_peers:
  k8s-node1:9333,
  k8s-node2:9333,
  k8s-node3:9333

否则后面 Raft 会出现你之前遇到的:

复制代码
raft.Server: Not current leader

3 ansible-playbook执行报错:主机名解析失败

/usr/local/bin/weed master \ > -mdir=/mnt/allin_data/allin/seaweedfs/master/m9333 \ > -ip=k8s-node3 \ > -port=9333 \ > -port.grpc=19333 \ > -peers=k8s-node1:9333,k8s-node2:9333,k8s-node3:9333 I0904 16:01:48.528425 file_util.go:29 Folder /mnt/allin_data/allin/seaweedfs/master/m9333 Permission: -rwxr-xr-x I0904 16:01:48.528910 master.go:394 current: k8s-node3:9333 peers:k8s-node1:9333,k8s-node2:9333,k8s-node3:9333 I0904 16:01:48.529062 master_server.go:156 Volume Size Limit is 30720 MB I0904 16:01:48.529123 collector.go:83 Reporting anonymous cluster statistics to https://telemetry.seaweedfs.com/api/collect every 24h0m0s, use -telemetry=false to opt out I0904 16:01:48.529343 master.go:203 Start Seaweed Master 30GB 4.45 79b87202136cebdaaa7db4d94eaa5915ad381276 at k8s-node3:9333 F0904 16:01:48.530369 master.go:206 Master startup error: listen tcp: lookup k8s-node3 on 10.80.0.2:53: no such host goroutine 1 running: github.com/seaweedfs/seaweedfs/weed/glog.stacks(0x0) /github/workspace/weed/glog/glog.go:832 +0x85 github.com/seaweedfs/seaweedfs/weed/glog.(*loggingT).output(0x8dabce0, 0x3, 0x39c03801b960, {0x566ff77?, 0x39c0382c9a28?}, 0x1?, 0x0) /github/workspace/weed/glog/glog.go:783 +0x43d github.com/seaweedfs/seaweedfs/weed/glog.(*loggingT).printf(0x8dabce0, 0x3, {0x413f3de, 0x18}, {0x39c0382c9a28, 0x1, 0x1}) /github/workspace/weed/glog/glog.go:680 +0x12a github.com/seaweedfs/seaweedfs/weed/glog.Fatalf(...) /github/workspace/weed/glog/glog.go:1242 github.com/seaweedfs/seaweedfs/weed/command.startMaster({0x39c038507ed8, 0x39c038507f10, 0x39c038464830, 0x39c038464850, 0x39c038464860, 0x39c038464880, 0x0, 0x39c03857c280, 0x39c03857c295, 0x39c03857c2e0, ...}, ...) /github/workspace/weed/command/master.go:206 +0x3d9 github.com/seaweedfs/seaweedfs/weed/command.runMaster(0x8d74790?, {0x39c03762c100?, 0x5?, 0x5?}) /github/workspace/weed/command/master.go:177 +0x667 main.main() /github/workspace/weed/weed.go:97 +0x469 root@k8s-node3 \~#

报错已经定位,不是 SeaweedFS 参数问题,而是 主机名解析失败

关键错误:

复制代码
Master startup error: listen tcp: lookup k8s-node3 on 10.80.0.2:53: no such host

意思:

SeaweedFS 执行:

复制代码
-ip=k8s-node3

启动监听时,需要解析:

复制代码
k8s-node3

但是 DNS:

复制代码
10.80.0.2

无法解析这个 hostname。


解决方案一(推荐):配置 /etc/hosts

三台节点都配置。

k8s-node1

复制代码
vim /etc/hosts

增加:

复制代码
10.40.0.77 k8s-node1
10.40.0.78 k8s-node2
10.40.0.79 k8s-node3

k8s-node2

同样:

复制代码
10.40.0.77 k8s-node1
10.40.0.78 k8s-node2
10.40.0.79 k8s-node3

k8s-node3

同样:

复制代码
10.40.0.77 k8s-node1
10.40.0.78 k8s-node2
10.40.0.79 k8s-node3

测试:

复制代码
ping k8s-node1
ping k8s-node2
ping k8s-node3

应该返回对应 IP。


解决方案二:直接使用 IP(更稳定)

SeaweedFS Master 集群生产环境建议使用 IP。

例如:

k8s-node3

改:

复制代码
-ip=k8s-node3

为:

复制代码
-ip=10.40.0.79

完整:

复制代码
/usr/local/bin/weed master \
-mdir=/mnt/allin_data/allin/seaweedfs/master/m9333 \
-ip=10.40.0.79 \
-port=9333 \
-port.grpc=19333 \
-peers=10.40.0.77:9333,10.40.0.78:9333,10.40.0.79:9333

另外两个节点:

k8s-node1

复制代码
weed master \
-mdir=/mnt/allin_data/allin/seaweedfs/master/m9333 \
-ip=10.40.0.77 \
-port=9333 \
-port.grpc=19333 \
-peers=10.40.0.77:9333,10.40.0.78:9333,10.40.0.79:9333

k8s-node2

复制代码
weed master \
-mdir=/mnt/allin_data/allin/seaweedfs/master/m9333 \
-ip=10.40.0.78 \
-port=9333 \
-port.grpc=19333 \
-peers=10.40.0.77:9333,10.40.0.78:9333,10.40.0.79:9333

另外检查你的 Ansible inventory

你现在应该不要写:

复制代码
[master]
k8s-node1
k8s-node2
k8s-node3

然后变量:

复制代码
master_peers: k8s-node1:9333,...

除非你保证 /etc/hosts

建议改成 IP:

defaults/main.yml

复制代码
master_peers: "10.40.0.77:9333,10.40.0.78:9333,10.40.0.79:9333"

然后模板:

复制代码
ExecStart={{weed_bin}} master \
-mdir={{master_dir}} \
-ip={{ansible_host}} \
-port=9333 \
-port.grpc=19333 \
-peers={{master_peers}}

这样 Ansible 部署不会依赖 DNS。


当前隐藏问题

之前你删除过:

复制代码
/mnt/allin_data/allin/seaweedfs/master/m9333

现在重新初始化时:

启动顺序必须:

1. k8s-node1

先启动:

复制代码
weed master ...

看到:

复制代码
is the leader

保持运行。

2. k8s-node2

3. k8s-node3

然后验证:

复制代码
curl http://10.40.0.77:9333/cluster/status

正常应该看到 Leader。


建议你现在先执行:

复制代码
cat /etc/hosts
hostname -I

三台都检查一下。

你的问题目前 99% 是 k8s-node1/2/3 没有做 hosts 解析

相关推荐
Tassel_YUE2 天前
非 ansible 主机批量执行命令方法:psssh 和 pscp(随手记)
linux·网络·ssh·scp·ansible
亿朝亿夕11 天前
ansible安装
ansible
.柒宇.12 天前
运维常见面试题_07_配置管理与监控(Ansible · Prometheus · Alertmanager)· 容器编排(Kubernetes)
运维·k8s·ansible·prometheus
相顾若初见12 天前
Ansible部署k8s
容器·kubernetes·ansible
wzq11_66612 天前
ansible部署项目——基于tidb数据库安装zabbix_server
ansible·zabbix·tidb
吃不吃早饭13 天前
RHEL9 搭建 Harbor 私有仓库与 Ansible 自动化部署 Kubernetes 运行环境完整实践
kubernetes·自动化·ansible
wzq11_66613 天前
Ansible自动部署Redis——六步自动安装Redis数据库
ansible
mohesashou13 天前
红帽linux的K8S部署
linux·kubernetes·ansible
lxw202302711614 天前
k8s安装部署(利用ansible)
linux·kubernetes·ansible