Fastdfs V6.12.1集群部署(arm/x86均可用)

礁之2024-12-21 17:33

文章目录

一、 Fastdfs 介绍

FastDFS 是一款高性能的分布式文件系统，特别适合用于存储和管理大量的文件

二、部署 信息

  • 使用 docker 部署 Fastdfs 集群
角色 服务器 IP 服务器系统 服务器配置 docker /docker-compose版本
tracker/storage 10.10.11.35 麒麟 V10 127G + 96C 18.09.0/v2.25.0
tracker/storage 10.10.11.56 麒麟 V10 254G + 64C 20.10.17/v2.5.0
  • 默认 tracker 端口为 22122， storage 端口为 23000，文件访问端口为 8080
  • 此文档使用镜像在 dockerhub 上有，x86 与 arm 双架构均存在，可以直接 pull

三、步骤

tracker/storage 机器的 compose 内容

  • 由于此容器内部启动使用 sed，所以根据内部命令来启动，编辑环境变量以及域名解析，多个 tracker 使用 \n 换行符实现
yaml 复制代码 
version: '3.8'
services:
  tracker:
    container_name: tracker
    image: ygqygq2/fastdfs-nginx:V6.12.1
    command: tracker
    network_mode: host
    privileged: true
    restart: always
    extra_hosts:
      - "tracker1:10.10.11.56"
      - "tracker2:10.10.11.35"
    environment:
      - TRACKER_SERVER=tracker1:22122\ntracker_server = tracker2:22122
      - GROUP_NAME=group111
    volumes:
      - ./data/tracker:/var/fdfs
      - /etc/localtime:/etc/localtime
  storage:
    container_name: storage
    image: ygqygq2/fastdfs-nginx:V6.12.1
    command: storage
    network_mode: host
    privileged: true
    restart: always
    extra_hosts:
      - "tracker1:10.10.11.56"
      - "tracker2:10.10.11.35"
    environment:
      - TRACKER_SERVER=tracker1:22122\ntracker_server = tracker2:22122
      - GROUP_NAME=group111
    volumes:
      - ./data/storage:/var/fdfs/
      - /etc/localtime:/etc/localtime

storage 机器的 compose

yaml 复制代码 
version: '3.8'
services:
  storage:
    container_name: storage
    image: ygqygq2/fastdfs-nginx:V6.12.1
    command: storage
    network_mode: host
    privileged: true
    restart: always
    extra_hosts:
      - "tracker1:10.10.11.56"
      - "tracker2:10.10.11.35"
    environment:
      - TRACKER_SERVER=tracker1:22122\ntracker_server = tracker2:22122
      - GROUP_NAME=group111
    volumes:
      - ./data/storage:/var/fdfs/
      - /etc/localtime:/etc/localtime

tracker 与 storage 启动

shell 复制代码 
# 在142上查看 storage 日志，可以看到会输出其他节点连接成功的信息
docker  logs -f storage
[2024-12-16 16:50:46] INFO - file: storage_func.c, line: 338, tracker_client_ip: 10.10.11.35, my_server_id_str: 10.10.11.35, g_server_id_in_filename: 587926026
[2024-12-16 16:50:46] INFO - file: tracker_client_thread.c, line: 315, successfully connect to tracker server 10.10.11.56:22122, as a tracker client, my ip is 10.10.11.35
[2024-12-16 16:50:46] INFO - file: tracker_client_thread.c, line: 2077, tracker server: #0. 10.10.11.35:22122, my_report_result: -1
[2024-12-16 16:50:46] INFO - file: tracker_client_thread.c, line: 315, successfully connect to tracker server 10.10.11.35:22122, as a tracker client, my ip is 10.10.11.35
[2024-12-16 16:50:47] INFO - file: tracker_client_thread.c, line: 2077, tracker server: #0. 10.10.11.35:22122, my_report_result: -1
[2024-12-16 16:51:16] INFO - file: tracker_client_thread.c, line: 1441, tracker server 10.10.11.56:22122, set tracker leader: 10.10.11.56:22122
[2024-12-16 16:51:16] INFO - file: tracker_client_thread.c, line: 790, my status: 6 (ONLINE) from tracker #0  != my status: 0 (INIT) from leader tracker #1, set report_my_status to true
[2024-12-16 16:51:16] INFO - file: storage_sync_func.c, line: 126, successfully connect to storage server 10.10.11.56:23000
[2024-12-16 16:51:16] INFO - file: tracker_client_thread.c, line: 2484, tracker server: 10.10.11.35:22122, try to set storage status from 6 (ONLINE) to 7 (ACTIVE)
[2024-12-16 16:51:17] INFO - file: tracker_client_thread.c, line: 315, successfully connect to tracker server 10.10.11.35:22122, as a tracker client, my ip is 10.10.11.35
[2024-12-16 16:51:46] INFO - file: storage_sync_func.c, line: 126, successfully connect to storage server 10.10.11.56:23000

# 在139查看tracker日志
[2024-12-16 16:50:45] INFO - FastDFS v6.12.1, base_path=/var/fdfs, max_connections=1024, connect_timeout=5, network_timeout=60, thread_stack_size=256 KB, max_pkg_size=256 KB, min_buff_size=8 KB, max_buff_size=256 KB, tcp_quick_ack=1, log_level=INFO, run_by_group=, run_by_user=, error-log: {sync_log_buff_interval=1, rotate_everyday=1, rotate_time=00:00, rotate_on_size=0, compress_old=0, compress_days_before=1, keep_days=0, delete_old_time=01:30}, port=22122, bind_addr=, address_family=auto, accept_threads=1, work_threads=4, store_lookup=2, store_group=, store_server=0, store_path=0, reserved_storage_space=1.00%, download_server=0, allow_ip_count=-1, check_active_interval=120s, storage_ip_changed_auto_adjust=1, storage_sync_file_max_delay=86400s, storage_sync_file_max_time=300s, use_trunk_file=0, slot_min_size=256, slot_max_size=1024 KB, trunk_alloc_alignment_size=256, trunk_file_size=64 MB, trunk_create_file_advance=0, trunk_create_file_time_base=02:00, trunk_create_file_interval=86400, trunk_create_file_space_threshold=20 GB, trunk_init_check_occupying=0, trunk_init_reload_from_binlog=0, trunk_free_space_merge=1, delete_unused_trunk_files=0, trunk_compress_binlog_min_interval=86400, trunk_compress_binlog_interval=86400, trunk_compress_binlog_time_base=03:00, trunk_binlog_max_backups=7, use_storage_id=0, id_type_in_filename=ip, trust_storage_server_id=1, storage_id/ip_count=0 / 0, store_slave_file_use_link=0, use_connection_pool=0, g_connection_pool_max_idle_time=3600s
[2024-12-16 16:50:46] ERROR - file: tracker_service.c, line: 686, client ip: 10.10.11.56, leader 10.10.11.56:22122 not exist
[2024-12-16 16:50:47] INFO - file: tracker_mem.c, line: 4393, sys files loaded from tracker server 10.10.11.56:22122
[2024-12-16 16:50:47] INFO - file: tracker_relationship.c, line: 356, the tracker leader is 10.10.11.56:22122
[2024-12-16 16:51:15] WARNING - file: tracker_mem.c, line: 4963, storage server: 10.10.11.56:23000, dest status: 2, my status: 1, should change my status!
[2024-12-16 16:51:15] WARNING - file: tracker_mem.c, line: 4963, storage server: 10.10.11.56:23000, dest status: 5, my status: 2, should change my status!
[2024-12-16 16:51:16] INFO - file: tracker_service.c, line: 1229, client ip: 10.10.11.35, do NOT change storage status, old status: 7 (ACTIVE), new status: 7 (ACTIVE)

目录层级与配置文件

shell 复制代码 
# 目录层级如下
data
  ├── storage
  └── tracker
      ├── data
      └── logs

查看集群信息

shell 复制代码 
# 查看当前集群信息，任意节点的 storage 容器查看即可	
docker exec -it storage fdfs_monitor /etc/fdfs/client.conf  
# server_count = 2 表示 tracker 有两个
server_count=2, server_index=0

tracker server is 10.10.11.35:22122

group count: 1

Group 1:
group name = group111
disk total space = 400,088 MB
disk free space = 15,674 MB
trunk free space = 0 MB
storage server count = 2
active server count = 2
storage server port = 23000
storage HTTP port = 8080
store path count = 1
subdir count per path = 256
current write server index = 0
current trunk file id = 0
。。。。。。
Storage 1:  # 表示第一个storage的信息
                id = 10.10.11.35
                ip_addr = 10.10.11.35  ACTIVE
。。。。。。
Storage 2:  # 第二个storage的信息
                id = 10.10.11.56
                ip_addr = 10.10.11.56  ACTIVE

测试

shell 复制代码 
# 通过访问tracker接口上传，但有可能会报错文件过大
curl -F "file=@./test.txt" http://10.10.11.140:22122/upload

# 通过容器内部命令上传
docker exec -it storage bash 
cd                      
echo 11111 > test.txt 
fdfs_test /etc/fdfs/client.conf upload test.txt

This is FastDFS client test program v6.12.1

Copyright (C) 2008, Happy Fish / YuQing

FastDFS may be copied only under the terms of the GNU General
Public License V3, which may be found in the FastDFS source kit.
Please visit the FastDFS Home Page http://www.fastken.com/ 
for more detail.

tracker_query_storage_store_list_without_group: 
        server 1. group_name=, ip_addr=10.10.11.35, port=23000
        server 2. group_name=, ip_addr=10.10.11.56, port=23000

group_name=group111, ip_addr=10.10.11.56, port=23000
storage_upload_by_filename
group_name=group111, remote_filename=M00/00/00/CgoLOGdf6m6AeFMrAAAABqkTGlI497.txt
source ip address: 10.10.11.56
file timestamp=2024-12-16 16:53:02
file size=6
file crc32=2836601426
example file url: http://10.10.11.56/group111/M00/00/00/CgoLOGdf6m6AeFMrAAAABqkTGlI497.txt
storage_upload_slave_by_filename
group_name=group111, remote_filename=M00/00/00/CgoLOGdf6m6AeFMrAAAABqkTGlI497_big.txt
source ip address: 10.10.11.56
file timestamp=2024-12-16 16:53:02
file size=6
file crc32=2836601426
example file url: http://10.10.11.56/group111/M00/00/00/CgoLOGdf6m6AeFMrAAAABqkTGlI497_big.txt
root@localhost:~# curl http://10.10.11.56:8080/group111/M00/00/00/CgoLOGdf6m6AeFMrAAAABqkTGlI497_big.txt
11111
root@localhost:~# curl http://10.10.11.35:8080/group111/M00/00/00/CgoLOGdf6m6AeFMrAAAABqkTGlI497_big.txt
11111
  • fastdfs 集群直接的 storage 会自动同步数据，更换 ip 再次访问，也可以成功访问

测试集群扩容与缩减

  • 扩容：目前有两种方法，扩容容量需要增加硬盘，挂载分区，扩容节点需要增加新的 group，修改新节点的配置后就可以加入集群，原有集群有数据后是无法直接增加节点的，除了上面增加 group 的方法就只能清除数据重新部署节点，所以推荐在部署集群之前最好先确认架构，防止后续因为架构调整带来的不必要的麻烦
  • 缩减：由于集群会自动同步数据，要缩减直接直接关闭storage、修改nginx代理即可

nginx 负载配置（示例配置）

nginx 复制代码 
upstream storage {
   server 10.10.11.56:23000;
   server 10.10.11.35:23000;
}

upstream storage_file {
   server 10.10.11.56:8080;
   server 10.10.11.35:8080;
}
upstream tracker {
   server 10.10.11.56:22122;
   server 10.10.11.35:22122;
}

server {
    listen 23001 ;
    server_name _;

    location / {
        proxy_set_header Host $http_host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_set_header X-Forwarded-Host $http_host;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_pass http://storage;
    }
}

server {
    listen 8889;
    server_name _;
    location / {
        proxy_set_header Host $http_host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_set_header X-Forwarded-Host $http_host;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_pass http://storage_file;
    }
}

server {
    listen 22123;
    server_name _;
    location / {
        proxy_set_header Host $http_host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_set_header X-Forwarded-Host $http_host;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_pass http://tracker;
    }
}

注意事项

  • fastdfs 默认保留的磁盘空间为 10%，当至少有一个节点的磁盘存储小于或等于 10%时，文件无法写入，可以更改保留空间，修改 tracker 容器中 /etc/fdfs/tracker.conf，进入容器使用 echo "reserved_storage_space = 1%" >> /etc/fdfs/tracker.conf 追加，所有的 tracker 都需要更改，然后单独重启 tracker，清空原有数据，即 data 下的所有内容，重启 storage 即可
  • 如果遇到一直不同步的清空，则删除 等待节点，重新启动，或删除全部 数据，重新创建容器
相关推荐
eight *38 分钟前
docker部署elk+filebeat日志收集分析系统
elk·docker·容器
小股虫1 小时前
分布式事务：在增长中台，我们如何做到“发出去的内容”和“记录的数据”不打架？
分布式·微服务·云原生·架构·团队建设·方法论
自己的九又四分之三站台2 小时前
docker安装pgvector、age和postgis
运维·docker·容器
忧郁蓝调262 小时前
Redis不停机数据迁移：基于 redis-shake 的跨实例 / 跨集群同步方案
运维·数据库·redis·阿里云·缓存·云原生·paas
java1234_小锋3 小时前
ZooKeeper集群中服务器之间是怎样通信的？
分布式·zookeeper·云原生
幺零九零零3 小时前
Docker底层-IPC Namespace（进程间通信隔离）
运维·docker·容器
easy_coder3 小时前
从“未知故障”到“自治诊断”：基于双路召回与RAG的智能诊断系统构建
人工智能·云原生·云计算
eddy-原4 小时前
ELKStack 与 Kubernetes 核心基础知识点综合作业
云原生·容器·kubernetes
V胡桃夹子4 小时前
Docker快速部署apollo
运维·docker·容器
oMcLin5 小时前
如何利用 Podman 替代 Docker：无 root 权限的容器管理实践
docker·dubbo·podman
热门推荐
01GitHub 镜像站点02Linux下V2Ray安装配置指南03jdk21下载、安装（Windows、Linux、macOS）04Claude Code Skills 实用使用手册05从快手“12·22”直播攻击事件看：一次教科书式的业务层饱和攻击06UV安装并设置国内源072025 最新教程：注册并切换到美区 Apple ID08【踩坑笔记】50系显卡适配的 PyTorch 安装09手把手教你通过Gemini3 pro 学生认证，白用一年，手慢无！10祝大家 2026 年新年快乐，代码无 bug，需求一次过