Fastdfs V6.12.1集群部署(arm/x86均可用)

礁之2024-12-21 17:33

文章目录

一、 Fastdfs 介绍

FastDFS 是一款高性能的分布式文件系统，特别适合用于存储和管理大量的文件

二、部署 信息

  • 使用 docker 部署 Fastdfs 集群
角色 服务器 IP 服务器系统 服务器配置 docker /docker-compose版本
tracker/storage 10.10.11.35 麒麟 V10 127G + 96C 18.09.0/v2.25.0
tracker/storage 10.10.11.56 麒麟 V10 254G + 64C 20.10.17/v2.5.0
  • 默认 tracker 端口为 22122， storage 端口为 23000，文件访问端口为 8080
  • 此文档使用镜像在 dockerhub 上有，x86 与 arm 双架构均存在，可以直接 pull

三、步骤

tracker/storage 机器的 compose 内容

  • 由于此容器内部启动使用 sed，所以根据内部命令来启动，编辑环境变量以及域名解析，多个 tracker 使用 \n 换行符实现
yaml 复制代码 
version: '3.8'
services:
  tracker:
    container_name: tracker
    image: ygqygq2/fastdfs-nginx:V6.12.1
    command: tracker
    network_mode: host
    privileged: true
    restart: always
    extra_hosts:
      - "tracker1:10.10.11.56"
      - "tracker2:10.10.11.35"
    environment:
      - TRACKER_SERVER=tracker1:22122\ntracker_server = tracker2:22122
      - GROUP_NAME=group111
    volumes:
      - ./data/tracker:/var/fdfs
      - /etc/localtime:/etc/localtime
  storage:
    container_name: storage
    image: ygqygq2/fastdfs-nginx:V6.12.1
    command: storage
    network_mode: host
    privileged: true
    restart: always
    extra_hosts:
      - "tracker1:10.10.11.56"
      - "tracker2:10.10.11.35"
    environment:
      - TRACKER_SERVER=tracker1:22122\ntracker_server = tracker2:22122
      - GROUP_NAME=group111
    volumes:
      - ./data/storage:/var/fdfs/
      - /etc/localtime:/etc/localtime

storage 机器的 compose

yaml 复制代码 
version: '3.8'
services:
  storage:
    container_name: storage
    image: ygqygq2/fastdfs-nginx:V6.12.1
    command: storage
    network_mode: host
    privileged: true
    restart: always
    extra_hosts:
      - "tracker1:10.10.11.56"
      - "tracker2:10.10.11.35"
    environment:
      - TRACKER_SERVER=tracker1:22122\ntracker_server = tracker2:22122
      - GROUP_NAME=group111
    volumes:
      - ./data/storage:/var/fdfs/
      - /etc/localtime:/etc/localtime

tracker 与 storage 启动

shell 复制代码 
# 在142上查看 storage 日志，可以看到会输出其他节点连接成功的信息
docker  logs -f storage
[2024-12-16 16:50:46] INFO - file: storage_func.c, line: 338, tracker_client_ip: 10.10.11.35, my_server_id_str: 10.10.11.35, g_server_id_in_filename: 587926026
[2024-12-16 16:50:46] INFO - file: tracker_client_thread.c, line: 315, successfully connect to tracker server 10.10.11.56:22122, as a tracker client, my ip is 10.10.11.35
[2024-12-16 16:50:46] INFO - file: tracker_client_thread.c, line: 2077, tracker server: #0. 10.10.11.35:22122, my_report_result: -1
[2024-12-16 16:50:46] INFO - file: tracker_client_thread.c, line: 315, successfully connect to tracker server 10.10.11.35:22122, as a tracker client, my ip is 10.10.11.35
[2024-12-16 16:50:47] INFO - file: tracker_client_thread.c, line: 2077, tracker server: #0. 10.10.11.35:22122, my_report_result: -1
[2024-12-16 16:51:16] INFO - file: tracker_client_thread.c, line: 1441, tracker server 10.10.11.56:22122, set tracker leader: 10.10.11.56:22122
[2024-12-16 16:51:16] INFO - file: tracker_client_thread.c, line: 790, my status: 6 (ONLINE) from tracker #0  != my status: 0 (INIT) from leader tracker #1, set report_my_status to true
[2024-12-16 16:51:16] INFO - file: storage_sync_func.c, line: 126, successfully connect to storage server 10.10.11.56:23000
[2024-12-16 16:51:16] INFO - file: tracker_client_thread.c, line: 2484, tracker server: 10.10.11.35:22122, try to set storage status from 6 (ONLINE) to 7 (ACTIVE)
[2024-12-16 16:51:17] INFO - file: tracker_client_thread.c, line: 315, successfully connect to tracker server 10.10.11.35:22122, as a tracker client, my ip is 10.10.11.35
[2024-12-16 16:51:46] INFO - file: storage_sync_func.c, line: 126, successfully connect to storage server 10.10.11.56:23000

# 在139查看tracker日志
[2024-12-16 16:50:45] INFO - FastDFS v6.12.1, base_path=/var/fdfs, max_connections=1024, connect_timeout=5, network_timeout=60, thread_stack_size=256 KB, max_pkg_size=256 KB, min_buff_size=8 KB, max_buff_size=256 KB, tcp_quick_ack=1, log_level=INFO, run_by_group=, run_by_user=, error-log: {sync_log_buff_interval=1, rotate_everyday=1, rotate_time=00:00, rotate_on_size=0, compress_old=0, compress_days_before=1, keep_days=0, delete_old_time=01:30}, port=22122, bind_addr=, address_family=auto, accept_threads=1, work_threads=4, store_lookup=2, store_group=, store_server=0, store_path=0, reserved_storage_space=1.00%, download_server=0, allow_ip_count=-1, check_active_interval=120s, storage_ip_changed_auto_adjust=1, storage_sync_file_max_delay=86400s, storage_sync_file_max_time=300s, use_trunk_file=0, slot_min_size=256, slot_max_size=1024 KB, trunk_alloc_alignment_size=256, trunk_file_size=64 MB, trunk_create_file_advance=0, trunk_create_file_time_base=02:00, trunk_create_file_interval=86400, trunk_create_file_space_threshold=20 GB, trunk_init_check_occupying=0, trunk_init_reload_from_binlog=0, trunk_free_space_merge=1, delete_unused_trunk_files=0, trunk_compress_binlog_min_interval=86400, trunk_compress_binlog_interval=86400, trunk_compress_binlog_time_base=03:00, trunk_binlog_max_backups=7, use_storage_id=0, id_type_in_filename=ip, trust_storage_server_id=1, storage_id/ip_count=0 / 0, store_slave_file_use_link=0, use_connection_pool=0, g_connection_pool_max_idle_time=3600s
[2024-12-16 16:50:46] ERROR - file: tracker_service.c, line: 686, client ip: 10.10.11.56, leader 10.10.11.56:22122 not exist
[2024-12-16 16:50:47] INFO - file: tracker_mem.c, line: 4393, sys files loaded from tracker server 10.10.11.56:22122
[2024-12-16 16:50:47] INFO - file: tracker_relationship.c, line: 356, the tracker leader is 10.10.11.56:22122
[2024-12-16 16:51:15] WARNING - file: tracker_mem.c, line: 4963, storage server: 10.10.11.56:23000, dest status: 2, my status: 1, should change my status!
[2024-12-16 16:51:15] WARNING - file: tracker_mem.c, line: 4963, storage server: 10.10.11.56:23000, dest status: 5, my status: 2, should change my status!
[2024-12-16 16:51:16] INFO - file: tracker_service.c, line: 1229, client ip: 10.10.11.35, do NOT change storage status, old status: 7 (ACTIVE), new status: 7 (ACTIVE)

目录层级与配置文件

shell 复制代码 
# 目录层级如下
data
  ├── storage
  └── tracker
      ├── data
      └── logs

查看集群信息

shell 复制代码 
# 查看当前集群信息，任意节点的 storage 容器查看即可	
docker exec -it storage fdfs_monitor /etc/fdfs/client.conf  
# server_count = 2 表示 tracker 有两个
server_count=2, server_index=0

tracker server is 10.10.11.35:22122

group count: 1

Group 1:
group name = group111
disk total space = 400,088 MB
disk free space = 15,674 MB
trunk free space = 0 MB
storage server count = 2
active server count = 2
storage server port = 23000
storage HTTP port = 8080
store path count = 1
subdir count per path = 256
current write server index = 0
current trunk file id = 0
。。。。。。
Storage 1:  # 表示第一个storage的信息
                id = 10.10.11.35
                ip_addr = 10.10.11.35  ACTIVE
。。。。。。
Storage 2:  # 第二个storage的信息
                id = 10.10.11.56
                ip_addr = 10.10.11.56  ACTIVE

测试

shell 复制代码 
# 通过访问tracker接口上传，但有可能会报错文件过大
curl -F "file=@./test.txt" http://10.10.11.140:22122/upload

# 通过容器内部命令上传
docker exec -it storage bash 
cd                      
echo 11111 > test.txt 
fdfs_test /etc/fdfs/client.conf upload test.txt

This is FastDFS client test program v6.12.1

Copyright (C) 2008, Happy Fish / YuQing

FastDFS may be copied only under the terms of the GNU General
Public License V3, which may be found in the FastDFS source kit.
Please visit the FastDFS Home Page http://www.fastken.com/ 
for more detail.

tracker_query_storage_store_list_without_group: 
        server 1. group_name=, ip_addr=10.10.11.35, port=23000
        server 2. group_name=, ip_addr=10.10.11.56, port=23000

group_name=group111, ip_addr=10.10.11.56, port=23000
storage_upload_by_filename
group_name=group111, remote_filename=M00/00/00/CgoLOGdf6m6AeFMrAAAABqkTGlI497.txt
source ip address: 10.10.11.56
file timestamp=2024-12-16 16:53:02
file size=6
file crc32=2836601426
example file url: http://10.10.11.56/group111/M00/00/00/CgoLOGdf6m6AeFMrAAAABqkTGlI497.txt
storage_upload_slave_by_filename
group_name=group111, remote_filename=M00/00/00/CgoLOGdf6m6AeFMrAAAABqkTGlI497_big.txt
source ip address: 10.10.11.56
file timestamp=2024-12-16 16:53:02
file size=6
file crc32=2836601426
example file url: http://10.10.11.56/group111/M00/00/00/CgoLOGdf6m6AeFMrAAAABqkTGlI497_big.txt
root@localhost:~# curl http://10.10.11.56:8080/group111/M00/00/00/CgoLOGdf6m6AeFMrAAAABqkTGlI497_big.txt
11111
root@localhost:~# curl http://10.10.11.35:8080/group111/M00/00/00/CgoLOGdf6m6AeFMrAAAABqkTGlI497_big.txt
11111
  • fastdfs 集群直接的 storage 会自动同步数据，更换 ip 再次访问，也可以成功访问

测试集群扩容与缩减

  • 扩容：目前有两种方法，扩容容量需要增加硬盘，挂载分区，扩容节点需要增加新的 group，修改新节点的配置后就可以加入集群，原有集群有数据后是无法直接增加节点的，除了上面增加 group 的方法就只能清除数据重新部署节点，所以推荐在部署集群之前最好先确认架构，防止后续因为架构调整带来的不必要的麻烦
  • 缩减：由于集群会自动同步数据，要缩减直接直接关闭storage、修改nginx代理即可

nginx 负载配置（示例配置）

nginx 复制代码 
upstream storage {
   server 10.10.11.56:23000;
   server 10.10.11.35:23000;
}

upstream storage_file {
   server 10.10.11.56:8080;
   server 10.10.11.35:8080;
}
upstream tracker {
   server 10.10.11.56:22122;
   server 10.10.11.35:22122;
}

server {
    listen 23001 ;
    server_name _;

    location / {
        proxy_set_header Host $http_host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_set_header X-Forwarded-Host $http_host;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_pass http://storage;
    }
}

server {
    listen 8889;
    server_name _;
    location / {
        proxy_set_header Host $http_host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_set_header X-Forwarded-Host $http_host;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_pass http://storage_file;
    }
}

server {
    listen 22123;
    server_name _;
    location / {
        proxy_set_header Host $http_host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_set_header X-Forwarded-Host $http_host;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_pass http://tracker;
    }
}

注意事项

  • fastdfs 默认保留的磁盘空间为 10%，当至少有一个节点的磁盘存储小于或等于 10%时，文件无法写入，可以更改保留空间，修改 tracker 容器中 /etc/fdfs/tracker.conf，进入容器使用 echo "reserved_storage_space = 1%" >> /etc/fdfs/tracker.conf 追加，所有的 tracker 都需要更改，然后单独重启 tracker，清空原有数据，即 data 下的所有内容，重启 storage 即可
  • 如果遇到一直不同步的清空，则删除 等待节点，重新启动，或删除全部 数据，重新创建容器
相关推荐
Jack_Kuo29 分钟前
【docker】如何打包前端并运行
前端·docker·容器
終不似少年遊*1 小时前
云计算HCIP-OpenStack01
云原生·云计算·学习笔记·openstack·hcip·虚拟化
Adellle1 小时前
docker容器镜像拉取失败解决方案
java·docker·容器
大熊程序猿1 小时前
apisix docker 安装
运维·docker·容器
于顾而言1 小时前
【Enjoy Kubernetes】1. 基础入门
开发语言·云原生·k8s
Karoku0662 小时前
【自动化部署】Ansible循环
linux·运维·数据库·docker·容器·自动化·ansible
杨浦老苏2 小时前
命令行音乐库管理工具Beets
docker·群晖·多媒体
m0_515790412 小时前
docker打包镜像并迁移：如何从A服务器打包docker镜像到B服务器上容器中运行
服务器·docker
KeyBordkiller3 小时前
OpenWRT——官方镜像安装Docker(网络环境需设置)并配置Sun-Panel
运维·docker·容器
热门推荐
01HCIA-datacom数通题库和录播视频资料02文心一言最新重磅发布！03(欧拉)openEuler系统添加网卡文件配置流程、(欧拉)openEuler系统手动配置ipv6地址流程、(欧拉)openEuler系统网络管理说明04MacBook外接键盘修改键位05如何解决电脑无声问题：排除故障的几种常见方法06玄机平台应急响应—webshell查杀07校验 GPT-4 真实性的三个经典问题：快速区分 GPT-3.5 与 GPT-4，并提供免费测试网站08WPS-0DAY-20230809的分析和利用复现09Docker 夺命连环 15 问10怎样让音频速度变慢？请跟随以下方法进行操作