最近有个需求是要下载ftp接近十个T的数据，在调研过多个工具后发现还是lftp的mirror最省事
mirror参数
Mirror specified source directory to local target directory. If target directory ends with a slash, the source base name is appended to target directory name. Source
and/or target can be URLs pointing to directories.
-c, --continue continue a mirror job if possible
-e, --delete delete files not present at remote site
--delete-first delete old files before transferring new ones
--depth-first descend into subdirectories before transferring files
-s, --allow-suid set suid/sgid bits according to remote site
--allow-chown try to set owner and group on files
--ascii use ascii mode transfers (implies --ignore-size)
--ignore-time ignore time when deciding whether to download
--ignore-size ignore size when deciding whether to download
--only-missing download only missing files
--only-existing download only files already existing at target
-n, --only-newer download only newer files (-c won't work)
--no-empty-dirs don't create empty directories (implies --depth-first)
-r, --no-recursion don't go to subdirectories
--no-symlinks don't create symbolic links
-p, --no-perms don't set file permissions
--no-umask don't apply umask to file modes
-R, --reverse reverse mirror (put files)
-L, --dereference download symbolic links as files
-N, --newer-than=SPEC download only files newer than specified time
--on-change=CMD execute the command if anything has been changed
--older-than=SPEC download only files older than specified time
--size-range=RANGE download only files with size in specified range
-P, --parallel[=N] download N files in parallel
--use-pget[-n=N] use pget to transfer every single file
--loop loop until no changes found
-i RX, --include RX include matching files
-x RX, --exclude RX exclude matching files
-I GP, --include-glob GP include matching files
-X GP, --exclude-glob GP exclude matching files
-v, --verbose[=level] verbose operation
--log=FILE write lftp commands being executed to FILE
--script=FILE write lftp commands to FILE, but don't execute them
--just-print, --dry-run same as --script=-
--use-cache use cached directory listings
--Remove-source-files remove files after transfer (use with caution)
-a same as --allow-chown --allow-suid --no-umask问题记录
1.虽然mirror支持多线程，我们也是针对三个大目录(其中很多子目录)下载，但是整个过程中list列表比较费时间，建议是直接mirror子目录 这样线程会多一些。
2.注意使用--only-missing参数，其他的参数比如only-newer 不太清楚原因但是会先删掉本地再下载一遍
#!/bin/bash
# FTP服务器信息
FTP_HOST="xxxxx"
FTP_USER="xxxx"
FTP_PASS="xxxxxxx"
# 定义要同步的远程和本地目录对
declare -A DIR_MAP=(
["/fumulu/zimulu1"]="/data/0/bendi/fumulu/"
["/fumulu/zimulu2"]="/data/0/bendi/fumulu/"
["/fumulu/zimulu3"]="/data/0/bendi/fumulu/"
["/fumulu/zimulu4"]="/data/0/bendi/fumulu/"
["/fumulu/zimulu5"]="/data/0/bendi/fumulu/"
["/fumulu/zimulu6"]="/data/0/bendi/fumulu/"
["/fumulu/zimulu7"]="/data/0/bendi/fumulu/"
)
# 创建日志目录
LOG_DIR="sync_logs"
mkdir -p "$LOG_DIR"
sync_directory() {
local remote_dir=$1
local local_dir=$2
# 生成日志文件名（将目录分隔符替换为下划线）
local log_name=$(echo "${remote_dir}" | tr '/' '_')
local log_file="$LOG_DIR/${log_name}sync.log"
# 确保本地目录存在
mkdir -p "$local_dir"
echo "开始同步 $remote_dir 到 $local_dir..." | tee -a "$log_file"
echo "同步开始时间: $(date)" >> "$log_file"
# 使用lftp进行同步操作，添加 --size-only 参数
temp_log=$(mktemp)
lftp -c "open -u $FTP_USER,$FTP_PASS $FTP_HOST; \
mirror --parallel=1000 --verbose --only-missing $remote_dir $local_dir" 2>&1 | tee -a "$temp_log" "$log_file"
# 检查文件下载失败的情况
if grep -i "File not available" "$temp_log" > /dev/null; then
echo "发现文件下载失败，记录到 shibai.txt..."
# 提取并记录失败的文件信息
grep -i "File not available" "$temp_log" | while read -r line; do
# 提取完整的文件路径和文件名
full_path=$(echo "$line" | grep -o "@.*" | cut -d' ' -f1)
echo "$full_path" >> shibai.txt
done
fi
echo "同步结束时间: $(date)" >> "$log_file"
echo "----------------------------------------" >> "$log_file"
# 清理临时日志文件
rm -f "$temp_log"
}
# 同时启动所有同步任务
for remote_dir in "${!DIR_MAP[@]}"; do
local_dir=${DIR_MAP[$remote_dir]}
sync_directory "$remote_dir" "$local_dir" &
done
# 等待所有后台任务完成
wait
echo "所有同步任务已完成。"