1.什么是SFTP?
SFTP(SSH File Transfer Protocol,也称 Secret File Transfer Protocol),是一种基于SSH(安全外壳)的安全的文件传输协议。使用SFTP协议可以在文件传输过程中提供一种安全的加密算法,从而保证数据的安全传输,所以SFTP是非常安全的。但是,由于这种传输方式使用了加密/解密技术,所以传输效率比普通的FTP要低。 SFTP是SSH的一部分,SFTP没有单独的守护进程,它必须使用SSHD守护进程(端口号默认是22)来完成相应的连接操作,sftp服务作为ssh的一个子服务,是通过 /etc/ssh/sshd_config
配置文件中的 Subsystem
实现的,如果没有配置 Subsystem
参数,则系统是不能进行sftp访问的。所以,要分离ssh和sftp服务的话,基本的思路是创建两个sshd进程,分别监听在不同的端口,一个作为ssh服务的deamon,另一个作为sftp服务的deamon。
Spring Integration核心组件
- SftpSessionFactory: sftp 客户端与服务端的会话工厂。客户端每次访问服务器时都会创建一个 session 对象,且可以通过 SftpSessionCaching 将 session 对象缓存起来,支持 session 共享,即可以在一个会话上进行多个 channel 的操作。如果 session 被重置,则在最后一次 channel 关闭之后,将断开连接。isSharedSession 为 true 时 session 将共享。
- SftpSessionCaching: sftp 会话缓存工厂。通过 poolSize 和 sessionWaiteTimeout 来设置缓存池大小和会话等待超时时间。缓存池默认是无限大,超时时间默认是 Integer.MAX_VALUE。
- SftpRemoteFileTemplate: 基于 SftpSessionFactory 创建的 sftp 文件操作模板类。其父类是 RemoteFileTemplate。支持上传、下载、追加、删除、重命名、列表、是否存在。基于输入输出流实现。
- SftpInboundChannelAdapter: sftp 入站通道适配器。可同步远程目录到本地,且可监听远程文件的操作,可实现下载。
- SftpOutboundChannelAdapter: sftp 出站通道适配器。实际是一个 sftp 消息处理器,将在服务器与客户端之间创建一个消息传输通道。此处的消息指的是 Message 的 payload,其支持 File、byte[]、String。其支持 ls、nlst、get、rm、mget、mv、put、mput 操作。
- Channel Adapter: 通道适配器,实际上是适配消息在客户端和服务器之间的传输。inbound adapter 是接收其它系统的消息,outbound adapter 是发送消息到其它系统。
- @ServiceActivator: 将注解作用的方法注册为处理消息的站点,inputChannel 表示接收消息的通道。
2.环境搭建
docker run -p 22:22 -d atmoz/sftp foo:pass:::upload
验证环境
说明已经可以连接上去了
3.代码工程
实验目标
实现文件上传和下载
pom.xml
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<parent>
<artifactId>springboot-demo</artifactId>
<groupId>com.et</groupId>
<version>1.0-SNAPSHOT</version>
</parent>
<modelVersion>4.0.0</modelVersion>
<artifactId>sftp</artifactId>
<properties>
<maven.compiler.source>8</maven.compiler.source>
<maven.compiler.target>8</maven.compiler.target>
</properties>
<dependencies>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-web</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-autoconfigure</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-test</artifactId>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.springframework.integration</groupId>
<artifactId>spring-integration-sftp</artifactId>
<!-- <version>5.4.1</version>-->
<version>5.2.8.RELEASE</version>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-configuration-processor</artifactId>
<optional>true</optional>
</dependency>
<dependency>
<groupId>org.projectlombok</groupId>
<artifactId>lombok</artifactId>
<optional>true</optional>
</dependency>
</dependencies>
</project>
service
package com.et.sftp.service.impl;
import com.et.sftp.config.SftpConfiguration;
import com.et.sftp.service.SftpService;
import lombok.extern.slf4j.Slf4j;
import org.springframework.integration.file.remote.FileInfo;
import org.springframework.integration.sftp.session.SftpRemoteFileTemplate;
import org.springframework.stereotype.Component;
import org.springframework.web.multipart.MultipartFile;
import javax.annotation.Resource;
import java.io.*;
import java.util.List;
@Slf4j
@Component
public class SftpServiceImpl implements SftpService {
@Resource
private SftpRemoteFileTemplate remoteFileTemplate;
@Resource
private SftpConfiguration.SftpGateway gateway;
/**
* single file upload
*
* @param file File
*/
@Override
public void uploadFile(File file) {
gateway.upload(file);
}
/**
* single file upload by byte[]
*
* @param bytes bytes
*/
@Override
public void uploadFile(byte[] bytes, String name) {
try {
gateway.upload(bytes, name);
} catch (Exception e) {
log.error("error:", e);
}
}
/**
* uopload by path
*
* @param bytes
* @param filename
* @param path
*/
@Override
public void upload(byte[] bytes, String filename, String path) {
try {
gateway.upload(bytes, filename, path);
} catch (Exception e) {
log.error("error:", e);
}
}
/**
* list files by path
*
* @param path
* @return List<String>
*/
@Override
public String[] listFile(String path) {
try {
return remoteFileTemplate.execute(session -> {
return session.listNames(path);
});
} catch (Exception e) {
log.error("error:", e);
}
return null;
}
/**
* list file and directory by path
*
* @param path
* @return List<String>
*/
@Override
public List<FileInfo> listALLFile(String path) {
return gateway.listFile(path);
}
/**
* download
*
* @param fileName
* @param savePath
* @return File
*/
@Override
public File downloadFile(String fileName, String savePath) {
try {
return remoteFileTemplate.execute(session -> {
remoteFileTemplate.setAutoCreateDirectory(true);
boolean existFile = session.exists(fileName);
if (existFile) {
InputStream is = session.readRaw(fileName);
return convertInputStreamToFile(is, savePath);
} else {
return null;
}
});
} catch (Exception e) {
log.error("error:", e);
}
return null;
}
/**
* read file
*
* @param fileName
* @return InputStream
*/
@Override
public InputStream readFile(String fileName) {
return remoteFileTemplate.execute(session -> {
return session.readRaw(fileName);
});
}
/**
* files is exists
*
* @param filePath
* @return boolean
*/
@Override
public boolean existFile(String filePath) {
try {
return remoteFileTemplate.execute(session ->
session.exists(filePath));
} catch (Exception e) {
log.error("error:", e);
}
return false;
}
public void renameFile(String file1, String file2) {
try {
remoteFileTemplate.execute(session -> {
session.rename(file1, file2);
return true;
});
} catch (Exception e) {
log.error("error:", e);
}
}
/**
* create directory
*
* @param dirName
* @return
*/
@Override
public boolean mkdir(String dirName) {
return remoteFileTemplate.execute(session -> {
if (!existFile(dirName)) {
return session.mkdir(dirName);
} else {
return false;
}
});
}
/**
* delete file
*
* @param fileName
* @return boolean
*/
@Override
public boolean deleteFile(String fileName) {
return remoteFileTemplate.execute(session -> {
boolean existFile = session.exists(fileName);
if (existFile) {
return session.remove(fileName);
} else {
log.info("file : {} not exist", fileName);
return false;
}
});
}
/**
* batch upload (MultipartFile)
*
* @param files List<MultipartFile>
* @throws IOException
*/
@Override
public void uploadFiles(List<MultipartFile> files, boolean deleteSource) throws IOException {
try {
for (MultipartFile multipartFile : files) {
if (multipartFile.isEmpty()) {
continue;
}
File file = convert(multipartFile);
gateway.upload(file);
if (deleteSource) {
file.delete();
}
}
} catch (Exception e) {
log.error("error:", e);
}
}
/**
* batch upload (MultipartFile)
*
* @param files List<MultipartFile>
* @throws IOException
*/
@Override
public void uploadFiles(List<MultipartFile> files) throws IOException {
uploadFiles(files, true);
}
/**
* single file upload (MultipartFile)
*
* @param multipartFile MultipartFile
* @throws IOException
*/
@Override
public void uploadFile(MultipartFile multipartFile) throws IOException {
gateway.upload(convert(multipartFile));
}
@Override
public String listFileNames(String dir) {
return gateway.nlstFile(dir);
}
@Override
public File getFile(String dir) {
return null;
}
@Override
public List<File> mgetFile(String dir) {
return null;
}
@Override
public boolean rmFile(String file) {
return false;
}
@Override
public boolean mv(String sourceFile, String targetFile) {
return false;
}
@Override
public File putFile(String dir) {
return null;
}
@Override
public List<File> mputFile(String dir) {
return null;
}
@Override
public String nlstFile(String dir) {
return gateway.nlstFile(dir);
}
private static File convertInputStreamToFile(InputStream inputStream, String savePath) {
OutputStream outputStream = null;
File file = new File(savePath);
try {
outputStream = new FileOutputStream(file);
int read;
byte[] bytes = new byte[1024];
while ((read = inputStream.read(bytes)) != -1) {
outputStream.write(bytes, 0, read);
}
log.info("convert InputStream to file done, savePath is : {}", savePath);
} catch (IOException e) {
log.error("error:", e);
} finally {
if (inputStream != null) {
try {
inputStream.close();
} catch (IOException e) {
e.printStackTrace();
}
}
if (outputStream != null) {
try {
outputStream.close();
} catch (IOException e) {
log.error("error:", e);
}
}
}
return file;
}
private static File convert(MultipartFile file) throws IOException {
File convertFile = new File(file.getOriginalFilename());
convertFile.createNewFile();
FileOutputStream fos = new FileOutputStream(convertFile);
fos.write(file.getBytes());
fos.close();
return convertFile;
}
}
package com.et.sftp.service;
import org.springframework.integration.file.remote.FileInfo;
import org.springframework.web.multipart.MultipartFile;
import java.io.File;
import java.io.IOException;
import java.io.InputStream;
import java.util.List;
public interface SftpService {
void uploadFile(File file);
void uploadFile(byte[] bytes, String name);
void upload(byte[] bytes, String filename, String path);
String[] listFile(String path);
List<FileInfo> listALLFile(String path);
File downloadFile(String fileName, String savePath);
InputStream readFile(String fileName);
boolean existFile(String filePath);
boolean mkdir(String dirName);
boolean deleteFile(String fileName);
void uploadFiles(List<MultipartFile> files, boolean deleteSource) throws IOException;
void uploadFiles(List<MultipartFile> files) throws IOException;
void uploadFile(MultipartFile multipartFile) throws IOException;
String listFileNames(String dir);
File getFile(String dir);
List<File> mgetFile(String dir);
boolean rmFile(String file);
boolean mv(String sourceFile, String targetFile);
File putFile(String dir);
List<File> mputFile(String dir);
//void upload(File file);
//void upload(byte[] inputStream, String name);
//List<File> downloadFiles(String dir);
String nlstFile(String dir);
}
config
在配置SFTP adapters之前,需要配置SFTP Session Factory;Spring Integration提供了如下xml和spring boot的定义方式。 每次使用 SFTP adapter,都需要Session Factory会话对象,一般情况,都会创建一个新的SFTP会话。同时还提供了Session的缓存功能。Spring integration中的Session Factory是依赖于JSch库来提供。 JSch支持在一个连接配置上多个channel的操作。原生的JSch技术开发,在打开一个channel操作之前,需要建立Session的连接。同样的,默认情况,Spring Integration为每一个channel操作使用单独的物理连接。在3.0版本发布之后,Cache Session Factory 出现 (CachingSessionFactory),将Session Factory包装在缓存中,支持Session共享,可以在一个连接上支持多个JSch Channel的操作。如果缓存被重置,在最后一次channel关闭之后,才会断开连接。
package com.et.sftp.config;
import com.jcraft.jsch.ChannelSftp;
import org.springframework.beans.factory.BeanFactory;
import org.springframework.boot.context.properties.EnableConfigurationProperties;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.expression.common.LiteralExpression;
import org.springframework.expression.spel.standard.SpelExpressionParser;
import org.springframework.integration.annotation.Gateway;
import org.springframework.integration.annotation.MessagingGateway;
import org.springframework.integration.annotation.ServiceActivator;
import org.springframework.integration.core.MessagingTemplate;
import org.springframework.integration.file.FileNameGenerator;
import org.springframework.integration.file.remote.FileInfo;
import org.springframework.integration.file.remote.session.CachingSessionFactory;
import org.springframework.integration.file.remote.session.SessionFactory;
import org.springframework.integration.file.support.FileExistsMode;
import org.springframework.integration.sftp.gateway.SftpOutboundGateway;
import org.springframework.integration.sftp.outbound.SftpMessageHandler;
import org.springframework.integration.sftp.session.DefaultSftpSessionFactory;
import org.springframework.integration.sftp.session.SftpRemoteFileTemplate;
import org.springframework.messaging.Message;
import org.springframework.messaging.MessageHandler;
import org.springframework.messaging.handler.annotation.Header;
import org.springframework.messaging.handler.annotation.Payload;
import javax.annotation.Resource;
import java.io.File;
import java.util.List;
@Configuration
@EnableConfigurationProperties(SftpProperties.class)
public class SftpConfiguration {
@Resource
private SftpProperties properties;
@Bean
public MessagingTemplate messagingTemplate(BeanFactory beanFactory) {
MessagingTemplate messagingTemplate = new MessagingTemplate();
messagingTemplate.setBeanFactory(beanFactory);
return messagingTemplate;
}
@Bean
public SessionFactory<ChannelSftp.LsEntry> sftpSessionFactory() {
DefaultSftpSessionFactory factory = new DefaultSftpSessionFactory(true);
factory.setHost(properties.getHost());
factory.setPort(properties.getPort());
factory.setUser(properties.getUsername());
factory.setPassword(properties.getPassword());
factory.setAllowUnknownKeys(true);
// factory.setTestSession(true);
// return factory;
return new CachingSessionFactory<ChannelSftp.LsEntry>(factory);
}
@Bean
public SftpRemoteFileTemplate remoteFileTemplate(SessionFactory<ChannelSftp.LsEntry> sftpSessionFactory) {
return new SftpRemoteFileTemplate(sftpSessionFactory);
}
@Bean
@ServiceActivator(inputChannel = "downloadChannel")
public MessageHandler downloadHandler(SessionFactory<ChannelSftp.LsEntry> sftpSessionFactory) {
SftpOutboundGateway sftpOutboundGateway = new SftpOutboundGateway(sftpSessionFactory, "mget", "payload");
sftpOutboundGateway.setOptions("-R");
sftpOutboundGateway.setFileExistsMode(FileExistsMode.REPLACE_IF_MODIFIED);
sftpOutboundGateway.setLocalDirectory(new File(properties.getLocalDir()));
sftpOutboundGateway.setAutoCreateLocalDirectory(true);
return sftpOutboundGateway;
}
@Bean
@ServiceActivator(inputChannel = "uploadChannel", outputChannel = "testChannel")
public MessageHandler uploadHandler(SessionFactory<ChannelSftp.LsEntry> sftpSessionFactory) {
SftpMessageHandler handler = new SftpMessageHandler(sftpSessionFactory);
handler.setRemoteDirectoryExpression(new LiteralExpression(properties.getRemoteDir()));
// handler.setChmod();
handler.setFileNameGenerator(message -> {
if (message.getPayload() instanceof File) {
return ((File) message.getPayload()).getName();
} else {
throw new IllegalArgumentException("File expected as payload.");
}
});
return handler;
}
@Bean
@ServiceActivator(inputChannel = "uploadByteChannel")
public MessageHandler multiTypeHandler(SessionFactory<ChannelSftp.LsEntry> sftpSessionFactory) {
SftpMessageHandler handler = new SftpMessageHandler(sftpSessionFactory);
handler.setRemoteDirectoryExpression(new LiteralExpression(properties.getRemoteDir()));
handler.setFileNameGenerator(message -> {
if (message.getPayload() instanceof byte[]) {
return (String) message.getHeaders().get("name");
} else {
throw new IllegalArgumentException("byte[] expected as payload.");
}
});
return handler;
}
@Bean
@ServiceActivator(inputChannel = "lsChannel")
public MessageHandler lsHandler(SessionFactory<ChannelSftp.LsEntry> sftpSessionFactory) {
SftpOutboundGateway sftpOutboundGateway = new SftpOutboundGateway(sftpSessionFactory, "ls", "payload");
sftpOutboundGateway.setOptions("-R");
return sftpOutboundGateway;
}
@Bean
@ServiceActivator(inputChannel = "nlstChannel")
public MessageHandler listFileNamesHandler(SessionFactory<ChannelSftp.LsEntry> sftpSessionFactory) {
SftpOutboundGateway sftpOutboundGateway = new SftpOutboundGateway(sftpSessionFactory, "nlst", "payload");
return sftpOutboundGateway;
}
@Bean
@ServiceActivator(inputChannel = "getChannel")
public MessageHandler getFileHandler(SessionFactory<ChannelSftp.LsEntry> sftpSessionFactory) {
SftpOutboundGateway sftpOutboundGateway = new SftpOutboundGateway(sftpSessionFactory, "get", "payload");
sftpOutboundGateway.setOptions("-R");
sftpOutboundGateway.setFileExistsMode(FileExistsMode.REPLACE_IF_MODIFIED);
sftpOutboundGateway.setLocalDirectory(new File(properties.getLocalDir()));
sftpOutboundGateway.setAutoCreateLocalDirectory(true);
return sftpOutboundGateway;
}
/**
* create by: qiushicai
* create time: 2020/11/20
*
* @return
*/
@Bean
@ServiceActivator(inputChannel = "abc")
public MessageHandler abcHandler(SessionFactory<ChannelSftp.LsEntry> sftpSessionFactory) {
SftpMessageHandler handler = new SftpMessageHandler(sftpSessionFactory);
handler.setRemoteDirectoryExpression(new LiteralExpression(properties.getRemoteDir()));
handler.setFileNameGenerator(message -> {
if (message.getPayload() instanceof byte[]) {
System.out.println("receive message:" + new String((byte[]) message.getPayload()));
message.getHeaders().forEach((k, v) -> System.out.println("\t\t|---" + k + "=" + v));
return "ok";
} else {
throw new IllegalArgumentException("byte[] expected as payload.");
}
});
return handler;
}
/**
*
* the #root object is the Message, which has two properties (headers and payload) that allow such expressions as payload, payload.thing, headers['my.header'], and so on
*
* link{ https://stackoverflow.com/questions/46650004/spring-integration-ftp-create-dynamic-directory-with-remote-directory-expressi}
* link{ https://docs.spring.io/spring-integration/reference/html/spel.html}
* @return
*/
@Bean
@ServiceActivator(inputChannel = "toPathChannel")
public MessageHandler pathHandler() {
SftpMessageHandler handler = new SftpMessageHandler(sftpSessionFactory());
// automatically create the remote directory
handler.setAutoCreateDirectory(true);
handler.setRemoteDirectoryExpression(new SpelExpressionParser().parseExpression("headers[path]"));
handler.setFileNameGenerator(new FileNameGenerator() {
@Override
public String generateFileName(Message<?> message) {
return (String) message.getHeaders().get("filename");
}
});
return handler;
}
/**
* <ul>
* <li>ls (list files)
* <li> nlst (list file names)
* <li> get (retrieve a file)
* <li> mget (retrieve multiple files)
* <li> rm (remove file(s))
* <li> mv (move and rename file)
* <li> put (send a file)
* <li> mput (send multiple files)
* </ul>
*
* @author :qiushicai
* @date :Created in 2020/11/20
* @description: outbound gateway API
* @version:
*/
@MessagingGateway
public interface SftpGateway {
//ls (list files)
@Gateway(requestChannel = "lsChannel")
List<FileInfo> listFile(String dir);
@Gateway(requestChannel = "nlstChannel")
String nlstFile(String dir);
@Gateway(requestChannel = "getChannel")
File getFile(String dir);
@Gateway(requestChannel = "mgetChannel")
List<File> mgetFile(String dir);
@Gateway(replyChannel = "rmChannel")
boolean rmFile(String file);
@Gateway(replyChannel = "mvChannel")
boolean mv(String sourceFile, String targetFile);
@Gateway(requestChannel = "putChannel")
File putFile(String dir);
@Gateway(requestChannel = "mputChannel")
List<File> mputFile(String dir);
@Gateway(requestChannel = "uploadChannel")
void upload(File file);
@Gateway(requestChannel = "uploadByteChannel")
void upload(byte[] inputStream, String name);
@Gateway(requestChannel = "toPathChannel")
void upload(@Payload byte[] file, @Header("filename") String filename, @Header("path") String path);
@Gateway(requestChannel = "downloadChannel")
List<File> downloadFiles(String dir);
}
}
从Spring Integration 3.0开始,通过SftpSession对象提供了一个新类Remote File Template。提供了Sftp文件发送、检索、删除和重命名文件的方法。此外,还提供了一个执行方法,允许调用者在会话上执行多个操作。在所有情况下,模板负责可靠地关闭会话。 SftpRemoteFileTemplate继承Remote File Template,可以很容易的实现对SFTP文件的发送(包括文件追加,替换等),检索,删除和重命名。
package com.et.sftp.config;
import lombok.Data;
import org.springframework.boot.context.properties.ConfigurationProperties;
@Data
@ConfigurationProperties(prefix = "sftp")
public class SftpProperties {
private String host;
private Integer port;
private String password;
private String username;
private String remoteDir;
private String localDir;
}
application.properties
##sftp properties
sftp.host=127.0.0.1
sftp.port=22
sftp.username=foo
sftp.password=pass
sftp.remoteDir=/upload
sftp.localDir=D:\\tmp\\sync-files
以上只是一些关键代码,所有代码请参见下面代码仓库
代码仓库
4.测试
在测试包里面有专门的测试类,具体可以查看源代码
文件是否存在
@Test
void testExistFile() {
boolean existFile = sftpService.existFile("/upload/home222.js");
System.out.println(existFile);
}
列出目录下的文件
@Test
void listFileTest() {
sftpService.listALLFile("/upload").stream().forEach(System.out::println);
}
下载文件
@Test
void testDownLoad() throws Exception {
sftpService.downloadFile("/upload/home.js", "D:\\tmp\\c222c.js");
//
// sftpService.uploadFile(new File("D:\\tmp\\cc.js"));
// InputStream inputStream = sftpService.readFile("/upload/cc.js");
//
// IOUtils.copy(inputStream, new FileOutputStream(new File("D:\\tmp\\" + UUID.randomUUID() + ".js")));
}
上传文件
@Test
void uploadFile() {
sftpService.uploadFile(new File("D:\\tmp\\cc.js"));
}