关于文件的上传和下载前面已经讲了2节课,今天我们主要讲一下如何分片下载,历史文章详解下面链接
1.分片下载的好处
- 使用分片下载: 将大文件分割成多个小块进行下载,可以降低内存占用和网络传输中断的风险。这样可以避免一次性下载整个大文件造成的性能问题。
- 断点续传: 实现断点续传功能,即在下载中途中断后,可以从已下载的部分继续下载,而不需要重新下载整个文件。
- 进度条显示: 在页面上展示下载进度,让用户清晰地看到文件下载的进度。如果一次全部下载可以从process中直接拿到参数计算得出(很精细),如果是分片下载,也是计算已下载的和总大小,只不过已下载的会成片成片的增加(不是很精细)。
分片下载原理
其实和大文件的分片上传原理一样,就是将一个大的文件对象进行切片,然后并发下载分片,最后再进行组装。 只是下载文件需要服务端进行文件的分片,客户端(浏览器)进行文件的组装合并。
HTTP 范围请求
分片下载文件需要用到一个 HTTP 范围请求:
HTTP 协议范围请求允许服务器只发送 HTTP 消息的一部分到客户端。
范围请求在传送大的媒体文件,或者与文件下载的断点续传功能搭配使用时非常有用。如果在响应中存在 Accept-Ranges
首部(并且它的值不为 "none"),那么表示该服务器支持范围请求。 在一个 Range 首部中,可以一次性请求多个部分,服务器会以 multipart 文件的形式将其返回。如果服务器返回的是范围响应,需要使用 206 Partial Content 状态码。假如所请求的范围不合法,那么服务器会返回 416 Range Not Satisfiable 状态码,表示客户端错误。服务器允许忽略 Range 首部,从而返回整个文件,状态码用 200 。
Range 请求语法
Range: <unit>=<range-start>-
Range: <unit>=<range-start>-<range-end>
Range: <unit>=<range-start>-<range-end>, <range-start>-<range-end>
Range: <unit>=<range-start>-<range-end>, <range-start>-<range-end>, <range-start>-<range-end>
unit
:范围请求所采用的单位,通常是字节(bytes)。<range-start>
:一个整数,表示在特定单位下,范围的起始值。<range-end>
:一个整数,表示在特定单位下,范围的结束值。这个值是可选的,如果不存在,表示此范围一直延伸到文档结束。
2.代码工程
实验目的
利用http range实现分片下载
pom.xml
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<parent>
<artifactId>springboot-demo</artifactId>
<groupId>com.et</groupId>
<version>1.0-SNAPSHOT</version>
</parent>
<modelVersion>4.0.0</modelVersion>
<artifactId>file</artifactId>
<properties>
<maven.compiler.source>8</maven.compiler.source>
<maven.compiler.target>8</maven.compiler.target>
</properties>
<dependencies>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-web</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-autoconfigure</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-test</artifactId>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.apache.httpcomponents</groupId>
<artifactId>httpclient</artifactId>
</dependency>
<dependency>
<groupId>org.apache.httpcomponents</groupId>
<artifactId>httpmime</artifactId>
</dependency>
<dependency>
<groupId>org.projectlombok</groupId>
<artifactId>lombok</artifactId>
</dependency>
<dependency>
<groupId>cn.hutool</groupId>
<artifactId>hutool-core</artifactId>
<version>5.8.15</version>
</dependency>
<dependency>
<groupId>commons-fileupload</groupId>
<artifactId>commons-fileupload</artifactId>
<version>1.3.1</version>
</dependency>
<dependency>
<groupId>commons-io</groupId>
<artifactId>commons-io</artifactId>
<version>2.4</version>
</dependency>
</dependencies>
</project>
分片下载
package com.et.controller;
import com.et.bean.DownLoadFileInfo;
import org.apache.commons.io.FileUtils;
import org.apache.http.HttpEntity;
import org.apache.http.HttpResponse;
import org.apache.http.client.HttpClient;
import org.apache.http.client.methods.HttpGet;
import org.apache.http.impl.client.HttpClients;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.web.bind.annotation.RequestMapping;
import org.springframework.web.bind.annotation.RestController;
import javax.servlet.http.HttpServletRequest;
import javax.servlet.http.HttpServletResponse;
import java.io.*;
import java.net.HttpURLConnection;
import java.net.URL;
import java.net.URLDecoder;
import java.net.URLEncoder;
import java.util.concurrent.*;
@RestController
public class DownloadController {
private static final Logger LOGGER = LoggerFactory.getLogger(DownloadController.class);
private final static long PER_PAGE = 1024L * 1024L * 50L;
private final static String DOWN_PATH = "D:\\tmp";
private static final String UTF8 = "UTF-8";
ExecutorService taskExecutor = Executors.newFixedThreadPool(10);
@RequestMapping("/download")
public void downLoadFile(HttpServletRequest request, HttpServletResponse response) throws IOException {
File file = new File("D:\\SoftWare\\oss-browser-win32-ia32.zip");
response.setCharacterEncoding(UTF8);
InputStream is = null;
OutputStream os = null;
try {
// chunk download Range bytes=100-1000 100-
long fSize = file.length();
response.setContentType("application/x-download");
String fileName = URLEncoder.encode(file.getName(), UTF8);
response.addHeader("Content-Disposition", "attachment;filename=" + fileName);
// Accept-Range
response.setHeader("Accept-Range", "bytes");
response.setHeader("fSize", String.valueOf(fSize));
response.setHeader("fName", fileName);
long pos = 0, last = fSize - 1, sum = 0;
if (null != request.getHeader("Range")) {
response.setStatus(HttpServletResponse.SC_PARTIAL_CONTENT);
String numberRange = request.getHeader("Range").replaceAll("bytes=", "");
String[] strRange = numberRange.split("-");
if (strRange.length == 2) {
pos = Long.parseLong(strRange[0].trim());
last = Long.parseLong(strRange[1].trim());
if (last > fSize-1) {
last = fSize - 1;
}
} else {
pos = Long.parseLong(numberRange.replaceAll("-", "").trim());
}
}
long rangeLength = last - pos + 1;
String contentRange = new StringBuffer("bytes").append(pos).append("-").append(last).append("/").append(fSize).toString();
response.setHeader("Content-Range", contentRange);
response.setHeader("Content-Length", String.valueOf(rangeLength));
os = new BufferedOutputStream(response.getOutputStream());
is = new BufferedInputStream(new FileInputStream(file));
is.skip(pos);
byte[] buffer = new byte[1024];
int length = 0;
while (sum < rangeLength) {
int readLength = (int) (rangeLength - sum);
length = is.read(buffer, 0, (rangeLength - sum) <= buffer.length ? readLength : buffer.length);
sum += length;
os.write(buffer,0, length);
}
System.out.println("download finish");
}finally {
if (is != null){
is.close();
}
if (os != null){
os.close();
}
}
}
@RequestMapping("/downloadFile")
public String downloadFile() {
DownLoadFileInfo fileInfo = download(0, 10, -1, null);
if (fileInfo != null) {
long pages = fileInfo.fSize / PER_PAGE;
for (long i = 0; i <= pages; i++) {
Future<DownLoadFileInfo> future = taskExecutor.submit(new DownloadThread(i * PER_PAGE, (i + 1) * PER_PAGE - 1, i, fileInfo.fName));
if (!future.isCancelled()) {
try {
fileInfo = future.get();
} catch (InterruptedException | ExecutionException e) {
e.printStackTrace();
}
}
}
return System.getProperty("user.home") + "\\Downloads\\" + fileInfo.fName;
}
return null;
}
private DownLoadFileInfo download(long start, long end, long page, String fName) {
File dir = new File(DOWN_PATH);
if (!dir.exists()) {
dir.mkdirs();
}
File file = new File(DOWN_PATH, page + "-" + fName);
if (file.exists() && page != -1 && file.length() == PER_PAGE) {
return null;
}
try {
HttpClient client = HttpClients.createDefault();
HttpGet httpGet = new HttpGet("http://127.0.0.1:8080/download");
httpGet.setHeader("Range", "bytes=" + start + "-" + end);
HttpResponse response = client.execute(httpGet);
String fSize = response.getFirstHeader("fSize").getValue();
fName = URLDecoder.decode(response.getFirstHeader("fName").getValue(), "UTF-8");
HttpEntity entity = response.getEntity();
InputStream is = entity.getContent();
FileOutputStream fos = new FileOutputStream(file);
byte[] buffer = new byte[1024];
int ch;
while ((ch = is.read(buffer)) != -1) {
fos.write(buffer, 0, ch);
}
is.close();
fos.flush();
fos.close();
// last part
if (end - Long.parseLong(fSize) > 0) {
// merge file
mergeFile(fName, page);
}
return new DownLoadFileInfo(Long.parseLong(fSize), fName);
} catch (IOException e) {
e.printStackTrace();
}
return null;
}
private void mergeFile(String fName, long page) {
File file = new File(DOWN_PATH, fName);
try {
BufferedOutputStream os = new BufferedOutputStream(new FileOutputStream(file));
for (long i = 0; i <= page; i++) {
File tempFile = new File(DOWN_PATH, i + "-" + fName);
while (!file.exists() || (i != page && tempFile.length() < PER_PAGE)) {
try {
Thread.sleep(100);
} catch (InterruptedException e) {
e.printStackTrace();
}
}
byte[] bytes = FileUtils.readFileToByteArray(tempFile);
os.write(bytes);
os.flush();
tempFile.delete();
}
File testFile = new File(DOWN_PATH, -1 + "-null");
testFile.delete();
os.flush();
os.close();
} catch (IOException e) {
e.printStackTrace();
}
}
/**
* get file size
*/
private long getRemoteFileSize(String remoteFileUrl) throws IOException {
long fileSize = 0;
HttpURLConnection httpConnection = (HttpURLConnection) new URL(remoteFileUrl).openConnection();
//使用HEAD方法
httpConnection.setRequestMethod("HEAD");
int responseCode = httpConnection.getResponseCode();
if (responseCode >= 400) {
LOGGER.debug("Web responsible fail!");
return 0;
}
String sHeader;
for (int i = 1;; i++) {
sHeader = httpConnection.getHeaderFieldKey(i);
if (sHeader != null && sHeader.equals("Content-Length")) {
LOGGER.debug("file size ContentLength:" + httpConnection.getContentLength());
fileSize = Long.parseLong(httpConnection.getHeaderField(sHeader));
break;
}
}
return fileSize;
}
class DownloadThread implements Callable<DownLoadFileInfo> {
long start;
long end;
long page;
String fName;
public DownloadThread(long start, long end, long page, String fName) {
this.start = start;
this.end = end;
this.page = page;
this.fName = fName;
}
@Override
public DownLoadFileInfo call() {
return download(start, end, page, fName);
}
}
}
以上只是一些关键代码,所有代码请参见下面代码仓库
代码仓库
3.测试
-
启动Spring Boot应用
-
查看日志,发现分为3块下载
download finish
download finish
download finish