比起上传资源更应该懂得如何资源回收

背景

如今上传资源是非常普遍的功能，往往数据操作与关联的上传的文件间属于两步操作，比如：修改用户头像，上传头像是一个接口，保存用户profile（包括头像信息）是另外一个接口，如果这时N次调用上传头像接口重复上传头像，之后在调用保存用户profile接口，这时有效的头像资源只需要一次，但会产生许多无效资源头像资源，久而久之会占用很大磁盘空间

解决方案

思路其实很简单：上传的资源都会先放在临时的资源目录，当上传资源与数据绑定时会复制存储到个有效资源目录下，而临时资源目录会跑任务定时清空。

本文将介绍如何在 NestJS 中管理上传的头像资源，将其存储到临时资源目录，当保存用户接口时绑定的头像资源将保存到有效的资源目录下。而每天生成的临时文件夹以日期命名，通过系统定时任务每天凌晨触发 NestJS 应用删除两天前的资源临时文件夹。具体时序图

sequenceDiagram participant User as 用户 participant Frontend as 前端 participant Backend as 后端 participant TempStorage as 临时存储 participant ValidStorage as 有效存储 participant CronJob as 定时任务 User->>Frontend: 选择头像 Frontend->>Backend: 上传头像请求 Backend->>TempStorage: 存储头像到临时目录 Backend-->>Frontend: 返回头像临时URL User->>Frontend: 点击保存按钮 Frontend->>Backend: 提交用户数据（包括头像临时URL） Backend->>ValidStorage: 将头像从临时目录复制到有效存储目录 Backend-->>Frontend: 保存成功 Frontend-->>User: 通知保存成功 CronJob->>Backend: 每天凌晨触发清理临时文件 Backend->>TempStorage: 删除两天前的临时文件夹

需预先了解：

避免文章太过，这里在之前的文章的基础上进行阐述，请先了解：《上传资源管理进阶：优化资源映射》

具体优化步骤

1. 资源上传到临时目录

首先需要再ResourceMapping 中添加两个字段isValid，createdAt，用于后期回收资源标识

diff 复制代码

// src/upload/resource-mapping.entity.ts

import {
    Entity,
    Column,
    PrimaryGeneratedColumn,
+    CreateDateColumn
} from 'typeorm';

@Entity()
export class ResourceMapping {
    @PrimaryGeneratedColumn()
    id: number;

    @Column()
    name: string;

    @Column()
    url: string;

+   @Column({ name: 'is_valid', default: false })
+   isValid: boolean; // 是否有效

+   @CreateDateColumn({ name: 'created_at' })
+   createdAt: Date;

}

在upload文件下创建文件：constants.ts, 设置临时目录，有效目录地址

ts 复制代码

// src/upload/constants.ts

import { join } from 'path';
// 临时目录
export const tempUploadDir = join(process.cwd(), 'uploads/temp');

// 有效目录
export const validUploadDir = join(process.cwd(), 'uploads/valid');

MulterModule 上传资源时候需要创建每天对应的临时目录，目录格式：YYYYMMDD，具体代码：

ts 复制代码

// src/upload/upload.module.ts

import { Module } from '@nestjs/common';
import { MulterModule } from '@nestjs/platform-express';
import { diskStorage } from 'multer';
import { extname, join } from 'path';
import { TypeOrmModule } from '@nestjs/typeorm';
import { UploadController } from './upload.controller';
import { UploadService } from './upload.service';
import { ResourceMapping } from './resource-mapping.entity';
import { existsSync, mkdirSync } from 'fs';
import * as dayjs from 'dayjs';
import { tempUploadDir } from './constants';

@Module({
  imports: [
    TypeOrmModule.forFeature([ResourceMapping]),
    MulterModule.register({
      storage: diskStorage({
        destination: (req, file, cb) => {
          const dateFolder = dayjs().format('YYYYMMDD');
          const uploadPath = join(tempUploadDir, dateFolder);

          // 检查日期文件夹是否存在，不存在则创建
          if (!existsSync(uploadPath)) {
            mkdirSync(uploadPath, { recursive: true });
          }

          cb(null, uploadPath);
        },
        filename: (req, file, cb) => {
          const ext = extname(file.originalname);
          const filename = `${Date.now()}${ext}`;
          cb(null, filename);
        },
      }),
    }),
  ],
  controllers: [UploadController],
  providers: [UploadService],
})
export class UploadModule {}

创建个新处理资源上传的API接口：uploadFile

diff 复制代码

// src/upload/upload.controller.ts

+  @Post('/uploadFile')
+   @UseInterceptors(FileInterceptor('file'))
+   async uploadFile(@UploadedFile() file) {
+        const { originalname = "", path } = file
+        const relativePath = path.replace(process.cwd(), '');
+        const url = `http://localhost:3000${relativePath}`
+        return this.uploadService.create({ name: originalname, url })
+    }

使用Postman上传资源可得路径：

从返回的url路径：xxx/uploads/temp/YYYYMMDD/xxxx，可以看出文件存放到了临时路径,该资源返回的id=13

2. 复制临时资源到有效资源目录下

在upload.service.ts 中创建复制资源方法：copyFileToValid，通过url copy资源到valid文件夹下

ts 复制代码

// src/upload/upload.service.ts

import { Injectable } from '@nestjs/common';
import { InjectRepository } from '@nestjs/typeorm';
import { Repository } from 'typeorm';
import { join } from 'path';
import { copyFileSync, existsSync, mkdirSync } from 'fs';
import { tempUploadDir, validUploadDir } from './constants';
import { ResourceMapping } from './resource-mapping.entity';
import { ResourceMappingDto } from './upload.dto';

@Injectable()
export class UploadService {
  constructor(
    @InjectRepository(ResourceMapping)
    private readonly resourceMappingRepository: Repository<ResourceMapping>,
  ) {}

  // 创建
  async create(dto: ResourceMappingDto) {
    const createRecord = await this.resourceMappingRepository.save(dto);
    return createRecord;
  }

  // 复制有效资源
  async copyFileToValid(url: string): Promise<string> {
    try {
      const [filename, dateFolder, ...rest] = url.split('/').reverse();
      const validPath = join(validUploadDir, dateFolder);
      if (!existsSync(validPath)) {
        mkdirSync(validPath, { recursive: true });
      }
      const fileRelativePath = `${dateFolder}/${filename}`;
      const tempFilePath = join(tempUploadDir, fileRelativePath);
      const validFilePath = join(validPath, filename);

      copyFileSync(tempFilePath, validFilePath);

      return url.replace('/temp/', '/valid/');
    } catch (error) {
      throw new Error('Failed to copy file to valid directory');
    }
  }
}

3. user sava profile

需要先修改user.entity.ts 支持头像avatar的映射，同时删除上篇文字中的attachments代码简化代码

diff 复制代码

// src/user/user.entity.ts

import {
    Entity,
    Column,
    PrimaryGeneratedColumn,
-   ManyToOne,
+    OneToOne,
    JoinColumn
} from 'typeorm';
import { ResourceMapping } from '../upload/resource-mapping.entity';

@Entity()
export class User {
    @PrimaryGeneratedColumn()
    id: number;

    @Column()
    name: string;

+    @Column({ nullable: true })
+   avatarId: number;

+    @OneToOne(() => ResourceMapping, { eager: false,nullable: true })
+   @JoinColumn({ name: 'avatarId' })
+    avatar: ResourceMapping;

-    @Column()
-   attachmentId: number;

-    @ManyToOne(() => ResourceMapping, { eager: false })
-    @JoinColumn({ name: 'attachmentId' })
-    attachment: ResourceMapping;

}

这里 avatar 与 ResourceMapping 为一对一的关系，eager=ture 允许查询时候自动关联的对象

user.module中引入UploadService 方便调用copyFileToValid方法

diff 复制代码

// src/user/user.module.ts

import { Module } from '@nestjs/common';
import { TypeOrmModule } from '@nestjs/typeorm';
import { UserService } from './user.service';
+ import { UploadService } from 'src/upload/upload.service';
import { UserController } from './user.controller';
import { User } from './user.entity';
import { ResourceMapping } from '../upload/resource-mapping.entity';

@Module({
  imports: [TypeOrmModule.forFeature([
    User,
    ResourceMapping
  ])],
  providers: [
    UserService,
+    UploadService
  ],
  controllers: [UserController],
})
export class UserModule { }

创建API接口：saveProfile 同时删除上篇文中的attachments代码

diff 复制代码

// src/user/user.controller.ts

import { Controller, Post, Body, Param, Get } from '@nestjs/common';
import { UserService } from './user.service';
import { User } from './user.entity';

@Controller('users')
export class UserController {
    constructor(private readonly userService: UserService) { }

    @Post()
    create(@Body() user: User): Promise<User> {
        return this.userService.create(user);
    }

+   @Post(':id/saveProfile')
+    async saveProfile(
+       @Param('id') userId: number,
+        @Body('resourceMappingId') resourceMappingId: number,
+    ) {
+        return await this.userService.saveProfile(userId, resourceMappingId);
+   }

-    @Post(':id/attachments')
-   async addAttachment(
-       @Param('id') userId: number,
-       @Body('resourceMappingId') resourceMappingId: number,
-    ) {
-       return await this.userService.addAttachment(userId, resourceMappingId);
-    }

    @Get(':id')
    async getUserInfo(@Param('id') id: number) {
        return await this.userService.getUserInfo(id);
    }
}

对应添加 userService 中saveProfile方法，同时删除上篇文中的attachments代码

diff 复制代码

// src/user/user.service.ts

import { Injectable } from '@nestjs/common';
import { InjectRepository } from '@nestjs/typeorm';
import { Repository } from 'typeorm';
+ import { UploadService } from 'src/upload/upload.service';
import { User } from './user.entity';
import { ResourceMapping } from '../upload/resource-mapping.entity';

@Injectable()
export class UserService {
    constructor(
        @InjectRepository(User)
        private usersRepository: Repository<User>,
        @InjectRepository(ResourceMapping)
        private readonly resourceMappingRepository: Repository<ResourceMapping>,
+        private readonly uploadService: UploadService,
    ) { }

    create(user: User): Promise<User> {
        return this.usersRepository.save(user);
    }

+    async saveProfile(userId: number, resourceMappingId: number) {
+       return await this.usersRepository.manager.transaction(async (manager) => {
+            const userProfile = await manager.findOne(User, { where: { id: userId } });
+            const resourceMapping = await manager.findOne(ResourceMapping, { where: { id: resourceMappingId } });

+            if (!userProfile || !resourceMapping) {
+                throw new Error('User or resource not found');
+            }
+            const validUrl = await this.uploadService.copyFileToValid(resourceMapping.url);

+           resourceMapping.url = validUrl;
+           resourceMapping.isValid = true;
+           await manager.save(ResourceMapping, resourceMapping);

+            userProfile.avatar = resourceMapping;
+            userProfile.avatarId = resourceMappingId;

+            await manager.save(User, userProfile);

+            return userProfile;
+        });
+   }


-    async addAttachment(userId: number, resourceMappingId: number) {
-        const userProfile = await this.usersRepository.findOne({
-            where: { id: userId },
-        });
-        const resourceMapping = await this.resourceMappingRepository.findOne({
-            where: { id: resourceMappingId },
-       });

-        if (userProfile && resourceMapping) {
-            userProfile.attachment = resourceMapping;
-            userProfile.attachmentId = resourceMappingId;
-            await this.usersRepository.save(userProfile);
-        }

-        return userProfile;
-   }


    async getUserInfo(id: number) {
        const userProfile = await this.usersRepository.findOne({
            where: { id },
            relations: ['attachment'],
        });
        return userProfile;
    }

}

开启事务transaction确保同时操作User和ResourceMapping成功时保存数据，现在通过postman Post 请求：localhost:3000/users/:id/saveProfile

这里的 user id = 1，可以看到saveProfile成功后资源目录从 /temp 复制到了 /valid中，由返回的结果也可以查看

json 复制代码

{
  "id": 1,
  "name": "Neo Luo",
  "avatarId": "13",
  "avatar": {
    "id": 13,
    "name": "psycho.png",
    "url": "http://localhost:3000/uploads/valid/20240810/1723268463646.png"
  }
}

4. 定时任务schedule删除临时资源

引入系统定时任务对临时资源进行删除，使用到@nestjs/schedule来支持改功能，首先安装对应依赖

shell 复制代码

npm install --save @nestjs/schedule

再配置进根app.module 中

diff 复制代码

// src/app.module.ts

import { Module } from '@nestjs/common';
import { TypeOrmModule } from '@nestjs/typeorm';
+import { ScheduleModule } from '@nestjs/schedule';
import { AppController } from './app.controller';
import { AppService } from './app.service';
import { UploadModule } from './upload/upload.module';
import { UserModule } from './user/user.module';


@Module({
  imports: [
    TypeOrmModule.forRoot({
      type: 'mysql',
      host: 'localhost',
      port: 3307,
      username: 'root',
      password: 'example',
      database: 'testdb',
      entities: [__dirname + '/**/*.entity{.ts,.js}'],
      synchronize: true,
    }),
+    ScheduleModule.forRoot(),
    UserModule,
    UploadModule
  ],
  controllers: [AppController],
  providers: [AppService],
})
export class AppModule { }

接下来就是写具体 schedule task，通过nest cli 快速创建 task模块

shell 复制代码

 nest g mo /task
 nest g s /task --no-spec

task.module 具体代码：

ts 复制代码

// src/task/task.module.ts

import { Module } from '@nestjs/common';
import { TypeOrmModule } from '@nestjs/typeorm';
import { TaskService } from './task.service';
import { UploadService } from 'src/upload/upload.service';
import { ResourceMapping } from '../upload/resource-mapping.entity';

@Module({
  imports: [TypeOrmModule.forFeature([ResourceMapping])],
  providers: [TaskService, UploadService],
  controllers: [],
})
export class TaskModule {}

task.service 具体代码：

ts 复制代码

// src/task/task.service.ts

import { Injectable } from '@nestjs/common';
import { Cron } from '@nestjs/schedule';
import { promises as fs } from 'fs';
import { join } from 'path';
import { tempUploadDir } from 'src/upload/constants';
import { UploadService } from 'src/upload/upload.service';
import * as dayjs from 'dayjs';

@Injectable()
export class TaskService {
  constructor(private readonly uploadService: UploadService) {}

  @Cron('0 0 0 * * *')
  async handleCron() {
    const twoDaysAgo = dayjs().subtract(2, 'day').format('YYYYMMDD');
    const dirPath = join(tempUploadDir, twoDaysAgo);
    try {
      await this.uploadService.deleteOldResourceMappings();
      await fs.access(dirPath);
      await fs.rm(dirPath, { recursive: true, force: true });
    } catch (err) {
      if (err.code !== 'ENOENT') {
        console.error(`Error deleting directory: ${err.message}`);
      }
    }
  }
}

每天凌晨0点0分0秒跑定时任务，删除两天前的临时文件和resourceMapping中无效的资源记录，其中@Cron（）具体参数说明：

shell 复制代码

* * * * * *
| | | | | |
| | | | | day of week
| | | | months
| | | day of month
| | hours
| minutes
seconds (optional)

当需要测试定时任务时可以写成每分钟触发：0 * * * * * ，这样方便测试调试，但是上线时候记得改回

还需要在upload service 中配置删除对应资源记录

diff 复制代码

// src/upload/upload.service.ts

import { Repository,
+ LessThanOrEqual
} from 'typeorm';


+ async deleteOldResourceMappings(): Promise<void> {

+        const twoDaysAgoDate = dayjs().subtract(2, 'day').toDate();

+       await this.resourceMappingRepository.delete({
+           createdAt: LessThanOrEqual(twoDaysAgoDate),
+           isValid: false,
+       });
+   }

对 resourceMapping表中删除两天前且isValid = false的记录，至此临时目录以及对应的记录删除完成

总结

通过本文的介绍，我们实现了上传资源到临时目录、在保存数据时将资源移动到有效目录、并使用定时任务来清理临时目录的方案。这一流程不仅优化了资源管理，也减少了存储空间的浪费，使系统更加高效地运行。