文章目录
- day4学习内容
- 自媒体文章自动审核
- [1 自媒体文章自动审核](#1 自媒体文章自动审核)
-
- [1.1 审核流程](#1.1 审核流程)
- [1.2 内容安全第三方接口](#1.2 内容安全第三方接口)
- [1.3 引入阿里云内容安全接口](#1.3 引入阿里云内容安全接口)
-
- [1.3.1 添加依赖](#1.3.1 添加依赖)
- [1.3.2 导入aliyun模块](#1.3.2 导入aliyun模块)
- [1.3.3 注入Bean测试](#1.3.3 注入Bean测试)
- [2 app端文章保存接口](#2 app端文章保存接口)
-
- [2.1 表结构说明](#2.1 表结构说明)
- [2.2 分布式id](#2.2 分布式id)
-
- [2.2.1 分布式id-技术选型](#2.2.1 分布式id-技术选型)
- [2.2.2 雪花算法](#2.2.2 雪花算法)
- [2.2.3 配置雪花算法](#2.2.3 配置雪花算法)
- [2.3 保存app端文章-思路分析](#2.3 保存app端文章-思路分析)
- [2.4 实现接口](#2.4 实现接口)
-
- [2.4.1 实现步骤](#2.4.1 实现步骤)
- [2.4.2 定义feign接口](#2.4.2 定义feign接口)
-
- [2.4.2.1 导入feign远程调用依赖](#2.4.2.1 导入feign远程调用依赖)
- [2.4.2.2 定义文章端远程接口](#2.4.2.2 定义文章端远程接口)
- [2.4.2.3 导入ArticleDto](#2.4.2.3 导入ArticleDto)
- [2.4.3 实现feign接口](#2.4.3 实现feign接口)
- [2.4.4 创建mapper](#2.4.4 创建mapper)
- [2.4.5 为AparticleConfig设置默认参数](#2.4.5 为AparticleConfig设置默认参数)
- [2.4.6 在ApArticleService的实现类ApArticleServiceImpl中实现方法](#2.4.6 在ApArticleService的实现类ApArticleServiceImpl中实现方法)
- [2.4.7 启动ArticleApplication](#2.4.7 启动ArticleApplication)
- [3 自媒体文章审核实现](#3 自媒体文章审核实现)
-
- [3.1 创建审核接口](#3.1 创建审核接口)
- [3.2 实现审核接口](#3.2 实现审核接口)
- [3.3 启动类扫描feign](#3.3 启动类扫描feign)
- [3.4 测试](#3.4 测试)
- [4 自媒体调用文章微服务feign远程调用服务降级](#4 自媒体调用文章微服务feign远程调用服务降级)
-
- [4.1 feign远程调用服务降级处理的逻辑](#4.1 feign远程调用服务降级处理的逻辑)
- [4.2 编写降级逻辑](#4.2 编写降级逻辑)
- [4.3 指定IArticleClient接口指向Feign降级逻辑](#4.3 指定IArticleClient接口指向Feign降级逻辑)
- [4.4 加载feign降级逻辑](#4.4 加载feign降级逻辑)
- [4.5 配置降级策略](#4.5 配置降级策略)
- [4.6 测试](#4.6 测试)
- [5 文章审核异步调用](#5 文章审核异步调用)
-
- [5.1 在自动审核的方法加上@Async注解](#5.1 在自动审核的方法加上@Async注解)
- [5.2 在文章发布后调用自动审核方法](#5.2 在文章发布后调用自动审核方法)
- [5.3 在启动类中添加注解开启异步调用](#5.3 在启动类中添加注解开启异步调用)
- [5.4 综合测试](#5.4 综合测试)
- [5.5 使用rabbit MQ来完成异步调用](#5.5 使用rabbit MQ来完成异步调用)
-
- [5.5.1 引入依赖](#5.5.1 引入依赖)
- [5.5.2 为微服务配置MQ](#5.5.2 为微服务配置MQ)
- [5.5.3 改造方法,创建监听队列](#5.5.3 改造方法,创建监听队列)
- [5.5.4 序列化MQ消息](#5.5.4 序列化MQ消息)
- [5.5.5 加上mq后的综合测试](#5.5.5 加上mq后的综合测试)
- [6 自管理敏感词过滤](#6 自管理敏感词过滤)
-
- [6.1 DFA实现原理](#6.1 DFA实现原理)
- [6.2 DFA检索过程](#6.2 DFA检索过程)
- [6.3 实现步骤](#6.3 实现步骤)
-
- [6.3.1 创建敏感词表](#6.3.1 创建敏感词表)
- [6.3.2 将wm_sensitive对应的实体类和mapper导入](#6.3.2 将wm_sensitive对应的实体类和mapper导入)
- [6.3.3 在阿里云接口前自行进行审查](#6.3.3 在阿里云接口前自行进行审查)
- [6.3.4 测试](#6.3.4 测试)
- [7 图片文字敏感词过滤](#7 图片文字敏感词过滤)
-
- [7.1 文字图片识别](#7.1 文字图片识别)
- [7.2 Tesseract-OCR](#7.2 Tesseract-OCR)
- [7.3 Tess4j案例](#7.3 Tess4j案例)
-
- [7.3.1 导入依赖](#7.3.1 导入依赖)
- [7.3.2 将训练好的分类器放入资源中](#7.3.2 将训练好的分类器放入资源中)
- [7.3.3 demo](#7.3.3 demo)
- [7.3.4 结果](#7.3.4 结果)
- [7.4 图片文字敏感词过滤实现](#7.4 图片文字敏感词过滤实现)
-
- [7.4.1 创建工具类](#7.4.1 创建工具类)
- [7.4.2 工具类被其他微服务使用](#7.4.2 工具类被其他微服务使用)
- [7.4.3 在微服务中配置](#7.4.3 在微服务中配置)
- [7.4.4 添加实现](#7.4.4 添加实现)
- [8 静态文件生成](#8 静态文件生成)
-
- [8.1 实现思路](#8.1 实现思路)
-
- [8.1.1 生成minio接口和实现,并且异步调用](#8.1.1 生成minio接口和实现,并且异步调用)
- [8.1.2 修改saveArticle逻辑](#8.1.2 修改saveArticle逻辑)
- [8.1.3 开启异步调用](#8.1.3 开启异步调用)
- [8.1.4 测试](#8.1.4 测试)
day4学习内容
自媒体文章自动审核
今日内容
1 自媒体文章自动审核
1.1 审核流程
1.2 内容安全第三方接口
1.3 引入阿里云内容安全接口
1.3.1 添加依赖
在heima-leadnews-common包下引入依赖
xml
<dependency>
<groupId>com.aliyun</groupId>
<artifactId>aliyun-java-sdk-core</artifactId>
<version>4.1.1</version>
</dependency>
<dependency>
<groupId>com.aliyun</groupId>
<artifactId>aliyun-java-sdk-green</artifactId>
<version>3.6.6</version>
</dependency>
<dependency>
<groupId>com.alibaba.fastjson2</groupId>
<artifactId>fastjson2</artifactId>
<version>2.0.9</version>
</dependency>
<dependency>
<groupId>com.aliyun.oss</groupId>
<artifactId>aliyun-sdk-oss</artifactId>
<version>2.8.3</version>
</dependency>
1.3.2 导入aliyun模块
放入heima-leadnews-common模块下的com.heima.common
哪个微服务使用,就在哪个微服务的nacos中配置
在heima-leadnews-wemedia中的nacos配置中心添加以下配置:
aliyun:
accessKeyId: LTAI5tCWHCcfvqQzu8k2oKmX
secret: auoKUFsghimbfVQHpy7gtRyBkoR4vc
#aliyun.scenes=porn,terrorism,ad,qrcode,live,logo
scenes: terrorism
1.3.3 注入Bean测试
在resource中META-INF的spring-factories中自动配置
org.springframework.boot.autoconfigure.EnableAutoConfiguration=\
com.heima.common.exception.ExceptionCatch,\
com.heima.common.aliyun.GreenTextScan,\
com.heima.common.aliyun.GreenImageScan
在测试类中进行测试
java
@SpringBootTest(classes = WemediaApplication.class)
@RunWith(SpringRunner.class)
public class AliyunTest {
@Autowired
private GreenTextScan greenTextScan;
@Autowired
private GreenImageScan greenImageScan;
@Autowired
private FileStorageService fileStorageService;
@Test
public void testScanText() throws Exception {
Map map = greenTextScan.greeTextScan("我是一个好人,冰毒");
System.out.println(map);
}
@Test
public void testScanImage() throws Exception {
byte[] bytes = fileStorageService.downLoadFile("http://192.168.200.130:9000/leadnews/2021/04/26/ef3cbe458db249f7bd6fb4339e593e55.jpg");
Map map = greenImageScan.imageScan(Arrays.asList(bytes));
System.out.println(map);
}
}
2 app端文章保存接口
2.1 表结构说明
2.2 分布式id
2.2.1 分布式id-技术选型
2.2.2 雪花算法
2.2.3 配置雪花算法
第一:在实体类中的id上加入如下配置,指定类型为id_worker
java
@TableId(value = "id",type = IdType.ID_WORKER)
private Long id;
第二:在application.yml文件中配置数据中心id和机器id
在文章的微服务的nacos配置中leadnews-article中添加
yaml
spring:
datasource:
driver-class-name: com.mysql.jdbc.Driver
url: jdbc:mysql://localhost:3306/leadnews_article?useUnicode=true&characterEncoding=UTF-8&serverTimezone=UTC&useSSL=false
username: root
password: 123sjbsjb
# 设置Mapper接口所对应的XML文件位置,如果你在Mapper接口中有自定义方法,需要进行该配置
mybatis-plus:
mapper-locations: classpath*:mapper/*.xml
# 设置别名包扫描路径,通过该属性可以给包中的类注册别名
type-aliases-package: com.heima.model.article.pojos
#雪花算法
global-config:
datacenter-id: 1
workerId: 1
minio:
accessKey: minio
secretKey: minio123
bucket: leadnews
endpoint: http://192.168.204.129:9000
readPath: http://192.168.204.129:9000
2.3 保存app端文章-思路分析
2.4 实现接口
2.4.1 实现步骤
2.4.2 定义feign接口
2.4.2.1 导入feign远程调用依赖
在heima-leadnews-feign-api的pom.xml中导入依赖
xml
<dependency>
<groupId>org.springframework.cloud</groupId>
<artifactId>spring-cloud-starter-openfeign</artifactId>
</dependency>
2.4.2.2 定义文章端远程接口
heima-leadnews-feign-api定义com.heima.apis.article.IArticleClient接口
@FeignClient(value = "leadnews-article")
@FeignClient指定文章远程调用接口名称
java
@FeignClient(value = "leadnews-article")
public interface IArticleClient {
@PostMapping("/api/v1/article/save")
public ResponseResult saveArticle(@RequestBody ArticleDto dto) ;
}
2.4.2.3 导入ArticleDto
在heima-leadnews-model模块下com.heima.model.article.dto中导入ArticleDto类
java
@Data
public class ArticleDto extends ApArticle {
/**
* 文章内容
*/
private String content;
}
2.4.3 实现feign接口
在heima-leadnews-service模块下的heima-leadnews-article模块下创建com.heima.article.feign.ArticleClient类
java
@RestController
public class ArticleClient implements IArticleClient {
@Autowired
private ApArticleService apArticleService;
@PostMapping("/api/v1/article/save")
@Override
public ResponseResult saveArticle(@RequestBody ArticleDto dto) {
return apArticleService.saveArticle(dto);
}
}
2.4.4 创建mapper
在heima-leadnews-service模块下的heima-leadnews-article模块下创建com.heima.article.mapper.ApArticleConfigMapper接口
java
@Mapper
public interface ApArticleConfigMapper extends BaseMapper<ApArticleConfig> {
}
2.4.5 为AparticleConfig设置默认参数
添加@NoArgsConstructor
java
public ApArticleConfig(Long articleId) {
this.articleId = articleId;
this.isDelete = false;
this.isDown = false;
this.isForward = true;
this.isComment = true;
}
添加有参构造
2.4.6 在ApArticleService的实现类ApArticleServiceImpl中实现方法
ApArticleService接口
java
public interface ApArticleService extends IService<ApArticle>{
/**
* 加载文章列表
* @param dto
* @param type 1 加载更多 2 加载最新
* @return
*/
public ResponseResult load(ArticleHomeDto dto, Short type);
/**
* 保存文章
* @param dto
* @return
*/
public ResponseResult saveArticle(ArticleHomeDto dto);
}
实现类,实现saveArticle方法
java
@Autowired
private ApArticleConfigMapper apArticleConfigMapper;
@Autowired
private ApArticleContentMapper apArticleContentMapper;
/**
* 保存文章
* @param dto
* @return
*/
@Override
public ResponseResult saveArticle(ArticleDto dto) {
//1.参数检查
if(dto == null){
return ResponseResult.errorResult(AppHttpCodeEnum.PARAM_INVALID);
}
ApArticle apArticle = new ApArticle();
//org.springframework.beans
BeanUtils.copyProperties(dto, apArticle);
//2.判断是否存在id
if(dto.getId() == null) {
//2.1 不存在id ,新增 文章、内容、配置
save(apArticle);
//2.1.2 保存文章配置
ApArticleConfig apArticleConfig = new ApArticleConfig(apArticle.getId());
apArticleConfigMapper.insert(apArticleConfig);
//2.1.3 保存文章内容
ApArticleContent apArticleContent = new ApArticleContent();
apArticleContent.setArticleId(apArticle.getId());
apArticleContent.setContent(dto.getContent());
apArticleContentMapper.insert(apArticleContent);
}
else {
//2.2 存在id,更新 文章、内容
//2.2.1 更新文章
updateById(apArticle);
//2.2.2 更新文章内容
ApArticleContent apArticleContent = apArticleContentMapper.selectOne(Wrappers
.<ApArticleContent>lambdaQuery()
.eq(ApArticleContent::getArticleId, dto.getId()));
apArticleContent.setContent(dto.getContent());
apArticleContentMapper.updateById(apArticleContent);
}
//3.返回结果 文章的id
return ResponseResult.okResult(apArticle.getId());
}
2.4.7 启动ArticleApplication
刚刚是新增,如果是修改。
就会在json中传入id
成功修改
3 自媒体文章审核实现
3.1 创建审核接口
在heima-leadnews-service中heima-leadnews-wemedia中的service新增WmNewAutoScanService接口
java
public interface WmNewAutoScanService {
/**
* 自动审核媒体文章
*/
public void autoScanMediaNews(Integer id);
}
3.2 实现审核接口
java
@Service
@Slf4j
@Transactional
public class WmNewAutoScanServiceImpl implements WmNewAutoScanService {
@Autowired
private WmNewsMapper wmNewsMapper;
@Qualifier("com.heima.apis.article.IArticleClient")
@Autowired
private IArticleClient iArticleClient;
@Autowired
private WmChannelMapper wmChannelMapper;
@Autowired
private WmUserMapper wmUserMapper;
@Override
public void autoScanMediaNews(Integer id) {
//1.查询自媒体文章
WmNews wmNews = wmNewsMapper.selectById(id);
if (wmNews == null) {
throw new RuntimeException("自媒体文章不存在");
}
if(wmNews.getStatus().equals(WmNews.Status.SUBMIT.getCode())){
Map<String,List<String>> scanMaterialsList = extractImageAndContent(wmNews);
//2.调用阿里云接口审核文本内容
List<String> contentTexts =scanMaterialsList.get("contentTexts");
boolean isTextScan =true;
if(!isTextScan)return;
//3.调用阿里云接口审核图片内容
List<String> imagesUrls =scanMaterialsList.get("imagesUrls");
boolean isImageScan =true;
if(!isImageScan)return;
if(isTextScan && isImageScan) {
//审核通过
wmNews.setStatus((short) 9);
wmNews.setReason("审核通过");
}
}
//4.审核成功保存app端的相关文章数据
ArticleDto dto=new ArticleDto();
BeanUtils.copyProperties(wmNews,dto);
//布局
dto.setLayout(wmNews.getType());
//频道
dto.setChannelId(wmNews.getChannelId());
//频道名称
WmChannel wmChannel = wmChannelMapper.selectById(wmNews.getChannelId());
if(wmChannel!=null){
dto.setChannelName(wmChannel.getName());
}
//作者
dto.setAuthorId(Long.valueOf(wmNews.getUserId()));
//作者名称
WmUser wmUser= wmUserMapper.selectById(wmNews.getUserId());
if(wmUser!=null){
dto.setAuthorName(wmUser.getName());
}
//设置文章id
if(wmNews.getArticleId()!=null){
dto.setId(wmNews.getArticleId());
}
dto.setCreatedTime(new Date());
ResponseResult responseResult = iArticleClient.saveArticle(dto);
if(responseResult.getCode().equals(200)){
//保存成功
wmNews.setArticleId((Long)responseResult.getData());
wmNewsMapper.updateById(wmNews);
}
else{
//保存失败
throw new RuntimeException("保存app端文章失败");
}
}
private Map<String,List<String>> extractImageAndContent(WmNews wmNews) {
//提取文章内容
String content = wmNews.getContent();
List<String> imagesUrls =new ArrayList<>();
List<String> contentTexts =new ArrayList<>();
Map<String,List<String>> scanMaterialsList =new HashMap<>();
List<Map> maps = JSON.parseArray(content, Map.class);
//提取文章图片
for (Map map : maps) {
if(map.get("type").equals("image")){
String imgUrl = (String) map.get("value");
imagesUrls.add(imgUrl);
}
if(map.get("type").equals("text")){
String text = (String) map.get("value");
contentTexts.add(text);
}
}
scanMaterialsList.put("imagesUrls",imagesUrls);
scanMaterialsList.put("contentTexts",contentTexts);
return scanMaterialsList;
}
}
3.3 启动类扫描feign
调用Feign远程接口时,要在启动类中加入@EnableFeignClients(basePackages = "com.heima.apis")来对feign的api进行扫描,同时也要引入feign-api模块的依赖
xml
<dependency>
<groupId>com.heima</groupId>
<artifactId>heima-leadnews-feign-api</artifactId>
</dependency>
java
@SpringBootApplication
@EnableDiscoveryClient
@MapperScan("com.heima.wemedia.mapper")
@EnableFeignClients(basePackages = "com.heima.apis")
public class WemediaApplication {
public static void main(String[] args) {
SpringApplication.run(WemediaApplication.class,args);
}
@Bean
public MybatisPlusInterceptor mybatisPlusInterceptor() {
MybatisPlusInterceptor interceptor = new MybatisPlusInterceptor();
interceptor.addInnerInterceptor(new PaginationInnerInterceptor(DbType.MYSQL));
return interceptor;
}
}
3.4 测试
转到WmNewAutoScanService接口中,ctrl+shift+T创建测试类
java
@SpringBootTest(classes = WemediaApplication.class)
@RunWith(SpringRunner.class)
class WmNewAutoScanServiceTest {
@Autowired
private WmNewAutoScanService wmNewAutoScanService;
@Test
void autoScanMediaNews() {
wmNewAutoScanService.autoScanMediaNews(6236);
}
}
4 自媒体调用文章微服务feign远程调用服务降级
4.1 feign远程调用服务降级处理的逻辑
4.2 编写降级逻辑
在heima-leadnews-feign-api模块下编写降级逻辑com.heima.apis.article.fallback.IArticleClientFallback类,实现IArticleClient接口
java
@Component
public class IArticleClientFallback implements IArticleClient {
@Override
public ResponseResult saveArticle(ArticleDto dto) {
return ResponseResult.errorResult(AppHttpCodeEnum.SERVER_ERROR,"获取数据失败");
}
}
4.3 指定IArticleClient接口指向Feign降级逻辑
@FeignClient(value = "leadnews-article",fallback = IArticleClientFallback.class)
java
@FeignClient(value = "leadnews-article",fallback = IArticleClientFallback.class)
public interface IArticleClient {
@PostMapping("/api/v1/article/save")
public ResponseResult saveArticle(@RequestBody ArticleDto dto) ;
}
4.4 加载feign降级逻辑
因为IArticleClientFallback是在com.heima.apis.article.fallback包下,并不能被spring通过@Component直接加载
因此需要在使用的微服务中加载feign
如使用的微服务是heima-leadnews-wemedia,所以要在com.heima.wemedia.config下创建InitConfig类加载feign降级策略
java
@Configuration
@ComponentScan("com.heima.apis.article.fallback")
public class InitConfig {
}
4.5 配置降级策略
要么在bootstrap中开启,要么在nacos中实现热更新
这里采用nacos热更新
yaml
feign:
# 开启feign对hystrix熔断降级的支持
hystrix:
enabled: true
# 修改调用超时时间
client:
config:
default:
connectTimeout: 2000
readTimeout: 2000
4.6 测试
当前设置超时2s进行降级,测试一下
在com.heima.article.service.impl.ApArticleServiceImpl类中的saveArticle方法添加睡眠3秒进行测试
java
@Override
public ResponseResult saveArticle(ArticleDto dto) {
try {
Thread.sleep(3000);
} catch (InterruptedException e) {
throw new RuntimeException(e);
}
//1.参数检查
if(dto == null){
这次审核6239
java
@SpringBootTest(classes = WemediaApplication.class)
@RunWith(SpringRunner.class)
class WmNewAutoScanServiceTest {
@Autowired
private WmNewAutoScanService wmNewAutoScanService;
@Test
void autoScanMediaNews() {
wmNewAutoScanService.autoScanMediaNews(6239);
}
}
5 文章审核异步调用
5.1 在自动审核的方法加上@Async注解
Springboot集成异步线程调用
java
@Override
@Async//表明这是一个异步方法
public void autoScanMediaNews(Integer id) {
try {
Thread.sleep(1000);
} catch (InterruptedException e) {
throw new RuntimeException(e);
}
5.2 在文章发布后调用自动审核方法
//5.审核文章
wmNewAutoScanService.autoScanMediaNews(wmNews.getId());
java
@Autowired
private WmNewAutoScanService wmNewAutoScanService;
@Override
public ResponseResult submitNews(WmNewsDto wmNewsDto) {
// 0.参数检查
if(wmNewsDto == null||wmNewsDto.getContent()==null){
return ResponseResult.errorResult(AppHttpCodeEnum.PARAM_INVALID);
}
//1. 保存或修改文章
WmNews wmNews = new WmNews();
BeanUtils.copyProperties(wmNewsDto,wmNews);
//1.1 封面
if(wmNewsDto.getImages()!=null&& wmNewsDto.getImages().size()>0){
String imageStr = StringUtils.join(wmNewsDto.getImages(), ",");
wmNews.setImages(imageStr);
}
//1.2 如果封面为自动-1,则需要手动设置封面规则
if(wmNewsDto.getType().equals(WemediaConstants.WM_NEWS_TYPE_AUTO)){
wmNews.setType(null);
}
saveOrUpdateWmNews(wmNews);
//2.判断是否为草稿,如果为草稿结束当前方法
if(wmNews.getStatus().equals(WmNews.Status.NORMAL.getCode())){
return ResponseResult.okResult(AppHttpCodeEnum.SUCCESS);
}
//3.不是草稿,保存文章内容与图片素材的关系
//3.1 获取文章内容的图片素材
List<String> imageList=extractUrlInfo(wmNewsDto.getContent());
saveRelativeInfoForContent(imageList,wmNews.getId());
//4.不是草稿,保存文章封面图片与图片素材的关系
saveRelativeInfoForCover(wmNewsDto,wmNews,imageList);
//5.审核文章
wmNewAutoScanService.autoScanMediaNews(wmNews.getId());
return ResponseResult.okResult(AppHttpCodeEnum.SUCCESS);
}
5.3 在启动类中添加注解开启异步调用
在自媒体引导类中使用@EnableAsync注解开启异步调用
java
@SpringBootApplication
@EnableDiscoveryClient
@MapperScan("com.heima.wemedia.mapper")
@EnableFeignClients(basePackages = "com.heima.apis")
@EnableAsync//开启异步
public class WemediaApplication {
public static void main(String[] args) {
SpringApplication.run(WemediaApplication.class,args);
}
@Bean
public MybatisPlusInterceptor mybatisPlusInterceptor() {
MybatisPlusInterceptor interceptor = new MybatisPlusInterceptor();
interceptor.addInnerInterceptor(new PaginationInnerInterceptor(DbType.MYSQL));
return interceptor;
}
}
5.4 综合测试
5.5 使用rabbit MQ来完成异步调用
我的异步调用只要在启动类中加入@EnableAsync就报错,迫不得已采用rabbitMQ
5.5.1 引入依赖
在heima-leadnews-service中引入依赖
xml
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-amqp</artifactId>
</dependency>
<dependency>
<groupId>com.fasterxml.jackson.core</groupId>
<artifactId>jackson-databind</artifactId>
</dependency>
5.5.2 为微服务配置MQ
在heima-leadnews-article和wemedia的配置文件中添加配置
yaml
spring:
rabbitmq:
host: 192.168.204.129
port: 5672
virtual-host: /
username: itcast
password: 123321
5.5.3 改造方法,创建监听队列
修改heima-leadnews-wemedia下的com.heima.wemedia.service.impl.WmNewAutoScanServiceImpl类中的autoScanMediaNews方法
java
@Autowired
private RabbitTemplate rabbitTemplate;
@Override
public void autoScanMediaNews(Integer id) {
//1.查询自媒体文章
WmNews wmNews = wmNewsMapper.selectById(id);
if (wmNews == null) {
throw new RuntimeException("自媒体文章不存在");
}
if(wmNews.getStatus().equals(WmNews.Status.SUBMIT.getCode())){
Map<String,List<String>> scanMaterialsList = extractImageAndContent(wmNews);
//2.调用阿里云接口审核文本内容
List<String> contentTexts =scanMaterialsList.get("contentTexts");
boolean isTextScan =true;
if(!isTextScan)return;
//3.调用阿里云接口审核图片内容
List<String> imagesUrls =scanMaterialsList.get("imagesUrls");
boolean isImageScan =true;
if(!isImageScan)return;
if(isTextScan && isImageScan) {
//审核通过
wmNews.setStatus((short) 9);
wmNews.setReason("审核通过");
}
}
//4.审核成功保存app端的相关文章数据
ArticleDto dto=new ArticleDto();
BeanUtils.copyProperties(wmNews,dto);
//布局
dto.setLayout(wmNews.getType());
//频道
dto.setChannelId(wmNews.getChannelId());
//频道名称
WmChannel wmChannel = wmChannelMapper.selectById(wmNews.getChannelId());
if(wmChannel!=null){
dto.setChannelName(wmChannel.getName());
}
//作者
dto.setAuthorId(Long.valueOf(wmNews.getUserId()));
//作者名称
WmUser wmUser= wmUserMapper.selectById(wmNews.getUserId());
if(wmUser!=null){
dto.setAuthorName(wmUser.getName());
}
//设置文章id
if(wmNews.getArticleId()!=null){
dto.setId(wmNews.getArticleId());
}
dto.setCreatedTime(new Date());
//2.rabbitmq异步处理
Map<String,Object> map=new HashMap<>();
map.put("dto",dto);
map.put("wmNewsId",id);
rabbitTemplate.convertAndSend("article.queue", map);
/*ResponseResult responseResult = iArticleClient.saveArticle(dto);
if(responseResult.getCode().equals(200)){
//保存成功
wmNews.setArticleId((Long)responseResult.getData());
wmNewsMapper.updateById(wmNews);
}
else{
//保存失败
log.error("保存app端文章失败,responseResult: {}", responseResult);
throw new RuntimeException("保存app端文章失败");
}*/
}
rabbitTemplate.convertAndSend("article.queue", map);
发送到article.queue队列
在heima-leadnews-article模块下创建com.heima.article.mq.ArticleMessageConsumer消费者监听类监听article.queue
java
@Slf4j
@Component
public class ArticleMessageConsumer {
@Autowired
private IArticleClient iArticleClient;
@Autowired
private RabbitTemplate rabbitTemplate;
@RabbitListener(bindings =@QueueBinding(
value=@Queue(name="article.queue"),
exchange=@Exchange(name="article.direct",type= ExchangeTypes.FANOUT)))
public void processMessage(Map<String,Object> map) {
ObjectMapper objectMapper = new ObjectMapper();
Object dto = map.get("dto");
Integer id= (Integer) map.get("wmNewsId");
LinkedHashMap<String, Object> linkedHashMap = (LinkedHashMap<String, Object>) dto;
ArticleDto articleDto = objectMapper.convertValue(linkedHashMap, ArticleDto.class);
// 异步处理文章数据
ResponseResult responseResult = iArticleClient.saveArticle(articleDto);
if(responseResult.getCode().equals(200)){
WmNews wmNews = new WmNews();
BeanUtils.copyProperties(dto, wmNews);
wmNews.setArticleId((Long)responseResult.getData());
Map<String,Object> params = new HashMap<>();
params.put("id", id);
params.put("wmNews", wmNews);
params.put("articleId",(Long)responseResult.getData());
rabbitTemplate.convertAndSend("wmNews.queue", params);
log.info("发送params成功,param: {}", params);
}
else{
//保存失败
log.error("保存app端文章失败,responseResult: {}", responseResult);
throw new RuntimeException("保存app端文章失败");
}
}
}
将ResponseResult responseResult = iArticleClient.saveArticle(articleDto);
回填的id发到wmNews.queue
在heima-leadnews-wemedia模块下创建com.heima.wemedia.mq.ReceiveWmNewsId消费者监听类监听wmNews.queue
java
@Component
@Slf4j
public class ReceiveWmNewsId {
@Autowired
private WmNewsMapper wmNewsMapper;
@RabbitListener(bindings =@QueueBinding(
value=@Queue(name="wmNews.queue"),
exchange=@Exchange(name="wmNews.direct",type= ExchangeTypes.FANOUT)))
public void processMessage(Map<String,Object> map) {
ObjectMapper objectMapper = new ObjectMapper();
Integer id= (Integer)map.get("id");
Object wmNews= map.get("wmNews");
Long articleId= (Long)map.get("articleId");
LinkedHashMap<String, Object> linkedHashMap = (LinkedHashMap<String, Object>) wmNews;
WmNews articleDto = objectMapper.convertValue(linkedHashMap, WmNews.class);
WmNews oldwmNews = wmNewsMapper.selectById(id);
BeanUtils.copyProperties(wmNews,oldwmNews);
oldwmNews.setStatus((short) 9);
oldwmNews.setReason("审核通过");
oldwmNews.setArticleId(articleId);
int i = wmNewsMapper.updateById(oldwmNews);
if(i == 0){
log.error("更新自媒体文章失败,wmNews: {}", oldwmNews);
throw new RuntimeException("更新自媒体文章失败");
}
}
}
5.5.4 序列化MQ消息
在heima-leadnews-article和wemedia的启动类中添加序列化器
java
@SpringBootApplication
@EnableDiscoveryClient
@MapperScan("com.heima.wemedia.mapper")
@EnableFeignClients(basePackages = "com.heima.apis")
public class WemediaApplication {
public static void main(String[] args) {
SpringApplication.run(WemediaApplication.class,args);
}
@Bean
public MybatisPlusInterceptor mybatisPlusInterceptor() {
MybatisPlusInterceptor interceptor = new MybatisPlusInterceptor();
interceptor.addInnerInterceptor(new PaginationInnerInterceptor(DbType.MYSQL));
return interceptor;
}
@Bean
public MessageConverter messageConverter(){
return new Jackson2JsonMessageConverter();
}
@Bean
public RabbitTemplate rabbitTemplate(ConnectionFactory connectionFactory, MessageConverter messageConverter) {
RabbitTemplate rabbitTemplate = new RabbitTemplate(connectionFactory);
rabbitTemplate.setMessageConverter(messageConverter);
return rabbitTemplate;
}
}
java
@SpringBootApplication
@EnableDiscoveryClient
@MapperScan("com.heima.article.mapper")
public class ArticleApplication {
public static void main(String[] args) {
SpringApplication.run(ArticleApplication.class,args);
}
@Bean
public MybatisPlusInterceptor mybatisPlusInterceptor() {
MybatisPlusInterceptor interceptor = new MybatisPlusInterceptor();
interceptor.addInnerInterceptor(new PaginationInnerInterceptor(DbType.MYSQL));
return interceptor;
}
@Bean
public MessageConverter messageConverter(){
return new Jackson2JsonMessageConverter();
}
@Bean
public RabbitTemplate rabbitTemplate(ConnectionFactory connectionFactory, MessageConverter messageConverter) {
RabbitTemplate rabbitTemplate = new RabbitTemplate(connectionFactory);
rabbitTemplate.setMessageConverter(messageConverter);
return rabbitTemplate;
}
}
5.5.5 加上mq后的综合测试
测试通过在MQ上也检测到消息
6 自管理敏感词过滤
6.1 DFA实现原理
6.2 DFA检索过程
6.3 实现步骤
6.3.1 创建敏感词表
在leadnews-wemedia数据库中到入wm_sensitive.sql
6.3.2 将wm_sensitive对应的实体类和mapper导入
java
@Data
@TableName("wm_sensitive")
public class WmSensitive implements Serializable {
private static final long serialVersionUID = 1L;
/**
* 主键
*/
@TableId(value = "id", type = IdType.AUTO)
private Integer id;
/**
* 敏感词
*/
@TableField("sensitives")
private String sensitives;
/**
* 创建时间
*/
@TableField("created_time")
private Date createdTime;
}
java
@Mapper
public interface WmSensitiveMapper extends BaseMapper<WmSensitive> {
}
6.3.3 在阿里云接口前自行进行审查
boolean isSensitive= handleSensitiveWords(contentTexts,wmNews);
java
if(wmNews.getStatus().equals(WmNews.Status.SUBMIT.getCode())){
Map<String,List<String>> scanMaterialsList = extractImageAndContent(wmNews);
//2.调用阿里云接口审核文本内容
List<String> contentTexts =scanMaterialsList.get("contentTexts");
//2.1 敏感词过滤
boolean isSensitive= handleSensitiveWords(contentTexts,wmNews);
boolean isTextScan =true;
if(!isTextScan)return;
//3.调用阿里云接口审核图片内容
List<String> imagesUrls =scanMaterialsList.get("imagesUrls");
boolean isImageScan =true;
java
@Autowired
private WmSensitiveMapper wmSensitiveMapper;
private boolean handleSensitiveWords(List<String> contentTexts, WmNews wmNews) {
boolean isSensitive = true;
//1.获取所有敏感词
List<WmSensitive> wmSensitiveList = wmSensitiveMapper.selectList(Wrappers.<WmSensitive>lambdaQuery().select(WmSensitive::getSensitives));
List<String> collect = wmSensitiveList.stream().map(WmSensitive::getSensitives).collect(Collectors.toList());
//2.初始化敏感词库
SensitiveWordUtil.initMap(collect);
//3.遍历文章内容查看是否包含敏感词
for(String contentText:contentTexts){
Map<String, Integer> map = SensitiveWordUtil.matchWords(contentText);
if(map.size()>0){
//4.如果包含敏感词,修改文章状态
wmNews.setStatus((short) 2);
wmNews.setReason("文章内容包含敏感词");
wmNewsMapper.updateById(wmNews);
isSensitive = false;
break;
}
}
return isSensitive;
}
6.3.4 测试
7 图片文字敏感词过滤
7.1 文字图片识别
7.2 Tesseract-OCR
7.3 Tess4j案例
7.3.1 导入依赖
在heima-leadnews-test模块下的tess4j-demo的模块下导入依赖
xml
<dependency>
<groupId>net.sourceforge.tess4j</groupId>
<artifactId>tess4j</artifactId>
<version>4.1.1</version>
</dependency>
7.3.2 将训练好的分类器放入资源中
7.3.3 demo
在tess4j-demo的Applcation中
java
public class Application {
/**
* 识别图片中的文字
* @param args
*/
public static void main(String[] args) {
// 1.创建Tesseract对象
Tesseract tesseract = new Tesseract();
// 2.设置训练库的位置
tesseract.setDatapath("D:\\Code\\JavaCode\\HeimaToutiao\\heima-leadnews\\heima-leadnews-test\\tess4j-demo\\src\\main\\resources\\tessdata");
// 3.设置识别语言
tesseract.setLanguage("chi_sim");
// 4.设置识别图片
File file = new File("D:\\Code\\JavaCode\\HeimaToutiao\\heima-leadnews\\heima-leadnews-test\\tess4j-demo\\src\\main\\resources\\testdata\\testImage.png");
// 5.识别图片
try {
String result = tesseract.doOCR(file);
System.out.println(result.replace("\\n|\\r", ""));
} catch (TesseractException e) {
e.printStackTrace();
}
}
}
7.3.4 结果
7.4 图片文字敏感词过滤实现
7.4.1 创建工具类
在heima-leadnews-common中创建com.heima.common.tess4j.Tess4jClient工具类,封装tess4j
java
@Getter
@Setter
@Component
@ConfigurationProperties(prefix = "tess4j")
public class Tess4jClient {
private String dataPath;
private String language;
public String doOCR(BufferedImage image) throws TesseractException {
//创建Tesseract对象
ITesseract tesseract = new Tesseract();
//设置字体库路径
tesseract.setDatapath(dataPath);
//中文识别
tesseract.setLanguage(language);
//执行ocr识别
String result = tesseract.doOCR(image);
//替换回车和tal键 使结果为一行
result = result.replaceAll("\\r|\\n", "-").replaceAll(" ", "");
return result;
}
}
7.4.2 工具类被其他微服务使用
想让工具类被其他微服务使用就要拷贝全路径,在当前的resource中的META-INF的spring.factories中添加配置
org.springframework.boot.autoconfigure.EnableAutoConfiguration=\
com.heima.common.exception.ExceptionCatch,\
com.heima.common.aliyun.GreenTextScan,\
com.heima.common.aliyun.GreenImageScan,\
com.heima.common.tess4j.Tess4jClient
7.4.3 在微服务中配置
在heima-leadnews-wemedia中的resource的boostrap.yml中进行配置
yaml
tess4j:
data-path: D:\Code\JavaCode\HeimaToutiao\heima-leadnews\heima-leadnews-test\tess4j-demo\src\main\resources\tessdata
language: chi_sim
7.4.4 添加实现
在WmNewsAutoScanServiceImpl中的handleImageScan方法上添加如下代码
java
try {
for (String image : images) {
byte[] bytes = fileStorageService.downLoadFile(image);
//图片识别文字审核---begin-----
//从byte[]转换为butteredImage
ByteArrayInputStream in = new ByteArrayInputStream(bytes);
BufferedImage imageFile = ImageIO.read(in);
//识别图片的文字
String result = tess4jClient.doOCR(imageFile);
//审核是否包含自管理的敏感词
boolean isSensitive = handleSensitiveScan(result, wmNews);
if(!isSensitive){
return isSensitive;
}
//图片识别文字审核---end-----
imageList.add(bytes);
}
}catch (Exception e){
e.printStackTrace();
}
8 静态文件生成
8.1 实现思路
我们在保存/修改文章时就应该同时异步的的生成静态文件,生成静态文件上传到minio中
8.1.1 生成minio接口和实现,并且异步调用
在com.heima.article.service.ArticleFreemarkerService接口
生成静态文件,上传到minio中
java
public interface ArticleFreemarkerService {
/**
* 生成静态化页面
* @param apArticle
* @param content
*/
public void buildArticleToMinio(ApArticle apArticle,String content);
}
java
@Service
@Slf4j
@Transactional
public class ArticleFreemarkerServiceImpl implements ArticleFreemarkerService {
@Autowired
private ApArticleContentMapper apArticleContentMapper;
@Autowired
private Configuration configuration;
@Autowired
private FileStorageService fileStorageService;
@Autowired
private ApArticleService apArticleService;
/**
* 生成静态化页面
* @param apArticle
* @param content
*/
@Async
@Override
public void buildArticleToMinio(ApArticle apArticle, String content) {
if(StringUtils.isNotBlank(content)){
//1.文章内容通过freemarker生成静态html页面
Template template = null;
//2 输出流
StringWriter writer = new StringWriter();
try {
template = configuration.getTemplate("article.ftl");
//2.1 创建模型
Map<String,Object> contentDataModel=new HashMap();
//content是固定的,因为article.ftl中有<#if content??>${content}</#if>
//因为apArticleContent.getContent()获取的是字符串,所以需要转换成对象
contentDataModel.put("content", JSONArray.parseArray(content));
//2.2 合成方法
template.process(contentDataModel,writer);
} catch (Exception e) {
throw new RuntimeException(e);
}
//3.把静态页面上传到minio
//3.1 文件流
InputStream inputStream = new ByteArrayInputStream(writer.toString().getBytes());
String path = fileStorageService.uploadHtmlFile("",apArticle.getId()+".html",inputStream);
//4.把静态页面的路径保存到数据库
apArticleService.update(Wrappers
.<ApArticle>lambdaUpdate()
.eq(ApArticle::getId,apArticle.getId())
.set(ApArticle::getStaticUrl,path));
}
}
}
8.1.2 修改saveArticle逻辑
修改com.heima.article.service.impl.ApArticleServiceImpl的saveArticle方法,添加buildArticleToMinio
java
articleFreemarkerService.buildArticleToMinio(apArticle, dto.getContent());
java
@Autowired
private ApArticleConfigMapper apArticleConfigMapper;
@Autowired
private ApArticleContentMapper apArticleContentMapper;
@Autowired
private ArticleFreemarkerService articleFreemarkerService;
/**
* 保存文章
* @param dto
* @return
*/
@Override
public ResponseResult saveArticle(ArticleDto dto) {
//1.参数检查
if(dto == null){
return ResponseResult.errorResult(AppHttpCodeEnum.PARAM_INVALID);
}
ApArticle apArticle = new ApArticle();
//org.springframework.beans
BeanUtils.copyProperties(dto, apArticle);
//2.判断是否存在id
if(dto.getId() == null) {
//2.1 不存在id ,新增 文章、内容、配置
save(apArticle);
//2.1.2 保存文章配置
ApArticleConfig apArticleConfig = new ApArticleConfig(apArticle.getId());
apArticleConfigMapper.insert(apArticleConfig);
//2.1.3 保存文章内容
ApArticleContent apArticleContent = new ApArticleContent();
apArticleContent.setArticleId(apArticle.getId());
apArticleContent.setContent(dto.getContent());
apArticleContentMapper.insert(apArticleContent);
}
else {
//2.2 存在id,更新 文章、内容
//2.2.1 更新文章
updateById(apArticle);
//2.2.2 更新文章内容
ApArticleContent apArticleContent = apArticleContentMapper.selectOne(Wrappers
.<ApArticleContent>lambdaQuery()
.eq(ApArticleContent::getArticleId, dto.getId()));
apArticleContent.setContent(dto.getContent());
apArticleContentMapper.updateById(apArticleContent);
}
//异步调用 生成静态文件上传到minio中
articleFreemarkerService.buildArticleToMinio(apArticle, dto.getContent());
//3.返回结果 文章的id
return ResponseResult.okResult(apArticle.getId());
}
}
8.1.3 开启异步调用
引导类加上@EnableAsyn
java
@SpringBootApplication
@EnableDiscoveryClient
@MapperScan("com.heima.article.mapper")
@EnableAsync
public class ArticleApplication {
public static void main(String[] args) {
SpringApplication.run(ArticleApplication.class,args);
}
@Bean
public MybatisPlusInterceptor mybatisPlusInterceptor() {
MybatisPlusInterceptor interceptor = new MybatisPlusInterceptor();
interceptor.addInnerInterceptor(new PaginationInnerInterceptor(DbType.MYSQL));
return interceptor;
}
@Bean
public MessageConverter messageConverter(){
return new Jackson2JsonMessageConverter();
}
@Bean
public RabbitTemplate rabbitTemplate(ConnectionFactory connectionFactory, MessageConverter messageConverter) {
RabbitTemplate rabbitTemplate = new RabbitTemplate(connectionFactory);
rabbitTemplate.setMessageConverter(messageConverter);
return rabbitTemplate;
}
}
8.1.4 测试
查看minio有没有生成
生成成功,查看数据库,有html生成,说明功能成功