WebRTC音视频通话-实现GPUImage视频美颜滤镜效果iOS

WebRTC音视频通话-实现GPUImage视频美颜滤镜效果

在WebRTC音视频通话的GPUImage美颜效果图如下

可以看下

之前搭建ossrs服务,可以查看:https://blog.csdn.net/gloryFlow/article/details/132257196

之前实现iOS端调用ossrs音视频通话,可以查看:https://blog.csdn.net/gloryFlow/article/details/132262724

之前WebRTC音视频通话高分辨率不显示画面问题,可以查看:https://blog.csdn.net/gloryFlow/article/details/132262724

修改SDP中的码率Bitrate,可以查看:https://blog.csdn.net/gloryFlow/article/details/132263021

一、GPUImage是什么?

GPUImage是iOS上一个基于OpenGL进行图像处理的开源框架,内置大量滤镜,架构灵活,可以在其基础上很轻松地实现各种图像处理功能。

GPUImage中包含各种滤镜,这里我不会使用那么多,使用的是GPUImageLookupFilter及GPUImagePicture

GPUImage中有一个专门针对lookup table进行处理的滤镜函数GPUImageLookupFilter,使用这个函数就可以直接对图片进行滤镜添加操作了。代码如下

objectivec 复制代码
/**
 GPUImage中有一个专门针对lookup table进行处理的滤镜函数GPUImageLookupFilter,使用这个函数就可以直接对图片进行滤镜添加操作了。
 originalImg是你希望添加滤镜的原始图片
 
 @param image 原图
 @return 处理后的图片
 */
+ (UIImage *)applyLookupFilter:(UIImage *)image lookUpImage:(UIImage *)lookUpImage {
    
    if (lookUpImage == nil) {
        return image;
    }
    
    UIImage *inputImage = image;
    
    UIImage *outputImage = nil;
    
    GPUImagePicture *stillImageSource = [[GPUImagePicture alloc] initWithImage:inputImage];
    //添加滤镜
    GPUImageLookupFilter *lookUpFilter = [[GPUImageLookupFilter alloc] init];
    
    //导入之前保存的NewLookupTable.png文件
    GPUImagePicture *lookupImg = [[GPUImagePicture alloc] initWithImage:lookUpImage];
    
    [lookupImg addTarget:lookUpFilter atTextureLocation:1];
    
    [stillImageSource addTarget:lookUpFilter atTextureLocation:0];
    [lookUpFilter useNextFrameForImageCapture];
    
    if([lookupImg processImageWithCompletionHandler:nil] && [stillImageSource processImageWithCompletionHandler:nil]) {
        outputImage= [lookUpFilter imageFromCurrentFramebuffer];
        
    }
    
    return outputImage;
}

这个需要lookUpImage,图列表如下

由于暂时没有整理demo的git

这里在使用applyLomofiFilter再试下效果

SDApplyFilter.m中的几个方法

objectivec 复制代码
+ (UIImage *)applyBeautyFilter:(UIImage *)image {
    GPUImageBeautifyFilter *filter = [[GPUImageBeautifyFilter alloc] init];
    [filter forceProcessingAtSize:image.size];
    GPUImagePicture *pic = [[GPUImagePicture alloc] initWithImage:image];
    [pic addTarget:filter];
    [pic processImage];
    [filter useNextFrameForImageCapture];
    
    return [filter imageFromCurrentFramebuffer];
}


/**
 Amatorka滤镜 Rise滤镜,可以使人像皮肤得到很好的调整

 @param image image
 @return 处理后的图片
 */
+ (UIImage *)applyAmatorkaFilter:(UIImage *)image
{
    GPUImageAmatorkaFilter *filter = [[GPUImageAmatorkaFilter alloc] init];
    [filter forceProcessingAtSize:image.size];
    GPUImagePicture *pic = [[GPUImagePicture alloc] initWithImage:image];
    [pic addTarget:filter];
    
    [pic processImage];
    [filter useNextFrameForImageCapture];
    return [filter imageFromCurrentFramebuffer];
}

/**
 复古型滤镜,感觉像旧上海滩

 @param image image
 @return 处理后的图片
 */
+ (UIImage *)applySoftEleganceFilter:(UIImage *)image
{
    GPUImageSoftEleganceFilter *filter = [[GPUImageSoftEleganceFilter alloc] init];
    [filter forceProcessingAtSize:image.size];
    GPUImagePicture *pic = [[GPUImagePicture alloc] initWithImage:image];
    [pic addTarget:filter];
    
    [pic processImage];
    [filter useNextFrameForImageCapture];
    return [filter imageFromCurrentFramebuffer];
}


/**
 图像黑白化,并有大量噪点

 @param image 原图
 @return 处理后的图片
 */
+ (UIImage *)applyLocalBinaryPatternFilter:(UIImage *)image
{
    GPUImageLocalBinaryPatternFilter *filter = [[GPUImageLocalBinaryPatternFilter alloc] init];
    
    [filter forceProcessingAtSize:image.size];
    GPUImagePicture *pic = [[GPUImagePicture alloc] initWithImage:image];
    [pic addTarget:filter];
    [pic processImage];
    [filter useNextFrameForImageCapture];
    
    return [filter imageFromCurrentFramebuffer];
}

/**
 单色滤镜
 
 @param image 原图
 @return 处理后的图片
 */
+ (UIImage *)applyMonochromeFilter:(UIImage *)image
{
    GPUImageMonochromeFilter *filter = [[GPUImageMonochromeFilter alloc] init];
    
    [filter forceProcessingAtSize:image.size];
    GPUImagePicture *pic = [[GPUImagePicture alloc] initWithImage:image];
    [pic addTarget:filter];
    [pic processImage];
    [filter useNextFrameForImageCapture];
    
    return [filter imageFromCurrentFramebuffer];
}

使用GPUImageSoftEleganceFilter复古型滤镜,感觉像旧上海滩效果图如下

使用GPUImageLocalBinaryPatternFilter图像黑白化效果图如下

使用GPUImageMonochromeFilter 效果图如下

二、WebRTC实现音视频通话中视频滤镜处理

之前实现iOS端调用ossrs音视频通话,可以查看:https://blog.csdn.net/gloryFlow/article/details/132262724

这个已经有完整的代码了,这里暂时做一下调整。

为RTCCameraVideoCapturer的delegate指向代理

objectivec 复制代码
- (RTCVideoTrack *)createVideoTrack {
    RTCVideoSource *videoSource = [self.factory videoSource];
    self.localVideoSource = videoSource;

    // 如果是模拟器
    if (TARGET_IPHONE_SIMULATOR) {
        if (@available(iOS 10, *)) {
            self.videoCapturer = [[RTCFileVideoCapturer alloc] initWithDelegate:self];
        } else {
            // Fallback on earlier versions
        }
    } else{
        self.videoCapturer = [[RTCCameraVideoCapturer alloc] initWithDelegate:self];
    }
    
    RTCVideoTrack *videoTrack = [self.factory videoTrackWithSource:videoSource trackId:@"video0"];
    
    return videoTrack;
}

实现RTCVideoCapturerDelegate的方法didCaptureVideoFrame

objectivec 复制代码
#pragma mark - RTCVideoCapturerDelegate处理代理
- (void)capturer:(RTCVideoCapturer *)capturer didCaptureVideoFrame:(RTCVideoFrame *)frame {
//    DebugLog(@"capturer:%@ didCaptureVideoFrame:%@", capturer, frame);

// 调用SDWebRTCBufferFliter的滤镜处理
    RTCVideoFrame *aFilterVideoFrame;
    if (self.delegate && [self.delegate respondsToSelector:@selector(webRTCClient:didCaptureVideoFrame:)]) {
        aFilterVideoFrame = [self.delegate webRTCClient:self didCaptureVideoFrame:frame];
    }
    
    //  操作C 需要手动释放  否则内存暴涨
//      CVPixelBufferRelease(_buffer)
    //    拿到pixelBuffer
//        ((RTCCVPixelBuffer*)frame.buffer).pixelBuffer
    
    if (!aFilterVideoFrame) {
        aFilterVideoFrame = frame;
    }
    
    [self.localVideoSource capturer:capturer didCaptureVideoFrame:frame];
}

之后调用SDWebRTCBufferFliter,实现滤镜效果。

实现将((RTCCVPixelBuffer *)frame.buffer).pixelBuffer进行渲染,这里用到了EAGLContext、CIContext

EAGLContext是OpenGL绘制句柄或者上下文,在绘制试图之前,需要指定使用创建的上下文绘制。

CIContext是用来渲染CIImage,将作用在CIImage上的滤镜链应用到原始的图片数据中。我这里需要将UIImage转换成CIImage。

具体代码实现如下

SDWebRTCBufferFliter.h

objectivec 复制代码
#import <Foundation/Foundation.h>
#import "WebRTCClient.h"

@interface SDWebRTCBufferFliter : NSObject

- (RTCVideoFrame *)webRTCClient:(WebRTCClient *)client didCaptureVideoFrame:(RTCVideoFrame *)frame;

@end

SDWebRTCBufferFliter.m

objectivec 复制代码
#import "SDWebRTCBufferFliter.h"
#import <VideoToolbox/VideoToolbox.h>
#import "SDApplyFilter.h"

@interface SDWebRTCBufferFliter ()
// 滤镜
@property (nonatomic, strong) EAGLContext *eaglContext;

@property (nonatomic, strong) CIContext *coreImageContext;

@property (nonatomic, strong) UIImage *lookUpImage;

@end

@implementation SDWebRTCBufferFliter

- (instancetype)init
{
    self = [super init];
    if (self) {
        self.eaglContext = [[EAGLContext alloc] initWithAPI:kEAGLRenderingAPIOpenGLES3];
        self.coreImageContext = [CIContext contextWithEAGLContext:self.eaglContext options:nil];
        self.lookUpImage = [UIImage imageNamed:@"lookup_jiari"];
    }
    return self;
}

- (RTCVideoFrame *)webRTCClient:(WebRTCClient *)client didCaptureVideoFrame:(RTCVideoFrame *)frame {
    
    CVPixelBufferRef pixelBufferRef = ((RTCCVPixelBuffer *)frame.buffer).pixelBuffer;

//    CFRetain(pixelBufferRef);
    if (pixelBufferRef) {
        CIImage *inputImage = [CIImage imageWithCVPixelBuffer:pixelBufferRef];

        CGImageRef imgRef = [_coreImageContext createCGImage:inputImage fromRect:[inputImage extent]];

        UIImage *fromImage = nil;
        if (!fromImage) {
            fromImage = [UIImage imageWithCGImage:imgRef];
        }

        UIImage *toImage;
        toImage = [SDApplyFilter applyMonochromeFilter:fromImage];
//
//        if (toImage == nil) {
//            toImage = [SDApplyFilter applyLookupFilter:fromImage lookUpImage:self.lookUpImage];
//        } else {
//            toImage = [SDApplyFilter applyLookupFilter:fromImage lookUpImage:self.lookUpImage];
//        }

        if (toImage == nil) {
            toImage = fromImage;
        }

        CGImageRef toImgRef = toImage.CGImage;
        CIImage *ciimage = [CIImage imageWithCGImage:toImgRef];
        [_coreImageContext render:ciimage toCVPixelBuffer:pixelBufferRef];

        CGImageRelease(imgRef);//必须释放
        fromImage = nil;
        toImage = nil;
        ciimage = nil;
        inputImage = nil;
    }

    RTCCVPixelBuffer *rtcPixelBuffer =
    [[RTCCVPixelBuffer alloc] initWithPixelBuffer:pixelBufferRef];
    RTCVideoFrame *filteredFrame =
    [[RTCVideoFrame alloc] initWithBuffer:rtcPixelBuffer
                                 rotation:frame.rotation
                              timeStampNs:frame.timeStampNs];
    
    return filteredFrame;
}

@end

至此可以看到在WebRTC音视频通话中GPUImage视频美颜滤镜的具体效果了。

三、小结

WebRTC音视频通话-实现GPUImage视频美颜滤镜效果。主要用到GPUImage处理视频画面CVPixelBufferRef,将处理后的CVPixelBufferRef生成RTCVideoFrame,通过调用localVideoSource中实现的didCaptureVideoFrame方法。内容较多,描述可能不准确,请见谅。

本文地址:https://blog.csdn.net/gloryFlow/article/details/132265842

学习记录,每天不停进步。

相关推荐
小曲曲1 小时前
接口上传视频和oss直传视频到阿里云组件
javascript·阿里云·音视频
安静读书3 小时前
Python解析视频FPS(帧率)、分辨率信息
python·opencv·音视频
佑华硬盘拷贝机3 小时前
音频档案批量拷贝:专业SD拷贝机解决方案
音视频
EasyNVR3 小时前
NVR管理平台EasyNVR多个NVR同时管理:全方位安防监控视频融合云平台方案
安全·音视频·监控·视频监控
比格丽巴格丽抱7 小时前
flutter项目苹果编译运行打包上线
flutter·ios
网络安全-老纪8 小时前
iOS应用网络安全之HTTPS
web安全·ios·https
xcLeigh11 小时前
HTML5超酷响应式视频背景动画特效(六种风格,附源码)
前端·音视频·html5
1024小神11 小时前
tauri2.0版本开发苹果ios和安卓android应用,环境搭建和最后编译为apk
android·ios·tauri
lzhdim12 小时前
iPhone 17 Air看点汇总:薄至6mm 刷新苹果轻薄纪录
ios·iphone
安和昂12 小时前
【iOS】知乎日报第四周总结
ios