WebRTC音视频通话-实现GPUImage视频美颜滤镜效果iOS

WebRTC音视频通话-实现GPUImage视频美颜滤镜效果

在WebRTC音视频通话的GPUImage美颜效果图如下

可以看下

之前搭建ossrs服务,可以查看:https://blog.csdn.net/gloryFlow/article/details/132257196

之前实现iOS端调用ossrs音视频通话,可以查看:https://blog.csdn.net/gloryFlow/article/details/132262724

之前WebRTC音视频通话高分辨率不显示画面问题,可以查看:https://blog.csdn.net/gloryFlow/article/details/132262724

修改SDP中的码率Bitrate,可以查看:https://blog.csdn.net/gloryFlow/article/details/132263021

一、GPUImage是什么?

GPUImage是iOS上一个基于OpenGL进行图像处理的开源框架,内置大量滤镜,架构灵活,可以在其基础上很轻松地实现各种图像处理功能。

GPUImage中包含各种滤镜,这里我不会使用那么多,使用的是GPUImageLookupFilter及GPUImagePicture

GPUImage中有一个专门针对lookup table进行处理的滤镜函数GPUImageLookupFilter,使用这个函数就可以直接对图片进行滤镜添加操作了。代码如下

objectivec 复制代码
/**
 GPUImage中有一个专门针对lookup table进行处理的滤镜函数GPUImageLookupFilter,使用这个函数就可以直接对图片进行滤镜添加操作了。
 originalImg是你希望添加滤镜的原始图片
 
 @param image 原图
 @return 处理后的图片
 */
+ (UIImage *)applyLookupFilter:(UIImage *)image lookUpImage:(UIImage *)lookUpImage {
    
    if (lookUpImage == nil) {
        return image;
    }
    
    UIImage *inputImage = image;
    
    UIImage *outputImage = nil;
    
    GPUImagePicture *stillImageSource = [[GPUImagePicture alloc] initWithImage:inputImage];
    //添加滤镜
    GPUImageLookupFilter *lookUpFilter = [[GPUImageLookupFilter alloc] init];
    
    //导入之前保存的NewLookupTable.png文件
    GPUImagePicture *lookupImg = [[GPUImagePicture alloc] initWithImage:lookUpImage];
    
    [lookupImg addTarget:lookUpFilter atTextureLocation:1];
    
    [stillImageSource addTarget:lookUpFilter atTextureLocation:0];
    [lookUpFilter useNextFrameForImageCapture];
    
    if([lookupImg processImageWithCompletionHandler:nil] && [stillImageSource processImageWithCompletionHandler:nil]) {
        outputImage= [lookUpFilter imageFromCurrentFramebuffer];
        
    }
    
    return outputImage;
}

这个需要lookUpImage,图列表如下

由于暂时没有整理demo的git

这里在使用applyLomofiFilter再试下效果

SDApplyFilter.m中的几个方法

objectivec 复制代码
+ (UIImage *)applyBeautyFilter:(UIImage *)image {
    GPUImageBeautifyFilter *filter = [[GPUImageBeautifyFilter alloc] init];
    [filter forceProcessingAtSize:image.size];
    GPUImagePicture *pic = [[GPUImagePicture alloc] initWithImage:image];
    [pic addTarget:filter];
    [pic processImage];
    [filter useNextFrameForImageCapture];
    
    return [filter imageFromCurrentFramebuffer];
}


/**
 Amatorka滤镜 Rise滤镜,可以使人像皮肤得到很好的调整

 @param image image
 @return 处理后的图片
 */
+ (UIImage *)applyAmatorkaFilter:(UIImage *)image
{
    GPUImageAmatorkaFilter *filter = [[GPUImageAmatorkaFilter alloc] init];
    [filter forceProcessingAtSize:image.size];
    GPUImagePicture *pic = [[GPUImagePicture alloc] initWithImage:image];
    [pic addTarget:filter];
    
    [pic processImage];
    [filter useNextFrameForImageCapture];
    return [filter imageFromCurrentFramebuffer];
}

/**
 复古型滤镜,感觉像旧上海滩

 @param image image
 @return 处理后的图片
 */
+ (UIImage *)applySoftEleganceFilter:(UIImage *)image
{
    GPUImageSoftEleganceFilter *filter = [[GPUImageSoftEleganceFilter alloc] init];
    [filter forceProcessingAtSize:image.size];
    GPUImagePicture *pic = [[GPUImagePicture alloc] initWithImage:image];
    [pic addTarget:filter];
    
    [pic processImage];
    [filter useNextFrameForImageCapture];
    return [filter imageFromCurrentFramebuffer];
}


/**
 图像黑白化,并有大量噪点

 @param image 原图
 @return 处理后的图片
 */
+ (UIImage *)applyLocalBinaryPatternFilter:(UIImage *)image
{
    GPUImageLocalBinaryPatternFilter *filter = [[GPUImageLocalBinaryPatternFilter alloc] init];
    
    [filter forceProcessingAtSize:image.size];
    GPUImagePicture *pic = [[GPUImagePicture alloc] initWithImage:image];
    [pic addTarget:filter];
    [pic processImage];
    [filter useNextFrameForImageCapture];
    
    return [filter imageFromCurrentFramebuffer];
}

/**
 单色滤镜
 
 @param image 原图
 @return 处理后的图片
 */
+ (UIImage *)applyMonochromeFilter:(UIImage *)image
{
    GPUImageMonochromeFilter *filter = [[GPUImageMonochromeFilter alloc] init];
    
    [filter forceProcessingAtSize:image.size];
    GPUImagePicture *pic = [[GPUImagePicture alloc] initWithImage:image];
    [pic addTarget:filter];
    [pic processImage];
    [filter useNextFrameForImageCapture];
    
    return [filter imageFromCurrentFramebuffer];
}

使用GPUImageSoftEleganceFilter复古型滤镜,感觉像旧上海滩效果图如下

使用GPUImageLocalBinaryPatternFilter图像黑白化效果图如下

使用GPUImageMonochromeFilter 效果图如下

二、WebRTC实现音视频通话中视频滤镜处理

之前实现iOS端调用ossrs音视频通话,可以查看:https://blog.csdn.net/gloryFlow/article/details/132262724

这个已经有完整的代码了,这里暂时做一下调整。

为RTCCameraVideoCapturer的delegate指向代理

objectivec 复制代码
- (RTCVideoTrack *)createVideoTrack {
    RTCVideoSource *videoSource = [self.factory videoSource];
    self.localVideoSource = videoSource;

    // 如果是模拟器
    if (TARGET_IPHONE_SIMULATOR) {
        if (@available(iOS 10, *)) {
            self.videoCapturer = [[RTCFileVideoCapturer alloc] initWithDelegate:self];
        } else {
            // Fallback on earlier versions
        }
    } else{
        self.videoCapturer = [[RTCCameraVideoCapturer alloc] initWithDelegate:self];
    }
    
    RTCVideoTrack *videoTrack = [self.factory videoTrackWithSource:videoSource trackId:@"video0"];
    
    return videoTrack;
}

实现RTCVideoCapturerDelegate的方法didCaptureVideoFrame

objectivec 复制代码
#pragma mark - RTCVideoCapturerDelegate处理代理
- (void)capturer:(RTCVideoCapturer *)capturer didCaptureVideoFrame:(RTCVideoFrame *)frame {
//    DebugLog(@"capturer:%@ didCaptureVideoFrame:%@", capturer, frame);

// 调用SDWebRTCBufferFliter的滤镜处理
    RTCVideoFrame *aFilterVideoFrame;
    if (self.delegate && [self.delegate respondsToSelector:@selector(webRTCClient:didCaptureVideoFrame:)]) {
        aFilterVideoFrame = [self.delegate webRTCClient:self didCaptureVideoFrame:frame];
    }
    
    //  操作C 需要手动释放  否则内存暴涨
//      CVPixelBufferRelease(_buffer)
    //    拿到pixelBuffer
//        ((RTCCVPixelBuffer*)frame.buffer).pixelBuffer
    
    if (!aFilterVideoFrame) {
        aFilterVideoFrame = frame;
    }
    
    [self.localVideoSource capturer:capturer didCaptureVideoFrame:frame];
}

之后调用SDWebRTCBufferFliter,实现滤镜效果。

实现将((RTCCVPixelBuffer *)frame.buffer).pixelBuffer进行渲染,这里用到了EAGLContext、CIContext

EAGLContext是OpenGL绘制句柄或者上下文,在绘制试图之前,需要指定使用创建的上下文绘制。

CIContext是用来渲染CIImage,将作用在CIImage上的滤镜链应用到原始的图片数据中。我这里需要将UIImage转换成CIImage。

具体代码实现如下

SDWebRTCBufferFliter.h

objectivec 复制代码
#import <Foundation/Foundation.h>
#import "WebRTCClient.h"

@interface SDWebRTCBufferFliter : NSObject

- (RTCVideoFrame *)webRTCClient:(WebRTCClient *)client didCaptureVideoFrame:(RTCVideoFrame *)frame;

@end

SDWebRTCBufferFliter.m

objectivec 复制代码
#import "SDWebRTCBufferFliter.h"
#import <VideoToolbox/VideoToolbox.h>
#import "SDApplyFilter.h"

@interface SDWebRTCBufferFliter ()
// 滤镜
@property (nonatomic, strong) EAGLContext *eaglContext;

@property (nonatomic, strong) CIContext *coreImageContext;

@property (nonatomic, strong) UIImage *lookUpImage;

@end

@implementation SDWebRTCBufferFliter

- (instancetype)init
{
    self = [super init];
    if (self) {
        self.eaglContext = [[EAGLContext alloc] initWithAPI:kEAGLRenderingAPIOpenGLES3];
        self.coreImageContext = [CIContext contextWithEAGLContext:self.eaglContext options:nil];
        self.lookUpImage = [UIImage imageNamed:@"lookup_jiari"];
    }
    return self;
}

- (RTCVideoFrame *)webRTCClient:(WebRTCClient *)client didCaptureVideoFrame:(RTCVideoFrame *)frame {
    
    CVPixelBufferRef pixelBufferRef = ((RTCCVPixelBuffer *)frame.buffer).pixelBuffer;

//    CFRetain(pixelBufferRef);
    if (pixelBufferRef) {
        CIImage *inputImage = [CIImage imageWithCVPixelBuffer:pixelBufferRef];

        CGImageRef imgRef = [_coreImageContext createCGImage:inputImage fromRect:[inputImage extent]];

        UIImage *fromImage = nil;
        if (!fromImage) {
            fromImage = [UIImage imageWithCGImage:imgRef];
        }

        UIImage *toImage;
        toImage = [SDApplyFilter applyMonochromeFilter:fromImage];
//
//        if (toImage == nil) {
//            toImage = [SDApplyFilter applyLookupFilter:fromImage lookUpImage:self.lookUpImage];
//        } else {
//            toImage = [SDApplyFilter applyLookupFilter:fromImage lookUpImage:self.lookUpImage];
//        }

        if (toImage == nil) {
            toImage = fromImage;
        }

        CGImageRef toImgRef = toImage.CGImage;
        CIImage *ciimage = [CIImage imageWithCGImage:toImgRef];
        [_coreImageContext render:ciimage toCVPixelBuffer:pixelBufferRef];

        CGImageRelease(imgRef);//必须释放
        fromImage = nil;
        toImage = nil;
        ciimage = nil;
        inputImage = nil;
    }

    RTCCVPixelBuffer *rtcPixelBuffer =
    [[RTCCVPixelBuffer alloc] initWithPixelBuffer:pixelBufferRef];
    RTCVideoFrame *filteredFrame =
    [[RTCVideoFrame alloc] initWithBuffer:rtcPixelBuffer
                                 rotation:frame.rotation
                              timeStampNs:frame.timeStampNs];
    
    return filteredFrame;
}

@end

至此可以看到在WebRTC音视频通话中GPUImage视频美颜滤镜的具体效果了。

三、小结

WebRTC音视频通话-实现GPUImage视频美颜滤镜效果。主要用到GPUImage处理视频画面CVPixelBufferRef,将处理后的CVPixelBufferRef生成RTCVideoFrame,通过调用localVideoSource中实现的didCaptureVideoFrame方法。内容较多,描述可能不准确,请见谅。

本文地址:https://blog.csdn.net/gloryFlow/article/details/132265842

学习记录,每天不停进步。

相关推荐
HarderCoder8 小时前
iOS 知识积累第一弹:从 struct 到 APP 生命周期的全景复盘
ios
MagicSakuraD18 小时前
LiveKit 的核心概念
webrtc
叽哥19 小时前
Flutter Riverpod上手指南
android·flutter·ios
用户092 天前
SwiftUI Charts 函数绘图完全指南
ios·swiftui·swift
YungFan2 天前
iOS26适配指南之UIColor
ios·swift
权咚2 天前
阿权的开发经验小集
git·ios·xcode
用户092 天前
TipKit与CloudKit同步完全指南
ios·swift
法的空间3 天前
Flutter JsonToDart 支持 JsonSchema
android·flutter·ios
kaixin_啊啊3 天前
突破限制:Melody远程音频管理新体验
音视频
2501_915918413 天前
iOS 上架全流程指南 iOS 应用发布步骤、App Store 上架流程、uni-app 打包上传 ipa 与审核实战经验分享
android·ios·小程序·uni-app·cocoa·iphone·webview