WebRTC音视频通话-RTC直播本地视频及相册视频文件

WebRTC音视频通话-RTC直播本地视频文件效果图如下

WebRTC音视频通话-RTC直播本地视频文件时候，用到了AVPlayer、CADisplayLink。

一、通过AVPlayer播放本地视频

AVPlayer是什么？

AVPlayer是基于AVFoundation框架的一个类，很接近底层，灵活性强，可以自定义视频播放样式。

AVPlayerLayer是什么？

AVPlayerLayer是视频播放时候的画面展示层。

CADisplayLink是什么？

CADisplayLink和NSTimer一样，是一个定时器。但是CADisplayLink会和屏幕的刷新率始终保持一致(很多时候会使用CADisplayLink来检测屏幕的帧率)。

AVPlayerItemVideoOutput是什么？

AVPlayerItemVideoOutput是PlayerItem视频的输出，通过AVPlayerItemVideoOutput可以获取视频的CVPixelBufferRef

下面就实现本地视频播放过程中结合ossrs进行WebRTC直播。

AVPlayer设置播放本地视频

objectivec 复制代码

 AVPlayerItem *playerItem = [[AVPlayerItem alloc] initWithURL:[NSURL fileURLWithPath:videoPath]];

objectivec 复制代码

 (void)reloadPlayItem:(AVPlayerItem *)playerItem {
    self.playerItem = playerItem;
    [self initPlayerVideoOutput];
        
    self.player = [AVPlayer playerWithPlayerItem:self.playerItem];
}

开始配置播放展示AVPlayerLayer

objectivec 复制代码

- (void)startPlay {
    if (self.isPlaying) {
        return;
    }
    
    AVPlayerLayer *playerLayer = [AVPlayerLayer playerLayerWithPlayer:self.player];
    self.playerLayer = playerLayer;
    self.playerLayer.backgroundColor = [UIColor clearColor].CGColor;
    self.playerLayer.videoGravity = AVLayerVideoGravityResizeAspectFill;
    
    [self.superView.layer addSublayer:self.playerLayer];
    self.playerLayer.frame = self.superView.bounds;

    [self.player seekToTime:CMTimeMake(0, 1)];
    [self.player play];
    [self startAnimating];
}

通过KVO监控播放器状态

objectivec 复制代码

/**
 *  通过KVO监控播放器状态
 */
- (void)observeValueForKeyPath:(NSString *)keyPath {
    DebugLog(@"observeValueForKeyPath：%@", keyPath);
    
    AVPlayerItem *videoItem = self.playerItem;
    if ([keyPath isEqualToString:@"timeControlStatus"]) {
        
        /**
         typedef NS_ENUM(NSInteger, AVPlayerTimeControlStatus) {
             AVPlayerTimeControlStatusPaused = 0,
             AVPlayerTimeControlStatusWaitingToPlayAtSpecifiedRate = 1,
             AVPlayerTimeControlStatusPlaying = 2
         } API_AVAILABLE(macos(10.12), ios(10.0), tvos(10.0), watchos(3.0));
         */
        // 监听播放器timeControlStatus 指示当前是否正在播放，无限期暂停播放，或在等待适当的网络条件时暂停播放
        if (@available(iOS 10.0, *)) {
            switch (self.player.timeControlStatus) {
                case AVPlayerTimeControlStatusPaused: {
                    NSLog(@"AVPlayerTimeControlStatusPaused");
                    // 暂停
                    self.isPlaying = NO;
                }
                    break;
                case AVPlayerTimeControlStatusWaitingToPlayAtSpecifiedRate: {
                    NSLog(@"AVPlayerTimeControlStatusWaitingToPlayAtSpecifiedRate");
                    // 等待
                }
                    break;
                case AVPlayerTimeControlStatusPlaying: {
                    NSLog(@"AVPlayerTimeControlStatusPlaying");
                    // 播放
                    self.isPlaying = YES;
                }
                    break;
                default:
                    break;
            }
        } else {
            // Fallback on earlier versions
        }
    }
}

设置关键的AVPlayerItemVideoOutput

objectivec 复制代码

- (void)initPlayerVideoOutput {
    //输出yuv 420格式
    NSDictionary *pixBuffAttributes = @{(id)kCVPixelBufferPixelFormatTypeKey: @(kCVPixelFormatType_420YpCbCr8BiPlanarVideoRange)};
    AVPlayerItemVideoOutput *output = [[AVPlayerItemVideoOutput alloc] initWithPixelBufferAttributes:pixBuffAttributes];
    [self.playerItem addOutput:output];
    self.playerItemVideoOutput = output;
    
    [self.playerItemVideoOutput setDelegate:self queue:dispatch_get_main_queue()];
    
    // 如果将 AVPlayerItemVideoOutput 类的输出（对于 suppressesPlayerRendering 的值为 YES）添加到 AVPlayerItem，则该项目的视频媒体将不会由 AVPlayer 呈现，而音频媒体、字幕媒体和其他类型的媒体（如果存在） , 将被渲染。
    self.playerItemVideoOutput.suppressesPlayerRendering = NO;
}

之后通过displayLink开启实时调用AVPlayerItemVideoOutput得到视频画面CVPixelBufferRef

objectivec 复制代码

#pragma mark - DisplayLink
- (void)startDisplayLink {
    if (self.displayLink) {
        return;
    }
    self.displayLink = [CADisplayLink displayLinkWithTarget:[YYWeakProxy proxyWithTarget:self]
     selector:@selector(handleDisplayLink:)];
    [self.displayLink addToRunLoop:[NSRunLoop currentRunLoop] forMode:NSDefaultRunLoopMode];
    // self.displayLink.preferredFramesPerSecond = 2;
    self.displayLink.paused = NO;
}

- (void)handleDisplayLink:(CADisplayLink *)displayLink {
    //do something
    CMTime outputItemTime = kCMTimeInvalid;
    CFTimeInterval nextVSync = ([displayLink timestamp] + [displayLink duration]);
    outputItemTime = [[self playerItemVideoOutput] itemTimeForHostTime:nextVSync];
    if ([[self playerItemVideoOutput] hasNewPixelBufferForItemTime:outputItemTime]) {
        CVPixelBufferRef pixelBuffer = NULL;
        pixelBuffer = [[self playerItemVideoOutput] copyPixelBufferForItemTime:outputItemTime itemTimeForDisplay:NULL];
        // ..... do something with pixbuffer
        if (self.delegate && [self.delegate respondsToSelector:@selector(videoLivePlayerPixelBufferRef:)]) {
            [self.delegate videoLivePlayerPixelBufferRef:pixelBuffer];
        }
        
        if (pixelBuffer != NULL) {
            CFRelease(pixelBuffer);
        }
    }
}

- (void)stopDisplayLink {
    [self.displayLink invalidate];
    self.displayLink = nil;
}

这样在播放过程中，通过CADisplayLink获取到CVPixelBufferRef，之后使用WebRTC来进行直播。

完整播放视频过程中获得CVPixelBufferRef代码如下

SDVideoLivePlayer.h

objectivec 复制代码

#import <Foundation/Foundation.h>
#import <AudioToolbox/AudioToolbox.h>
#import <AVFoundation/AVFoundation.h>

@protocol SDVideoLivePlayerDelegate;
@interface SDVideoLivePlayer : NSObject

@property (nonatomic, strong) UIView *superView;

@property (nonatomic, weak) id<SDVideoLivePlayerDelegate> delegate;

- (instancetype)initWithSuperView:(UIView *)superView;

- (void)reloadPlayItem:(AVPlayerItem *)playerItem;

/// 开始播放
- (void)startPlay;

/// 结束播放
- (void)stopPlay;

@end

@protocol SDVideoLivePlayerDelegate <NSObject>

- (void)videoLivePlayerPixelBufferRef:(CVPixelBufferRef)pixelBufferRef;

@end

SDVideoLivePlayer.m

objectivec 复制代码

#import "SDVideoLivePlayer.h"

@interface SDVideoLivePlayer ()<AVPlayerItemOutputPullDelegate> {

}

@property (nonatomic, strong) AVPlayer *player;
@property (nonatomic, strong) AVPlayerLayer *playerLayer;
@property (nonatomic, strong) AVPlayerItem *playerItem;
@property (nonatomic, assign) BOOL isPlaying;
@property (nonatomic, strong) AVPlayerItemVideoOutput *playerItemVideoOutput;
@property (nonatomic, strong) CADisplayLink *displayLink;

@end

@implementation SDVideoLivePlayer

- (instancetype)initWithSuperView:(UIView *)superView
{
    self = [super init];
    if (self) {
        self.superView = superView;
        self.isPlaying = NO;
        [self addNotifications];
        [self initPlayerVideoOutput];
    }
    return self;
}

- (void)initPlayerVideoOutput {
    //输出yuv 420格式
    NSDictionary *pixBuffAttributes = @{(id)kCVPixelBufferPixelFormatTypeKey: @(kCVPixelFormatType_420YpCbCr8BiPlanarVideoRange)};
    AVPlayerItemVideoOutput *output = [[AVPlayerItemVideoOutput alloc] initWithPixelBufferAttributes:pixBuffAttributes];
    [self.playerItem addOutput:output];
    self.playerItemVideoOutput = output;
    
    [self.playerItemVideoOutput setDelegate:self queue:dispatch_get_main_queue()];
    
    // 如果将 AVPlayerItemVideoOutput 类的输出（对于 suppressesPlayerRendering 的值为 YES）添加到 AVPlayerItem，则该项目的视频媒体将不会由 AVPlayer 呈现，而音频媒体、字幕媒体和其他类型的媒体（如果存在） , 将被渲染。
    self.playerItemVideoOutput.suppressesPlayerRendering = NO;
}

- (void)reloadPlayItem:(AVPlayerItem *)playerItem {
    
    self.playerItem = playerItem;
    [self initPlayerVideoOutput];
        
    self.player = [AVPlayer playerWithPlayerItem:self.playerItem];
}

- (void)startPlay {
    if (self.isPlaying) {
        return;
    }
    
    AVPlayerLayer *playerLayer = [AVPlayerLayer playerLayerWithPlayer:self.player];
    self.playerLayer = playerLayer;
    self.playerLayer.backgroundColor = [UIColor clearColor].CGColor;
    self.playerLayer.videoGravity = AVLayerVideoGravityResizeAspectFill;
    
    [self.superView.layer addSublayer:self.playerLayer];
    self.playerLayer.frame = self.superView.bounds;

    [self.player seekToTime:CMTimeMake(0, 1)];
    [self.player play];
    [self startAnimating];
}

- (void)stopPlay {
    self.isPlaying = NO;
    [self.player pause];
    [self stopAnimating];
}

/**
 *  通过KVO监控播放器状态
 */
- (void)observeValueForKeyPath:(NSString *)keyPath {
    DebugLog(@"observeValueForKeyPath：%@", keyPath);
    
    AVPlayerItem *videoItem = self.playerItem;
    if ([keyPath isEqualToString:@"timeControlStatus"]) {
        
        /**
         typedef NS_ENUM(NSInteger, AVPlayerTimeControlStatus) {
             AVPlayerTimeControlStatusPaused = 0,
             AVPlayerTimeControlStatusWaitingToPlayAtSpecifiedRate = 1,
             AVPlayerTimeControlStatusPlaying = 2
         } API_AVAILABLE(macos(10.12), ios(10.0), tvos(10.0), watchos(3.0));
         */
        // 监听播放器timeControlStatus 指示当前是否正在播放，无限期暂停播放，或在等待适当的网络条件时暂停播放
        if (@available(iOS 10.0, *)) {
            switch (self.player.timeControlStatus) {
                case AVPlayerTimeControlStatusPaused: {
                    NSLog(@"AVPlayerTimeControlStatusPaused");
                    // 暂停
                    self.isPlaying = NO;
                }
                    break;
                case AVPlayerTimeControlStatusWaitingToPlayAtSpecifiedRate: {
                    NSLog(@"AVPlayerTimeControlStatusWaitingToPlayAtSpecifiedRate");
                    // 等待
                }
                    break;
                case AVPlayerTimeControlStatusPlaying: {
                    NSLog(@"AVPlayerTimeControlStatusPlaying");
                    // 播放
                    self.isPlaying = YES;
                }
                    break;
                default:
                    break;
            }
        } else {
            // Fallback on earlier versions
        }
    }
}

- (void)audioSessionInterrupted:(NSNotification *)notification{
    //通知类型
    NSDictionary * info = notification.userInfo;
    if ([[info objectForKey:AVAudioSessionInterruptionTypeKey] integerValue] == 1) {
        [self.player pause];
    } else {
        [self.player play];
    }
}

- (void)startAnimating {
    [self startDisplayLink];
    self.displayLink.paused = NO;
}

- (void)stopAnimating {
    self.displayLink.paused = YES;
    [self stopDisplayLink];
}

- (void)pauseAnimating {
    self.displayLink.paused = YES;
    [self stopDisplayLink];
}

- (void)resumeAnimating {
    if (!self.displayLink) {
        [self startDisplayLink];
    }
    self.displayLink.paused = NO;
}

#pragma mark - DisplayLink
- (void)startDisplayLink {
    if (self.displayLink) {
        return;
    }
    self.displayLink = [CADisplayLink displayLinkWithTarget:[YYWeakProxy proxyWithTarget:self]
     selector:@selector(handleDisplayLink:)];
    [self.displayLink addToRunLoop:[NSRunLoop currentRunLoop] forMode:NSDefaultRunLoopMode];
    // self.displayLink.preferredFramesPerSecond = 2;
    self.displayLink.paused = NO;
}

- (void)handleDisplayLink:(CADisplayLink *)displayLink {
    //do something
    CMTime outputItemTime = kCMTimeInvalid;
    CFTimeInterval nextVSync = ([displayLink timestamp] + [displayLink duration]);
    outputItemTime = [[self playerItemVideoOutput] itemTimeForHostTime:nextVSync];
    if ([[self playerItemVideoOutput] hasNewPixelBufferForItemTime:outputItemTime]) {
        CVPixelBufferRef pixelBuffer = NULL;
        pixelBuffer = [[self playerItemVideoOutput] copyPixelBufferForItemTime:outputItemTime itemTimeForDisplay:NULL];
        // ..... do something with pixbuffer
        if (self.delegate && [self.delegate respondsToSelector:@selector(videoLivePlayerPixelBufferRef:)]) {
            [self.delegate videoLivePlayerPixelBufferRef:pixelBuffer];
        }
        
        if (pixelBuffer != NULL) {
            CFRelease(pixelBuffer);
        }
    }
}

- (void)stopDisplayLink {
    [self.displayLink invalidate];
    self.displayLink = nil;
}

#pragma mark - AVPlayerItemOutputPullDelegate
- (void)outputMediaDataWillChange:(AVPlayerItemOutput *)sender {
    [self stopPlay];
}

#pragma mark - Observers
- (void)addNotifications {
    [[NSNotificationCenter defaultCenter] addObserver:self selector:@selector(replay:) name:AVPlayerItemDidPlayToEndTimeNotification object:nil];
    
    // 音频播放被中断
    [[NSNotificationCenter defaultCenter] addObserver:self selector:@selector(audioSessionInterrupted:) name:AVAudioSessionInterruptionNotification object:nil];
    
    __weak typeof(self) weakSelf = self;
    if (@available(iOS 10.0, *)) {
        [self.KVOController observe:self.player keyPath:@"timeControlStatus" options:NSKeyValueObservingOptionOld|NSKeyValueObservingOptionNew block:^(id  _Nullable observer, id  _Nonnull object, NSDictionary<NSString *,id> * _Nonnull change) {
            __strong typeof(weakSelf) strongSelf = weakSelf;
            [strongSelf observeValueForKeyPath:@"timeControlStatus"];
        }];
    }
}

- (void)removeNotifications {
    [[NSNotificationCenter defaultCenter] removeObserver:self];
    [self.KVOController unobserveAll];
}

- (void)replay:(NSNotification *)notification {
    if (notification.object == self.player.currentItem) {
        [self.player seekToTime:CMTimeMake(0, 1)];
        [self.player play];
    }
}

- (void)dealloc {
    [self removeNotifications];
}

@end

二、获取相册视频

获取相册视频代码如下，得到AVPlayerItem，调用- (void)reloadPlayItem:(AVPlayerItem *)playerItem;

objectivec 复制代码

- (void)startPlayAlbumVideo:(SDMediaModel *)mediaModel {
    if (!(mediaModel && mediaModel.phasset)) {
        return;
    }
    __weak typeof(self) weakSelf = self;
    [[PhotoKitManager shareInstance] requestPlayerItemForVideo:mediaModel.phasset completion:^(AVPlayerItem *playerItem) {
        __strong typeof(weakSelf) strongSelf = weakSelf;
        [strongSelf.videoLivePlayer reloadPlayItem:playerItem];
        [strongSelf.videoLivePlayer startPlay];
    } failure:^{
        __strong typeof(weakSelf) strongSelf = weakSelf;
    }];
}

三、WebRTC来进行RTC直播视频

CADisplayLink获取到CVPixelBufferRef，我们使用WebRTC来进行直播，相当于我们直播我们预先准备好的一个视频。

在之前实现了GPUImage视频通话视频美颜滤镜，这里用到了RTCVideoFrame，我们需要将得到的视频的CVPixelBufferRef，通过封装成RTCVideoFrame，通过调用RTCVideoSource的didCaptureVideoFrame方法实现最终的效果。

RTCVideoSource设置如下

objectivec 复制代码

- (RTCVideoTrack *)createVideoTrack {
    RTCVideoSource *videoSource = [self.factory videoSource];
    self.localVideoSource = videoSource;

    // 如果是模拟器
    if (TARGET_IPHONE_SIMULATOR) {
        if (@available(iOS 10, *)) {
            self.videoCapturer = [[RTCFileVideoCapturer alloc] initWithDelegate:self];
        } else {
            // Fallback on earlier versions
        }
    } else{
        self.videoCapturer = [[RTCCameraVideoCapturer alloc] initWithDelegate:self];
    }
    
    RTCVideoTrack *videoTrack = [self.factory videoTrackWithSource:videoSource trackId:@"video0"];
    
    return videoTrack;
}


- (void)createMediaSenders {
    if (!self.isPublish) {
        return;
    }
    
    NSString *streamId = @"stream";
    
    // Audio
    RTCAudioTrack *audioTrack = [self createAudioTrack];
    self.localAudioTrack = audioTrack;
    
    RTCRtpTransceiverInit *audioTrackTransceiver = [[RTCRtpTransceiverInit alloc] init];
    audioTrackTransceiver.direction = RTCRtpTransceiverDirectionSendOnly;
    audioTrackTransceiver.streamIds = @[streamId];
    
    [self.peerConnection addTransceiverWithTrack:audioTrack init:audioTrackTransceiver];
    
    // Video
    RTCVideoTrack *videoTrack = [self createVideoTrack];
    self.localVideoTrack = videoTrack;
    RTCRtpTransceiverInit *videoTrackTransceiver = [[RTCRtpTransceiverInit alloc] init];
    videoTrackTransceiver.direction = RTCRtpTransceiverDirectionSendOnly;
    videoTrackTransceiver.streamIds = @[streamId];
    [self.peerConnection addTransceiverWithTrack:videoTrack init:videoTrackTransceiver];
}

详细内容请看iOS端调用ossrs音视频通话部分。

将得到的视频的CVPixelBufferRef，通过封装成RTCVideoFrame，再通过调用RTCVideoSource的didCaptureVideoFrame方法。

objectivec 复制代码

- (RTCVideoFrame *)webRTCClient:(WebRTCClient *)client didCaptureVideoFrame:(RTCVideoFrame *)frame videoPixelBufferRef:(CVPixelBufferRef)videoPixelBufferRef {
    RTCCVPixelBuffer *rtcPixelBuffer =
    [[RTCCVPixelBuffer alloc] initWithPixelBuffer:videoPixelBufferRef];
    RTCVideoFrame *rtcVideoFrame =
    [[RTCVideoFrame alloc] initWithBuffer:rtcPixelBuffer
                                 rotation:frame.rotation
                              timeStampNs:frame.timeStampNs];
    return rtcVideoFrame;
}

使用RTCVideoSource调用didCaptureVideoFrame

[self.localVideoSource capturer:capturer didCaptureVideoFrame:frame];

其他

之前搭建ossrs服务，可以查看：https://blog.csdn.net/gloryFlow/article/details/132257196

之前实现iOS端调用ossrs音视频通话，可以查看：https://blog.csdn.net/gloryFlow/article/details/132262724

之前WebRTC音视频通话高分辨率不显示画面问题，可以查看：https://blog.csdn.net/gloryFlow/article/details/132262724

修改SDP中的码率Bitrate，可以查看：https://blog.csdn.net/gloryFlow/article/details/132263021

GPUImage视频通话视频美颜滤镜，可以查看：https://blog.csdn.net/gloryFlow/article/details/132265842

四、小结

WebRTC音视频通话-RTC直播本地视频文件。主要用到AVPlayer播放视频，AVPlayerItemVideoOutput得到CVPixelBufferRef，将处理后的CVPixelBufferRef生成RTCVideoFrame，通过调用WebRTC的localVideoSource中实现的didCaptureVideoFrame方法。内容较多，描述可能不准确，请见谅。

本文地址：https://blog.csdn.net/gloryFlow/article/details/132267068

学习记录，每天不停进步。