Android-RTC系列软重启，改变以往细读源代码的方式 改为 带上实际问题分析代码。增加实用性，方便形成肌肉记忆。同时不分种类、不分难易程度，在线征集问题切入点。
问题：webrtc是如何创建视频编码器选用软编码or硬编码的？如何扩展支持OpenH264，OpenH265？
其实这个问题，是上篇文章的一个补充。在上篇文章【WebRTC-13】我们确认创建编码器是如何创建的，找到了编码器的实体类，并大致的勾画出其调用链路，如下所示：
cpp
SdpOfferAnswerHandler:: ApplyLocalDescription / ApplyRemoteDescription（sdp信息）
SdpOfferAnswerHandler::UpdateTransceiversAndDataChannels -> UpdateTransceiverChannel（创建RtpTransceiver->Video/VoiceChannel）
SdpOfferAnswerHandler::UpdateSessionState
SdpOfferAnswerHandler::PushdownMediaDescription
BaseChannel::SetLocalContent(const MediaContentDescription* content, ..)
VoiceChannel/VideoChannel::SetLocalContent_w
BaseChannel::UpdateLocalStreams_w(const std::vector<StreamParams>& streams, ..)
WebRtcVideoSendChannel::AddSendStream
WebRtcVideoSendChannel::WebRtcVideoSendStream::WebRtcVideoSendStream（Constructor）
WebRtcVideoSendChannel::WebRtcVideoSendStream::SetCodec|::RecreateWebRtcStream|::SetSenderParameters|::ReconfigureEncoder
Call::CreateVideoSendStream
VideoSendStreamImpl() -> VideoStreamEncoder(Interface)我们可以在VideoStreamEncoder实体类中快速定位成员变量std::unique_ptr<VideoEncoder> encoder_ 代码创建的地方。
cpp
VideoStreamEncoderSettings settings_;
VideoEncoderConfig encoder_config_;
void VideoStreamEncoder::ReconfigureEncoder() {
std::unique_ptr<VideoEncoder> encoder_;
encoder_ = MaybeCreateFrameDumpingEncoderWrapper(
settings_.encoder_factory->CreateVideoEncoder(encoder_config_.video_format)，
field_trials_);
}随着调用链路的回溯，我们定位到WebRtcVideoSendChannel::WebRtcVideoSendStream的构造函数初始化了VideoSendStreamParameters，VideoStreamEncoder需要使用以下的三个输入参数，都是WebRtcVideoSendStream构造函数前在上层传入并创建好了。
webrtc::VideoSendStream::Config parameters_.config
webrtc::VideoEncoderConfig parameters_.encoder_config
VideoCodecSettings parameters_.codec_settings;
也就是在WebRtcVideoSendStream创建的地方，WebRtcVideoSendChannel::AddSendStream。
cpp
// cpp\media\engine\webrtc_video_engine.cc
bool WebRtcVideoSendChannel::AddSendStream(const StreamParams& sp) {
webrtc::VideoSendStream::Config config(transport());
config.encoder_settings.encoder_factory = encoder_factory_;
config.encoder_settings.bitrate_allocator_factory = bitrate_allocator_factory_;
...
WebRtcVideoSendStream* stream = new WebRtcVideoSendStream(
call_, sp, std::move(config), default_send_options_,
video_config_.enable_cpu_adaptation, bitrate_config_.max_bitrate_bps,
send_codec(), send_rtp_extensions_, send_params_);
}【记住上面出现的transport() 以后会重点讲】显然encoder_factory也是从WebRtcVideoSendChannel外部持有者，webrtc_media_engine传进来的。经过了前面几篇文章的分析，这里直接跳过无聊的跟踪代码，直接给出encoder_factory的根持有者调用链------由PeerConnectionFactory持有的，实际对象位于Java层的DefaultVideoEncoderFactory。
cpp
|CreatePeerConnectionFactoryForJava
|---CreateModularPeerConnectionFactory( PeerConnectionFactoryDependencies )
|------ConnectionContext( dep )::EnableMeida = dep->media_factory->CreateMediaEngine
| auto audio_engine = std::make_unique<WebRtcVoiceEngine>(
| &env.task_queue_factory(), deps.adm.get(), env.field_trials(),
| std::move(deps.audio_encoder_factory),
| std::move(deps.audio_decoder_factory), std::move(deps.audio_mixer),
| std::move(deps.audio_processing), std::move(deps.audio_frame_processor) );
| auto video_engine = std::make_unique<WebRtcVideoEngine>(
| std::move(deps.video_encoder_factory),
| std::move(deps.video_decoder_factory), env.field_trials());
|---------CreateMediaEngine( deps ) with audio_engine / video_engine现在我们找到问题答案的入口了，由于C++和Java之间存在一层JNI接口，所以我们先顺着调用接口，看看JNI层的接口对象类。
cpp
//文件位置 \sdk\android\src\jni\pc\peer_connection_factory.cc
ScopedJavaLocalRef<jobject> CreatePeerConnectionFactoryForJava(...) {
PeerConnectionFactoryDependencies dependencies;
...
dependencies.adm = std::move(audio_device_module);
dependencies.audio_encoder_factory = std::move(audio_encoder_factory);
dependencies.audio_decoder_factory = std::move(audio_decoder_factory);
dependencies.audio_processing = std::move(audio_processor);
dependencies.video_encoder_factory =
absl::WrapUnique(CreateVideoEncoderFactory(jni, jencoder_factory));
dependencies.video_decoder_factory =
absl::WrapUnique(CreateVideoDecoderFactory(jni, jdecoder_factory));
EnableMedia(dependencies);
rtc::scoped_refptr<PeerConnectionFactoryInterface> factory =
CreateModularPeerConnectionFactory(std::move(dependencies));
}
//文件位置 \sdk\android\src\jni\pc\video.cc
VideoEncoderFactory* CreateVideoEncoderFactory(
JNIEnv* jni,
const JavaRef<jobject>& j_encoder_factory) {
return IsNull(jni, j_encoder_factory)
? nullptr
: new VideoEncoderFactoryWrapper(jni, j_encoder_factory);
}明显这个VideoEncoderFactoryWrapper就是Jni层的接口类，并且看到了它是继承了C++的VideoEncoderFactory 纯虚类。
cpp
// 文件位置 sdk\android\src\jni\video_encoder_factory_wrapper.h .cc
// Wrapper for Java VideoEncoderFactory class. Delegates method calls through
// JNI and wraps the encoder inside VideoEncoderWrapper.
class VideoEncoderFactoryWrapper : public VideoEncoderFactory {
public:
VideoEncoderFactoryWrapper(JNIEnv* jni,
const JavaRef<jobject>& encoder_factory);
~VideoEncoderFactoryWrapper() override;
std::unique_ptr<VideoEncoder> CreateVideoEncoder(
const SdpVideoFormat& format) override;
// Returns a list of supported codecs in order of preference.
std::vector<SdpVideoFormat> GetSupportedFormats() const override;
std::vector<SdpVideoFormat> GetImplementations() const override;
std::unique_ptr<EncoderSelectorInterface> GetEncoderSelector() const override;
private:
const ScopedJavaGlobalRef<jobject> encoder_factory_;
std::vector<SdpVideoFormat> supported_formats_;
std::vector<SdpVideoFormat> implementations_;
};
std::unique_ptr<VideoEncoder> VideoEncoderFactoryWrapper::CreateVideoEncoder(
const SdpVideoFormat& format) {
JNIEnv* jni = AttachCurrentThreadIfNeeded();
ScopedJavaLocalRef<jobject> j_codec_info =
SdpVideoFormatToVideoCodecInfo(jni, format);
ScopedJavaLocalRef<jobject> encoder = Java_VideoEncoderFactory_createEncoder(
jni, encoder_factory_, j_codec_info);
if (!encoder.obj())
return nullptr;
return JavaToNativeVideoEncoder(jni, encoder);
}我们也看到了VideoEncoderFactoryWrapper::CreateVideoEncoder的实现，就是调用Java层的DefaultVideoEncoderFactory.createEncoder。现在到Java层看看实现。
java
/** Helper class that combines HW and SW encoders. */
public class DefaultVideoEncoderFactory implements VideoEncoderFactory {
private final VideoEncoderFactory hardwareVideoEncoderFactory;
private final VideoEncoderFactory softwareVideoEncoderFactory = new SoftwareVideoEncoderFactory();
/** Create encoder factory using default hardware encoder factory. */
public DefaultVideoEncoderFactory(
EglBase.Context eglContext, boolean enableIntelVp8Encoder, boolean enableH264HighProfile) {
this.hardwareVideoEncoderFactory =
new HardwareVideoEncoderFactory(eglContext, enableIntelVp8Encoder, enableH264HighProfile);
}
@Nullable
@Override
public VideoEncoder createEncoder(VideoCodecInfo info) {
final VideoEncoder softwareEncoder = softwareVideoEncoderFactory.createEncoder(info);
final VideoEncoder hardwareEncoder = hardwareVideoEncoderFactory.createEncoder(info);
if (hardwareEncoder != null && softwareEncoder != null) {
// Both hardware and software supported, wrap it in a software fallback
return new VideoEncoderFallback(
/* fallback= */ softwareEncoder, /* primary= */ hardwareEncoder);
}
return hardwareEncoder != null ? hardwareEncoder : softwareEncoder;
}
}
// A combined video encoder that falls back on a secondary encoder if the primary encoder fails.
public class VideoEncoderFallback extends WrappedNativeVideoEncoder {
private final VideoEncoder fallback;
private final VideoEncoder primary;
public VideoEncoderFallback(VideoEncoder fallback, VideoEncoder primary) {
this.fallback = fallback;
this.primary = primary;
}
@Override
public long createNativeVideoEncoder() {
return nativeCreateEncoder(fallback, primary);
}
@Override
public boolean isHardwareEncoder() {
return primary.isHardwareEncoder();
}
}Java层的DefaultVideoEncoderFactory是同时创建硬编码器和软编码器，并以Fallback备份的形式，打包成一个WrapperNativeVideoEncoder。上述流程各种类关系如下：
搞清楚各种对象类的上下文之后，我们就得继续深入探究软编码是怎么产生，要如何扩展。也就是上图加粗的SoftwareVideoEncoderFactory -> WrappedNativeVideoEncoder
节省篇幅，以下是 Java.SoftwareVideoEncoderFactory到Cxx.internal_encoder_factory具体实现的依赖链路。
cpp
|-org/webrtc/SoftwareVideoEncoderFactory.java
|--sdk/android/src/jni/software_video_encoder_factory.cc
|---api/video_codecs/builtin_video_encoder_factory.cc
|----media/engine/internal_encoder_factory.cc接下来就是本次问题的答案了，在Cxx.internal_encoder_factory我们可以看到一个动态模板的Factory。这里的VideoEncoderFactoryTemplate是VideoEncoderFactory多模板实现类，
cpp
// media/engine/internal_encoder_factory.cc
namespace {
using Factory =
VideoEncoderFactoryTemplate<webrtc::LibvpxVp8EncoderTemplateAdapter,
#if defined(WEBRTC_USE_H264)
webrtc::OpenH264EncoderTemplateAdapter,
#endif
#if defined(RTC_USE_LIBAOM_AV1_ENCODER)
webrtc::LibaomAv1EncoderTemplateAdapter,
#endif
webrtc::LibvpxVp9EncoderTemplateAdapter>;
}
std::unique_ptr<VideoEncoder> InternalEncoderFactory::CreateVideoEncoder(
const SdpVideoFormat& format) {
auto original_format =
FuzzyMatchSdpVideoFormat(Factory().GetSupportedFormats(), format);
return original_format ? Factory().CreateVideoEncoder(*original_format)
: nullptr;
}
// api/video_codecs/video_encoder_factory_template.h
// api/video_codecs/video_encoder_factory_template_open_h264_adapter.h
// api/video_codecs/video_encoder_factory_template_libvpx_vp8_adapter.h
// api/video_codecs/video_encoder_factory_template_libaom_av1_adapter.h
template <typename V, typename... Vs>
std::unique_ptr<VideoEncoder> CreateVideoEncoderInternal(
const SdpVideoFormat& format) {
if (IsFormatInList(format, V::SupportedFormats())) {
return V::CreateEncoder(format);
}
if constexpr (sizeof...(Vs) > 0) {
return CreateVideoEncoderInternal<Vs...>(format);
}
return nullptr;
}仔细看创建编码器对象的方法实现，逻辑上很好理解，其实就是递归检查模板列表的编码器工厂类(适配器)，命中格式就直接调用模板类的方法实现；不命中则继续往后递归，直至最后一个候选。
比起用STL容器去实现适配器列表，用元模板的实现方式能从编译器层面上去优化实现的代码，提高性能。
而且到这里我们也找到了如何扩展软编码器的入口：仿照OpenH264EncoderTemplateAdapter实现模板静态结构体（VideoEncoderFactory）接口方法；仿照H264Encoder继承VideoEncoder实现功能方法。但其实这里只是找到了 扩展软编解码器的入口，要真正实现扩展软编解码，还需要做大量的调查&分析工作，了解webrtc内部大量的调度算法才能得以实现。