鸿蒙5.0实战案例:基于OpenGL渲染视频画面帧

往期推文全新看点(文中附带全新鸿蒙5.0全栈学习笔录)

✏️ 鸿蒙(HarmonyOS)北向开发知识点记录~

✏️ 鸿蒙(OpenHarmony)南向开发保姆级知识点汇总~

✏️ 鸿蒙应用开发与鸿蒙系统开发哪个更有前景?

✏️ 嵌入式开发适不适合做鸿蒙南向开发?看完这篇你就了解了~

✏️ 对于大前端开发来说,转鸿蒙开发究竟是福还是祸?

✏️ 鸿蒙岗位需求突增!移动端、PC端、IoT到底该怎么选?

✏️ 记录一场鸿蒙开发岗位面试经历~

✏️ 持续更新中......


场景描述

在直播场景中,会有礼物、魔法等表情临时出现在画面,需要获取视频画面帧进行纹理更新后再渲染

通过OpenGL渲染视频画面帧。

⦁ 在ArkTS侧调用createAVPlayer()创建AVPlayer实例,初始化进入idle状态。设置业务需要的监听事件,设置资源:设置属性url,AVPlayer进入initialized状态。

⦁ 设置窗口:获取并设置属性SurfaceID,该surfaceId是native侧nativeImage的surfaceID,调用play方法开始播放。

⦁ 添加XComponent组件,在native侧通过OH_NativeXComponent_RegisterCallback注册XComponent回调。

⦁ 在napi 接口init的时候在Xcomponent的OnSurfaceCreatedCB回调中初始化渲染线程,通过OH_NativeImage_Create创建nativeImage。

⦁ 通过OH_NativeImage_GetSurfaceId获取surfaceID并传递到arkts侧。

⦁ 通过OH_NativeImage_SetOnFrameAvailableListener设置帧可用回调,通过NativeVsync接收系统信号,控制渲染。

⦁ 通过OH_NativeImage_UpdateSurfaceImage获取最新帧更新相关联的OpenGL ES纹理,通过eglSwapBuffers将纹理渲染上屏。

核心代码

创建nativeImage,获取nativeImage的id,设置帧可用回调。

复制代码
bool RenderThread::CreateNativeImage() 
{ 
  nativeImage_ = OH_NativeImage_Create(-1, GL_TEXTURE_EXTERNAL_OES); 
  if (nativeImage_ == nullptr) { 
    OH_LOG_Print(LOG_APP, LOG_ERROR, LOG_PRINT_DOMAIN, "RenderThread", "OH_NativeImage_Create failed."); 
    return false; 
  } 
  int ret = 0; 
  { 
    std::lock_guard<std::mutex> lock(nativeImageSurfaceIdMutex_); 
    ret = OH_NativeImage_GetSurfaceId(nativeImage_, &nativeImageSurfaceId_); 
  } 
  if (ret != 0) { 
    OH_LOG_Print(LOG_APP, LOG_ERROR, LOG_PRINT_DOMAIN, "RenderThread", 
      "OH_NativeImage_GetSurfaceId failed, ret is %{public}d.", ret); 
    return false; 
  } 
  nativeImageFrameAvailableListener_.context = this; 
  nativeImageFrameAvailableListener_.onFrameAvailable = &RenderThread::OnNativeImageFrameAvailable; 
  ret = OH_NativeImage_SetOnFrameAvailableListener(nativeImage_, nativeImageFrameAvailableListener_); 
  if (ret != 0) { 
    OH_LOG_Print(LOG_APP, LOG_ERROR, LOG_PRINT_DOMAIN, "RenderThread", 
      "OH_NativeImage_SetOnFrameAvailableListener failed, ret is %{public}d.", ret); 
    return false; 
  } 
  return true; 
} 
void RenderThread::OnNativeImageFrameAvailable(void *data) 
{ 
  OH_LOG_Print(LOG_APP, LOG_DEBUG, LOG_PRINT_DOMAIN, "RenderThread", "OnNativeImageFrameAvailable."); 
  auto renderThread = reinterpret_cast<RenderThread *>(data); 
  if (renderThread == nullptr) { 
    return; 
  } 
  renderThread->availableFrameCnt_++; 
  renderThread->wakeUpCond_.notify_one(); 
}
NativeVsync接收系统发送的Vsync信号,控制渲染节奏。
bool RenderThread::InitNativeVsync() 
{ 
  nativeVsync_ = OH_NativeVSync_Create(DEMO_NAME, strlen(DEMO_NAME)); 
  if (nativeVsync_ == nullptr) { 
    OH_LOG_Print(LOG_APP, LOG_ERROR, LOG_PRINT_DOMAIN, "RenderThread", "Create NativeVSync failed."); 
    return false; 
  } 
  (void)OH_NativeVSync_RequestFrame(nativeVsync_, &RenderThread::OnVsync, this); 
  return true; 
} 
void RenderThread::OnVsync(long long timestamp, void *data) 
{ 
  OH_LOG_Print(LOG_APP, LOG_DEBUG, LOG_PRINT_DOMAIN, "RenderThread", "OnVsync %{public}llu.", timestamp); 
  auto renderThread = reinterpret_cast<RenderThread *>(data); 
  if (renderThread == nullptr) { 
    return; 
  } 
  renderThread->vSyncCnt_++; 
  renderThread->wakeUpCond_.notify_one(); 
}

在渲染线程中更新画面帧数据到OpenGL纹理,并做对应的opengl处理后,通过SwapBuffers()方法上屏。

复制代码
void RenderThread::DrawImage() 
{ 
  if(nativeImageTexId_ == 9999) { 
    glGenTextures(1, &nativeImageTexId_); 
    glBindTexture(GL_TEXTURE_EXTERNAL_OES, nativeImageTexId_); 
    // set the texture wrapping parameters 
    glTexParameteri(GL_TEXTURE_EXTERNAL_OES, GL_TEXTURE_WRAP_S, GL_REPEAT); 
    glTexParameteri(GL_TEXTURE_EXTERNAL_OES, GL_TEXTURE_WRAP_T, GL_REPEAT); 
    // set texture filtering parameters 
    glTexParameteri(GL_TEXTURE_EXTERNAL_OES, GL_TEXTURE_MIN_FILTER, GL_LINEAR); 
    glTexParameteri(GL_TEXTURE_EXTERNAL_OES, GL_TEXTURE_MAG_FILTER, GL_LINEAR); 
  } 
  OH_LOG_Print(LOG_APP, LOG_DEBUG, LOG_PRINT_DOMAIN, "RenderThread", "DrawImage."); 
  if (eglSurface_ == EGL_NO_SURFACE) { 
    OH_LOG_Print(LOG_APP, LOG_WARN, LOG_PRINT_DOMAIN, "RenderThread", "eglSurface_ is EGL_NO_SURFACE"); 
    return; 
  } 
  OH_NativeImage_AttachContext(nativeImage_, nativeImageTexId_); 
  renderContext_->MakeCurrent(eglSurface_); 
  int32_t ret = OH_NativeImage_UpdateSurfaceImage(nativeImage_); 
  if (ret != 0) { 
    OH_LOG_Print(LOG_APP, LOG_ERROR, LOG_PRINT_DOMAIN, "RenderThread", 
      "OH_NativeImage_UpdateSurfaceImage failed, ret: %{public}d, texId: %{public}d", 
      ret, nativeImageTexId_); 
    return; 
  } 
  OH_LOG_Print(LOG_APP, LOG_DEBUG, LOG_PRINT_DOMAIN, "RenderThread", "OH_NativeImage_UpdateSurfaceImage succeed."); 
  float matrix[16]; 
  ret = OH_NativeImage_GetTransformMatrix(nativeImage_, matrix); 
  if (ret != 0) { 
    OH_LOG_Print(LOG_APP, LOG_ERROR, LOG_PRINT_DOMAIN, "RenderThread", 
      "OH_NativeImage_GetTransformMatrix failed, ret: %{public}d", ret); 
    return; 
  } 
  glClearColor(1.0f, 1.0f, 1.0f, 1.0f); 
  glClear(GL_COLOR_BUFFER_BIT); 
  shaderProgram_->Use(); 
  shaderProgram_->SetInt("texture", 0); 
  glActiveTexture(GL_TEXTURE0); 
  glBindTexture(GL_TEXTURE_EXTERNAL_OES, nativeImageTexId_); 
  shaderProgram_->SetMatrix4v("matTransform", matrix, 16, false); 
  glBindVertexArray(vertexArrayObject_); 
  glEnable(GL_DEPTH_TEST); 
  glDrawElements(GL_TRIANGLES, 6, GL_UNSIGNED_INT, Detail::indices); 
  renderContext_->SwapBuffers(eglSurface_); 
}
相关推荐
lqj_本人2 小时前
鸿蒙OS&UniApp 开发支持图片和视频的多媒体展示组件#三方框架 #Uniapp
uni-app·音视频·harmonyos
zhangfeng11332 小时前
绘制音频信号的各种频谱图,包括Mel频谱图、STFT频谱图等。它不仅能够绘制频谱图librosa.display.specshow
音视频
清月电子4 小时前
KT6368A通过蓝牙芯片获取手机时间详细说明,对应串口指令举例
单片机·嵌入式硬件·物联网·智能手机·音视频
lqj_本人6 小时前
鸿蒙OS&UniApp 实现的数据可视化图表组件#三方框架 #Uniapp
信息可视化·uni-app·harmonyos
zhangfeng113311 小时前
通过对音频信号提取梅尔频谱图并转换为对数梅尔频谱图得到的。它的形状主要由以下参数决定 转换成图片 64*64像素
算法·机器学习·音视频
lyitcc11 小时前
moviepy视频添加中文水印
音视频·moviepy·python添加视频水印
远瞻。12 小时前
【论文精读】2024 arXiv --VEnhancer现实世界视频超分辨率(RealWorld VSR)
论文阅读·stable diffusion·音视频·超分辨率重建
lqj_本人15 小时前
鸿蒙OS&UniApp 制作美观的文章列表展示组件#三方框架 #Uniapp
华为·uni-app·harmonyos
郑知鱼15 小时前
【拥抱鸿蒙】Flutter+Cursor轻松打造HarmonyOS应用(一)
flutter·华为·harmonyos·鸿蒙·cursor·移动端·鸿蒙next·ohos
bestadc16 小时前
鸿蒙 Form Kit(卡片开发服务)
harmonyos