[rdk系列之情绪识别算法上板运行]

qq_457924292025-12-04 8:26

rdk系列之情绪识别算法上板运行

前言：
这一篇文章主要是记录一下当训练好一个pytorch模型之后如何经过量化然后在rdk x5上丝滑运行，包括具体的量化过程、ros包封装过程以及tros结合进行网页端的展示。
首发： RDK社区

一、前期准备

这里主要是给出需要的所有东西的相关链接，包括模型、数据集、docker镜像、地平线OE开发包。

1.1. 模型

情绪识别模型采用的是开源的一个github项目，链接在此

这个项目应该是基于yolo11n在fer2013数据集上进行训练的，详情可以查看仓库中的readme文档

1.2. 数据集

下载链接

该数据集中包含七类类别，分别为：愤怒，厌恶，恐惧，开心，悲伤，惊讶，中性

分辨率为48×48,均为灰度图

1.3. docker相关

相关下载链接
docker官网提供的离线包网址
docker官网安装链接

1.3.1. docker安装

  1. 离线下载：wget https://download.docker.com/linux/static/stable/x86_64/docker-19.03.8.tgz

  2. 解压：tar zxf docker-19.03.8.tgz

  3. 移动相关目录：sudo mv docker/* /usr/bin/

  4. 服务配置：sudo vi /etc/systemd/system/docker.service 在service文件中添加下列内容：

bash 复制代码 
[Unit]
Description=Docker Application Container Engine
Documentation=https://docs.docker.com
After=network-online.target firewalld.service
Wants=network-online.target
[Service]
Type=notify
# the default is not to use systemd for cgroups because the delegate issues still
# exists and systemd currently does not support the cgroup feature set required
# for containers run by docker
ExecStart=/usr/bin/dockerd
ExecReload=/bin/kill -s HUP $MAINPID
# Having non-zero Limit*s causes performance problems due to accounting overhead
# in the kernel. We recommend using cgroups to do container-local accounting.
LimitNOFILE=infinity
LimitNPROC=infinity
LimitCORE=infinity
# Uncomment TasksMax if your systemd version supports it.
# Only systemd 226 and above support this version.
#TasksMax=infinity
TimeoutStartSec=0
# set delegate yes so that systemd does not reset the cgroups of docker containers
Delegate=yes
# kill only the docker process, not all processes in the cgroup
KillMode=process
# restart the docker process if it exits prematurely
Restart=on-failure
StartLimitBurst=3
StartLimitInterval=60s
[Install]
WantedBy=multi-user.target
  1. 配置加载：systemctl daemon-reload
  2. 启动配置：systemctl start docker systemctl enable docker
  3. 添加至用户组：
bash 复制代码 
 sudo groupadd docker
 sudo gpasswd -a ${USER} docker
 sudo service docker restart

1.4. 工具链相关

注意这里都需要在主机中进行下载

  1. OE工具包下载：
    wget -c ftp://x5ftp@vrftp.horizon.ai/OpenExplorer/v1.2.8_release/horizon_x5_open_explorer_v1.2.8-py310_20240926.tar.gz --ftp-password=x5ftp@123$%
  2. docker包下载（CPU和GPU二选一即可）：
    CPU：
    wget -c ftp://x5ftp@vrftp.horizon.ai/OpenExplorer/v1.2.8_release/docker_openexplorer_ubuntu_20_x5_cpu_v1.2.8.tar.gz --ftp-password=x5ftp@123$%
    GPU:
    wget -c ftp://x5ftp@vrftp.horizon.ai/OpenExplorer/v1.2.8_release/docker_openexplorer_ubuntu_20_x5_gpu_v1.2.8.tar.gz --ftp-password=x5ftp@123$%
  3. 解压OE工具包：tar -xvf horizon_x5_open_explorer_v1.2.8-py310_20240926.tar.gz
  4. 由于量化工具接受的是onnx模型，因此需要提前把pt权重转换为onnx格式，yolo系列转换的指令为：model.export(format='onnx', opset=11)
    注意：这里的model就是加载的pt权重，此外由于工具链限制需要将opset设置为11

1.5 相关资料

地瓜官方的github仓库Model_zoo中提供了大多数主流算法的部署例程

若所采用的模型不在Model_Zoo中，所需配置文件的参考模板在此
官方用户手册

二、正式开始量化

在开始下列的步骤前，需要确保在主机环境中已经准备好了下列内容

  • 解压好的OE工具包
  • onnx格式的模型权重
  • 数据集
  • docker包

接下来就是正式量化的步骤

2.1. 进入docker环境

记得修改ai_toolchain_package_path和dataset_path的路径

bash 复制代码 
export version=v1.2.8-py310
export ai_toolchain_package_path=/home/xxx/horizon_x5_open_explorer_v1.2.8-py310_20240926#请自行修改路径
export dataset_path=/home/xxx/dataset   #请自行修改路径，没有dataset请自行创建

# load镜像
docker load < docker_openexplorer_ubuntu_20_x5_cpu_v1.2.8.tar.gz

# 进入镜像
#CPU版本
docker run -it --rm \
    -v "$ai_toolchain_package_path":/open_explorer \
    -v "$dataset_path":/data \
    openexplorer/ai_toolchain_ubuntu_20_x5_cpu:"${version}"
#GPU版本
docker run -it --rm \
    --gpus all \ 
    --shm-size=15g \ 
    -v "$ai_toolchain_package_path":/open_explorer \
    -v "$dataset_path":/data \
    openexplorer/ai_toolchain_ubuntu_20_x5_gpu:"${version}"

2.2. 把自己的模型放入环境中

可以直接把onnx权重文件复制到horizon_x5_open_explorer_v1.2.8-py310_20240926文件夹下，我在该文件夹下新建了一个model文件用于放置模型权重

接下来进行模型验证； hb_mapper checker --model-type onnx --march bayes-e --model /path/to/model

会得到下列输出：

若模型检测没有问题那么就可以进行量化的步骤

2.3 准备校准数据

校准数据的来源为训练集或验证集，从中抽取100张进行后续的操作，有两种不同的预处理格式，在这里一一列出

方式一：设置process_on参数为True

若设置该参数为True，则不需要对图片进行额外的处理，转换工具在读取这些图片后，会自动进行处理，将其缩放到模型输入节点要求的尺寸大小，但可能对量化后的精度有一点影响。

对应的yaml文件如下：
记得修改模型路径、输出路径、输出模型文件名称、校准数据集路径

python 复制代码 
model_parameters:
  onnx_model: 'model/best.onnx' #模型路径
  march: "bayes-e"
  layer_out_dump: False
  working_dir: 'model-output' #输出路径
  output_model_file_prefix: 'yolo11-emo' #保存的名称
input_parameters:
  input_name: ""
  input_type_rt: 'nv12'
  input_type_train: 'rgb'
  input_layout_train: 'NCHW'
  norm_type: 'data_scale'
  scale_value: 0.003921568627451
calibration_parameters:
  cal_data_dir: 'dataset' #这里是校准数据的路径
  cal_data_type: 'float32'
  preprocess_on： True
  calibration_type: 'default'
  optimization: set_Softmax_input_int8,set_Softmax_output_int8
compiler_parameters:
  jobs: 2
  compile_mode: 'latency'
  debug: true
  optimize_level: 'O3'

方式二：设置process_on参数为False（即保持默认）

若将该参数设置为False，那么就需要对校准数据进行额外的处理，处理的代码如下：

python 复制代码 
img_names = os.listdir('data')#原始数据
cnt_total = len(img_names)
for cnt, img_name in enumerate(img_names, 1):
    img_path = os.path.join('data', img_name)
    img = cv2.imread(img_path)
    input_tensor = cv2.cvtColor(img, cv2.COLOR_BGR2RGB)     # BGR2RGB
    input_tensor = cv2.resize(img, (640, 640)) # resize
    input_tensor = np.transpose(input_tensor, (2, 0, 1))    # HWC2CHW
    input_tensor = np.expand_dims(input_tensor, axis=0).astype(np.float32)  # CHW -> NCHW
    dst_path = os.path.join('result1', img_name[:-4] + '.rgb') # tofile
    input_tensor.tofile(dst_path)

注意：
1. 此处的处理和训练时的前处理保持一致
2. 如果yaml中有配置mean和scale, 则此处无须计算mean和scale.

随后修改yaml文件中相关的参数，修改后的yaml文件如下：

python 复制代码 
model_parameters:
  onnx_model: 'model/best.onnx'
  march: "bayes-e"
  layer_out_dump: False
  working_dir: 'model-output-off'
  output_model_file_prefix: 'yolo11-emo-off'
input_parameters:
  input_name: ""
  input_type_rt: 'nv12'
  input_type_train: 'rgb'
  input_layout_train: 'NCHW'
  norm_type: 'data_scale'
  scale_value: 0.003921568627451
calibration_parameters:
  cal_data_dir: 'dataset-off'
  cal_data_type: 'float32'
  preprocess_on: False
  calibration_type: 'default'
  optimization: set_Softmax_input_int8,set_Softmax_output_int8
compiler_parameters:
  jobs: 2
  compile_mode: 'latency'
  debug: true
  optimize_level: 'O3'

2.4 量化

量化指令如下(记得修改config的路径为自己的yaml路径)：
hb_mapper makertbin --model-type onnx --config yolo11.yaml

量化完成后会有如下的输出，相关log文件保存在/open_explorer/hb_mapper_makertbin.log：

通过hb_perf model-output/yolo11-emo.bin可视化量化之后的模型：

量化结束之后，会产生多个中间文件，

每个转换文件的具体解读可参考：模型转换物解读

2.4 bin模型上板运行

图片推理

在x5板上运行下列代码，采用hobot_dnn来加载bin模型并进行推理

python 复制代码 
from hobot_dnn import pyeasy_dnn as dnn
import cv2
import numpy as np

def bgr2nv12(image):
    """
    输入: image (H, W, 3) BGR uint8
    输出: nv12 (H*3/2, W) uint8
    """
    H, W, _ = image.shape
    # BGR -> YUV I420
    yuv_i420 = cv2.cvtColor(image, cv2.COLOR_BGR2YUV_I420)  # shape=(H*3/2, W)
    
    Y = yuv_i420[0:H, :]
    U = yuv_i420[H:H + H//4, :].reshape(H//2, W//2)
    V = yuv_i420[H + H//4:H + H//2, :].reshape(H//2, W//2)

    UV = np.empty((H//2, W), dtype=np.uint8)
    UV[:, 0::2] = U
    UV[:, 1::2] = V

    nv12 = np.vstack((Y, UV))
    return nv12

models = dnn.load('best.bin')

model = models[0]
image_path = '10.png'
image = cv2.imread(image_path)  # BGR
image = cv2.resize(image, (640, 640), interpolation=cv2.INTER_LINEAR)
nv12_image = bgr2nv12(image)
nv12_image_batch = np.expand_dims(nv12_image, axis=0)  # 添加 batch 维度
nv12_image_batch = nv12_image_batch.astype(np.uint8)
emotion_results = model.forward(nv12_image_batch)
result = emotion_results[0].buffer
output = np.squeeze(result).transpose(1, 0)
cls_scores = output[:, 5:]
obj_conf = output[:,4]
scores = obj_conf[:, None] * cls_scores  # shape -> (8400, 7)
flat_idx = np.argmax(scores)
top_idx, emotion_class_id = divmod(flat_idx, cls_scores.shape[1])
emotion_names = [
    "Anger",
    "Disgust",
    "Fear",
    "Happy",
    "Neutral",
    "Sad",
    "Surprise",
  ]
emotion_label = emotion_names[emotion_class_id]
print("Predicted emotion:", emotion_label)

视频流推理

下列代码是采用视频流做实时的检测

python 复制代码 
from hobot_dnn import pyeasy_dnn as dnn
import numpy as np
import cv2
from PIL import Image 
import time
models = dnn.load('best.bin')
def bgr2nv12(image):
    H, W, _ = image.shape
    # BGR -> YUV I420
    yuv_i420 = cv2.cvtColor(image, cv2.COLOR_BGR2YUV_I420)  # shape=(H*3/2, W)
    
    Y = yuv_i420[0:H, :]
    U = yuv_i420[H:H + H//4, :].reshape(H//2, W//2)
    V = yuv_i420[H + H//4:H + H//2, :].reshape(H//2, W//2)

    UV = np.empty((H//2, W), dtype=np.uint8)
    UV[:, 0::2] = U
    UV[:, 1::2] = V

    nv12 = np.vstack((Y, UV))
    return nv12

model = models[0]
cap = cv2.VideoCapture(0)
while True:
    _ ,frame = cap.read()
    image = cv2.resize(frame, (640, 640), interpolation=cv2.INTER_LINEAR)
    nv12_image = bgr2nv12(image)
    nv12_image_batch = np.expand_dims(nv12_image, axis=0)  # 添加 batch 维度
    nv12_image_batch = nv12_image_batch.astype(np.uint8)
    t0 = time.time()
    emotion_results = model.forward(nv12_image_batch)
    t1 = time.time()
    result = emotion_results[0].buffer
    output = np.squeeze(result).transpose(1, 0)
    cls_scores = output[:, 5:]
    obj_conf = output[:,4]
    scores = obj_conf[:, None] * cls_scores  # shape -> (8400, 7)
    flat_idx = np.argmax(scores)
    top_idx, emotion_class_id = divmod(flat_idx, cls_scores.shape[1])
    emotion_names = [
        "Anger",
        "Disgust",
        "Fear",
        "Happy",
        "Neutral",
        "Sad",
        "Surprise",
    ]
    emotion_label = emotion_names[emotion_class_id]
    print("Predicted emotion:", emotion_label)
    print("FPS:",1.0/(t1-t0))

实时检测结果及FPS指标

三、与TROS结合实现网页端显示

3.1 封装为ROS2节点

  1. 工作空间创建
    若对ros2存在疑问，可查询胡老师ROS2 21讲
bash 复制代码 
mkdir -p ~/emo_ws/src
cd ~/emo_ws

#创建功能包
ros2 pkg create emo --build-type ament_python  --dependencies rclpy std_msgs
  1. 节点文件编写
python 复制代码 
#!/usr/bin/env python3
import rclpy
from rclpy.node import Node
from sensor_msgs.msg import CompressedImage
from std_msgs.msg import String   
from cv_bridge import CvBridge
import cv2
import numpy as np
from hobot_dnn import pyeasy_dnn as dnn
import time

class FaceExtractor(Node):
    def __init__(self):
        super().__init__('face_extractor')
        self.bridge = CvBridge()
        self.last_image = None

        self.emotion_model = dnn.load('best.bin')
        self.emotion_model = self.emotion_model[0]

        self.image_sub = self.create_subscription(
            CompressedImage,
            '/image',
            self.image_callback,
            10
        )

        self.emotion_pub = self.create_publisher(
            String,
            '/emotion_result',
            10
        )

        self.get_logger().info("✅ FaceExtractor 已启动，等待图像...")

    def image_callback(self, msg):
        np_arr = np.frombuffer(msg.data, np.uint8)
        image = cv2.imdecode(np_arr, cv2.IMREAD_COLOR)
        if image is None:
            self.get_logger().warn("Decoded image is None.")
            return
        emotion = self.run_emotion_model(image)
        msg_pub = String()
        msg_pub.data = emotion
        self.emotion_pub.publish(msg_pub)
        self.get_logger().info(f"已发布情绪结果: {emotion}")
   
    def bgr2nv12(self, image):
        H, W, _ = image.shape
        yuv_i420 = cv2.cvtColor(image, cv2.COLOR_BGR2YUV_I420)
        Y = yuv_i420[0:H, :]
        U = yuv_i420[H:H + H//4, :].reshape(H//2, W//2)
        V = yuv_i420[H + H//4:H + H//2, :].reshape(H//2, W//2)
        UV = np.empty((H//2, W), dtype=np.uint8)
        UV[:, 0::2] = U
        UV[:, 1::2] = V
        nv12 = np.vstack((Y, UV))
        return nv12

    def run_emotion_model(self, face_img):
        image = cv2.resize(face_img, (640, 640), interpolation=cv2.INTER_LINEAR)
        nv12_image = self.bgr2nv12(image)
        nv12_image_batch = np.expand_dims(nv12_image, axis=0).astype(np.uint8)

        start_time = time.time()
        emotion_results = self.emotion_model.forward(nv12_image_batch)
        end_time = time.time()
        inference_time = end_time - start_time
        fps = 1.0 / inference_time if inference_time > 0 else 0.0
        self.get_logger().debug(f"fps {fps:.2f}")

        result = emotion_results[0].buffer
        output = np.squeeze(result).transpose(1, 0)
        cls_scores = output[:, 5:]
        obj_conf = output[:, 4]
        scores = obj_conf[:, None] * cls_scores  # shape -> (8400, 7)
        flat_idx = np.argmax(scores)
        box_idx, emotion_class_id = divmod(flat_idx, scores.shape[1])
        emotion_names = [
            "Anger",
            "Disgust",
            "Fear",
            "Happy",
            "Neutral",
            "Sad",
            "Surprise",
        ]
        return emotion_names[int(emotion_class_id)]

def main(args=None):
    rclpy.init(args=args)
    node = FaceExtractor()
    try:
        rclpy.spin(node)
    except KeyboardInterrupt:
        pass
    finally:
        node.destroy_node()
        rclpy.shutdown()
  1. launch文件编写
python 复制代码 
from launch import LaunchDescription
from launch_ros.actions import Node
from launch.actions import IncludeLaunchDescription, DeclareLaunchArgument
from launch.launch_description_sources import PythonLaunchDescriptionSource
from launch.substitutions import LaunchConfiguration
from ament_index_python import get_package_share_directory
import os
from launch.substitutions import TextSubstitution

def generate_launch_description():
    camera_node = IncludeLaunchDescription(
        PythonLaunchDescriptionSource(
            os.path.join(get_package_share_directory('hobot_usb_cam'),
                            'launch/hobot_usb_cam.launch.py')
        ),
        launch_arguments={
            'usb_image_width': '640',
            'usb_image_height': '480'
        }.items()
    )
    emo_node = Node(
        package='emo',
        executable='emo_ros',
        name='face_extractor',
        output='screen',
        arguments=['--ros-args', '--log-level', 'warn']
    )
    return LaunchDescription([
            camera_node,
            emo_node,
        ])
  1. setup.py文件
python 复制代码 
from setuptools import find_packages, setup

package_name = 'emo'

setup(
    name=package_name,
    version='0.0.0',
    packages=find_packages(exclude=['test']),
    data_files=[
        ('share/ament_index/resource_index/packages',
            ['resource/' + package_name]),
        ('share/' + package_name, ['package.xml']),
        ('share/' + package_name + '/launch', ['launch/emo_launch.py']),
    ],
    install_requires=['setuptools'],
    zip_safe=True,
    maintainer='sunrise',
    maintainer_email='sunrise@todo.todo',
    description='TODO: Package description',
    license='TODO: License declaration',
    tests_require=['pytest'],
    entry_points={
        'console_scripts': [
            'emo_ros = emo.emo_ros:main',
        ],
    },
)
  1. 编译
bash 复制代码 
cd ~/emo_ws
colcon build --symlink-install
source instsll/setup.bash
  1. 执行ros2 launch emo emo_launch.py,随后查看节点可看到emotion_result

3.2 与TROS的webstock结合可视化

注意：
1.发布情绪识别结果的话题类型要为PerceptionTargets
2.为了顺利可视化，最好添加header时间戳（如果不添加的话可能会出现网页端无输出）
相较于上一小节中代码，改动较多的为节点代码和launch文件，在这里全部给出

若相对webstock进行进一步的了解，可查阅rdk 用户手册介绍

  1. emo_ros.py
python 复制代码 
#!/usr/bin/env python3
import rclpy
from rclpy.node import Node
from sensor_msgs.msg import CompressedImage
from ai_msgs.msg import PerceptionTargets, Target, Roi, Attribute

from cv_bridge import CvBridge
import cv2
import numpy as np
from hobot_dnn import pyeasy_dnn as dnn
import time
from builtin_interfaces.msg import Time

class FaceExtractor(Node):
    def __init__(self):
        super().__init__('face_extractor')
        self.bridge = CvBridge()
        self.last_image = None

        self.emotion_model = dnn.load('/home/sunrise/yolo-emo/test/yolo11-emo.bin')
        self.emotion_model = self.emotion_model[0]

        self.image_sub = self.create_subscription(
            CompressedImage,
            '/image',
            self.image_callback,
            10
        )

        self.perception_pub = self.create_publisher(
            PerceptionTargets,
            '/emotion_result',
            10
        )

        self.get_logger().info("✅ FaceExtractor 已启动，等待图像...")
    def image_callback(self, msg):
        try:
            np_arr = np.frombuffer(msg.data, np.uint8)
            image = cv2.imdecode(np_arr, cv2.IMREAD_COLOR)
            if image is None:
                self.get_logger().warn("Decoded image is None.")
                return

            try:
                emotion_label, emotion_class_id, box = self.run_emotion_model(image)
            except Exception as e:
                self.get_logger().error(f"Emotion model error: {e}")
                emotion_label = "unknown"
                emotion_class_id = -1
                box = (0,0,0,0)

            x_c, y_c, box_w, box_h = box

            msg_out = PerceptionTargets()
            msg_out.header = msg.header

            target = Target()
            roi = Roi()
            roi.type = "face"

            H, W = image.shape[:2]

            if box_w == 0 or box_h == 0:
                roi_w, roi_h = int(W * 0.4), int(H * 0.4)
                x1 = (W - roi_w) // 2
                y1 = (H - roi_h) // 2
            else:
                x1 = max(0, int(x_c - box_w // 2))
                y1 = max(0, int(y_c - box_h // 2))
                roi_w = int(box_w)
                roi_h = int(box_h)

            roi.rect.x_offset = x1
            roi.rect.y_offset = y1
            roi.rect.width = roi_w
            roi.rect.height = roi_h

            target.rois.append(roi)

            attr = Attribute()
            attr.type = emotion_label
            attr.value = float(emotion_class_id)
            target.attributes.append(attr)

            msg_out.targets.append(target)

            self.perception_pub.publish(msg_out)

        except Exception as e:
            self.get_logger().error(f"Unexpected error in callback: {e}")


    def bgr2nv12(self, image):
        H, W, _ = image.shape
        yuv_i420 = cv2.cvtColor(image, cv2.COLOR_BGR2YUV_I420)
        Y = yuv_i420[0:H, :]
        U = yuv_i420[H:H + H//4, :].reshape(H//2, W//2)
        V = yuv_i420[H + H//4:H + H//2, :].reshape(H//2, W//2)
        UV = np.empty((H//2, W), dtype=np.uint8)
        UV[:, 0::2] = U
        UV[:, 1::2] = V
        nv12 = np.vstack((Y, UV))
        return nv12

    def run_emotion_model(self, face_img):
        image = cv2.resize(face_img, (640, 640), interpolation=cv2.INTER_LINEAR)
        nv12_image = self.bgr2nv12(image)
        nv12_image_batch = np.expand_dims(nv12_image, axis=0).astype(np.uint8)

        start_time = time.time()
        emotion_results = self.emotion_model.forward(nv12_image_batch)
        end_time = time.time()
        inference_time = end_time - start_time
        fps = 1.0 / inference_time if inference_time > 0 else 0.0
        self.get_logger().debug(f"fps {fps:.2f}")

        result = emotion_results[0].buffer
        output = np.squeeze(result).transpose(1, 0)
        cls_scores = output[:, 5:]
        obj_conf = output[:, 4]
        scores = obj_conf[:, None] * cls_scores  # shape -> (8400, 7)
        flat_idx = np.argmax(scores)
        box_idx, emotion_class_id = divmod(flat_idx, scores.shape[1])
        emotion_names = [
            "Anger",
            "Disgust",
            "Fear",
            "Happy",
            "Neutral",
            "Sad",
            "Surprise",
        ]
        box = output[box_idx, 0:4]  # x_center, y_center, w, h (相对 0-1 或像素，看模型)
        return emotion_names[int(emotion_class_id)],emotion_class_id,box

def main(args=None):
    rclpy.init(args=args)
    node = FaceExtractor()
    try:
        rclpy.spin(node)
    except KeyboardInterrupt:
        pass
    finally:
        node.destroy_node()
        rclpy.shutdown()
  1. launch文件
    注意：这里默认相机为usb相机，如果需要采用mipi相机可修改camera_type
python 复制代码 
from launch import LaunchDescription
from launch_ros.actions import Node
from launch.actions import IncludeLaunchDescription, DeclareLaunchArgument
from launch.launch_description_sources import PythonLaunchDescriptionSource
from launch.substitutions import LaunchConfiguration
from ament_index_python import get_package_share_directory
import os
from launch.substitutions import TextSubstitution

def generate_launch_description():
    # ------------------- Camera 类型 -------------------
    camera_type = "usb"  # 可选 'usb', 'fb', 'mipi'

    camera_node = None
    camera_device_arg = None

    if camera_type == "fb":
        # 本地 image publisher
        camera_node = IncludeLaunchDescription(
            PythonLaunchDescriptionSource(
                os.path.join(get_package_share_directory('hobot_image_publisher'),
                             'launch/hobot_image_publisher.launch.py')
            ),
            launch_arguments={
                'publish_message_topic_name': '/image',
                'publish_is_shared_mem': 'False',
                'publish_is_compressed_img_pub': 'True'
            }.items()
        )
    elif camera_type == "usb":
        camera_node = IncludeLaunchDescription(
            PythonLaunchDescriptionSource(
                os.path.join(get_package_share_directory('hobot_usb_cam'),
                             'launch/hobot_usb_cam.launch.py')
            ),
            launch_arguments={
                'usb_image_width': '640',
                'usb_image_height': '480'
            }.items()
        )
    else:  # mipi
        camera_device_arg = DeclareLaunchArgument(
            'device', default_value='F37', description='mipi camera device'
        )
        camera_node = IncludeLaunchDescription(
            PythonLaunchDescriptionSource(
                os.path.join(get_package_share_directory('mipi_cam'),
                             'launch/mipi_cam.launch.py')
            ),
            launch_arguments={
                'mipi_image_width': '960',
                'mipi_image_height': '544',
                'mipi_io_method': 'shared_mem',
                'mipi_frame_ts_type': 'realtime',
                'mipi_video_device': LaunchConfiguration('device')
            }.items()
        )

    # ------------------- Shared Memory -------------------
    shared_mem_node = IncludeLaunchDescription(
        PythonLaunchDescriptionSource(
            os.path.join(get_package_share_directory('hobot_shm'),
                         'launch/hobot_shm.launch.py')
        )
    )

    # ------------------- Codec 节点 -------------------
    # nv12 -> jpeg (发布 /image)
    nv12_codec_node = IncludeLaunchDescription(
        PythonLaunchDescriptionSource(
            os.path.join(
                get_package_share_directory('hobot_codec'),
                'launch/hobot_codec_decode.launch.py')),
        launch_arguments={
            'codec_in_mode': 'ros',
            'codec_out_mode': 'shared_mem',
            'codec_sub_topic': '/image',
            'codec_pub_topic': '/hbmem_img'
        }.items()
    )

    # ------------------- Emotion 节点 -------------------
    emo_node = Node(
        package='emotion_pkg',
        executable='emo_ros',
        name='face_extractor',
        output='screen',
        arguments=['--ros-args', '--log-level', 'warn']
    )

    # ------------------- Websocket 节点 -------------------
    web_smart_topic_arg = DeclareLaunchArgument(
        'smart_topic',
        default_value='/emotion_result',
        description='websocket smart topic')
    web_node = IncludeLaunchDescription(
        PythonLaunchDescriptionSource(
            os.path.join(
                get_package_share_directory('websocket'),
                'launch/websocket.launch.py')),
        launch_arguments={
            'websocket_image_topic': '/image',
            'websocket_smart_topic': LaunchConfiguration('smart_topic'),
            'websocket_only_show_image':'False' #如果是图像的话，必须要开这个
        }.items()
    )

    # ------------------- LaunchDescription -------------------
    return LaunchDescription([
            # 启动零拷贝环境配置node
            shared_mem_node,
            # image publish
            camera_node,
            # image codec
            nv12_codec_node,
            # body detection
            emo_node,
            # web display
            web_smart_topic_arg,
            web_node
        ])
  1. 编译
bash 复制代码 
cd ~/emo_ws
colcon build --symlink-install
source instsll/setup.bash
  1. 运行ros2 launch emo emo_launch.py后在网页端输入http:192.168.127.10:8000即可实时展示
    其中192.168.127.10需要更换为自己板卡的IP，x5默认的ip是这个
    网页端中会显示识别的区域、情绪识别模型识别的结果以及该结果对应的label组的id号，识别的区域名词和结果的id号这两个可在代码中修改

四、优化

如果想要进一步优化精度和性能的话，可以参考社区中新鲜出炉的文章

五、参考文献

参考一
参考二：开源代码链接
参考三：开源代码链接
参考四：rdk_model_zoo链接
参考五：社区之前的量化文章

六、中间文件

相关中间文件都在压缩包中

相关推荐
(●—●)橘子……29 分钟前
力扣344.反转字符串 练习理解
python·学习·算法·leetcode·职场和发展
Bdygsl33 分钟前
数字图像处理总结 Day 3 —— 图像增强与运算
图像处理·算法
田里的水稻34 分钟前
spline_curve
算法·几何学
X***C86242 分钟前
SpringMVC 请求参数接收
前端·javascript·算法
Bear on Toilet1 小时前
12 . 二叉树的直径
数据结构·算法·二叉树
惜.己1 小时前
数据结构与算法-数组异或操作
数据结构·算法
2301_807997381 小时前
代码随想录-day55
数据结构·c++·算法
程序员东岸11 小时前
《数据结构——排序（中）》选择与交换的艺术：从直接选择到堆排序的性能跃迁
数据结构·笔记·算法·leetcode·排序算法
程序员-King.11 小时前
day104—对向双指针—接雨水（LeetCode-42）
算法·贪心算法
热门推荐
01GitHub 镜像站点02UV安装并设置国内源03BongoCat - 跨平台键盘猫动画工具04【超详细教程】手把手教你从微软官网免费下载Windows 10官方原版ISO镜像（2025最新版）05安娜的档案(Anna’s Archive) 镜像网站/国内最新可访问入口（持续更新）06本地部署阿里最新开源的Z-Image07Linux下V2Ray安装配置指南08【保姆级教程】免费使用Gemini3的5种方法！免翻墙/国内直连09Labelme从安装到标注：零基础完整指南10Meta第三代“分割一切”模型——SAM 3本地部署教程：首支持文本提示分割，400万概念、30毫秒响应，检测分割追踪一网打尽