Ubuntu OpenCV C++ 获取MYNT EYE S1030-IR摄像头图像

0. 环境

ubuntu16台式机
MYNT EYE S1030-IR，固件版本号v2.0
opencv 3.4.1
MYNT EYE SDK v2.0

关于MYNT EYE摄像头的版本和SDK的版本号，根据《固件与 SDK 适配性》这个表格

https://mynt-eye-s-sdk.readthedocs.io/zh-cn/latest/src/firmware/applicable.html

对应的固件选择相同版本的SDK

1. opencv 3.4.1

1.1 准备源码

浏览器打开https://opencv.org/releases/，链接到github

下载了

opencv-3.4.1.tar.gz

放在

~/workspace/s1030ir_out_of_box/ext

1.2 安装依赖

复制代码

# compiler ✓
sudo apt-get install -y build-essential
# required ✓
sudo apt-get install -y cmake git libgtk2.0-dev pkg-config libavcodec-dev libavformat-dev libswscale-dev

1.3 自定义临时环境变量

复制代码

cd ~/workspace/s1030ir_out_of_box
export WORK_DIR=$PWD
export ARCH="x64"

export C_COMPILER="gcc"
export CXX_COMPILER="g++"
export AR="ar"

echo $WORK_DIR
echo $ARCH
echo $C_COMPILER
echo $CXX_COMPILER
echo $AR

1.4 命令行编译安装

复制粘贴到终端中，会自动执行。

复制代码

cd $WORK_DIR
cd ext/
mkdir -p install && mkdir -p install/$ARCH && mkdir -p install/$ARCH/lib && mkdir -p install/$ARCH/include
rm -rf opencv-3.4.1
tar -zvxf opencv-3.4.1.tar.gz && cd opencv-3.4.1
rm -rf build
mkdir build && cd build
cmake  -DCMAKE_C_COMPILER=$C_COMPILER -DCMAKE_CXX_COMPILER=$CXX_COMPILER \
-DCMAKE_BUILD_TYPE=RELEASE \
-DCMAKE_INSTALL_PREFIX=$WORK_DIR/ext/install/$ARCH \
-DBUILD_DOCS=OFF \
-DBUILD_EXAMPLES=OFF \
-DBUILD_TESTS=OFF \
-DBUILD_PERF_TESTS=OFF \
-DSOFTFP=OFF \
-DBUILD_SHARED_LIBS=OFF -DCMAKE_C_FLAGS="-L$WORK_DIR/ext/install/$ARCH/lib" ..
make -j8
make install

2. MYNT-EYE-S-SDK-Docs

Docs也可以得到本地版本的。曾经mynt eye的官网、docs、天猫店等等的都下线了。现在好像docs还在线，天猫也还有货源。

2.0 环境

ubuntu18（ubuntu16的python3.5编译失败）

2.1 源码

复制代码

git clone https://github.com/slightech/MYNT-EYE-S-SDK-Docs-zh_CN

切换目录

cd MYNT-EYE-S-SDK-Docs-zh_CN

cd docs

2.2 python 虚拟环境

安装

复制代码

sudo apt install python3-venv

建立环境

复制代码

python3 -m venv .venv

激活

复制代码

source .venv/bin/activate

列出包

复制代码

pip list

2.3 安装依赖

用默认pip源

复制代码

pip install -r requirements.txt

或者指定用aliyun的源

复制代码

pip install -r requirements.txt -i http://mirrors.aliyun.com/pypi/simple/  --trusted-host mirrors.aliyun.com

编译出html

复制代码

make html

退出虚拟环境

复制代码

deactivate

2.4 运行html测试

3. MYNT EYE SDK

sdk的版本号和摄像头的固件版本号对应，api的适配会较好。可以通过sample内的get_device_info获取设备固件号。

3.1 获取源码

https://github.com/slightech/MYNT-EYE-S-SDK/releases

下载得到

MYNT-EYE-S-SDK-2.0.1-rc2.tar.gz

解压到

cd ~/workspace/s1030ir_out_of_box/ext/MYNT-EYE-S-SDK-2.0.1-rc2

3.2 临时变量

复制代码

cd ~/workspace/s1030ir_out_of_box
export WORK_DIR=$PWD
export ARCH="x64"

export C_COMPILER="gcc"
export CXX_COMPILER="g++"
export AR="ar"

echo $WORK_DIR
echo $ARCH
echo $C_COMPILER
echo $CXX_COMPILER
echo $AR

3.3 编译sdk

复制代码

cd $WORK_DIR
cd ext/MYNT-EYE-S-SDK-2.0.1-rc2
	rm -rf build
	mkdir build && cd build 
	cmake  -DCMAKE_C_COMPILER=$C_COMPILER -DCMAKE_CXX_COMPILER=$CXX_COMPILER \
	-DCMAKE_INSTALL_PREFIX=$WORK_DIR/ext/install/$ARCH \
	-DOpenCV_DIR=$WORK_DIR/ext/install/$ARCH/share/OpenCV \
	-DBUILD_SHARED_LIBS=OFF -DCMAKE_C_FLAGS="-L$WORK_DIR/ext/install/$ARCH/lib" ..
make -j8
make install

3.4 编译samples

复制代码

cd $WORK_DIR
cd ext/MYNT-EYE-S-SDK-2.0.1-rc2
cd samples
rm -rf build
mkdir build && cd build 
cmake  -DCMAKE_C_COMPILER=$C_COMPILER -DCMAKE_CXX_COMPILER=$CXX_COMPILER \
	-DCMAKE_INSTALL_PREFIX=$WORK_DIR/ext/install/$ARCH \
	-Dmynteye_DIR=$WORK_DIR/ext/install/$ARCH/lib/cmake/mynteye \
	-DBUILD_SHARED_LIBS=OFF \
	-DCMAKE_C_FLAGS="-L$WORK_DIR/ext/install/$ARCH/lib" ..
make -j8
cp ../_output/bin/* ../../../install/$ARCH/bin/ -r

3.5 运行测试

3.5.1 获取设备信息

复制代码

cd ~/workspace/s1030ir_out_of_box
export WORK_DIR=$PWD
export ARCH="x64"

export C_COMPILER="gcc"
export CXX_COMPILER="g++"
export AR="ar"
cd $WORK_DIR/ext/install/$ARCH/bin/tutorials
sudo ./get_device_info

3.5.2 双目摄像头

复制代码

sudo ./get_stereo

3.5.3 获取深度信息

复制代码

sudo ./get_depth

3.5.4 获取IMU数据

复制代码

sudo ./get_imu

4. get_points

sdk中还有一个demo是get_points，这个是要用到PCL的。

VTK 7.1.1
metslib 0.5.3
pcl 1.8.1

4.0 环境变量

复制代码

cd ~/workspace/s1030ir_out_of_box
export WORK_DIR=$PWD
export ARCH="x64"

export C_COMPILER="gcc"
export CXX_COMPILER="g++"
export AR="ar"

echo $WORK_DIR
echo $ARCH
echo $C_COMPILER
echo $CXX_COMPILER
echo $AR

4.1 安装依赖

复制代码

sudo apt-get update
sudo apt-get install -y git build-essential linux-libc-dev
sudo apt-get install -y cmake cmake-gui 
sudo apt-get install -y libusb-1.0-0-dev libusb-dev libudev-dev
sudo apt-get install -y mpi-default-dev openmpi-bin openmpi-common  
sudo apt-get install -y libflann1.8 libflann-dev
sudo apt-get install -y libeigen3-dev
sudo apt-get install -y libboost-all-dev
# sudo apt-get install -y libvtk5.10-qt4 libvtk5.10 libvtk5-dev
sudo apt-get install -y libqhull* libgtest-dev
sudo apt-get install -y freeglut3-dev pkg-config
sudo apt-get install -y libxmu-dev libxi-dev 
sudo apt-get install -y mono-complete
sudo apt-get install -y qt-sdk openjdk-8-jdk openjdk-8-jre

sudo apt-get install -y libopenni-dev 
sudo apt-get install -y libopenni2-dev

sudo apt-get install libx11-dev libxext-dev libxtst-dev libxrender-dev libxmu-dev libxmuu-dev
sudo apt-get install build-essential libgl1-mesa-dev libglu1-mesa-dev
sudo apt-get install cmake cmake-gui

sudo apt-get install libx11-dev libxext-dev libxtst-dev libxrender-dev libxmu-dev libxmuu-dev libxi-dev
sudo apt-get install freeglut3-dev pkg-config
sudo apt-get install build-essential libgl1-mesa-dev libglu1-mesa-dev libglew-dev

4.2 vtk 7编译安装

复制代码

tar -zvxf VTK-7.1.1.tar.gz && cd VTK-7.1.1
mkdir build && cd build
cmake  -DCMAKE_C_COMPILER=$C_COMPILER -DCMAKE_CXX_COMPILER=$CXX_COMPILER \
-DCMAKE_INSTALL_PREFIX=$WORK_DIR/ext/install/$ARCH \
-DCMAKE_C_FLAGS="-L$WORK_DIR/ext/install/$ARCH/lib" ..
make -j4
make install

4.3 metslib

复制代码

cd $WORK_DIR
cd ext/
wget https://www.coin-or.org/download/source/metslib/metslib-0.5.3.tgz
tar xzvf metslib-0.5.3.tgz
cd metslib-0.5.3
#./configure
 ./configure --enable-static --prefix=$WORK_DIR/ext/install/$ARCH
make -j4
make install

4.4 pcl

pcl 1.8.1的编译，用i3的台式机，CPU 4核心，8G RAM可以编译。

复制代码

sudo apt-get install libpcap-dev

cd $WORK_DIR
cd ext/
mkdir -p install && mkdir -p install/$ARCH && mkdir -p install/$ARCH/lib && mkdir -p install/$ARCH/include
rm -rf pcl-pcl-1.8.1
tar -zvxf pcl-pcl-1.8.1.tar.gz && cd pcl-pcl-1.8.1
rm -rf build
mkdir build && cd build
cmake  -DCMAKE_C_COMPILER=$C_COMPILER -DCMAKE_CXX_COMPILER=$CXX_COMPILER \
-DCMAKE_INSTALL_PREFIX=$WORK_DIR/ext/install/$ARCH \
-DBUILD_GPU=ON -DBUILD_apps=ON -DBUILD_examples=ON \
-DVTK_DIR=$WORK_DIR/ext/install/x64/lib/cmake/vtk-7.1 \
-DBUILD_visualization=ON \
-DBUILD_SHARED_LIBS=OFF -DCMAKE_C_FLAGS="-L$WORK_DIR/ext/install/$ARCH/lib" ..
make -j8
make install

4.5 再次编译sdk的samples

复制代码

cd $WORK_DIR
cd ext/MYNT-EYE-S-SDK-2.0.1-rc2
cd samples
rm -rf build
mkdir build && cd build 
cmake \
-DCMAKE_C_COMPILER=$C_COMPILER -DCMAKE_CXX_COMPILER=$CXX_COMPILER \
-DOpenCV_DIR=$WORK_DIR/ext/install/$ARCH/share/OpenCV \
-Dmynteye_DIR=$WORK_DIR/ext/install/$ARCH/lib/cmake/mynteye \
-DPCL_DIR=$WORK_DIR/ext/install/$ARCH/share/pcl-1.8 \
-DCMAKE_C_FLAGS="-L$WORK_DIR/ext/install/$ARCH/lib" \
..
make -j8
cp ../_output/bin/* ../../../install/$ARCH/bin/ -r

4.6 运行测试

复制代码

cd ~/workspace/s1030ir_out_of_box
export WORK_DIR=$PWD
export ARCH="x64"
cd $WORK_DIR/ext/install/$ARCH/bin/tutorials
sudo ./get_points

5. 实现设备驱动

程序目录结构

config模块是用iniparser的，log模块用了log4cplus，不用也可以。

5.1 m_device.cpp

复制代码

//
// Created by qjfen on 2024/1/19.
//

#include "stdio.h"
#include <cstring>              // memset
#include <iostream>             // std::cout
#include "m_device.h"

#include "mynteye/context.h"
#include "mynteye/device.h"
#include "mynteye/logger.h"
#include "mynteye/api.h"
#include <opencv2/opencv.hpp>

MYNTEYE_USE_NAMESPACE

std::shared_ptr<API> handle;
bool g_enableDepth;
bool g_enableImu;

m_device::m_device()
{
	//ctor
}

m_device::~m_device()
{
	//dtor
}
m_device& m_device::getInstance()
{
	static m_device instance;
	return instance;
}
int m_device::init(int handle_index)
{

	int ret;

	LOG(INFO) << "Detecting MYNT EYE devices";
	Context context;
	auto &&devices = context.devices();

	size_t n = devices.size();
	if (n <= 0) {
		LOG(ERROR) << "No MYNT EYE devices :(";
		return -1;
	}

	for (size_t i = 0; i < n; i++) {
		auto &&device = devices[i];
		LOG(INFO) << "MYNT EYE device index: " << i;
		LOG(INFO) << "  Device name: " << device->GetInfo(Info::DEVICE_NAME);
		LOG(INFO) << "  Serial number: " << device->GetInfo(Info::SERIAL_NUMBER);
		LOG(INFO) << "  Firmware version: "
		          << device->GetInfo(Info::FIRMWARE_VERSION);
		LOG(INFO) << "  Hardware version: "
		          << device->GetInfo(Info::HARDWARE_VERSION);
		LOG(INFO) << "  Spec version: " << device->GetInfo(Info::SPEC_VERSION);
		LOG(INFO) << "  Lens type: " << device->GetInfo(Info::LENS_TYPE);
		LOG(INFO) << "  IMU type: " << device->GetInfo(Info::IMU_TYPE);
		LOG(INFO) << "  Nominal baseline: "
		          << device->GetInfo(Info::NOMINAL_BASELINE);
	}

	handle = API::Create();
	if (!handle)
		return -1;

	return 0;
}
int m_device::close()
{
	int ret;
	if (!g_enableImu) {
		handle->Stop(Source::VIDEO_STREAMING);
	} else {
		handle->Stop(Source::ALL);
	}
	return 0;
}
int m_device::config(bool enable_depth, bool enable_imu)
{
	g_enableDepth = enable_depth;
	g_enableImu = enable_imu;

	if(g_enableDepth) {
		handle->EnableStreamData(Stream::DEPTH);
	}
	if(g_enableImu) {
		handle->EnableMotionDatas();
	}
	return 0;
}
int m_device::open()
{

	if (!g_enableImu) {
		handle->Start(Source::VIDEO_STREAMING);
	} else {
		handle->Start(Source::ALL);
	}
}
int m_device::recvFrame(cv::Mat* mat_left, cv::Mat* mat_right, cv::Mat* mat_depth, std::vector<mynteye::api::MotionData>* imu_datas)
{
	handle->WaitForStreams();
	api::StreamData left_data = handle->GetStreamData(Stream::LEFT);
	api::StreamData right_data = handle->GetStreamData(Stream::RIGHT);

	*mat_left = left_data.frame.clone();
	*mat_right = right_data.frame.clone();
	if (left_data.frame.empty()) {
		LOG(INFO) << "left_data is empty";
		return -1;
	}
	if (right_data.frame.empty()) {
		LOG(INFO) << "right_data is empty";
		return -1;
	}

	if(g_enableDepth) {
		api::StreamData depth_data = handle->GetStreamData(Stream::DEPTH);
		*mat_depth = depth_data.frame.clone();
		if (depth_data.frame.empty()) {
			LOG(INFO) << "depth_data is empty";
			return -1;
		}
	}
	if(g_enableImu) {
		std::vector<api::MotionData> motion_datas = handle->GetMotionDatas();
		*imu_datas = motion_datas;
	}
	return 0;
}

5.2 m_device.h

复制代码

#ifndef __M_DEVICE_H
#define __M_DEVICE_H

#define device_inst            m_device::getInstance()


#include "stdint.h"                 // uint32_t
#include <opencv2/opencv.hpp>
#include "mynteye/api.h"

class m_device
{

public:
	m_device();
	virtual ~m_device();
	static m_device& getInstance();

	int init(int handle_index);
	int open();
	int close();
	int recvFrame(cv::Mat* mat_left, cv::Mat* mat_right, cv::Mat* mat_depth, std::vector<mynteye::api::MotionData>* imu_datas);
	int config(bool enable_depth, bool enable_imu);
	
protected:

private:
};


#endif //__M_DEVICE_H

5.3 demo_device.cpp

复制代码

#include <stdio.h>
#include <unistd.h>         // usleep

#include "m_device.h"

#include <opencv2/opencv.hpp>
#include "mynteye/api.h"
#include "mynteye/logger.h"

int main(void)
{
	int ret;
	printf("[main]device_inst.init...\r\n");
	ret = device_inst.init(0);

	printf("[main]device_inst.config...\r\n");
	bool enable_depth = true;
	bool enable_imu = true;
	ret = device_inst.config(enable_depth, enable_imu);

	printf("[main]device_inst.open...\r\n");
	ret = device_inst.open();

	printf("[main]device_inst.recvMat...\r\n");
	cv::Mat mat_left, mat_right, mat_depth;
	std::vector<mynteye::api::MotionData> motion_datas;
	while(1) {
		ret = device_inst.recvFrame(&mat_left, &mat_right, &mat_depth, &motion_datas);
		if (ret>=0) {
			break;
		}

		cv::waitKey(1);
	}
	printf("[main]device_inst.close...\r\n");
	device_inst.close();

	cv::imwrite("mat_left.png", mat_left);
	cv::imwrite("mat_right.png", mat_right);
	if(enable_depth) {
		cv::imwrite("mat_depth.png", mat_depth);
	}

	cv::imshow("mat_left", mat_left);
	cv::imshow("mat_right", mat_right);
	if(enable_depth) {
		cv::imshow("mat_depth", mat_depth);
	}
	cv::waitKey(0);
	cv::destroyAllWindows();

	if(enable_imu)
	{
		
    for (auto &&data : motion_datas) {
      LOG(INFO) << "Imu frame_id: " << data.imu->frame_id
                << ", timestamp: " << data.imu->timestamp
                << ", accel_x: " << data.imu->accel[0]
                << ", accel_y: " << data.imu->accel[1]
                << ", accel_z: " << data.imu->accel[2]
                << ", gyro_x: " << data.imu->gyro[0]
                << ", gyro_y: " << data.imu->gyro[1]
                << ", gyro_z: " << data.imu->gyro[2]
                << ", temperature: " << data.imu->temperature;
    }
		
	}
	return 0;
}

5.4 编译

复制代码

# 环境
cd ~/workspace/s1030ir_out_of_box
export WORK_DIR=$PWD
export ARCH="x64"

# 用户源码编译出库
$CXX_COMPILER -c src/m_device.cpp -Iinclude -Iext/install/$ARCH/include -Lext/install/$ARCH/lib-std=c++11 -lmynteye -lopencv_calib3d -lopencv_imgproc -lopencv_imgcodecs -lopencv_core -lopencv_photo  -lopencv_highgui -std=c++11
$AR -r libs1030ir.a *.o
rm *.o

# 编译可执行程序
$CXX_COMPILER -o demo_device demo/demo_device.cpp -Iext/install/$ARCH/include -Iinclude -L. -Lext/install/$ARCH/lib  -ls1030ir -lmynteye -lopencv_calib3d -lopencv_imgproc -lopencv_imgcodecs -lopencv_core -lopencv_photo  -lopencv_highgui -std=c++11

5.5 效果

获取到了左右摄像头的图像、深度图像、和IMU数据。