初识增强现实（AR）

笔记来源：

1.2023年中国增强现实（AR）行业研究报告

3.In-Depth Review of Augmented Reality: Tracking Technologies, Development Tools, AR Displays, Collaborative AR, and Security Concerns

4.Design and Implementation of Augmented Reality Browser for Mobile Terminal

5.FUSE: Towards AI-Based Future Services for Generating Augmented Reality Experiences

1.1 AR的简单定义

AR can be defined as a system that incorporates three basic features: a combination of real and virtual worlds, real-time interaction, and accurate 3D registration of virtual and real objects.

AR可以被定义为一个包含三个基本特征的系统：

(1) 虚实融合

(2) 实时交互

(3) 虚拟和现实对象的精确3D配准。

1.2 概念区分AR/MR/VR

1.3 AR所需的硬件

1.3.0 Chipset

1.3.1 Displays

AR光学显示系统大体包含两大块：

光学组合器
微型显示屏

1.3.2 Tracking

运动追踪所需硬件

motion tracking technologies： digital cameras and/or other optical sensors、accelerometers、GPS、gyroscopes、solid state compasses、radio-frequency identification (RFID)

下图来自：In-Depth Review of Augmented Reality: Tracking Technologies, Development Tools, AR Displays, Collaborative AR, and Security Concerns

1.3.3 Input Device

技术包括将用户的口语翻译成计算机指令的语音识别系统，以及通过视觉检测或嵌入在外围设备（例如魔杖、手写笔、指针、手套或其他身体穿戴设备）中的传感器来解释用户的身体动作的手势识别系统。

1.3.4 Networking

移动AR应用程序依赖强大计算能力，为弥补移动设备计算能力不足，将应用程序的计算传输给远程计算平台，这时就需要移动网络设备的支持对数据进行传输。

1.4 AR所需的软件和算法

1.4.1 Alighment (Registration) and Tracking

将虚拟物体和真实环境放在同一个坐标系下，在这种情况下虚拟物体相对真实环境就有了正确的位置和方向（配准问题）

Registration or alignment of partial 3D models is the process of bringing them all into the same reference frame by means of a suitable transformation.

部分 3D 模型的配准或对齐是通过适当的变换将它们全部带入同一参考系的过程。
Registration technology enables virtual images to be superimposed accurately in the proper environment. The main flow of 3d registration technology has two steps. First, determine the relationship between the virtual image, the model and the direction and position information of the camera or display device. Second, the virtual rendered image and model are accurately projected into the real environment, so the virtual image and model can be merged with the real environment. For the three-dimensional registration technology based on computer vision, it sets the reference point to realize the determination of the direction and position of the real scene by the camera or the display. --Research on Tracking and Registration Algorithm Based on Natural Feature Point

配准技术使虚拟图像能够在适当的环境中准确地叠加。

3D配准技术的主要流程有两个步骤。

首先，确定虚拟图像、模型以及摄像机或显示设备的方向和位置信息之间的关系。

其次，将虚拟渲染的图像和模型精确地投影到真实环境中，因此虚拟图像和模型可以与真实环境融合。

对于基于计算机视觉的三维配准技术来说，它通过设置参考点来实现摄像机或显示器对真实场景的方向和位置的确定。
To realize the three-dimensional registration process in augmented reality, the internal and external parameters of the camera are required. In fact, the camera calibration determines the intrinsic matrix, and the extrinsic matrix contains a translation component T and 3 rotation components. Therefore, the external parameter matrix M of each frame of the image obtained in the three-dimensional registration can be uniquely determined.The position of the point, that is, the accurate registration position of the virtual object in the real scene.--Research on Tracking and Registration Algorithm Based on Natural Feature Point

为了实现增强现实中的三维配准过程，需要相机的内部和外部参数。事实上，相机标定决定了内参矩阵，外参矩阵包含一个平移分量T和3个旋转分量。因此，三维配准得到的每一帧图像的外参矩阵可以唯一确定。点的位置，即虚拟物体在真实场景中的准确配准位置。

经过配准后，用户、虚拟物体和真实环境同处在一个坐标系下，当后续虚拟物体或和用户进行移动时，能够更加精准的追踪虚拟物体和用户（摄像头）相对于真实环境的位置

实现追踪所需的技术

下图来自：In-Depth Review of Augmented Reality: Tracking Technologies, Development Tools, AR Displays, Collaborative AR, and Security Concerns

许多增强现实的计算机视觉方法都是继承自视觉里程计（VO）

上述方法通常由两部分组成：

The first stage is to detect interest points, fiducial markers or optical flow in the camera images. This step can use feature detection methods like corner detection, blob detection, edge detection or thresholding, and other image processing methods.

第一阶段：检测相机图像中的兴特征点、基准标记或光流。此步骤可以使用特征检测方法，例如角点检测、斑点检测、边缘检测或阈值处理以及其他图像处理方法。
The second stage restores a real world coordinate system from the data obtained in the first stage. Some methods assume objects with known geometry (or fiducial markers) are present in the scene. In some of those cases the scene 3D structure should be calculated beforehand. If part of the scene is unknown simultaneous localization and mapping (SLAM) can map relative positions. If no information about scene geometry is available, structure from motion methods like bundle adjustment are used. Mathematical methods used in the second stage include: projective (epipolar) geometry, geometric algebra, rotation representation with exponential map, kalman and particle filters, nonlinear optimization, robust statistics.

第二阶段：根据第一阶段获得的数据恢复真实世界坐标系。一些方法假设场景中存在具有已知几何形状（或基准标记）的对象。在某些情况下，应事先计算场景 3D 结构。

如果场景的一部分未知，同时定位和建图 (SLAM) 可以绘制相对位置。

如果没有有关场景几何的信息可用，则使用类似BA的SfM方法

第二阶段使用的数学方法包括：对极几何、几何代数、指数图旋转表示（Connecting the Lie algebra to the Lie group is the exponential map）、卡尔曼和粒子滤波器、非线性优化、鲁棒统计。

1.4.2 Rendering (lighting、shading, etc.)

Rendering is the process of generating a visual representation of a 3D scene or object from a 3D model. In augmented reality, rendering is a critical step in creating realistic and immersive AR experiences. It involves the use of computer algorithms to calculate how virtual objects should appear in the user's view, considering factors like lighting, shadows, textures, and perspective.

AR rendering pipeline.

The camera or a set of sensors capture information about the environment.

The information is processed to:

(1) calculate camera extrinsic parameters and

(2) build understanding of the environment in order to generate augmentations in the form of text, images, 3D objects or neural scene representations.

(3) Finally, the extrinsic and graphic primitives are fed to the renderer which creates 2D images ready to be displayed.

以上只提到了配准、追踪、渲染，其实AR还涉及很多算法，接下来我们看看AR目前面临的几大问题

初识增强现实（AR）