【AI最前线】DP双像素sensor相关的AI算法全集：深度估计、图像去模糊去雨去雾恢复、图像重建、自动对焦

Dual Pixel 简介

双像素是成像系统的感光元器件中单帧同时生成的图像：通过双像素可以实现：深度估计、图像去模糊去雨去雾恢复、图像重建

成像原理来源如上，也有遮罩等方式的pd生成，如图双像素视图可以看到光圈的不同一半，这提供了一个深度提示。然而，由于基本的模糊性，如果相机的焦距（或光圈大小或焦距）发生变化，不同的场景可能会产生相同的双像素图像。在（a）中，具有焦距g1的相机在距离Z1处成像聚焦的蓝色点和离焦的橙色点。通过光圈左半部分折射的光（深蓝色和橙色光线）到达每个双像素的右半部分，反之亦然。这导致了一个双像素图像，其中失焦橙色点被d像素（a，"DP数据"）位移，被b像素模糊（a，"图像"）。在（b）中，不同的焦距和场景深度集产生相同的双像素和RGB图像。然而，如文中所示，该场景通过逆深度上的仿射变换与（a）中的场景相关。消费者迅速采用。双像素相机似乎代表了更雄心勃勃的光场相机和传统相机之间的一种有前景的折衷方案------DP相机牺牲了可忽略的空间分辨率来采样光场中的两个角度，而真正的单眼相机只采样一个角度，Lytro Illum等光场相机以牺牲显著的空间分辨率为代价采样196个角度。因此，它们在消费类相机和内窥镜等空间受限的应用中得到了更广泛的采用[6]。

消费类硬件的最新发展可能为深度估计的新方法提供机会。最近，通过使用密集的双像素（DP）传感器（图2），可以使用一台相机同时捕获两张类似于具有微小基线的立体对的图像（图1）。虽然这项技术最初是为相机自动对焦而开发的，但双像素图像也可以用来从单个相机中恢复密集的深度图，从而消除了对额外硬件、校准或同步的任何需求。例如，Wadhwa等人[55]使用经典的立体技术（块匹配和边缘感知平滑）从DP数据中恢复深度。但如图1所示，传统立体技术可以生成的深度图的质量是有限的，因为DP图像中视差和焦点之间的相互作用可能会导致经典立体匹配技术失败。现有的基于单眼学习的技术在这方面也表现不佳

接下来就介绍双像素相关的sota论文和代码，长期更新教学

Table of contents

• DualPixel
• Dateset

Dual Pixel

Year	Pub	Paper	App	Repo
2018	SIGGRAPH	Synthetic Depth-of-Field with a Single-Camera Mobile Phone	Depth / Segmenation / Synthetic DoF
2019	CVPR	Reflection Removal Using a Dual-Pixel Sensor	Reflection Removal	Code
2019	ICCV	Learning Single Camera Depth Estimation using Dual-Pixels	Depth	Code & Dataset
2020	ICCP	Modeling Defocus-Disparity in Dual-Pixel Sensors	Depth	Code & Dataset
2020	CVPR	Learning to Autofocus	Autofocus	Dataset
2020	ECCV	Du2Net: Learning Depth Estimation from Dual-Cameras and Dual-Pixels	Disparity (w/Stereo)
2020	ECCV	Defocus Deblurring Using Dual-Pixel Data	Deblur	Code & Dataset
2021	CVPR	Dual Pixel Exploration: Simultaneous Depth Estimation and Image Restoration	Depth / Deblur	Code & Dataset
2021	CVPRW	NTIRE 2021 Challenge for Defocus Deblurring Using Dual-pixel Images: Methods and Results	Deblur
2021	CVPRW	ATTSF Attention! Stay Focus!	Deblur	Code
2021	ICCV	Defocus Map Estimation and Deblurring From a Single Dual-Pixel Image	Deblur / Defocus Map	Code & Dataset
2021	ICCV	Learning to Reduce Defocus Blur by Realistically Modeling Dual-Pixel Data	Deblur	Code & Dataset
2021	IEEE	World Largest Mobile Image Sensor with All Directional Phase Detection Auto Focus Function	Depth
2021	ICTC	Disparity probability volume guided defocus deblurring using dual pixel data	Deblur
2021	Journal of Electronic Imaging	Defocus deblurring: a designed deep model based on CNN	Deblur
2021	IEEE	All-Directional Dual Pixel Auto Focus Technology in CMOS Image Sensors	Circuits
2022	WACV	Improving Single-Image Defocus Deblurring: How Dual-Pixel Images Help Through Multi-Task Learning	Deblur	Code & Dataset
2022	ISSCC	A 1/1.57-inch 50Mpixel CMOS Image Sensor With 1.0μm All-Directional Dual Pixel by 0.5μm-Pitch Full-Depth Deep-Trench Isolation Technology	Circuits
2022	ECCV	Facial Depth and Normal Estimation using Single Dual-Pixel Camera	Depth / Surface Normal / Anti-spoofing / Relighting	Code & Dataset
2023	CVPR	K3DN: Disparity-Aware Kernel Estimation for Dual-Pixel Defocus Deblurring	Deblur
2023	CVPR	Spatio-Focal Bidirectional Disparity Estimation From a Dual-Pixel Image	Depth / Disparity	Code
2023	ICCP	Learning to Synthesize Photorealistic Dual-pixel Images from RGBD frames	Simulator	Code & Dataset
2023	ICCV	Exploring Positional Characteristics of Dual-Pixel Data for Camera Autofocus	Autofocus

这其中

|------|------|-----------------------------------------------------------------------------------------------------------------------------------------------------------|-------|
| 2019 | ICCV | Learning Single Camera Depth Estimation using Dual-Pixels | Depth |

网络的代码被复现在：DualPixelFace/src at main · MinJunKang/DualPixelFace · GitHub ；GitHub - vyi/PdCapture: [google-research/dual-pixels](Forked from https://github.com/google-research/google-research/tree/master/dual_pixels)

GitHub - RugvedKatole/Learning-Single-Camera-Depth-Estimation-using-Dual-Pixels: This Repo is an implementation of paper titled "Learning Single Camera Depth Estimation using Dual-Pixels"

Dataset

Year	Pub	Paper	Detail
2019	ICCV	Learning Single Camera Depth Estimation using Dual-Pixels	Train:2506, Test:684, Res:1512x2016(DP), 16bit png, DP Raw / Depth
2020	ICCP	Modeling Defocus-Disparity in Dual-Pixel Sensors	Num:100, Res:5180x2940, RGB 8bit jpg / 16bit tif Depth, DP LR / Depth
2020	ECCV	Defocus Deblurring Using Dual-Pixel Data	Num:500, Res:1680x1120, 16bit, Used for NTIRE 2021 Challenge (CVPRW)
2021	CVPR	Dual Pixel Exploration: Simultaneous Depth Estimation and Image Restoration	DP Simulator from NYUD Dataset
2021	ICCV	Learning to Reduce Defocus Blur by Realistically Modeling Dual-Pixel Data	DP Simulator form SYNTHIA-SF dataset
2023	ICCP	Learning to Synthesize Photorealistic Dual-pixel Images from RGBD frames	Num:5130, Res:1680x1120, RGB(DP LRC) 8bit png / Depth 16bit png

小结

欢迎AI深度估计、AI重建、AI恢复图像：图像去雾去雨去模糊降噪方向的同学一起交流，cv君正在打造最大最前沿的AI社区，让你在浓厚的学习氛围中进步，让你在飞速发展的时代中砥砺前行；免费进群交流请私信cv君（免费）