Dancing under the stars: video denoising in starlight

Monakhova K, Richter S R, Waller L, et al. Dancing under the stars: video denoising in starlight $C$ //Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022: 16241-16251.

Introduction

hawkmoths and carpenter bees: navigate on the darkest moonless nights by the light of the stars
CMOS camera generally needs at least 3/4 moon illumination (>0.1 lux)
The reason for Challenge: the minuscule amounts of light in the environment

some tips

using long exposure times for photographers to collect enough light from the scene
working well for still images, but limiting temporal resolution and precluding imaging of moving objects
increasing the gain for camera--each pixel more sensitive to light
shorter exposures, but increasing the noise present in each frame

Method

classic methods

from classic methods to deep learning-based approaches:
extract the signal from the noise
based on assumptions about the statistical distributions of the image and noise
noise model (Gaussian or Poisson-Gaussian noise)
the understanding of the structure of the noise

deep learning-based approaches

from classic methods to deep learning-based approaches:
train a denoiser using clean/noisy image pairs
some issues:
thousands of training image pairs
noise is highly camera-specific
for non-moving objects

Method

a good camera
a physics-inspired noise generator
train a video denoiser

physics-inspired noise generator
shot noise, read noise, row noise, temporal row noise, quantization noise, fixed pattern, and periodic noise

Datasets

there are more detectable photons in near infrared (NIR) than at RGB wavelengths at night
Canon LI3030SAI Sensor
paired clean/noisy static scenes
clean videos of moving objects
noisy videos of moving objects
10 clips--2558 noisy images
67 clean/noisy image pairs --16 noisy bursts
10 video sequences:166 video clips for training, 10 for testing
329 video clips from MOT video $1$

Video denoising

white-balancing, histogram equalization, gamma correction

Experiments

low-light noise model (ELD)
CA-GAN
Noise Flow

single-image denoising
video denoising

$1$ Leal-Taixé L, Milan A, Reid I, et al. Motchallenge 2015: Towards a benchmark for multi-target tracking $J$ . arXiv preprint arXiv:1504.01942, 2015.