Monakhova K, Richter S R, Waller L, et al. Dancing under the stars: video denoising in starlight[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022: 16241-16251.
Introduction
- hawkmoths and carpenter bees: navigate on the darkest moonless nights by the light of the stars
- CMOS camera generally needs at least 3/4 moon illumination (>0.1 lux)
- The reason for Challenge: the minuscule amounts of light in the environment
some tips
- using long exposure times for photographers to collect enough light from the scene
- working well for still images, but limiting temporal resolution and precluding imaging of moving objects
- increasing the gain for camera--each pixel more sensitive to light
- shorter exposures, but increasing the noise present in each frame
Method
classic methods
- from classic methods to deep learning-based approaches:
- extract the signal from the noise
- based on assumptions about the statistical distributions of the image and noise
- noise model (Gaussian or Poisson-Gaussian noise)
- the understanding of the structure of the noise
deep learning-based approaches
- from classic methods to deep learning-based approaches:
- train a denoiser using clean/noisy image pairs
- some issues:
- thousands of training image pairs
- noise is highly camera-specific
- for non-moving objects
Method
- a good camera
- a physics-inspired noise generator
- train a video denoiser
- physics-inspired noise generator
- shot noise, read noise, row noise, temporal row noise, quantization noise, fixed pattern, and periodic noise
Datasets
- there are more detectable photons in near infrared (NIR) than at RGB wavelengths at night
- Canon LI3030SAI Sensor
- paired clean/noisy static scenes
- clean videos of moving objects
- noisy videos of moving objects
- 10 clips--2558 noisy images
- 67 clean/noisy image pairs --16 noisy bursts
- 10 video sequences:166 video clips for training, 10 for testing
- 329 video clips from MOT video[1]
Video denoising
- white-balancing, histogram equalization, gamma correction
Experiments
- low-light noise model (ELD)
- CA-GAN
- Noise Flow
- single-image denoising
- video denoising
1\]Leal-Taixé L, Milan A, Reid I, et al. Motchallenge 2015: Towards a benchmark for multi-target tracking\[J\]. arXiv preprint arXiv:1504.01942, 2015. *** ** * ** ***