Combining Pre- and Post-Demosaicking Noise Removal for RAW Video

TL;DR

This paper introduces a novel denoising approach for RAW video that combines pre- and post-demosaicking neural networks, leveraging self-similarity and temporal filtering to adapt to various noise levels and improve real-world video quality.

Contribution

It proposes a self-similarity-based scheme that weights pre- and post-demosaicking denoisers, with adaptive influence based on noise levels, and incorporates temporal prefiltering for enhanced texture reconstruction.

Findings

Higher noise levels benefit from increased pre-demosaicking influence.

The method adapts to any noise level using sensor noise estimation.

It achieves competitive results with state-of-the-art denoising techniques.

Abstract

Denoising is one of the fundamental steps of the processing pipeline that converts data captured by a camera sensor into a display-ready image or video. It is generally performed early in the pipeline, usually before demosaicking, although studies swapping their order or even conducting them jointly have been proposed. With the advent of deep learning, the quality of denoising algorithms has steadily increased. Even so, modern neural networks still have a hard time adapting to new noise levels and scenes, which is indispensable for real-world applications. With those in mind, we propose a self-similarity-based denoising scheme that weights both a pre- and a post-demosaicking denoiser for Bayer-patterned CFA video data. We show that a balance between the two leads to better image quality, and we empirically find that higher noise levels benefit from a higher influence pre-demosaicking. We also integrate temporal trajectory prefiltering steps before each denoiser, which further improve texture reconstruction. The proposed method only requires an estimation of the noise model at the sensor, accurately adapts to any noise level, and is competitive with the state of the art, making it suitable for real-world videography.