Multi-Stage Raw Video Denoising with Adversarial Loss and Gradient Mask

TL;DR

This paper introduces a multi-stage learning-based method for raw video denoising under low light, utilizing explicit frame alignment, fusion, and adversarial training with gradient masks to produce coherent, high-quality videos.

Contribution

It proposes a novel multi-stage framework combining alignment, fusion, and adversarial loss with gradient masks for improved raw video denoising.

Findings

Outperforms state-of-the-art denoising methods in quality.

Produces temporally coherent and realistic videos.

Effectively utilizes neighboring frames without direct alignment of distant frames.

Abstract

In this paper, we propose a learning-based approach for denoising raw videos captured under low lighting conditions. We propose to do this by first explicitly aligning the neighboring frames to the current frame using a convolutional neural network (CNN). We then fuse the registered frames using another CNN to obtain the final denoised frame. To avoid directly aligning the temporally distant frames, we perform the two processes of alignment and fusion in multiple stages. Specifically, at each stage, we perform the denoising process on three consecutive input frames to generate the intermediate denoised frames which are then passed as the input to the next stage. By performing the process in multiple stages, we can effectively utilize the information of neighboring frames without directly aligning the temporally distant frames. We train our multi-stage system using an adversarial loss with a conditional discriminator. Specifically, we condition the discriminator on a soft gradient mask to prevent introducing high-frequency artifacts in smooth regions. We show that our system is able to produce temporally coherent videos with realistic details. Furthermore, we demonstrate through extensive experiments that our approach outperforms state-of-the-art image and video denoising methods both numerically and visually.