Real-RawVSR: Real-World Raw Video Super-Resolution with a Benchmark Dataset

TL;DR

This paper introduces the first real-world raw video super-resolution dataset and a novel two-branch network method that effectively enhances raw videos to high-resolution sRGB outputs, outperforming existing methods.

Contribution

The creation of a real-world raw video SR dataset and the development of a two-branch network tailored for raw Bayer pattern videos are the key innovations.

Findings

The proposed method outperforms benchmark real and synthetic video SR methods.

The dataset includes 450 diverse video pairs captured with DSLR cameras.

The two-branch network effectively leverages raw Bayer data for super-resolution.

Abstract

In recent years, real image super-resolution (SR) has achieved promising results due to the development of SR datasets and corresponding real SR methods. In contrast, the field of real video SR is lagging behind, especially for real raw videos. Considering the superiority of raw image SR over sRGB image SR, we construct a real-world raw video SR (Real-RawVSR) dataset and propose a corresponding SR method. We utilize two DSLR cameras and a beam-splitter to simultaneously capture low-resolution (LR) and high-resolution (HR) raw videos with 2x, 3x, and 4x magnifications. There are 450 video pairs in our dataset, with scenes varying from indoor to outdoor, and motions including camera and object movements. To our knowledge, this is the first real-world raw VSR dataset. Since the raw video is characterized by the Bayer pattern, we propose a two-branch network, which deals with both the packed RGGB sequence and the original Bayer pattern sequence, and the two branches are complementary to each other. After going through the proposed co-alignment, interaction, fusion, and reconstruction modules, we generate the corresponding HR sRGB sequence. Experimental results demonstrate that the proposed method outperforms benchmark real and synthetic video SR methods with either raw or sRGB inputs. Our code and dataset are available at https://github.com/zmzhang1998/Real-RawVSR.