RefVSR++: Exploiting Reference Inputs for Reference-based Video Super-resolution

TL;DR

RefVSR++ is a novel video super-resolution method that independently aggregates reference and low-resolution inputs over time, significantly improving performance by over 1dB PSNR compared to previous approaches.

Contribution

It introduces a parallel temporal aggregation framework for reference and LR images, along with enhanced feature alignment mechanisms, advancing the state-of-the-art in reference-based VSR.

Findings

Outperforms previous methods by over 1dB PSNR

Establishes a new benchmark in reference-based VSR

Demonstrates the effectiveness of independent temporal aggregation

Abstract

Smartphones with multi-camera systems, featuring cameras with varying field-of-views (FoVs), are increasingly common. This variation in FoVs results in content differences across videos, paving the way for an innovative approach to video super-resolution (VSR). This method enhances the VSR performance of lower resolution (LR) videos by leveraging higher resolution reference (Ref) videos. Previous works, which operate on this principle, generally expand on traditional VSR models by combining LR and Ref inputs over time into a unified stream. However, we can expect that better results are obtained by independently aggregating these Ref image sequences temporally. Therefore, we introduce an improved method, RefVSR++, which performs the parallel aggregation of LR and Ref images in the temporal direction, aiming to optimize the use of the available data. RefVSR++ also incorporates improved mechanisms for aligning image features over time, crucial for effective VSR. Our experiments demonstrate that RefVSR++ outperforms previous works by over 1dB in PSNR, setting a new benchmark in the field.