Learning for Video Super-Resolution through HR Optical Flow Estimation

TL;DR

This paper introduces an end-to-end trainable video super-resolution framework that super-resolves both images and optical flows, demonstrating that high-resolution optical flows improve correspondence accuracy and overall SR quality.

Contribution

The paper presents a novel HR optical flow reconstruction network (OFRnet) and integrates it into a video SR framework, enhancing accuracy and consistency over existing methods.

Findings

HR optical flows outperform LR flows in correspondence accuracy.

The proposed framework achieves state-of-the-art results on Vid4 and DAVIS-10 datasets.

Super-resolving optical flows improves both accuracy and temporal consistency.

Abstract

Video super-resolution (SR) aims to generate a sequence of high-resolution (HR) frames with plausible and temporally consistent details from their low-resolution (LR) counterparts. The generation of accurate correspondence plays a significant role in video SR. It is demonstrated by traditional video SR methods that simultaneous SR of both images and optical flows can provide accurate correspondences and better SR results. However, LR optical flows are used in existing deep learning based methods for correspondence generation. In this paper, we propose an end-to-end trainable video SR framework to super-resolve both images and optical flows. Specifically, we first propose an optical flow reconstruction network (OFRnet) to infer HR optical flows in a coarse-to-fine manner. Then, motion compensation is performed according to the HR optical flows. Finally, compensated LR inputs are fed to a super-resolution network (SRnet) to generate the SR results. Extensive experiments demonstrate that HR optical flows provide more accurate correspondences than their LR counterparts and improve both accuracy and consistency performance. Comparative results on the Vid4 and DAVIS-10 datasets show that our framework achieves the state-of-the-art performance.