Towards Interpretable Video Super-Resolution via Alternating Optimization

TL;DR

This paper introduces an interpretable framework for space-time video super-resolution that jointly addresses deblurring, interpolation, and super-resolution by combining model-based and learning-based methods, improving video quality.

Contribution

It formulates STVSR as two interleaved sub-problems solved alternately, with an analytical Fourier transform layer and a recurrent enhancement module, offering interpretability and superior performance.

Findings

Outperforms existing methods in quantitative metrics

Produces higher visual quality in reconstructed videos

Effectively handles motion blur and aliasing

Abstract

In this paper, we study a practical space-time video super-resolution (STVSR) problem which aims at generating a high-framerate high-resolution sharp video from a low-framerate low-resolution blurry video. Such problem often occurs when recording a fast dynamic event with a low-framerate and low-resolution camera, and the captured video would suffer from three typical issues: i) motion blur occurs due to object/camera motions during exposure time; ii) motion aliasing is unavoidable when the event temporal frequency exceeds the Nyquist limit of temporal sampling; iii) high-frequency details are lost because of the low spatial sampling rate. These issues can be alleviated by a cascade of three separate sub-tasks, including video deblurring, frame interpolation, and super-resolution, which, however, would fail to capture the spatial and temporal correlations among video sequences. To address this, we propose an interpretable STVSR framework by leveraging both model-based and learning-based methods. Specifically, we formulate STVSR as a joint video deblurring, frame interpolation, and super-resolution problem, and solve it as two sub-problems in an alternate way. For the first sub-problem, we derive an interpretable analytical solution and use it as a Fourier data transform layer. Then, we propose a recurrent video enhancement layer for the second sub-problem to further recover high-frequency details. Extensive experiments demonstrate the superiority of our method in terms of quantitative metrics and visual quality.