Multiframe Motion Coupling for Video Super Resolution

TL;DR

This paper introduces a novel variational approach for video super resolution that efficiently incorporates motion information across multiple frames, achieving state-of-the-art results with linear growth in computational complexity.

Contribution

It presents the first variational super resolution method that jointly optimizes multiple frames with motion coupling, improving efficiency and temporal consistency.

Findings

Achieves state-of-the-art super resolution quality.

Computational complexity grows linearly with the number of frames.

Competitive with machine learning approaches.

Abstract

The idea of video super resolution is to use different view points of a single scene to enhance the overall resolution and quality. Classical energy minimization approaches first establish a correspondence of the current frame to all its neighbors in some radius and then use this temporal information for enhancement. In this paper, we propose the first variational super resolution approach that computes several super resolved frames in one batch optimization procedure by incorporating motion information between the high-resolution image frames themselves. As a consequence, the number of motion estimation problems grows linearly in the number of frames, opposed to a quadratic growth of classical methods and temporal consistency is enforced naturally. We use infimal convolution regularization as well as an automatic parameter balancing scheme to automatically determine the reliability of the motion information and reweight the regularization locally. We demonstrate that our approach yields state-of-the-art results and even is competitive with machine learning approaches.