Is There Tradeoff between Spatial and Temporal in Video Super-Resolution?

TL;DR

This paper investigates the potential tradeoff between spatial and temporal quality in video super-resolution, questioning whether optimizing for one inherently compromises the other and exploring joint optimization possibilities.

Contribution

It provides an analysis of the relationship between spatial and temporal quality in video SR and discusses the feasibility of jointly optimizing both metrics.

Findings

Identifies a potential tradeoff between spatial and temporal quality.

Highlights that optimizing for spatial quality alone may lead to temporal inconsistency.

Suggests the need for joint optimization strategies in video SR.

Abstract

Recent advances of deep learning lead to great success of image and video super-resolution (SR) methods that are based on convolutional neural networks (CNN). For video SR, advanced algorithms have been proposed to exploit the temporal correlation between low-resolution (LR) video frames, and/or to super-resolve a frame with multiple LR frames. These methods pursue higher quality of super-resolved frames, where the quality is usually measured frame by frame in e.g. PSNR. However, frame-wise quality may not reveal the consistency between frames. If an algorithm is applied to each frame independently (which is the case of most previous methods), the algorithm may cause temporal inconsistency, which can be observed as flickering. It is a natural requirement to improve both frame-wise fidelity and between-frame consistency, which are termed spatial quality and temporal quality, respectively. Then we may ask, is a method optimized for spatial quality also optimized for temporal quality? Can we optimize the two quality metrics jointly?