Spatio-Temporal Perception-Distortion Trade-off in Learned Video SR

TL;DR

This paper introduces a new measure and architecture for video super-resolution that explicitly accounts for the naturalness of motion and flow, improving the perception-distortion trade-off in spatio-temporal video quality.

Contribution

It proposes a novel spatio-temporal perceptual quality measure and a perceptual VSR architecture that enforces flow naturalness for better video quality trade-offs.

Findings

The new measure emphasizes optical flow naturalness.

The PSVR architecture enforces flow naturalness explicitly.

Experimental results support the importance of motion naturalness in perception-distortion trade-off.

Abstract

Perception-distortion trade-off is well-understood for single-image super-resolution. However, its extension to video super-resolution (VSR) is not straightforward, since popular perceptual measures only evaluate naturalness of spatial textures and do not take naturalness of flow (temporal coherence) into account. To this effect, we propose a new measure of spatio-temporal perceptual video quality emphasizing naturalness of optical flow via the perceptual straightness hypothesis (PSH) for meaningful spatio-temporal perception-distortion trade-off. We also propose a new architecture for perceptual VSR (PSVR) to explicitly enforce naturalness of flow to achieve realistic spatio-temporal perception-distortion trade-off according to the proposed measures. Experimental results with PVSR support the hypothesis that a meaningful perception-distortion tradeoff for video should account for the naturalness of motion in addition to naturalness of texture.