Learning Spatio-Temporal Downsampling for Effective Video Upscaling

TL;DR

This paper introduces a neural network framework that jointly learns spatio-temporal downsampling and upsampling to improve video reconstruction quality, effectively addressing aliasing issues and enabling applications like video resampling and storage.

Contribution

It proposes a novel joint learning approach for spatio-temporal downsampling and upsampling, with modules for explicit temporal propagation and space-time feature rearrangement.

Findings

Significantly improves space-time video reconstruction quality.

Preserves spatial textures and motion patterns during downsampling and upscaling.

Enables applications such as arbitrary resampling and efficient storage.

Abstract

Downsampling is one of the most basic image processing operations. Improper spatio-temporal downsampling applied on videos can cause aliasing issues such as moir\'e patterns in space and the wagon-wheel effect in time. Consequently, the inverse task of upscaling a low-resolution, low frame-rate video in space and time becomes a challenging ill-posed problem due to information loss and aliasing artifacts. In this paper, we aim to solve the space-time aliasing problem by learning a spatio-temporal downsampler. Towards this goal, we propose a neural network framework that jointly learns spatio-temporal downsampling and upsampling. It enables the downsampler to retain the key patterns of the original video and maximizes the reconstruction performance of the upsampler. To make the downsamping results compatible with popular image and video storage formats, the downsampling results are encoded to uint8 with a differentiable quantization layer. To fully utilize the space-time correspondences, we propose two novel modules for explicit temporal propagation and space-time feature rearrangement. Experimental results show that our proposed method significantly boosts the space-time reconstruction quality by preserving spatial textures and motion patterns in both downsampling and upscaling. Moreover, our framework enables a variety of applications, including arbitrary video resampling, blurry frame reconstruction, and efficient video storage.