Omnidirectional Video Super-Resolution using Deep Learning

TL;DR

This paper introduces a new deep learning model, S3PO, for 360-degree video super-resolution, addressing distortion issues and demonstrating superior performance over existing methods with a novel dataset and specialized loss function.

Contribution

The paper presents a novel spherical super-resolution model, S3PO, and a new 360-degree video dataset, 360VDS, improving super-resolution quality for immersive VR videos.

Findings

S3PO outperforms state-of-the-art VSR models on 360-degree videos.

The novel loss function effectively addresses spherical distortion.

The study demonstrates the extensibility of conventional VSR models to 360-degree videos.

Abstract

Omnidirectional Videos (or 360{\deg} videos) are widely used in Virtual Reality (VR) to facilitate immersive and interactive viewing experiences. However, the limited spatial resolution in 360{\deg} videos does not allow for each degree of view to be represented with adequate pixels, limiting the visual quality offered in the immersive experience. Deep learning Video Super-Resolution (VSR) techniques used for conventional videos could provide a promising software-based solution; however, these techniques do not tackle the distortion present in equirectangular projections of 360{\deg} video signals. An additional obstacle is the limited availability of 360{\deg} video datasets for study. To address these issues, this paper creates a novel 360{\deg} Video Dataset (360VDS) with a study of the extensibility of conventional VSR models to 360{\deg} videos. This paper further proposes a novel deep learning model for 360{\deg} Video Super-Resolution (360{\deg} VSR), called Spherical Signal Super-resolution with a Proportioned Optimisation (S3PO). S3PO adopts recurrent modelling with an attention mechanism, unbound from conventional VSR techniques like alignment. With a purpose-built feature extractor and a novel loss function addressing spherical distortion, S3PO outperforms most state-of-the-art conventional VSR models and 360{\deg}~specific super-resolution models on 360{\deg} video datasets. A step-wise ablation study is presented to understand and demonstrate the impact of the chosen architectural sub-components, targeted training and optimisation.