Spatio-Temporal Distortion Aware Omnidirectional Video Super-Resolution

TL;DR

This paper introduces STDAN, a novel spatio-temporal distortion-aware network designed to enhance the resolution and visual quality of omnidirectional videos by addressing their unique geometric and temporal challenges.

Contribution

The paper presents a new network architecture with joint spatio-temporal alignment and reconstruction, specifically tailored for omnidirectional videos, improving over existing super-resolution methods.

Findings

Outperforms state-of-the-art methods in visual fidelity.

Enhances temporal consistency and reduces geometric artifacts.

Operates efficiently with affordable computational costs.

Abstract

Omnidirectional videos (ODVs) provide an immersive visual experience by capturing the 360{\deg} scene. With the rapid advancements in virtual/augmented reality, metaverse, and generative artificial intelligence, the demand for high-quality ODVs is surging. However, ODVs often suffer from low resolution due to their wide field of view and limitations in capturing devices and transmission bandwidth. Although video super-resolution (SR) is a capable video quality enhancement technique, the performance ceiling and practical generalization of existing methods are limited when applied to ODVs due to their unique attributes. To alleviate spatial projection distortions and temporal flickering of ODVs, we propose a Spatio-Temporal Distortion Aware Network (STDAN) with joint spatio-temporal alignment and reconstruction. Specifically, we incorporate a spatio-temporal continuous alignment (STCA) to mitigate discrete geometric artifacts in parallel with temporal alignment. Subsequently, we introduce an interlaced multi-frame reconstruction (IMFR) to enhance temporal consistency. Furthermore, we employ latitude-saliency adaptive (LSA) weights to focus on regions with higher texture complexity and human-watching interest. By exploring a spatio-temporal jointly framework and real-world viewing strategies, STDAN effectively reinforces spatio-temporal coherence on a novel ODV-SR dataset and ensures affordable computational costs. Extensive experimental results demonstrate that STDAN outperforms state-of-the-art methods in improving visual fidelity and dynamic smoothness of ODVs.