360VFI: A Dataset and Benchmark for Omnidirectional Video Frame Interpolation

TL;DR

This paper introduces 360VFI, a new dataset and benchmark for omnidirectional video frame interpolation, addressing the unique distortions in 360-degree videos with a specialized neural network approach.

Contribution

The paper presents the first dataset and benchmark for omnidirectional video frame interpolation, along with a novel distortion-aware neural network architecture tailored for ERP format videos.

Findings

Effective interpolation performance on 360VFI dataset

Modeling omnidirectional distortion improves interpolation quality

Benchmark reveals challenges under various distortion conditions

Abstract

Head-mounted 360{\deg} displays and portable 360{\deg} cameras have significantly progressed, providing viewers a realistic and immersive experience. However, many omnidirectional videos have low frame rates that can lead to visual fatigue, and the prevailing plane frame interpolation methodologies are unsuitable for omnidirectional video interpolation because they are designed solely for traditional videos. This paper introduces the benchmark dataset, 360VFI, for Omnidirectional Video Frame Interpolation. We present a practical implementation that introduces a distortion prior from omnidirectional video into the network to modulate distortions. Specifically, we propose a pyramid distortion-sensitive feature extractor that uses the unique characteristics of equirectangular projection (ERP) format as prior information. Moreover, we devise a decoder that uses an affine transformation to further facilitate the synthesis of intermediate frames. 360VFI is the first dataset and benchmark that explores the challenge of Omnidirectional Video Frame Interpolation. Through our benchmark analysis, we present four different distortion condition scenes in the proposed 360VFI dataset to evaluate the challenges triggered by distortion during interpolation. Besides, experimental results demonstrate that Omnidirectional Video Interpolation can be effectively improved by modeling for omnidirectional distortion.