Learning to compose 6-DoF omnidirectional videos using multi-sphere images

TL;DR

This paper introduces a simplified method for generating 6-DoF omnidirectional videos using a 3D ConvNet and multi-sphere images, avoiding complex pre-processing steps like depth estimation.

Contribution

It presents a novel system that directly uses conventional VR footage to produce 6-DoF content without depth maps or segmentation, utilizing a new weighted sphere sweep volume fusion technique.

Findings

Compatible with most panoramic VR camera setups

Simplifies 6-DoF video generation process

Provides a high-quality artifact-free content generation approach

Abstract

Omnidirectional video is an essential component of Virtual Reality. Although various methods have been proposed to generate content that can be viewed with six degrees of freedom (6-DoF), existing systems usually involve complex depth estimation, image in-painting or stitching pre-processing. In this paper, we propose a system that uses a 3D ConvNet to generate a multi-sphere images (MSI) representation that can be experienced in 6-DoF VR. The system utilizes conventional omnidirectional VR camera footage directly without the need for a depth map or segmentation mask, thereby significantly simplifying the overall complexity of the 6-DoF omnidirectional video composition. By using a newly designed weighted sphere sweep volume (WSSV) fusing technique, our approach is compatible with most panoramic VR camera setups. A ground truth generation approach for high-quality artifact-free 6-DoF contents is proposed and can be used by the research and development community for 6-DoF content generation.