3D scene reconstruction from monocular spherical video with motion parallax

TL;DR

This paper presents a method to extract nearly complete 360-degree depth information from monocular spherical videos with motion parallax, enabling detailed 3D scene reconstruction from simple, widely available footage.

Contribution

The authors introduce a novel monocular spherical stereo technique that retrieves comprehensive depth data from standard 360 videos with motion parallax, without requiring specialized equipment.

Findings

Depth retrieval covers up to 97% of the sphere in solid angle.

Objects over 30 meters away can be estimated at 30 km/h.

Reconstructed 3D structures are clearly observable.

Abstract

In this paper, we describe a method to capture nearly entirely spherical (360 degree) depth information using two adjacent frames from a single spherical video with motion parallax. After illustrating a spherical depth information retrieval using two spherical cameras, we demonstrate monocular spherical stereo by using stabilized first-person video footage. Experiments demonstrated that the depth information was retrieved on up to 97% of the entire sphere in solid angle. At a speed of 30 km/h, we were able to estimate the depth of an object located over 30 m from the camera. We also reconstructed the 3D structures (point cloud) using the obtained depth data and confirmed the structures can be clearly observed. We can apply this method to 3D structure retrieval of surrounding environments such as 1) previsualization, location hunting/planning of a film, 2) real scene/computer graphics synthesis and 3) motion capture. Thanks to its simplicity, this method can be applied to various videos. As there is no pre-condition other than to be a 360 video with motion parallax, we can use any 360 videos including those on the Internet to reconstruct the surrounding environments. The cameras can be lightweight enough to be mounted on a drone. We also demonstrated such applications.