PatchMatch-Stereo-Panorama, a fast dense reconstruction from 360{\deg} video images

TL;DR

This paper introduces a real-time dense 3D reconstruction method from 360-degree video for UAVs, extending PatchMatch-Stereo for equirectangular images, optimized for speed and accuracy on consumer hardware.

Contribution

It adapts PatchMatch-Stereo for real-time use with equirectangular cameras and introduces a parallel algorithm optimized for low latency and high accuracy.

Findings

Dense 3D reconstruction achievable on consumer laptops.

Improved accuracy and completeness over offline MVS methods.

Supports equirectangular camera models for 360-degree videos.

Abstract

This work proposes a new method for real-time dense 3d reconstruction for common 360{\deg} action cams, which can be mounted on small scouting UAVs during USAR missions. The proposed method extends a feature based Visual monocular SLAM (OpenVSLAM, based on the popular ORB-SLAM) for robust long-term localization on equirectangular video input by adding an additional densification thread that computes dense correspondences for any given keyframe with respect to a local keyframe-neighboorhood using a PatchMatch-Stereo-approach. While PatchMatch-Stereo-types of algorithms are considered state of the art for large scale Mutli-View-Stereo they had not been adapted so far for real-time dense 3d reconstruction tasks. This work describes a new massively parallel variant of the PatchMatch-Stereo-algorithm that differs from current approaches in two ways: First it supports the equirectangular camera model while other solutions are limited to the pinhole camera model. Second it is optimized for low latency while keeping a high level of completeness and accuracy. To achieve this it operates only on small sequences of keyframes, but employs techniques to compensate for the potential loss of accuracy due to the limited number of frames. Results demonstrate that dense 3d reconstruction is possible on a consumer grade laptop with a recent mobile GPU and that it is possible with improved accuracy and completeness over common offline-MVS solutions with comparable quality settings.