DeepFusion: Real-Time Dense 3D Reconstruction for Monocular SLAM using Single-View Depth and Gradient Predictions

TL;DR

DeepFusion is a real-time dense 3D reconstruction system for monocular SLAM that combines CNN-based depth predictions with multiview stereo, enabling accurate, scale-aware reconstructions on a GPU.

Contribution

The paper introduces DeepFusion, a novel system that fuses CNN-predicted depth maps with multiview stereo for real-time dense 3D reconstruction in monocular SLAM.

Findings

Performs at least as well as comparable systems on synthetic and real datasets.

Produces metric-scale dense reconstructions in real-time on GPU.

Effectively fuses learned depth uncertainties with geometric constraints.

Abstract

While the keypoint-based maps created by sparse monocular simultaneous localisation and mapping (SLAM) systems are useful for camera tracking, dense 3D reconstructions may be desired for many robotic tasks. Solutions involving depth cameras are limited in range and to indoor spaces, and dense reconstruction systems based on minimising the photometric error between frames are typically poorly constrained and suffer from scale ambiguity. To address these issues, we propose a 3D reconstruction system that leverages the output of a convolutional neural network (CNN) to produce fully dense depth maps for keyframes that include metric scale. Our system, DeepFusion, is capable of producing real-time dense reconstructions on a GPU. It fuses the output of a semi-dense multiview stereo algorithm with the depth and gradient predictions of a CNN in a probabilistic fashion, using learned uncertainties produced by the network. While the network only needs to be run once per keyframe, we are able to optimise for the depth map with each new frame so as to constantly make use of new geometric constraints. Based on its performance on synthetic and real-world datasets, we demonstrate that DeepFusion is capable of performing at least as well as other comparable systems.