CodeMapping: Real-Time Dense Mapping for Sparse SLAM using Compact Scene Representations

TL;DR

This paper introduces CodeMapping, a real-time dense mapping framework that enhances sparse SLAM systems by predicting dense depth images using a variational autoencoder, enabling improved scene understanding and reconstruction.

Contribution

It presents a novel dense mapping method that integrates with sparse SLAM using a VAE conditioned on sparse data, allowing real-time dense depth prediction without disrupting SLAM performance.

Findings

Achieves accurate dense depth estimation with ORB-SLAM3 integration.

Enables globally consistent 3D reconstruction via TSDF fusion.

Runs in parallel to SLAM for real-time performance.

Abstract

We propose a novel dense mapping framework for sparse visual SLAM systems which leverages a compact scene representation. State-of-the-art sparse visual SLAM systems provide accurate and reliable estimates of the camera trajectory and locations of landmarks. While these sparse maps are useful for localization, they cannot be used for other tasks such as obstacle avoidance or scene understanding. In this paper we propose a dense mapping framework to complement sparse visual SLAM systems which takes as input the camera poses, keyframes and sparse points produced by the SLAM system and predicts a dense depth image for every keyframe. We build on CodeSLAM and use a variational autoencoder (VAE) which is conditioned on intensity, sparse depth and reprojection error images from sparse SLAM to predict an uncertainty-aware dense depth map. The use of a VAE then enables us to refine the dense depth images through multi-view optimization which improves the consistency of overlapping frames. Our mapper runs in a separate thread in parallel to the SLAM system in a loosely coupled manner. This flexible design allows for integration with arbitrary metric sparse SLAM systems without delaying the main SLAM process. Our dense mapper can be used not only for local mapping but also globally consistent dense 3D reconstruction through TSDF fusion. We demonstrate our system running with ORB-SLAM3 and show accurate dense depth estimation which could enable applications such as robotics and augmented reality.