Neural Implicit Dense Semantic SLAM

TL;DR

This paper introduces a neural implicit dense semantic SLAM system that combines classical tracking with neural field-based mapping to produce dense 3D geometry and semantic maps in real-time, even in challenging conditions.

Contribution

It presents a novel RGBD vSLAM algorithm integrating neural fields for dense, semantic, and scalable scene mapping in an online manner.

Findings

Robust tracking and mapping in noisy or sparse data

Effective semantic labeling of indoor scenes

Scalable to large environments using multiple local networks

Abstract

Visual Simultaneous Localization and Mapping (vSLAM) is a widely used technique in robotics and computer vision that enables a robot to create a map of an unfamiliar environment using a camera sensor while simultaneously tracking its position over time. In this paper, we propose a novel RGBD vSLAM algorithm that can learn a memory-efficient, dense 3D geometry, and semantic segmentation of an indoor scene in an online manner. Our pipeline combines classical 3D vision-based tracking and loop closing with neural fields-based mapping. The mapping network learns the SDF of the scene as well as RGB, depth, and semantic maps of any novel view using only a set of keyframes. Additionally, we extend our pipeline to large scenes by using multiple local mapping networks. Extensive experiments on well-known benchmark datasets confirm that our approach provides robust tracking, mapping, and semantic labeling even with noisy, sparse, or no input depth. Overall, our proposed algorithm can greatly enhance scene perception and assist with a range of robot control problems.