Volume-based Semantic Labeling with Signed Distance Functions

TL;DR

This paper introduces a novel method for integrating semantic labels into dense 3D reconstructions generated by volume-based SLAM systems, enabling semantically labeled environment maps from RGB-D data.

Contribution

It presents the first approach to embed semantic information into dense maps from volume-based SLAM, combining semantic segmentation with 3D reconstruction.

Findings

Successfully labeled dense reconstructions with semantic information.

Validated with ground truth, noisy, and CNN-based labels.

Demonstrated effectiveness on publicly available RGB-D datasets.

Abstract

Research works on the two topics of Semantic Segmentation and SLAM (Simultaneous Localization and Mapping) have been following separate tracks. Here, we link them quite tightly by delineating a category label fusion technique that allows for embedding semantic information into the dense map created by a volume-based SLAM algorithm such as KinectFusion. Accordingly, our approach is the first to provide a semantically labeled dense reconstruction of the environment from a stream of RGB-D images. We validate our proposal using a publicly available semantically annotated RGB-D dataset and a) employing ground truth labels, b) corrupting such annotations with synthetic noise, c) deploying a state of the art semantic segmentation algorithm based on Convolutional Neural Networks.