DA-RNN: Semantic Mapping with Data Associated Recurrent Neural Networks

TL;DR

This paper introduces DA-RNNs, a novel recurrent neural network framework that jointly performs 3D scene mapping and semantic labeling by integrating semantic information into 3D reconstructions from RGB-D videos.

Contribution

The work presents a new recurrent neural network architecture for semantic labeling in RGB-D videos and integrates it with mapping techniques like KinectFusion for semantic 3D scene mapping.

Findings

Effective semantic 3D scene mapping demonstrated on real and synthetic datasets.

DA-RNNs successfully integrate semantic labels into 3D reconstructions.

Improved understanding of 3D scenes for robotic interaction.

Abstract

3D scene understanding is important for robots to interact with the 3D world in a meaningful way. Most previous works on 3D scene understanding focus on recognizing geometrical or semantic properties of the scene independently. In this work, we introduce Data Associated Recurrent Neural Networks (DA-RNNs), a novel framework for joint 3D scene mapping and semantic labeling. DA-RNNs use a new recurrent neural network architecture for semantic labeling on RGB-D videos. The output of the network is integrated with mapping techniques such as KinectFusion in order to inject semantic information into the reconstructed 3D scene. Experiments conducted on a real world dataset and a synthetic dataset with RGB-D videos demonstrate the ability of our method in semantic 3D scene mapping.