Meaningful Maps With Object-Oriented Semantic Mapping

TL;DR

This paper presents a system that creates environmental maps combining geometric point clouds with object-level semantic information, enabling robots to understand scenes more meaningfully by integrating geometry and semantics.

Contribution

It introduces a novel approach that simultaneously builds geometric and semantic maps using RGB-D SLAM, deep learning object detection, and 3D segmentation.

Findings

Successfully maps unseen object instances with semantic labels

Integrates geometric and semantic data in a unified map

Enhances robot scene understanding capabilities

Abstract

For intelligent robots to interact in meaningful ways with their environment, they must understand both the geometric and semantic properties of the scene surrounding them. The majority of research to date has addressed these mapping challenges separately, focusing on either geometric or semantic mapping. In this paper we address the problem of building environmental maps that include both semantically meaningful, object-level entities and point- or mesh-based geometrical representations. We simultaneously build geometric point cloud models of previously unseen instances of known object classes and create a map that contains these object models as central entities. Our system leverages sparse, feature-based RGB-D SLAM, image-based deep-learning object detection and 3D unsupervised segmentation.