# Volumetric Instance-Aware Semantic Mapping and 3D Object Discovery

## TL;DR

This paper introduces a method for real-time, volumetric, instance-aware semantic mapping that detects, tracks, and models 3D objects, including unseen categories, during robotic exploration.

## Contribution

It presents a novel incremental approach combining geometric and semantic segmentation for building detailed object-centric maps in real-time.

## Key findings

- Competitive with state-of-the-art methods on public datasets
- Capable of discovering objects from unseen categories
- Effective in real-world robotic mapping scenarios

## Abstract

To autonomously navigate and plan interactions in real-world environments, robots require the ability to robustly perceive and map complex, unstructured surrounding scenes. Besides building an internal representation of the observed scene geometry, the key insight toward a truly functional understanding of the environment is the usage of higher-level entities during mapping, such as individual object instances. We propose an approach to incrementally build volumetric object-centric maps during online scanning with a localized RGB-D camera. First, a per-frame segmentation scheme combines an unsupervised geometric approach with instance-aware semantic object predictions. This allows us to detect and segment elements both from the set of known classes and from other, previously unseen categories. Next, a data association step tracks the predicted instances across the different frames. Finally, a map integration strategy fuses information about their 3D shape, location, and, if available, semantic class into a global volume. Evaluation on a publicly available dataset shows that the proposed approach for building instance-level semantic maps is competitive with state-of-the-art methods, while additionally able to discover objects of unseen categories. The system is further evaluated within a real-world robotic mapping setup, for which qualitative results highlight the online nature of the method.