Artifacts Mapping: Multi-Modal Semantic Mapping for Object Detection and 3D Localization

TL;DR

This paper introduces a multi-modal sensor fusion framework for semantic mapping that enables robots to accurately detect and localize objects in environments, improving over single-sensor methods by combining RGB, depth, and lidar data.

Contribution

The work presents a novel multi-modal sensor fusion framework for object detection and localization in semantic mapping, integrating RGB, depth, and lidar data for enhanced accuracy.

Findings

Achieves 98% object detection accuracy in real environments.

Outperforms single-sensor setups in obstacle detection.

Sensor fusion improves detection of near and far objects.

Abstract

Geometric navigation is nowadays a well-established field of robotics and the research focus is shifting towards higher-level scene understanding, such as Semantic Mapping. When a robot needs to interact with its environment, it must be able to comprehend the contextual information of its surroundings. This work focuses on classifying and localising objects within a map, which is under construction (SLAM) or already built. To further explore this direction, we propose a framework that can autonomously detect and localize predefined objects in a known environment using a multi-modal sensor fusion approach (combining RGB and depth data from an RGB-D camera and a lidar). The framework consists of three key elements: understanding the environment through RGB data, estimating depth through multi-modal sensor fusion, and managing artifacts (i.e., filtering and stabilizing measurements). The experiments show that the proposed framework can accurately detect 98% of the objects in the real sample environment, without post-processing, while 85% and 80% of the objects were mapped using the single RGBD camera or RGB + lidar setup respectively. The comparison with single-sensor (camera or lidar) experiments is performed to show that sensor fusion allows the robot to accurately detect near and far obstacles, which would have been noisy or imprecise in a purely visual or laser-based approach.