Multimodal Sensor-Based Semantic 3D Mapping for a Large-Scale Environment

TL;DR

This paper presents a multimodal sensor-based system combining 3D Lidar and camera data for large-scale semantic 3D mapping, improving accuracy and efficiency over camera-only methods in robotics applications.

Contribution

The study introduces a novel multimodal sensor fusion approach using GPS, IMU, Lidar, and CNN-based segmentation, with a 3D refinement process for enhanced semantic mapping.

Findings

Outperforms state-of-the-art in accuracy and IoU

Effective in large-scale challenging environments

Reduces computational complexity compared to camera-only methods

Abstract

Semantic 3D mapping is one of the most important fields in robotics, and has been used in many applications, such as robot navigation, surveillance, and virtual reality. In general, semantic 3D mapping is mainly composed of 3D reconstruction and semantic segmentation. As these technologies evolve, there has been great progress in semantic 3D mapping in recent years. Furthermore, the number of robotic applications requiring semantic information in 3D mapping to perform high-level tasks has increased, and many studies on semantic 3D mapping have been published. Existing methods use a camera for both 3D reconstruction and semantic segmentation. However, this is not suitable for large-scale environments and has the disadvantage of high computational complexity. To address this problem, we propose a multimodal sensor-based semantic 3D mapping system using a 3D Lidar combined with a camera. In this study, we build a 3D map by estimating odometry based on a global positioning system (GPS) and an inertial measurement unit (IMU), and use the latest 2D convolutional neural network (CNN) for semantic segmentation. To build a semantic 3D map, we integrate the 3D map with semantic information by using coordinate transformation and Bayes' update scheme. In order to improve the semantic 3D map, we propose a 3D refinement process to correct wrongly segmented voxels and remove traces of moving vehicles in the 3D map. Through experiments on challenging sequences, we demonstrate that our method outperforms state-of-the-art methods in terms of accuracy and intersection over union (IoU). Thus, our method can be used for various applications that require semantic information in 3D map.