FuseSeg: LiDAR Point Cloud Segmentation Fusing Multi-Modal Data

TL;DR

FuseSeg introduces a fusion approach combining LiDAR and RGB data for improved point cloud segmentation, achieving up to 18% IoU gain and real-time processing at 50 fps.

Contribution

It presents a novel fusion method that integrates multi-modal data into a single network for enhanced segmentation accuracy.

Findings

Up to 18% IoU improvement on KITTI benchmark

Real-time segmentation at 50 fps

Effective fusion of LiDAR and RGB data

Abstract

We introduce a simple yet effective fusion method of LiDAR and RGB data to segment LiDAR point clouds. Utilizing the dense native range representation of a LiDAR sensor and the setup calibration, we establish point correspondences between the two input modalities. Subsequently, we are able to warp and fuse the features from one domain into the other. Therefore, we can jointly exploit information from both data sources within one single network. To show the merit of our method, we extend SqueezeSeg, a point cloud segmentation network, with an RGB feature branch and fuse it into the original structure. Our extension called FuseSeg leads to an improvement of up to 18% IoU on the KITTI benchmark. In addition to the improved accuracy, we also achieve real-time performance at 50 fps, five times as fast as the KITTI LiDAR data recording speed.