PointPainting: Sequential Fusion for 3D Object Detection

TL;DR

PointPainting introduces a sequential sensor fusion method that enhances 3D object detection by appending semantic class scores from images to lidar points, significantly improving performance on benchmark datasets.

Contribution

The paper proposes PointPainting, a novel fusion approach that effectively combines image semantics with lidar data, outperforming existing methods and setting new state-of-the-art results.

Findings

Large improvements on multiple 3D detection methods

State-of-the-art results on KITTI for bird's-eye view detection

Effective fusion depends on semantic segmentation quality

Abstract

Camera and lidar are important sensor modalities for robotics in general and self-driving cars in particular. The sensors provide complementary information offering an opportunity for tight sensor-fusion. Surprisingly, lidar-only methods outperform fusion methods on the main benchmark datasets, suggesting a gap in the literature. In this work, we propose PointPainting: a sequential fusion method to fill this gap. PointPainting works by projecting lidar points into the output of an image-only semantic segmentation network and appending the class scores to each point. The appended (painted) point cloud can then be fed to any lidar-only method. Experiments show large improvements on three different state-of-the art methods, Point-RCNN, VoxelNet and PointPillars on the KITTI and nuScenes datasets. The painted version of PointRCNN represents a new state of the art on the KITTI leaderboard for the bird's-eye view detection task. In ablation, we study how the effects of Painting depends on the quality and format of the semantic segmentation output, and demonstrate how latency can be minimized through pipelining.