EPNet: Enhancing Point Features with Image Semantics for 3D Object Detection

TL;DR

EPNet is a novel framework that fuses LiDAR point clouds with image semantics to improve 3D object detection accuracy, addressing sensor fusion and confidence inconsistency issues.

Contribution

It introduces a fusion module for point-wise semantic enhancement and a confidence consistency loss, advancing 3D detection methods without requiring image annotations.

Findings

EPNet outperforms state-of-the-art methods on KITTI and SUN-RGBD datasets.

The fusion module effectively integrates image semantics into point features.

The confidence consistency loss improves localization and classification reliability.

Abstract

In this paper, we aim at addressing two critical issues in the 3D detection task, including the exploitation of multiple sensors~(namely LiDAR point cloud and camera image), as well as the inconsistency between the localization and classification confidence. To this end, we propose a novel fusion module to enhance the point features with semantic image features in a point-wise manner without any image annotations. Besides, a consistency enforcing loss is employed to explicitly encourage the consistency of both the localization and classification confidence. We design an end-to-end learnable framework named EPNet to integrate these two components. Extensive experiments on the KITTI and SUN-RGBD datasets demonstrate the superiority of EPNet over the state-of-the-art methods. Codes and models are available at: \url{https://github.com/happinesslz/EPNet}.