PTA-Det: Point Transformer Associating Point cloud and Image for 3D Object Detection

TL;DR

PTA-Det introduces a novel multi-modal 3D object detection method that effectively fuses LiDAR and image data using pseudo points and transformer-based feature fusion, improving detection performance in autonomous driving.

Contribution

The paper presents a new fusion framework with a pseudo point cloud generator and transformer-based feature fusion module for enhanced multi-modal 3D detection.

Findings

Achieves competitive results on KITTI dataset

Effectively fuses LiDAR and image features

Outperforms some existing multi-modal methods

Abstract

In autonomous driving, 3D object detection based on multi-modal data has become an indispensable approach when facing complex environments around the vehicle. During multi-modal detection, LiDAR and camera are simultaneously applied for capturing and modeling. However, due to the intrinsic discrepancies between the LiDAR point and camera image, the fusion of the data for object detection encounters a series of problems. Most multi-modal detection methods perform even worse than LiDAR-only methods. In this investigation, we propose a method named PTA-Det to improve the performance of multi-modal detection. Accompanied by PTA-Det, a Pseudo Point Cloud Generation Network is proposed, which can convert image information including texture and semantic features by pseudo points. Thereafter, through a transformer-based Point Fusion Transition (PFT) module, the features of LiDAR points and pseudo points from image can be deeply fused under a unified point-based representation. The combination of these modules can conquer the major obstacle in feature fusion across modalities and realizes a complementary and discriminative representation for proposal generation. Extensive experiments on the KITTI dataset show the PTA-Det achieves a competitive result and support its effectiveness.