M3DeTR: Multi-representation, Multi-scale, Mutual-relation 3D Object Detection with Transformers

TL;DR

M3DeTR introduces a transformer-based architecture that unifies multiple point cloud representations and scales for improved 3D object detection, achieving state-of-the-art results on KITTI and Waymo datasets.

Contribution

It is the first method to combine multiple point cloud representations, feature scales, and mutual relationships simultaneously using transformers for 3D detection.

Findings

Achieves state-of-the-art performance on KITTI and Waymo datasets.

Significantly improves baseline by 1.48% mAP on Waymo.

Ranks 1st on KITTI 3D Detection Benchmark for cars and cyclists.

Abstract

We present a novel architecture for 3D object detection, M3DeTR, which combines different point cloud representations (raw, voxels, bird-eye view) with different feature scales based on multi-scale feature pyramids. M3DeTR is the first approach that unifies multiple point cloud representations, feature scales, as well as models mutual relationships between point clouds simultaneously using transformers. We perform extensive ablation experiments that highlight the benefits of fusing representation and scale, and modeling the relationships. Our method achieves state-of-the-art performance on the KITTI 3D object detection dataset and Waymo Open Dataset. Results show that M3DeTR improves the baseline significantly by 1.48% mAP for all classes on Waymo Open Dataset. In particular, our approach ranks 1st on the well-known KITTI 3D Detection Benchmark for both car and cyclist classes, and ranks 1st on Waymo Open Dataset with single frame point cloud input. Our code is available at: https://github.com/rayguan97/M3DETR.