Multi-Modality Task Cascade for 3D Object Detection

TL;DR

This paper introduces a multi-modality cascade network that integrates 2D and 3D data for improved 3D object detection, significantly outperforming previous methods on the SUN RGB-D dataset.

Contribution

The paper proposes a novel multi-modality task cascade network (MTC-RCNN) that effectively fuses 2D and 3D information for enhanced object detection performance.

Findings

Achieved +3.8 mAP@0.5 improvement over state-of-the-art.

Demonstrated that integrated 2D-3D training improves both tasks.

Introduced dual-head 2D segmentation scheme for robustness.

Abstract

Point clouds and RGB images are naturally complementary modalities for 3D visual understanding - the former provides sparse but accurate locations of points on objects, while the latter contains dense color and texture information. Despite this potential for close sensor fusion, many methods train two models in isolation and use simple feature concatenation to represent 3D sensor data. This separated training scheme results in potentially sub-optimal performance and prevents 3D tasks from being used to benefit 2D tasks that are often useful on their own. To provide a more integrated approach, we propose a novel Multi-Modality Task Cascade network (MTC-RCNN) that leverages 3D box proposals to improve 2D segmentation predictions, which are then used to further refine the 3D boxes. We show that including a 2D network between two stages of 3D modules significantly improves both 2D and 3D task performance. Moreover, to prevent the 3D module from over-relying on the overfitted 2D predictions, we propose a dual-head 2D segmentation training and inference scheme, allowing the 2nd 3D module to learn to interpret imperfect 2D segmentation predictions. Evaluating our model on the challenging SUN RGB-D dataset, we improve upon state-of-the-art results of both single modality and fusion networks by a large margin ($\textbf{+3.8}$ mAP@0.5). Code will be released $\href{https://github.com/Divadi/MTC_RCNN}{\text{here.}}$