FULLER: Unified Multi-modality Multi-task 3D Perception via Multi-level Gradient Calibration

TL;DR

This paper introduces FULLER, a multi-level gradient calibration framework that improves multi-modality and multi-task 3D perception in autonomous driving by balancing task and modality contributions during training.

Contribution

The paper proposes a novel gradient calibration method that addresses modality bias and task conflict, enhancing multi-modality fusion and multi-task learning in 3D perception.

Findings

14.4% mIoU improvement on map segmentation

1.4% mAP improvement on 3D detection

Effective in large-scale benchmark nuScenes

Abstract

Multi-modality fusion and multi-task learning are becoming trendy in 3D autonomous driving scenario, considering robust prediction and computation budget. However, naively extending the existing framework to the domain of multi-modality multi-task learning remains ineffective and even poisonous due to the notorious modality bias and task conflict. Previous works manually coordinate the learning framework with empirical knowledge, which may lead to sub-optima. To mitigate the issue, we propose a novel yet simple multi-level gradient calibration learning framework across tasks and modalities during optimization. Specifically, the gradients, produced by the task heads and used to update the shared backbone, will be calibrated at the backbone's last layer to alleviate the task conflict. Before the calibrated gradients are further propagated to the modality branches of the backbone, their magnitudes will be calibrated again to the same level, ensuring the downstream tasks pay balanced attention to different modalities. Experiments on large-scale benchmark nuScenes demonstrate the effectiveness of the proposed method, eg, an absolute 14.4% mIoU improvement on map segmentation and 1.4% mAP improvement on 3D detection, advancing the application of 3D autonomous driving in the domain of multi-modality fusion and multi-task learning. We also discuss the links between modalities and tasks.