MTA: Multimodal Task Alignment for BEV Perception and Captioning

TL;DR

This paper introduces MTA, a framework that aligns BEV perception and captioning tasks to improve autonomous driving scene understanding without extra runtime costs.

Contribution

MTA proposes a novel multimodal alignment framework that enhances both BEV perception and captioning by integrating alignment mechanisms during training.

Findings

Significant performance improvements on nuScenes and TOD3Cap datasets.

10.7% better in rare perception scenarios.

9.2% improvement in captioning accuracy.

Abstract

Bird's eye view (BEV)-based 3D perception plays a crucial role in autonomous driving applications. The rise of large language models has spurred interest in BEV-based captioning to understand object behavior in the surrounding environment. However, existing approaches treat perception and captioning as separate tasks, focusing on the performance of only one task and overlooking the potential benefits of multimodal alignment. To bridge this gap between modalities, we introduce MTA, a novel multimodal task alignment framework that boosts both BEV perception and captioning. MTA consists of two key components: (1) BEV-Language Alignment (BLA), a contextual learning mechanism that aligns the BEV scene representations with ground-truth language representations, and (2) Detection-Captioning Alignment (DCA), a cross-modal prompting mechanism that aligns detection and captioning outputs. MTA seamlessly integrates into state-of-the-art baselines during training, adding no extra computational complexity at runtime. Extensive experiments on the nuScenes and TOD3Cap datasets show that MTA significantly outperforms state-of-the-art baselines in both tasks, achieving a 10.7% improvement in challenging rare perception scenarios and a 9.2% improvement in captioning. These results underscore the effectiveness of unified alignment in reconciling BEV-based perception and captioning.