Toward Unified Multimodal Representation Learning for Autonomous Driving

TL;DR

This paper introduces a unified multimodal pre-training framework for autonomous driving that aligns visual, textual, and 3D data in a shared space, improving scene understanding beyond pairwise methods.

Contribution

It proposes a Contrastive Tensor Pre-training (CTP) method that jointly aligns multiple modalities using a tensor loss, extending beyond traditional pairwise similarity approaches.

Findings

Achieves better multimodal alignment in autonomous driving scenarios.

Effective for both pretrained and from-scratch encoder training.

Constructed a new triplet dataset for validation.

Abstract

Contrastive Language-Image Pre-training (CLIP) has shown impressive performance in aligning visual and textual representations. Recent studies have extended this paradigm to 3D vision to improve scene understanding for autonomous driving. A common strategy is to employ pairwise cosine similarity between modalities to guide the training of a 3D encoder. However, considering the similarity between individual modality pairs rather than all modalities jointly fails to ensure consistent and unified alignment across the entire multimodal space. In this paper, we propose a Contrastive Tensor Pre-training (CTP) framework that simultaneously aligns multiple modalities in a unified embedding space to enhance end-to-end autonomous driving. Compared with pairwise cosine similarity alignment, our method extends the 2D similarity matrix into a multimodal similarity tensor. Furthermore, we introduce a tensor loss to enable joint contrastive learning across all modalities. For experimental validation of our framework, we construct a text-image-point cloud triplet dataset derived from existing autonomous driving datasets. The results show that our proposed unified multimodal alignment framework achieves favorable performance for both scenarios: (i) aligning a 3D encoder with pretrained CLIP encoders, and (ii) pretraining all encoders from scratch.