LargeAD: Large-Scale Cross-Sensor Data Pretraining for Autonomous Driving

TL;DR

LargeAD introduces a scalable framework that leverages vision foundation models for cross-modal 3D pretraining, significantly improving autonomous driving perception tasks through semantic alignment of 2D and 3D data.

Contribution

The paper presents a novel large-scale 3D pretraining method using VFMs for cross-modal alignment, superpixel generation, and temporal consistency in autonomous driving datasets.

Findings

Substantial improvements in LiDAR segmentation and detection accuracy.

Effective cross-modal feature alignment across diverse datasets.

Robustness and generalization demonstrated on 11 large-scale datasets.

Abstract

Recent advancements in vision foundation models (VFMs) have revolutionized visual perception in 2D, yet their potential for 3D scene understanding, particularly in autonomous driving applications, remains underexplored. In this paper, we introduce LargeAD, a versatile and scalable framework designed for large-scale 3D pretraining across diverse real-world driving datasets. Our framework leverages VFMs to extract semantically rich superpixels from 2D images, which are aligned with LiDAR point clouds to generate high-quality contrastive samples. This alignment facilitates cross-modal representation learning, enhancing the semantic consistency between 2D and 3D data. We introduce several key innovations: (i) VFM-driven superpixel generation for detailed semantic representation, (ii) a VFM-assisted contrastive learning strategy to align multimodal features, (iii) superpoint temporal consistency to maintain stable representations across time, and (iv) multi-source data pretraining to generalize across various LiDAR configurations. Our approach achieves substantial gains over state-of-the-art methods in linear probing and fine-tuning for LiDAR-based segmentation and object detection. Extensive experiments on 11 large-scale multi-sensor datasets highlight our superior performance, demonstrating adaptability, efficiency, and robustness in real-world autonomous driving scenarios.