VESPA: Towards un(Human)supervised Open-World Pointcloud Labeling for Autonomous Driving

TL;DR

VESPA introduces a multimodal autolabeling pipeline that combines LiDAR and camera data with vision-language models to enable open-vocabulary, high-quality 3D labeling for autonomous driving, reducing reliance on manual annotation.

Contribution

The paper presents VESPA, a novel autolabeling method that fuses LiDAR and camera data with vision-language models for open-vocabulary 3D object detection without ground-truth annotations.

Findings

Achieves 52.95% AP for object discovery on Nuscenes

Reaches up to 46.54% AP for multiclass detection

Supports discovery of novel categories in 3D scenes

Abstract

Data collection for autonomous driving is rapidly accelerating, but manual annotation, especially for 3D labels, remains a major bottleneck due to its high cost and labor intensity. Autolabeling has emerged as a scalable alternative, allowing the generation of labels for point clouds with minimal human intervention. While LiDAR-based autolabeling methods leverage geometric information, they struggle with inherent limitations of lidar data, such as sparsity, occlusions, and incomplete object observations. Furthermore, these methods typically operate in a class-agnostic manner, offering limited semantic granularity. To address these challenges, we introduce VESPA, a multimodal autolabeling pipeline that fuses the geometric precision of LiDAR with the semantic richness of camera images. Our approach leverages vision-language models (VLMs) to enable open-vocabulary object labeling and to refine detection quality directly in the point cloud domain. VESPA supports the discovery of novel categories and produces high-quality 3D pseudolabels without requiring ground-truth annotations or HD maps. On Nuscenes dataset, VESPA achieves an AP of 52.95% for object discovery and up to 46.54% for multiclass object detection, demonstrating strong performance in scalable 3D scene understanding. Code will be available upon acceptance.