Improving 3D Labeling in Self-Driving by Inferring Vehicle Information using Vision Language Models

TL;DR

This paper introduces a method that uses vision language models to improve 3D vehicle labeling in self-driving, reducing manual effort and increasing accuracy through zero-shot vehicle recognition and dimension inference.

Contribution

The authors propose a novel approach leveraging VLMs for zero-shot vehicle make, model, and dimension inference to enhance 3D labeling accuracy in autonomous driving.

Findings

VLMs improve 3D bounding box accuracy over baselines.

The approach mitigates occlusion-related labeling failures.

Integrating VLMs reduces manual labeling time while increasing quality.

Abstract

We present an approach to improve 3D vehicle labeling in self-driving applications through zero-shot inference of vehicle information, leveraging Vehicle Make and Model Recognition (VMMR) methods. The proposed approach utilizes a Vision Language Model (VLM) to both infer a vehicle's make, model, and generation from image crops, and output accurate 3D bounding box dimensions to seed manual labeling. We evaluate the impact of iterative prompt engineering and the choice of different VLMs on both vehicle bounding box inference and make/model/generation recognition. When compared to strong baselines, the proposed approach not only shows high accuracy, but also excels in mitigating specific failure modes where VLMs provide better dimensions than initial lidar-aided human annotated labels (e.g., in cases of significant vehicle occlusion). Experiments on both public and proprietary data strongly suggest that our conclusions are generalizable across different labelers and datasets. The results demonstrate that integrating VLMs into the labeling process can reduce manual labeling time while increasing label quality.