FOMO-3D: Using Vision Foundation Models for Long-Tailed 3D Object Detection

TL;DR

FOMO-3D introduces a multi-modal 3D detection approach that leverages vision foundation models to improve recognition of rare objects in complex traffic environments for autonomous vehicles.

Contribution

It is the first to utilize vision foundation models as priors in a multi-modal 3D detection framework for long-tailed object recognition.

Findings

Significant improvements in detecting rare traffic objects.

Effective multi-modal fusion of LiDAR and vision priors.

Enhanced generalization in real-world driving scenarios.

Abstract

In order to navigate complex traffic environments, self-driving vehicles must recognize many semantic classes pertaining to vulnerable road users or traffic control devices. However, many safety-critical objects (e.g., construction worker) appear infrequently in nominal traffic conditions, leading to a severe shortage of training examples from driving data alone. Recent vision foundation models, which are trained on a large corpus of data, can serve as a good source of external prior knowledge to improve generalization. We propose FOMO-3D, the first multi-modal 3D detector to leverage vision foundation models for long-tailed 3D detection. Specifically, FOMO-3D exploits rich semantic and depth priors from OWLv2 and Metric3Dv2 within a two-stage detection paradigm that first generates proposals with a LiDAR-based branch and a novel camera-based branch, and refines them with attention especially to image features from OWL. Evaluations on real-world driving data show that using rich priors from vision foundation models with careful multi-modal fusion designs leads to large gains for long-tailed 3D detection. Project website is at https://waabi.ai/fomo3d/.