Revisiting Birds Eye View Perception Models with Frozen Foundation Models: DINOv2 and Metric3Dv2

TL;DR

This paper demonstrates that integrating large foundation models like DINOv2 and Metric3Dv2 into birds eye view perception architectures significantly reduces training data needs and improves performance, including replacing traditional LiDAR with depth-based pseudo point clouds.

Contribution

It introduces a novel approach of using frozen foundation models for feature extraction and depth estimation in BEV perception, achieving higher accuracy with less data and replacing LiDAR with depth information.

Findings

Lift-Splat-Shoot with frozen DINOv2 and Metric3Dv2 exceeds baseline IoU by 7.4 using half the data.

Replacing LiDAR with depth-based PseudoLiDAR improves IoU by 3.

Foundation models can effectively enhance BEV perception with minimal training data.

Abstract

Birds Eye View perception models require extensive data to perform and generalize effectively. While traditional datasets often provide abundant driving scenes from diverse locations, this is not always the case. It is crucial to maximize the utility of the available training data. With the advent of large foundation models such as DINOv2 and Metric3Dv2, a pertinent question arises: can these models be integrated into existing model architectures to not only reduce the required training data but surpass the performance of current models? We choose two model architectures in the vehicle segmentation domain to alter: Lift-Splat-Shoot, and Simple-BEV. For Lift-Splat-Shoot, we explore the implementation of frozen DINOv2 for feature extraction and Metric3Dv2 for depth estimation, where we greatly exceed the baseline results by 7.4 IoU while utilizing only half the training data and iterations. Furthermore, we introduce an innovative application of Metric3Dv2's depth information as a PseudoLiDAR point cloud incorporated into the Simple-BEV architecture, replacing traditional LiDAR. This integration results in a +3 IoU improvement compared to the Camera-only model.