Benchmarking Vision Foundation Models for Input Monitoring in Autonomous Driving

TL;DR

This paper evaluates the effectiveness of Vision Foundation Models combined with density estimation techniques for unsupervised out-of-distribution detection in autonomous driving, demonstrating superior performance over existing methods.

Contribution

It introduces a systematic benchmark of VFMs with density modeling for OOD detection in complex driving scenarios, highlighting their potential for safety monitoring.

Findings

VFM embeddings with density estimation outperform existing OOD detection methods.

The proposed framework effectively detects high-risk inputs that could cause errors.

Systematic evaluation across diverse conditions confirms the robustness of the approach.

Abstract

Deep neural networks (DNNs) remain challenged by distribution shifts in complex open-world domains like automated driving (AD): Robustness against yet unknown novel objects (semantic shift) or styles like lighting conditions (covariate shift) cannot be guaranteed. Hence, reliable operation-time monitors for identification of out-of-training-data-distribution (OOD) scenarios are imperative. Current approaches for OOD classification are untested for complex domains like AD, are limited in the kinds of shifts they detect, or even require supervision with OOD samples. To prepare for unanticipated shifts, we instead establish a framework around a principled, unsupervised and model-agnostic method that unifies detection of semantic and covariate shifts: Find a full model of the training data's feature distribution, to then use its density at new points as in-distribution (ID) score. To implement this, we propose to combine Vision Foundation Models (VFMs) as feature extractors with density modeling techniques. Through a comprehensive benchmark of 4 VFMs with different backbone architectures and 5 density-modeling techniques against established baselines, we provide the first systematic evaluation of OOD classification capabilities of VFMs across diverse conditions. A comparison with state-of-the-art binary OOD classification methods reveals that VFM embeddings with density estimation outperform existing approaches in identifying OOD inputs. Additionally, we show that our method detects high-risk inputs likely to cause errors in downstream tasks, thereby improving overall performance. Overall, VFMs, when coupled with robust density modeling techniques, are promising to realize model-agnostic, unsupervised, reliable safety monitors in complex vision tasks