Real-World On-Vehicle Evaluation of Embedding-Based Anomaly Detection

TL;DR

This paper presents a real-time, adaptable anomaly detection method for autonomous vehicles using pretrained vision transformer embeddings and nearest-neighbor similarity, capable of localizing anomalies without explicit supervision.

Contribution

The authors introduce a simple, reference-based anomaly detection approach leveraging foundation models, suitable for real-world deployment without dataset-specific training.

Findings

Achieves good performance on the Road Anomaly benchmark.

Successfully localizes anomalies in diverse real-world scenes.

Demonstrates consistent qualitative behavior in practice.

Abstract

Detecting anomalies in traffic scenes is crucial for ensuring safety in autonomous driving, yet collecting representative anomalous data remains challenging. Existing anomaly detection methods are highly specialized and rely on normality as defined by the abstract semantic Cityscapes classes, making it difficult to adapt to diverse real-world scenarios. We propose an adaptable real-time anomaly detection method that leverages foundation models in the form of pretrained vision transformer embeddings to detect deviations via nearest-neighbor similarity in the latent semantic feature space. Based on patch-wise processing, the algorithm produces dense anomaly masks, allowing for the localization of detected anomalies. The method robustly models normality through a single reference image. This formulation avoids explicit supervision and dataset-specific training, making it suitable for real-world deployment. We evaluate the method on standard benchmarks and on an automated vehicle in real-world scenarios. Despite its simplicity, the method achieves good performance on the Road Anomaly benchmark and demonstrates consistent qualitative behavior in practice, successfully highlighting semantically unusual objects in diverse scenes. These results suggest that simple, reference-based methods can provide useful anomaly signals under realistic operating conditions.