Vision Foundation Model Embedding-Based Semantic Anomaly Detection

TL;DR

This paper presents a novel approach for detecting semantic anomalies in autonomous systems by leveraging vision foundation model embeddings, comparing runtime images to nominal scenarios, and using filtering to reduce false positives.

Contribution

It introduces a framework utilizing vision embeddings for semantic anomaly detection, including raw and object-centric variants, with a filtering mechanism for improved robustness.

Findings

Instance-based method with filtering matches GPT-4o performance

Framework provides precise anomaly localization

Effective in simulated autonomous driving scenarios

Abstract

Semantic anomalies are contextually invalid or unusual combinations of familiar visual elements that can cause undefined behavior and failures in system-level reasoning for autonomous systems. This work explores semantic anomaly detection by leveraging the semantic priors of state-of-the-art vision foundation models, operating directly on the image. We propose a framework that compares local vision embeddings from runtime images to a database of nominal scenarios in which the autonomous system is deemed safe and performant. In this work, we consider two variants of the proposed framework: one using raw grid-based embeddings, and another leveraging instance segmentation for object-centric representations. To further improve robustness, we introduce a simple filtering mechanism to suppress false positives. Our evaluations on CARLA-simulated anomalies show that the instance-based method with filtering achieves performance comparable to GPT-4o, while providing precise anomaly localization. These results highlight the potential utility of vision embeddings from foundation models for real-time anomaly detection in autonomous systems.