Can VLMs Unlock Semantic Anomaly Detection? A Framework for Structured Reasoning

TL;DR

This paper introduces SAVANT, a model-agnostic framework that enhances VLMs for semantic anomaly detection in autonomous driving by structured reasoning, significantly improving detection recall and enabling scalable data annotation.

Contribution

The paper presents SAVANT, a novel layered reasoning framework that improves VLM-based anomaly detection and facilitates large-scale data annotation for autonomous driving scenarios.

Findings

SAVANT improves VLM recall by approximately 18.5% over prompting baselines.

The framework enables automatic labeling of around 10,000 images with high confidence.

Fine-tuned models achieve 90.8% recall and 93.8% accuracy in anomaly detection.

Abstract

Autonomous driving systems remain critically vulnerable to the long-tail of rare, out-of-distribution semantic anomalies. While VLMs have emerged as promising tools for perception, their application in anomaly detection remains largely restricted to prompting proprietary models - limiting reliability, reproducibility, and deployment feasibility. To address this gap, we introduce SAVANT (Semantic Anomaly Verification/Analysis Toolkit), a novel model-agnostic reasoning framework that reformulates anomaly detection as a layered semantic consistency verification. By applying SAVANT's two-phase pipeline - structured scene description extraction and multi-modal evaluation - existing VLMs improve their scores in detecting anomalous driving scenarios from input images. Our approach replaces ad hoc prompting with semantic-aware reasoning, transforming VLM-based detection into a principled decomposition across four semantic domains. We show that across a balanced set of real-world driving scenarios, applying SAVANT improves VLM's absolute recall by approximately 18.5% compared to prompting baselines. Moreover, this gain enables reliable large-scale annotation: leveraging the best proprietary model within our framework, we automatically labeled around 10,000 real-world images with high confidence. We use the resulting high-quality dataset to fine-tune a 7B open-source model (Qwen2.5-VL) to perform single-shot anomaly detection, achieving 90.8% recall and 93.8% accuracy - surpassing all models evaluated while enabling local deployment at near-zero cost. By coupling structured semantic reasoning with scalable data curation, we provide a practical solution to data scarcity in semantic anomaly detection for autonomous systems. Supplementary material: https://TUM-AVS.github.io/SAVANT/.