Harnessing Large Language Models for Training-free Video Anomaly Detection

TL;DR

This paper introduces LAVAD, a training-free video anomaly detection method that uses pre-trained language and vision-language models to generate descriptions, score anomalies, and outperform traditional training-based approaches on surveillance datasets.

Contribution

LAVAD is the first training-free VAD approach leveraging pre-trained models for description and anomaly scoring, eliminating the need for domain-specific training.

Findings

Outperforms unsupervised and one-class methods on UCF-Crime and XD-Violence datasets.

Does not require any training or data collection for deployment.

Effective use of cross-modal similarity for caption cleaning and score refinement.

Abstract

Video anomaly detection (VAD) aims to temporally locate abnormal events in a video. Existing works mostly rely on training deep models to learn the distribution of normality with either video-level supervision, one-class supervision, or in an unsupervised setting. Training-based methods are prone to be domain-specific, thus being costly for practical deployment as any domain change will involve data collection and model training. In this paper, we radically depart from previous efforts and propose LAnguage-based VAD (LAVAD), a method tackling VAD in a novel, training-free paradigm, exploiting the capabilities of pre-trained large language models (LLMs) and existing vision-language models (VLMs). We leverage VLM-based captioning models to generate textual descriptions for each frame of any test video. With the textual scene description, we then devise a prompting mechanism to unlock the capability of LLMs in terms of temporal aggregation and anomaly score estimation, turning LLMs into an effective video anomaly detector. We further leverage modality-aligned VLMs and propose effective techniques based on cross-modal similarity for cleaning noisy captions and refining the LLM-based anomaly scores. We evaluate LAVAD on two large datasets featuring real-world surveillance scenarios (UCF-Crime and XD-Violence), showing that it outperforms both unsupervised and one-class methods without requiring any training or data collection.