MoniTor: Exploiting Large Language Models with Instruction for Online Video Anomaly Detection

TL;DR

MoniTor introduces a real-time, training-free online video anomaly detection method leveraging large vision-language models, a novel memory-based scoring queue, and LSTM-inspired prediction to effectively identify anomalies in surveillance videos.

Contribution

The paper presents MoniTor, a novel online VAD framework that uses pre-trained models, a memory-based scoring queue, and LSTM-inspired prediction for real-time anomaly detection without training.

Findings

Outperforms state-of-the-art methods on UCF-Crime and XD-Violence datasets.

Achieves competitive results with weakly supervised methods without training.

Effectively models temporal dependencies for anomaly detection.

Abstract

Video Anomaly Detection (VAD) aims to locate unusual activities or behaviors within videos. Recently, offline VAD has garnered substantial research attention, which has been invigorated by the progress in large language models (LLMs) and vision-language models (VLMs), offering the potential for a more nuanced understanding of anomalies. However, online VAD has seldom received attention due to real-time constraints and computational intensity. In this paper, we introduce a novel Memory-based online scoring queue scheme for Training-free VAD (MoniTor), to address the inherent complexities in online VAD. Specifically, MoniTor applies a streaming input to VLMs, leveraging the capabilities of pre-trained large-scale models. To capture temporal dependencies more effectively, we incorporate a novel prediction mechanism inspired by Long Short-Term Memory (LSTM) networks. This ensures the model can effectively model past states and leverage previous predictions to identify anomalous behaviors. Thereby, it better understands the current frame. Moreover, we design a scoring queue and an anomaly prior to dynamically store recent scores and cover all anomalies in the monitoring scenario, providing guidance for LLMs to distinguish between normal and abnormal behaviors over time. We evaluate MoniTor on two large datasets (i.e., UCF-Crime and XD-Violence) containing various surveillance and real-world scenarios. The results demonstrate that MoniTor outperforms state-of-the-art methods and is competitive with weakly supervised methods without training. Code is available at https://github.com/YsTvT/MoniTor.