Track Any Anomalous Object: A Granular Video Anomaly Detection Pipeline

TL;DR

This paper introduces TAO, a novel granular video anomaly detection framework that tracks multiple anomalous objects at the pixel level, enhancing localization accuracy and robustness in complex video sequences.

Contribution

The paper presents the first unified framework for fine-grained, pixel-level tracking of anomalous objects in videos, improving detection precision without threshold tuning.

Findings

Sets new benchmarks in accuracy and robustness.

Effectively localizes anomalies in complex videos.

Removes the need for threshold tuning.

Abstract

Video anomaly detection (VAD) is crucial in scenarios such as surveillance and autonomous driving, where timely detection of unexpected activities is essential. Although existing methods have primarily focused on detecting anomalous objects in videos -- either by identifying anomalous frames or objects -- they often neglect finer-grained analysis, such as anomalous pixels, which limits their ability to capture a broader range of anomalies. To address this challenge, we propose a new framework called Track Any Anomalous Object (TAO), which introduces a granular video anomaly detection pipeline that, for the first time, integrates the detection of multiple fine-grained anomalous objects into a unified framework. Unlike methods that assign anomaly scores to every pixel, our approach transforms the problem into pixel-level tracking of anomalous objects. By linking anomaly scores to downstream tasks such as segmentation and tracking, our method removes the need for threshold tuning and achieves more precise anomaly localization in long and complex video sequences. Experiments demonstrate that TAO sets new benchmarks in accuracy and robustness. Project page available online.