CueBench: Advancing Unified Understanding of Context-Aware Video Anomalies in Real-World

TL;DR

CueBench is a comprehensive benchmark for evaluating context-aware video anomaly understanding in real-world scenarios, highlighting current models' limitations and proposing a new fine-tuning approach that significantly improves performance.

Contribution

The paper introduces CueBench, a novel unified benchmark with a hierarchical taxonomy for context-aware video anomalies, and proposes Cue-R1, a reinforcement fine-tuning method that outperforms existing models.

Findings

Existing vision-language models perform poorly on real-world anomaly understanding.

Cue-R1 surpasses state-of-the-art methods by over 24% on CueBench.

CueBench provides a rigorous evaluation framework for diverse anomaly tasks.

Abstract

How far are deep models from real-world video anomaly understanding (VAU)? Current works typically emphasize on detecting unexpected occurrences deviated from normal patterns or comprehending anomalous events with interpretable descriptions. However, they exhibit only a superficial comprehension of real-world anomalies, with limited breadth in complex principles and subtle context that distinguish the anomalies from normalities, e.g., climbing cliffs with safety gear vs. without it. To this end, we introduce CueBench, the first of its kind Benchmark, devoted to Context-aware video anomalies within a Unified Evaluation framework. We comprehensively establish an event-centric hierarchical taxonomy that anchors two core event types: 14 conditional and 18 absolute anomaly events, defined by their refined semantics from diverse contexts across 174 scenes and 198 attributes. Based on this, we propose to unify and benchmark context-aware VAU with various challenging tasks across recognition, temporal grounding, detection, and anticipation. This also serves as a rigorous and fair probing evaluation suite for generative-discriminative as well as generalized-specialized vision-language models (VLMs). To address the challenges underlying CueBench, we further develop Cue-R1 based on R1-style reinforcement fine-tuning with verifiable, task-aligned, and hierarchy-refined rewards in a unified generative manner. Extensive results on CueBench reveal that, existing VLMs are still far from satisfactory real-world anomaly understanding, while our Cue-R1 surpasses these state-of-the-art approaches by over 24% on average.