UAL-Bench: The First Comprehensive Unusual Activity Localization Benchmark

TL;DR

This paper introduces UAL-Bench, a comprehensive benchmark for localizing unusual activities in videos, evaluating foundation models' capabilities and proposing new metrics and methods to improve localization accuracy.

Contribution

It presents the first extensive benchmark for unusual activity localization, evaluates multiple models including a novel VLM-LLM approach, and introduces a new evaluation metric R@1, TD <= p.

Findings

VLM-LLM outperforms Vid-LLMs in localizing short-span events.

The new metric R@1, TD <= p better captures localization accuracy.

Challenges remain in long-duration video localization, especially in autism diagnosis.

Abstract

Localizing unusual activities, such as human errors or surveillance incidents, in videos holds practical significance. However, current video understanding models struggle with localizing these unusual events likely because of their insufficient representation in models' pretraining datasets. To explore foundation models' capability in localizing unusual activity, we introduce UAL-Bench, a comprehensive benchmark for unusual activity localization, featuring three video datasets: UAG-OOPS, UAG-SSBD, UAG-FunQA, and an instruction-tune dataset: OOPS-UAG-Instruct, to improve model capabilities. UAL-Bench evaluates three approaches: Video-Language Models (Vid-LLMs), instruction-tuned Vid-LLMs, and a novel integration of Vision-Language Models and Large Language Models (VLM-LLM). Our results show the VLM-LLM approach excels in localizing short-span unusual events and predicting their onset (start time) more accurately than Vid-LLMs. We also propose a new metric, R@1, TD <= p, to address limitations in existing evaluation methods. Our findings highlight the challenges posed by long-duration videos, particularly in autism diagnosis scenarios, and the need for further advancements in localization techniques. Our work not only provides a benchmark for unusual activity localization but also outlines the key challenges for existing foundation models, suggesting future research directions on this important task.