Zero-Shot Action Recognition in Surveillance Videos

TL;DR

This paper demonstrates that Large Vision-Language Models, combined with a novel sampling method, significantly improve zero-shot action recognition in surveillance videos, addressing challenges of limited data and diverse conditions.

Contribution

It introduces the use of LVLMs like VideoLLaMA2 for surveillance video understanding and proposes Self-Reflective Sampling to enhance zero-shot performance.

Findings

VideoLLaMA2 achieves 20% boost over baseline in zero-shot performance.

Self-ReS increases zero-shot action recognition to 44.6%.

Results show LVLMs and sampling techniques are promising for surveillance analysis.

Abstract

The growing demand for surveillance in public spaces presents significant challenges due to the shortage of human resources. Current AI-based video surveillance systems heavily rely on core computer vision models that require extensive finetuning, which is particularly difficult in surveillance settings due to limited datasets and difficult setting (viewpoint, low quality, etc.). In this work, we propose leveraging Large Vision-Language Models (LVLMs), known for their strong zero and few-shot generalization, to tackle video understanding tasks in surveillance. Specifically, we explore VideoLLaMA2, a state-of-the-art LVLM, and an improved token-level sampling method, Self-Reflective Sampling (Self-ReS). Our experiments on the UCF-Crime dataset show that VideoLLaMA2 represents a significant leap in zero-shot performance, with 20% boost over the baseline. Self-ReS additionally increases zero-shot action recognition performance to 44.6%. These results highlight the potential of LVLMs, paired with improved sampling techniques, for advancing surveillance video analysis in diverse scenarios.