Human-centric Behavior Description in Videos: New Benchmark and Model

TL;DR

This paper introduces a new human-centric video surveillance dataset with detailed individual behavior annotations and proposes a novel captioning model that achieves state-of-the-art results in describing behaviors at the person level.

Contribution

The paper presents a new dataset with detailed annotations for individual behaviors in surveillance videos and a novel captioning approach for fine-grained behavior description.

Findings

Achieved state-of-the-art results in person-level behavior captioning.

Created a dataset with 7,820 individuals across 1,012 videos with detailed annotations.

Enabled linking individuals to their behaviors for better surveillance analysis.

Abstract

In the domain of video surveillance, describing the behavior of each individual within the video is becoming increasingly essential, especially in complex scenarios with multiple individuals present. This is because describing each individual's behavior provides more detailed situational analysis, enabling accurate assessment and response to potential risks, ensuring the safety and harmony of public places. Currently, video-level captioning datasets cannot provide fine-grained descriptions for each individual's specific behavior. However, mere descriptions at the video-level fail to provide an in-depth interpretation of individual behaviors, making it challenging to accurately determine the specific identity of each individual. To address this challenge, we construct a human-centric video surveillance captioning dataset, which provides detailed descriptions of the dynamic behaviors of 7,820 individuals. Specifically, we have labeled several aspects of each person, such as location, clothing, and interactions with other elements in the scene, and these people are distributed across 1,012 videos. Based on this dataset, we can link individuals to their respective behaviors, allowing for further analysis of each person's behavior in surveillance videos. Besides the dataset, we propose a novel video captioning approach that can describe individual behavior in detail on a person-level basis, achieving state-of-the-art results. To facilitate further research in this field, we intend to release our dataset and code.