Human Action Co-occurrence in Lifestyle Vlogs using Graph Link Prediction

TL;DR

This paper introduces a new dataset and models for automatically identifying co-occurring human actions in lifestyle videos, leveraging graph link prediction techniques to improve understanding of action relationships.

Contribution

The paper presents the ACE dataset and graph-based models for action co-occurrence prediction, advancing the understanding of action relations in videos.

Findings

Graphs effectively capture relations between human actions.

Graph representations improve co-occurrence prediction accuracy.

The ACE dataset enables further research in action understanding.

Abstract

We introduce the task of automatic human action co-occurrence identification, i.e., determine whether two human actions can co-occur in the same interval of time. We create and make publicly available the ACE (Action Co-occurrencE) dataset, consisting of a large graph of ~12k co-occurring pairs of visual actions and their corresponding video clips. We describe graph link prediction models that leverage visual and textual information to automatically infer if two actions are co-occurring. We show that graphs are particularly well suited to capture relations between human actions, and the learned graph representations are effective for our task and capture novel and relevant information across different data domains. The ACE dataset and the code introduced in this paper are publicly available at https://github.com/MichiganNLP/vlog_action_co-occurrence.