Visual Semantic Role Labeling for Video Understanding

TL;DR

This paper introduces a novel framework for video understanding using visual semantic role labeling, along with a large-scale annotated dataset called VidSitu, enabling detailed analysis of events and entities in movies.

Contribution

The paper presents the VidSitu benchmark dataset and a new framework for semantic role labeling in videos, advancing the understanding of complex, diverse movie clips.

Findings

VidSitu contains 29K annotated 10-second clips from movies.

Standard models show room for improvement on semantic role labeling in videos.

Comprehensive analysis highlights challenges and opportunities in video event understanding.

Abstract

We propose a new framework for understanding and representing related salient events in a video using visual semantic role labeling. We represent videos as a set of related events, wherein each event consists of a verb and multiple entities that fulfill various roles relevant to that event. To study the challenging task of semantic role labeling in videos or VidSRL, we introduce the VidSitu benchmark, a large-scale video understanding data source with $29K$ $10$-second movie clips richly annotated with a verb and semantic-roles every $2$ seconds. Entities are co-referenced across events within a movie clip and events are connected to each other via event-event relations. Clips in VidSitu are drawn from a large collection of movies (${\sim}3K$) and have been chosen to be both complex (${\sim}4.2$ unique verbs within a video) as well as diverse (${\sim}200$ verbs have more than $100$ annotations each). We provide a comprehensive analysis of the dataset in comparison to other publicly available video understanding benchmarks, several illustrative baselines and evaluate a range of standard video recognition models. Our code and dataset is available at vidsitu.org.