A Survey of Video Datasets for Grounded Event Understanding

TL;DR

This survey reviews 105 video datasets focused on event understanding, highlighting the need for better task framing and dataset curation to advance multimodal AI's common-sense reasoning capabilities.

Contribution

It provides a comprehensive analysis of existing datasets and tasks for video event understanding, proposing guidelines for future dataset creation and task design.

Findings

Many datasets focus on specific event types, limiting generalization.

Current tasks lack standardization, hindering progress.

Temporal aspects and ambiguity are crucial for effective event understanding.

Abstract

While existing video benchmarks largely consider specialized downstream tasks like retrieval or question-answering (QA), contemporary multimodal AI systems must be capable of well-rounded common-sense reasoning akin to human visual understanding. A critical component of human temporal-visual perception is our ability to identify and cognitively model "things happening", or events. Historically, video benchmark tasks have implicitly tested for this ability (e.g., video captioning, in which models describe visual events with natural language), but they do not consider video event understanding as a task in itself. Recent work has begun to explore video analogues to textual event extraction but consists of competing task definitions and datasets limited to highly specific event types. Therefore, while there is a rich domain of event-centric video research spanning the past 10+ years, it is unclear how video event understanding should be framed and what resources we have to study it. In this paper, we survey 105 video datasets that require event understanding capability, consider how they contribute to the study of robust event understanding in video, and assess proposed video event extraction tasks in the context of this body of research. We propose suggestions informed by this survey for dataset curation and task framing, with an emphasis on the uniquely temporal nature of video events and ambiguity in visual content.