GEB+: A Benchmark for Generic Event Boundary Captioning, Grounding and Retrieval

TL;DR

GEB+ introduces a large dataset and tasks for fine-grained understanding of status changes in videos, advancing research in event boundary captioning, grounding, and retrieval.

Contribution

The paper presents the Kinetic-GEB+ dataset with over 170k boundaries, new tasks for status change understanding, and a TPD modeling method that improves performance.

Findings

Significant performance gains with TPD modeling.

Current methods struggle with visual difference representation.

Dataset promotes development of more nuanced video understanding.

Abstract

Cognitive science has shown that humans perceive videos in terms of events separated by the state changes of dominant subjects. State changes trigger new events and are one of the most useful among the large amount of redundant information perceived. However, previous research focuses on the overall understanding of segments without evaluating the fine-grained status changes inside. In this paper, we introduce a new dataset called Kinetic-GEB+. The dataset consists of over 170k boundaries associated with captions describing status changes in the generic events in 12K videos. Upon this new dataset, we propose three tasks supporting the development of a more fine-grained, robust, and human-like understanding of videos through status changes. We evaluate many representative baselines in our dataset, where we also design a new TPD (Temporal-based Pairwise Difference) Modeling method for visual difference and achieve significant performance improvements. Besides, the results show there are still formidable challenges for current methods in the utilization of different granularities, representation of visual difference, and the accurate localization of status changes. Further analysis shows that our dataset can drive developing more powerful methods to understand status changes and thus improve video level comprehension. The dataset including both videos and boundaries is available at https://yuxuan-w.github.io/GEB-plus/