Charades-Ego: A Large-Scale Dataset of Paired Third and First Person Videos

TL;DR

Charades-Ego is a large, diverse dataset linking first and third-person videos with detailed annotations, enabling advancements in egocentric video understanding and cross-modal tasks.

Contribution

It introduces a comprehensive egocentric video dataset with extensive annotations, expanding resources for egocentric and cross-modal video research.

Findings

Largest egocentric dataset with 68,536 activity instances

Includes temporal annotations and textual descriptions

Facilitates research in classification, localization, and captioning

Abstract

In Actor and Observer we introduced a dataset linking the first and third-person video understanding domains, the Charades-Ego Dataset. In this paper we describe the egocentric aspect of the dataset and present annotations for Charades-Ego with 68,536 activity instances in 68.8 hours of first and third-person video, making it one of the largest and most diverse egocentric datasets available. Charades-Ego furthermore shares activity classes, scripts, and methodology with the Charades dataset, that consist of additional 82.3 hours of third-person video with 66,500 activity instances. Charades-Ego has temporal annotations and textual descriptions, making it suitable for egocentric video classification, localization, captioning, and new tasks utilizing the cross-modal nature of the data.