RefEgo: Referring Expression Comprehension Dataset from First-Person Perception of Ego4D

TL;DR

RefEgo is a new large-scale dataset derived from egocentric videos in Ego4D, designed to advance the understanding of referring expressions in first-person perspectives for real-world applications.

Contribution

The paper introduces RefEgo, a comprehensive video-based referring expression dataset from egocentric views, filling a gap in real-world, first-person perception data for grounding language in visual scenes.

Findings

Combines state-of-the-art models with object tracking for improved performance

Achieves effective object tracking even when objects are out-of-frame or multiple similar objects are present

Demonstrates the dataset's potential for advancing first-person referring expression comprehension

Abstract

Grounding textual expressions on scene objects from first-person views is a truly demanding capability in developing agents that are aware of their surroundings and behave following intuitive text instructions. Such capability is of necessity for glass-devices or autonomous robots to localize referred objects in the real-world. In the conventional referring expression comprehension tasks of images, however, datasets are mostly constructed based on the web-crawled data and don't reflect diverse real-world structures on the task of grounding textual expressions in diverse objects in the real world. Recently, a massive-scale egocentric video dataset of Ego4D was proposed. Ego4D covers around the world diverse real-world scenes including numerous indoor and outdoor situations such as shopping, cooking, walking, talking, manufacturing, etc. Based on egocentric videos of Ego4D, we constructed a broad coverage of the video-based referring expression comprehension dataset: RefEgo. Our dataset includes more than 12k video clips and 41 hours for video-based referring expression comprehension annotation. In experiments, we combine the state-of-the-art 2D referring expression comprehension models with the object tracking algorithm, achieving the video-wise referred object tracking even in difficult conditions: the referred object becomes out-of-frame in the middle of the video or multiple similar objects are presented in the video. Codes are available at https://github.com/shuheikurita/RefEgo