EgoMe: A New Dataset and Challenge for Following Me via Egocentric View in Real World

TL;DR

EgoMe introduces a large-scale egocentric dataset with paired videos, eye gaze, and sensor data to advance human imitation learning research from the imitator's perspective in real-world scenarios.

Contribution

The paper presents EgoMe, a novel dataset with paired exo-ego videos, eye gaze, and sensor data, enabling more realistic imitation learning research from the imitator's viewpoint.

Findings

EgoMe outperforms existing datasets in diversity and realism.

The dataset facilitates new benchmarks for imitation learning.

Analysis shows improved understanding of imitation processes.

Abstract

In human imitation learning, the imitator typically take the egocentric view as a benchmark, naturally transferring behaviors observed from an exocentric view to their owns, which provides inspiration for researching how robots can more effectively imitate human behavior. However, current research primarily focuses on the basic alignment issues of ego-exo data from different cameras, rather than collecting data from the imitator's perspective, which is inconsistent with the high-level cognitive process. To advance this research, we introduce a novel large-scale egocentric dataset, called EgoMe, which towards following the process of human imitation learning via the imitator's egocentric view in the real world. Our dataset includes 7902 paired exo-ego videos (totaling15804 videos) spanning diverse daily behaviors in various real-world scenarios. For each video pair, one video captures an exocentric view of the imitator observing the demonstrator's actions, while the other captures an egocentric view of the imitator subsequently following those actions. Notably, EgoMe uniquely incorporates exo-ego eye gaze, other multi-modal sensor IMU data and different-level annotations for assisting in establishing correlations between observing and imitating process. We further provide a suit of challenging benchmarks for fully leveraging this data resource and promoting the robot imitation learning research. Extensive analysis demonstrates significant advantages over existing datasets. Our EgoMe dataset and benchmarks are available at https://huggingface.co/datasets/HeqianQiu/EgoMe.