Ego-Body Pose Estimation via Ego-Head Pose Estimation

TL;DR

This paper introduces EgoEgo, a novel two-stage method for estimating full-body 3D human motion from egocentric videos by leveraging head pose estimation, thus avoiding the need for paired datasets and improving performance.

Contribution

The paper proposes a new disentangled approach that separates head and body pose estimation, enabling training on separate datasets and achieving superior results.

Findings

EgoEgo outperforms existing methods on synthetic and real datasets.

The approach effectively leverages large-scale unpaired datasets.

Systematic benchmarking with the new ARES dataset demonstrates state-of-the-art performance.

Abstract

Estimating 3D human motion from an egocentric video sequence plays a critical role in human behavior understanding and has various applications in VR/AR. However, naively learning a mapping between egocentric videos and human motions is challenging, because the user's body is often unobserved by the front-facing camera placed on the head of the user. In addition, collecting large-scale, high-quality datasets with paired egocentric videos and 3D human motions requires accurate motion capture devices, which often limit the variety of scenes in the videos to lab-like environments. To eliminate the need for paired egocentric video and human motions, we propose a new method, Ego-Body Pose Estimation via Ego-Head Pose Estimation (EgoEgo), which decomposes the problem into two stages, connected by the head motion as an intermediate representation. EgoEgo first integrates SLAM and a learning approach to estimate accurate head motion. Subsequently, leveraging the estimated head pose as input, EgoEgo utilizes conditional diffusion to generate multiple plausible full-body motions. This disentanglement of head and body pose eliminates the need for training datasets with paired egocentric videos and 3D human motion, enabling us to leverage large-scale egocentric video datasets and motion capture datasets separately. Moreover, for systematic benchmarking, we develop a synthetic dataset, AMASS-Replica-Ego-Syn (ARES), with paired egocentric videos and human motion. On both ARES and real data, our EgoEgo model performs significantly better than the current state-of-the-art methods.