EgoCast: Forecasting Egocentric Human Pose in the Wild

TL;DR

EgoCast is a novel bimodal approach for 3D egocentric human pose forecasting that leverages egocentric videos and proprioceptive data, improving real-life motion estimation in dynamic scenes.

Contribution

It introduces a current-frame estimation module that generates pseudo-groundtruth poses, removing the need for past groundtruth data during forecasting.

Findings

EgoCast outperforms state-of-the-art methods on Ego-Exo4D dataset.

The approach effectively handles real-world, unscripted activities.

Validated on multiple datasets with significant accuracy improvements.

Abstract

Accurately estimating and forecasting human body pose is important for enhancing the user's sense of immersion in Augmented Reality. Addressing this need, our paper introduces EgoCast, a bimodal method for 3D human pose forecasting using egocentric videos and proprioceptive data. We study the task of human pose forecasting in a realistic setting, extending the boundaries of temporal forecasting in dynamic scenes and building on the current framework for current pose estimation in the wild. We introduce a current-frame estimation module that generates pseudo-groundtruth poses for inference, eliminating the need for past groundtruth poses typically required by current methods during forecasting. Our experimental results on the recent Ego-Exo4D and Aria Digital Twin datasets validate EgoCast for real-life motion estimation. On the Ego-Exo4D Body Pose 2024 Challenge, our method significantly outperforms the state-of-the-art approaches, laying the groundwork for future research in human pose estimation and forecasting in unscripted activities with egocentric inputs.