3D Human Pose Perception from Egocentric Stereo Videos

TL;DR

This paper introduces a transformer-based framework for egocentric stereo 3D human pose estimation that leverages scene and temporal information, and provides new large-scale datasets for evaluation.

Contribution

The work presents a novel transformer-based approach utilizing scene reconstruction and temporal features, along with two new egocentric stereo datasets for comprehensive benchmarking.

Findings

Significantly outperforms previous methods in pose estimation accuracy.

Effectively handles challenging scenarios like crouching and sitting.

Provides large-scale datasets for future research.

Abstract

While head-mounted devices are becoming more compact, they provide egocentric views with significant self-occlusions of the device user. Hence, existing methods often fail to accurately estimate complex 3D poses from egocentric views. In this work, we propose a new transformer-based framework to improve egocentric stereo 3D human pose estimation, which leverages the scene information and temporal context of egocentric stereo videos. Specifically, we utilize 1) depth features from our 3D scene reconstruction module with uniformly sampled windows of egocentric stereo frames, and 2) human joint queries enhanced by temporal features of the video inputs. Our method is able to accurately estimate human poses even in challenging scenarios, such as crouching and sitting. Furthermore, we introduce two new benchmark datasets, i.e., UnrealEgo2 and UnrealEgo-RW (RealWorld). The proposed datasets offer a much larger number of egocentric stereo views with a wider variety of human motions than the existing datasets, allowing comprehensive evaluation of existing and upcoming methods. Our extensive experiments show that the proposed approach significantly outperforms previous methods. We will release UnrealEgo2, UnrealEgo-RW, and trained models on our project page.