Enhancing Egocentric 3D Pose Estimation with Third Person Views

TL;DR

This paper introduces a new multi-view embedding approach and dataset to improve egocentric 3D pose estimation from wearable camera videos, outperforming existing methods without domain adaptation.

Contribution

The paper presents First2Third-Pose, a synchronized multi-view dataset, and a semi-Siamese architecture for self-supervised learning of joint embeddings linking first- and third-person views.

Findings

Significant improvement over state-of-the-art egocentric pose estimation methods.

The joint embedding space effectively captures discriminative features from single-view videos.

The dataset enables training without domain adaptation or camera parameter knowledge.

Abstract

In this paper, we propose a novel approach to enhance the 3D body pose estimation of a person computed from videos captured from a single wearable camera. The key idea is to leverage high-level features linking first- and third-views in a joint embedding space. To learn such embedding space we introduce First2Third-Pose, a new paired synchronized dataset of nearly 2,000 videos depicting human activities captured from both first- and third-view perspectives. We explicitly consider spatial- and motion-domain features, combined using a semi-Siamese architecture trained in a self-supervised fashion. Experimental results demonstrate that the joint multi-view embedded space learned with our dataset is useful to extract discriminatory features from arbitrary single-view egocentric videos, without needing domain adaptation nor knowledge of camera parameters. We achieve significant improvement of egocentric 3D body pose estimation performance on two unconstrained datasets, over three supervised state-of-the-art approaches. Our dataset and code will be available for research purposes.