Egocentric Visibility-Aware Human Pose Estimation

TL;DR

This paper introduces a large-scale egocentric dataset with keypoint visibility labels and a new visibility-aware pose estimation method that improves accuracy by explicitly modeling keypoint invisibility.

Contribution

The paper presents Eva-3M, a large-scale visibility-annotated dataset, and EvaPose, a novel method that leverages visibility information for better egocentric human pose estimation.

Findings

Eva-3M contains over 3 million frames with visibility labels.

EvaPose outperforms existing methods on visibility-aware pose estimation.

Visibility labels significantly improve pose estimation accuracy.

Abstract

Egocentric human pose estimation (HPE) using a head-mounted device is crucial for various VR and AR applications, but it faces significant challenges due to keypoint invisibility. Nevertheless, none of the existing egocentric HPE datasets provide keypoint visibility annotations, and the existing methods often overlook the invisibility problem, treating visible and invisible keypoints indiscriminately during estimation. As a result, their capacity to accurately predict visible keypoints is compromised. In this paper, we first present Eva-3M, a large-scale egocentric visibility-aware HPE dataset comprising over 3.0M frames, with 435K of them annotated with keypoint visibility labels. Additionally, we augment the existing EMHI dataset with keypoint visibility annotations to further facilitate the research in this direction. Furthermore, we propose EvaPose, a novel egocentric visibility-aware HPE method that explicitly incorporates visibility information to enhance pose estimation accuracy. Extensive experiments validate the significant value of ground-truth visibility labels in egocentric HPE settings, and demonstrate that our EvaPose achieves state-of-the-art performance in both Eva-3M and EMHI datasets.