PoseMoE: Mixture-of-Experts Network for Monocular 3D Human Pose Estimation

TL;DR

PoseMoE introduces a mixture-of-experts network that disentangles 2D pose and depth features, improving monocular 3D human pose estimation accuracy by reducing the influence of uncertain depth features.

Contribution

The paper proposes PoseMoE, a novel mixture-of-experts architecture that separately refines 2D pose and depth features, with a cross-expert module for better feature aggregation, addressing limitations of previous entangled encoding methods.

Findings

Outperforms existing lifting-based methods on Human3.6M, MPI-INF-3DHP, and 3DPW datasets.

Effectively disentangles 2D pose and depth features, reducing depth uncertainty impact.

Enhances feature representation through cross-expert spatio-temporal aggregation.

Abstract

The lifting-based methods have dominated monocular 3D human pose estimation by leveraging detected 2D poses as intermediate representations. The 2D component of the final 3D human pose benefits from the detected 2D poses, whereas its depth counterpart must be estimated from scratch. The lifting-based methods encode the detected 2D pose and unknown depth in an entangled feature space, explicitly introducing depth uncertainty to the detected 2D pose, thereby limiting overall estimation accuracy. This work reveals that the depth representation is pivotal for the estimation process. Specifically, when depth is in an initial, completely unknown state, jointly encoding depth features with 2D pose features is detrimental to the estimation process. In contrast, when depth is initially refined to a more dependable state via network-based estimation, encoding it together with 2D pose information is beneficial. To address this limitation, we present a Mixture-of-Experts network for monocular 3D pose estimation named PoseMoE. Our approach introduces: (1) A mixture-of-experts network where specialized expert modules refine the well-detected 2D pose features and learn the depth features. This mixture-of-experts design disentangles the feature encoding process for 2D pose and depth, therefore reducing the explicit influence of uncertain depth features on 2D pose features. (2) A cross-expert knowledge aggregation module is proposed to aggregate cross-expert spatio-temporal contextual information. This step enhances features through bidirectional mapping between 2D pose and depth. Extensive experiments show that our proposed PoseMoE outperforms the conventional lifting-based methods on three widely used datasets: Human3.6M, MPI-INF-3DHP, and 3DPW.