MPM: A Unified 2D-3D Human Pose Representation via Masked Pose Modeling

TL;DR

This paper introduces MPM, a transformer-based framework that unifies 2D and 3D human pose representations through masked pose modeling, enabling versatile pose estimation tasks with state-of-the-art results.

Contribution

It proposes a novel unified 2D-3D pose representation framework using masked pose modeling and a single-stream transformer architecture, handling multiple pose estimation tasks.

Findings

Achieves state-of-the-art performance on MPI-INF-3DHP dataset.

Effectively models spatial and temporal pose relations with high masking ratio.

Handles multiple pose estimation tasks within a single framework.

Abstract

Estimating 3D human poses only from a 2D human pose sequence is thoroughly explored in recent years. Yet, prior to this, no such work has attempted to unify 2D and 3D pose representations in the shared feature space. In this paper, we propose \mpm, a unified 2D-3D human pose representation framework via masked pose modeling. We treat 2D and 3D poses as two different modalities like vision and language and build a single-stream transformer-based architecture. We apply two pretext tasks, which are masked 2D pose modeling, and masked 3D pose modeling to pre-train our network and use full-supervision to perform further fine-tuning. A high masking ratio of $71.8~\%$ in total with a spatio-temporal mask sampling strategy leads to better relation modeling both in spatial and temporal domains. \mpm~can handle multiple tasks including 3D human pose estimation, 3D pose estimation from occluded 2D pose, and 3D pose completion in a \textbf{single} framework. We conduct extensive experiments and ablation studies on several widely used human pose datasets and achieve state-of-the-art performance on MPI-INF-3DHP.