LangPose: Language-Aligned Motion for Robust 3D Human Pose Estimation

TL;DR

LangPose introduces a novel framework that aligns motion embeddings with text embeddings of action labels, effectively improving 3D human pose estimation by leveraging semantic information to address occlusion and ambiguity issues.

Contribution

The paper proposes LangPose, a two-stage framework that integrates action knowledge through language alignment, enhancing robustness and accuracy in 3D human pose estimation.

Findings

Achieves state-of-the-art performance on Human3.6M and MPI-INF-3DHP datasets.

Improves MPJPE to 36.7mm on Human3.6M with detected 2D poses.

Enhances robustness by incorporating semantic information and masked modeling techniques.

Abstract

2D-to-3D human pose lifting is an ill-posed problem due to depth ambiguity and occlusion. Existing methods relying on spatial and temporal consistency alone are insufficient to resolve these problems especially in the presence of significant occlusions or high dynamic actions. Semantic information, however, offers a complementary signal that can help disambiguate such cases. To this end, we propose LangPose, a framework that leverages action knowledge by aligning motion embeddings with text embeddings of fine-grained action labels. LangPose operates in two stages: pretraining and fine-tuning. In the pretraining stage, the model simultaneously learns to recognize actions and reconstruct 3D poses from masked and noisy 2D poses. During the fine-tuning stage, the model is further refined using real-world 3D human pose estimation datasets without action labels. Additionally, our framework incorporates masked body parts and masked time windows in motion modeling, encouraging the model to leverage semantic information when spatial and temporal consistency is unreliable. Experiments demonstrate the effectiveness of LangPose, achieving SOTA level performance in 3D pose estimation on public datasets, including Human3.6M and MPI-INF-3DHP. Specifically, LangPose achieves an MPJPE of 36.7mm on Human3.6M with detected 2D poses as input and 15.5mm on MPI-INF-3DHP with ground-truth 2D poses as input.