UniHPR: Unified Human Pose Representation via Singular Value Contrastive Learning

TL;DR

UniHPR introduces a novel contrastive learning approach to unify human pose representations across multiple modalities, significantly improving pose estimation and retrieval accuracy.

Contribution

The paper presents a new singular value-based contrastive loss for aligning multi-modal human pose embeddings, enhancing cross-modal fusion and downstream task performance.

Findings

Achieves 49.9mm MPJPE on Human3.6M dataset

Attains 51.6mm PA-MPJPE on 3DPW dataset

Enables accurate 2D and 3D pose retrieval

Abstract

In recent years, there has been a growing interest in developing effective alignment pipelines to generate unified representations from different modalities for multi-modal fusion and generation. As an important component of Human-Centric applications, Human Pose representations are critical in many downstream tasks, such as Human Pose Estimation, Action Recognition, Human-Computer Interaction, Object tracking, etc. Human Pose representations or embeddings can be extracted from images, 2D keypoints, 3D skeletons, mesh models, and lots of other modalities. Yet, there are limited instances where the correlation among all of those representations has been clearly researched using a contrastive paradigm. In this paper, we propose UniHPR, a unified Human Pose Representation learning pipeline, which aligns Human Pose embeddings from images, 2D and 3D human poses. To align more than two data representations at the same time, we propose a novel singular value-based contrastive learning loss, which better aligns different modalities and further boosts performance. To evaluate the effectiveness of the aligned representation, we choose 2D and 3D Human Pose Estimation (HPE) as our evaluation tasks. In our evaluation, with a simple 3D human pose decoder, UniHPR achieves remarkable performance metrics: MPJPE 49.9mm on the Human3.6M dataset and PA-MPJPE 51.6mm on the 3DPW dataset with cross-domain evaluation. Meanwhile, we are able to achieve 2D and 3D pose retrieval with our unified human pose representations in Human3.6M dataset, where the retrieval error is 9.24mm in MPJPE.