PoseBH: Prototypical Multi-Dataset Training Beyond Human Pose Estimation

TL;DR

PoseBH introduces a novel multi-dataset training framework for pose estimation that effectively handles skeletal heterogeneity and limited supervision, improving generalization across diverse datasets and transferring well to related tasks.

Contribution

It proposes nonparametric keypoint prototypes and a cross-type self-supervision mechanism to unify and enhance multi-dataset pose estimation beyond existing methods.

Findings

Improves generalization across multiple pose datasets.

Transfers effectively to hand and body shape estimation tasks.

Maintains performance on standard human pose benchmarks.

Abstract

We study multi-dataset training (MDT) for pose estimation, where skeletal heterogeneity presents a unique challenge that existing methods have yet to address. In traditional domains, \eg regression and classification, MDT typically relies on dataset merging or multi-head supervision. However, the diversity of skeleton types and limited cross-dataset supervision complicate integration in pose estimation. To address these challenges, we introduce PoseBH, a new MDT framework that tackles keypoint heterogeneity and limited supervision through two key techniques. First, we propose nonparametric keypoint prototypes that learn within a unified embedding space, enabling seamless integration across skeleton types. Second, we develop a cross-type self-supervision mechanism that aligns keypoint predictions with keypoint embedding prototypes, providing supervision without relying on teacher-student models or additional augmentations. PoseBH substantially improves generalization across whole-body and animal pose datasets, including COCO-WholeBody, AP-10K, and APT-36K, while preserving performance on standard human pose benchmarks (COCO, MPII, and AIC). Furthermore, our learned keypoint embeddings transfer effectively to hand shape estimation (InterHand2.6M) and human body shape estimation (3DPW). The code for PoseBH is available at: https://github.com/uyoung-jeong/PoseBH.