X-Pose: Detecting Any Keypoints

TL;DR

X-Pose introduces an end-to-end multi-modal prompt-based framework for detecting any keypoints across diverse objects and scenarios, supported by a large unified dataset, UniKPT, achieving significant accuracy improvements.

Contribution

The paper presents X-Pose, a novel multi-modal prompt-based keypoint detection framework and the UniKPT dataset, enabling accurate detection of diverse keypoints in complex real-world images.

Findings

X-Pose outperforms existing methods with 27.7 AP, 6.44 PCK, and 7.0 AP improvements.

The UniKPT dataset unifies 13 datasets with 338 keypoints across 1,237 categories.

X-Pose demonstrates strong generalization across styles, categories, and poses.

Abstract

This work aims to address an advanced keypoint detection problem: how to accurately detect any keypoints in complex real-world scenarios, which involves massive, messy, and open-ended objects as well as their associated keypoints definitions. Current high-performance keypoint detectors often fail to tackle this problem due to their two-stage schemes, under-explored prompt designs, and limited training data. To bridge the gap, we propose X-Pose, a novel end-to-end framework with multi-modal (i.e., visual, textual, or their combinations) prompts to detect multi-object keypoints for any articulated (e.g., human and animal), rigid, and soft objects within a given image. Moreover, we introduce a large-scale dataset called UniKPT, which unifies 13 keypoint detection datasets with 338 keypoints across 1,237 categories over 400K instances. Training with UniKPT, X-Pose effectively aligns text-to-keypoint and image-to-keypoint due to the mutual enhancement of multi-modal prompts based on cross-modality contrastive learning. Our experimental results demonstrate that X-Pose achieves notable improvements of 27.7 AP, 6.44 PCK, and 7.0 AP compared to state-of-the-art non-promptable, visual prompt-based, and textual prompt-based methods in each respective fair setting. More importantly, the in-the-wild test demonstrates X-Pose's strong fine-grained keypoint localization and generalization abilities across image styles, object categories, and poses, paving a new path to multi-object keypoint detection in real applications. Our code and dataset are available at https://github.com/IDEA-Research/X-Pose.