MK-Pose: Category-Level Object Pose Estimation via Multimodal-Based Keypoint Learning

TL;DR

MK-Pose introduces a multimodal framework combining RGB, point clouds, and text for category-level object pose estimation, improving accuracy and robustness against occlusion and variation.

Contribution

The paper presents a novel multimodal keypoint learning framework with attention and graph modules, enhancing pose estimation without shape priors.

Findings

Outperforms state-of-the-art in IoU and average precision

Effective cross-dataset generalization demonstrated

No reliance on shape priors needed

Abstract

Category-level object pose estimation, which predicts the pose of objects within a known category without prior knowledge of individual instances, is essential in applications like warehouse automation and manufacturing. Existing methods relying on RGB images or point cloud data often struggle with object occlusion and generalization across different instances and categories. This paper proposes a multimodal-based keypoint learning framework (MK-Pose) that integrates RGB images, point clouds, and category-level textual descriptions. The model uses a self-supervised keypoint detection module enhanced with attention-based query generation, soft heatmap matching and graph-based relational modeling. Additionally, a graph-enhanced feature fusion module is designed to integrate local geometric information and global context. MK-Pose is evaluated on CAMERA25 and REAL275 dataset, and is further tested for cross-dataset capability on HouseCat6D dataset. The results demonstrate that MK-Pose outperforms existing state-of-the-art methods in both IoU and average precision without shape priors. Codes will be released at \href{https://github.com/yangyifanYYF/MK-Pose}{https://github.com/yangyifanYYF/MK-Pose}.