RAG-6DPose: Retrieval-Augmented 6D Pose Estimation via Leveraging CAD as Knowledge Base

TL;DR

RAG-6DPose introduces a retrieval-augmented method for 6D pose estimation that leverages a multi-modal CAD knowledge base to improve accuracy and robustness, especially under occlusions and novel viewpoints.

Contribution

It is the first to integrate CAD models as a knowledge base with retrieval and decoding for 6D pose estimation, enhancing robustness and accuracy.

Findings

Outperforms existing methods on standard benchmarks.

Effectively handles occlusions and unseen viewpoints.

Demonstrates robustness in real-world robotic tasks.

Abstract

Accurate 6D pose estimation is key for robotic manipulation, enabling precise object localization for tasks like grasping. We present RAG-6DPose, a retrieval-augmented approach that leverages 3D CAD models as a knowledge base by integrating both visual and geometric cues. Our RAG-6DPose roughly contains three stages: 1) Building a Multi-Modal CAD Knowledge Base by extracting 2D visual features from multi-view CAD rendered images and also attaching 3D points; 2) Retrieving relevant CAD features from the knowledge base based on the current query image via our ReSPC module; and 3) Incorporating retrieved CAD information to refine pose predictions via retrieval-augmented decoding. Experimental results on standard benchmarks and real-world robotic tasks demonstrate the effectiveness and robustness of our approach, particularly in handling occlusions and novel viewpoints. Supplementary material is available on our project website: https://sressers.github.io/RAG-6DPose .