HIPPo: Harnessing Image-to-3D Priors for Model-free Zero-shot 6D Pose Estimation

TL;DR

HIPPo introduces a model-free, zero-shot 6D pose estimation framework that leverages image-to-3D priors from diffusion models, enabling rapid, accurate pose estimation without curated models or reference images.

Contribution

The paper presents HIPPo, a novel framework that uses diffusion-based image-to-3D priors for instant, model-free 6D pose estimation of unseen objects in real-time scenarios.

Findings

HIPPo outperforms state-of-the-art methods with limited prior references.

HIPPo can generate 3D meshes of unseen objects in seconds.

HIPPo effectively refines object meshes through online observations.

Abstract

This work focuses on model-free zero-shot 6D object pose estimation for robotics applications. While existing methods can estimate the precise 6D pose of objects, they heavily rely on curated CAD models or reference images, the preparation of which is a time-consuming and labor-intensive process. Moreover, in real-world scenarios, 3D models or reference images may not be available in advance and instant robot reaction is desired. In this work, we propose a novel framework named HIPPo, which eliminates the need for curated CAD models and reference images by harnessing image-to-3D priors from Diffusion Models, enabling model-free zero-shot 6D pose estimation. Specifically, we construct HIPPo Dreamer, a rapid image-to-mesh model built on a multiview Diffusion Model and a 3D reconstruction foundation model. Our HIPPo Dreamer can generate a 3D mesh of any unseen objects from a single glance in just a few seconds. Then, as more observations are acquired, we propose to continuously refine the diffusion prior mesh model by joint optimization of object geometry and appearance. This is achieved by a measurement-guided scheme that gradually replaces the plausible diffusion priors with more reliable online observations. Consequently, HIPPo can instantly estimate and track the 6D pose of a novel object and maintain a complete mesh for immediate robotic applications. Thorough experiments on various benchmarks show that HIPPo outperforms state-of-the-art methods in 6D object pose estimation when prior reference images are limited.