Zero123-6D: Zero-shot Novel View Synthesis for RGB Category-level 6D Pose Estimation

TL;DR

Zero123-6D leverages diffusion model-based novel-view synthesis combined with feature extraction and online optimization to improve zero-shot category-level 6D pose estimation from RGB images, reducing data needs and eliminating depth requirements.

Contribution

This work introduces the first application of diffusion models for zero-shot category-level 6D pose estimation, integrating novel view synthesis with online refinement for better accuracy.

Findings

Improves pose estimation accuracy on CO3D dataset.

Reduces data requirements and removes depth dependency.

Demonstrates effectiveness of diffusion models in unstructured environments.

Abstract

Estimating the pose of objects through vision is essential to make robotic platforms interact with the environment. Yet, it presents many challenges, often related to the lack of flexibility and generalizability of state-of-the-art solutions. Diffusion models are a cutting-edge neural architecture transforming 2D and 3D computer vision, outlining remarkable performances in zero-shot novel-view synthesis. Such a use case is particularly intriguing for reconstructing 3D objects. However, localizing objects in unstructured environments is rather unexplored. To this end, this work presents Zero123-6D, the first work to demonstrate the utility of Diffusion Model-based novel-view-synthesizers in enhancing RGB 6D pose estimation at category-level, by integrating them with feature extraction techniques. Novel View Synthesis allows to obtain a coarse pose that is refined through an online optimization method introduced in this work to deal with intra-category geometric differences. In such a way, the outlined method shows reduction in data requirements, removal of the necessity of depth information in zero-shot category-level 6D pose estimation task, and increased performance, quantitatively demonstrated through experiments on the CO3D dataset.