MegaPose: 6D Pose Estimation of Novel Objects via Render & Compare

TL;DR

MegaPose is a novel 6D pose estimation method for unseen objects that leverages synthetic data, render-and-compare refinement, and a coarse-to-fine approach, achieving state-of-the-art results without retraining on new objects.

Contribution

Introduces a render&compare based 6D pose refiner, a coarse pose classifier, and a large synthetic dataset for generalizing pose estimation to novel objects.

Findings

Achieves state-of-the-art on ModelNet and YCB-Video datasets.

Performs competitively on BOP challenge datasets.

Generalizes well to unseen objects without retraining.

Abstract

We introduce MegaPose, a method to estimate the 6D pose of novel objects, that is, objects unseen during training. At inference time, the method only assumes knowledge of (i) a region of interest displaying the object in the image and (ii) a CAD model of the observed object. The contributions of this work are threefold. First, we present a 6D pose refiner based on a render&compare strategy which can be applied to novel objects. The shape and coordinate system of the novel object are provided as inputs to the network by rendering multiple synthetic views of the object's CAD model. Second, we introduce a novel approach for coarse pose estimation which leverages a network trained to classify whether the pose error between a synthetic rendering and an observed image of the same object can be corrected by the refiner. Third, we introduce a large-scale synthetic dataset of photorealistic images of thousands of objects with diverse visual and shape properties and show that this diversity is crucial to obtain good generalization performance on novel objects. We train our approach on this large synthetic dataset and apply it without retraining to hundreds of novel objects in real images from several pose estimation benchmarks. Our approach achieves state-of-the-art performance on the ModelNet and YCB-Video datasets. An extensive evaluation on the 7 core datasets of the BOP challenge demonstrates that our approach achieves performance competitive with existing approaches that require access to the target objects during training. Code, dataset and trained models are available on the project page: https://megapose6d.github.io/.