NeuSurfEmb: A Complete Pipeline for Dense Correspondence-based 6D Object Pose Estimation without CAD Models

TL;DR

NeuSurfEmb introduces a CAD-model-free pipeline for 6D object pose estimation using a novel NeuS2 representation, semi-automated data generation, and a correspondence-based estimator, achieving competitive results with less setup.

Contribution

The paper presents a fully pipeline that eliminates the need for CAD models and PBR, leveraging NeuS2 and semi-automated photorealistic data synthesis for robust pose estimation.

Findings

Competitive performance on LINEMOD-Occlusion dataset.

Outperforms CAD-model-free approaches in accuracy and robustness.

Eases deployment in real-world scenarios with minimal manual setup.

Abstract

State-of-the-art approaches for 6D object pose estimation assume the availability of CAD models and require the user to manually set up physically-based rendering (PBR) pipelines for synthetic training data generation. Both factors limit the application of these methods in real-world scenarios. In this work, we present a pipeline that does not require CAD models and allows training a state-of-the-art pose estimator requiring only a small set of real images as input. Our method is based on a NeuS2 object representation, that we learn through a semi-automated procedure based on Structure-from-Motion (SfM) and object-agnostic segmentation. We exploit the novel-view synthesis ability of NeuS2 and simple cut-and-paste augmentation to automatically generate photorealistic object renderings, which we use to train the correspondence-based SurfEmb pose estimator. We evaluate our method on the LINEMOD-Occlusion dataset, extensively studying the impact of its individual components and showing competitive performance with respect to approaches based on CAD models and PBR data. We additionally demonstrate the ease of use and effectiveness of our pipeline on self-collected real-world objects, showing that our method outperforms state-of-the-art CAD-model-free approaches, with better accuracy and robustness to mild occlusions. To allow the robotics community to benefit from this system, we will publicly release it at https://www.github.com/ethz-asl/neusurfemb.