Finding NeMO: A Geometry-Aware Representation of Template Views for Few-Shot Perception

TL;DR

NeMO introduces a geometry-aware object representation that enables few-shot detection, segmentation, and pose estimation of unseen objects from RGB images, without retraining or camera-specific tuning.

Contribution

The paper proposes NeMO, a novel object-centric representation that uses minimal template views to perform multiple perception tasks on unseen objects efficiently.

Findings

Achieves state-of-the-art results on BOP benchmark datasets.

Enables quick object onboarding without retraining.

Supports multiple perception tasks with a single network.

Abstract

We present Neural Memory Object (NeMO), a novel object-centric representation that can be used to detect, segment and estimate the 6DoF pose of objects unseen during training using RGB images. Our method consists of an encoder that requires only a few RGB template views depicting an object to generate a sparse object-like point cloud using a learned UDF containing semantic and geometric information. Next, a decoder takes the object encoding together with a query image to generate a variety of dense predictions. Through extensive experiments, we show that our method can be used for few-shot object perception without requiring any camera-specific parameters or retraining on target data. Our proposed concept of outsourcing object information in a NeMO and using a single network for multiple perception tasks enhances interaction with novel objects, improving scalability and efficiency by enabling quick object onboarding without retraining or extensive pre-processing. We report competitive and state-of-the-art results on various datasets and perception tasks of the BOP benchmark, demonstrating the versatility of our approach. https://github.com/DLR-RM/nemo