Unseen Object Instance Segmentation with Fully Test-time RGB-D Embeddings Adaptation

TL;DR

This paper introduces a test-time adaptation method for unseen object segmentation that leverages RGB-D features and domain adaptation techniques, improving performance without additional annotations or synthetic data revisits.

Contribution

It proposes a novel non-parametric entropy objective and cross-modality knowledge distillation for efficient test-time adaptation across sim2real domains.

Findings

Achieves state-of-the-art unseen object segmentation performance.

Improves segmentation accuracy on overlap and boundary metrics.

Requires only test images without annotations or synthetic data revisits.

Abstract

Segmenting unseen objects is a crucial ability for the robot since it may encounter new environments during the operation. Recently, a popular solution is leveraging RGB-D features of large-scale synthetic data and directly applying the model to unseen real-world scenarios. However, the domain shift caused by the sim2real gap is inevitable, posing a crucial challenge to the segmentation model. In this paper, we emphasize the adaptation process across sim2real domains and model it as a learning problem on the BatchNorm parameters of a simulation-trained model. Specifically, we propose a novel non-parametric entropy objective, which formulates the learning objective for the test-time adaptation in an open-world manner. Then, a cross-modality knowledge distillation objective is further designed to encourage the test-time knowledge transfer for feature enhancement. Our approach can be efficiently implemented with only test images, without requiring annotations or revisiting the large-scale synthetic training data. Besides significant time savings, the proposed method consistently improves segmentation results on the overlap and boundary metrics, achieving state-of-the-art performance on unseen object instance segmentation.