Learning from SAM: Harnessing a Foundation Model for Sim2Real Adaptation by Regularization

TL;DR

This paper introduces a self-supervised domain adaptation method for semantic segmentation in robotics, leveraging a foundation model to improve performance on unannotated target data, outperforming prior approaches.

Contribution

The method uses a segmentation foundation model and a novel invariance-variance loss to enhance domain adaptation without target annotations, handling overlapping segments effectively.

Findings

Outperforms prior domain adaptation methods on YCB-Video and HomebrewedDB datasets.

Achieves superior results even compared to networks trained with real annotations on YCB-Video.

Demonstrates applicability to a custom robotic application.

Abstract

Domain adaptation is especially important for robotics applications, where target domain training data is usually scarce and annotations are costly to obtain. We present a method for self-supervised domain adaptation for the scenario where annotated source domain data (e.g. from synthetic generation) is available, but the target domain data is completely unannotated. Our method targets the semantic segmentation task and leverages a segmentation foundation model (Segment Anything Model) to obtain segment information on unannotated data. We take inspiration from recent advances in unsupervised local feature learning and propose an invariance-variance loss over the detected segments for regularizing feature representations in the target domain. Crucially, this loss structure and network architecture can handle overlapping segments and oversegmentation as produced by Segment Anything. We demonstrate the advantage of our method on the challenging YCB-Video and HomebrewedDB datasets and show that it outperforms prior work and, on YCB-Video, even a network trained with real annotations. Additionally, we provide insight through model ablations and show applicability to a custom robotic application.