Self-supervised Transfer Learning for Instance Segmentation through Physical Interaction

TL;DR

This paper introduces a self-supervised transfer learning method enabling robots to perform instance segmentation of unknown objects through physical interaction, reducing reliance on manual labeling and improving performance in cluttered scenes.

Contribution

The authors propose a novel self-supervised transfer learning approach that fine-tunes a segmentation network using robot-acquired interaction data, outperforming standard models trained on large labeled datasets.

Findings

SelfDeepMask outperforms DeepMask trained on COCO by 9.5% in average precision.

Robot interaction data can effectively generate training labels for segmentation.

Combining self-supervised learning with noisy label training improves robustness.

Abstract

Instance segmentation of unknown objects from images is regarded as relevant for several robot skills including grasping, tracking and object sorting. Recent results in computer vision have shown that large hand-labeled datasets enable high segmentation performance. To overcome the time-consuming process of manually labeling data for new environments, we present a transfer learning approach for robots that learn to segment objects by interacting with their environment in a self-supervised manner. Our robot pushes unknown objects on a table and uses information from optical flow to create training labels in the form of object masks. To achieve this, we fine-tune an existing DeepMask network for instance segmentation on the self-labeled training data acquired by the robot. We evaluate our trained network (SelfDeepMask) on a set of real images showing challenging and cluttered scenes with novel objects. Here, SelfDeepMask outperforms the DeepMask network trained on the COCO dataset by 9.5% in average precision. Furthermore, we combine our approach with recent approaches for training with noisy labels in order to better cope with induced label noise.