Self-Supervised Interactive Object Segmentation Through a Singulation-and-Grasping Approach

TL;DR

This paper introduces a robot learning approach that actively interacts with cluttered objects to improve instance segmentation performance without manual labeling, using a singulation-and-grasping policy trained via reinforcement learning.

Contribution

It proposes a novel singulation-and-grasping policy for active data collection and self-supervised learning to enhance segmentation of unseen objects in unstructured environments.

Findings

Achieves 70% success in object singulation in simulation.

Attains 87.8% average precision on toy blocks.

Outperforms baseline methods in real-world segmentation tasks.

Abstract

Instance segmentation with unseen objects is a challenging problem in unstructured environments. To solve this problem, we propose a robot learning approach to actively interact with novel objects and collect each object's training label for further fine-tuning to improve the segmentation model performance, while avoiding the time-consuming process of manually labeling a dataset. The Singulation-and-Grasping (SaG) policy is trained through end-to-end reinforcement learning. Given a cluttered pile of objects, our approach chooses pushing and grasping motions to break the clutter and conducts object-agnostic grasping for which the SaG policy takes as input the visual observations and imperfect segmentation. We decompose the problem into three subtasks: (1) the object singulation subtask aims to separate the objects from each other, which creates more space that alleviates the difficulty of (2) the collision-free grasping subtask; (3) the mask generation subtask to obtain the self-labeled ground truth masks by using an optical flow-based binary classifier and motion cue post-processing for transfer learning. Our system achieves 70% singulation success rate in simulated cluttered scenes. The interactive segmentation of our system achieves 87.8%, 73.9%, and 69.3% average precision for toy blocks, YCB objects in simulation and real-world novel objects, respectively, which outperforms several baselines.