RGB-D Object Detection and Semantic Segmentation for Autonomous Manipulation in Clutter

TL;DR

This paper presents a deep-learning approach combining object detection and semantic segmentation using RGB-D data for improved autonomous manipulation in cluttered environments, validated on challenging datasets including Amazon Picking Challenge and disaster scenarios.

Contribution

The paper introduces a novel RGB-D perception method that fuses depth information and leverages pretrained features to enhance object detection and segmentation in cluttered scenes.

Findings

Achieved high accuracy in object detection and segmentation in cluttered scenes.

Demonstrated robustness on Amazon Picking Challenge and disaster-response datasets.

Combined perception methods improve reliability of robotic manipulation.

Abstract

Autonomous robotic manipulation in clutter is challenging. A large variety of objects must be perceived in complex scenes, where they are partially occluded and embedded among many distractors, often in restricted spaces. To tackle these challenges, we developed a deep-learning approach that combines object detection and semantic segmentation. The manipulation scenes are captured with RGB-D cameras, for which we developed a depth fusion method. Employing pretrained features makes learning from small annotated robotic data sets possible. We evaluate our approach on two challenging data sets: one captured for the Amazon Picking Challenge 2016, where our team NimbRo came in second in the Stowing and third in the Picking task, and one captured in disaster-response scenarios. The experiments show that object detection and semantic segmentation complement each other and can be combined to yield reliable object perception.