Towards Exploratory and Focused Manipulation with Bimanual Active Perception: A New Problem, Benchmark and Strategy

TL;DR

This paper introduces the Exploratory and Focused Manipulation (EFM) problem, establishes the EFM-10 benchmark with related tasks, and proposes the Bimanual Active Perception (BAP) strategy utilizing dual-arm manipulation and perception to improve task performance.

Contribution

It defines a new fundamental problem in manipulation, creates a benchmark dataset, and proposes a novel bimanual active perception strategy for improved exploration and focus during manipulation.

Findings

BAP strategy effectively improves manipulation performance.

EFM-10 benchmark provides a new standard for exploration-focused tasks.

Dataset supports training and evaluation of active perception methods.

Abstract

Recently, active vision has reemerged as an important concept for manipulation, since visual occlusion occurs more frequently when main cameras are mounted on the robot heads. We reflect on the visual occlusion issue and identify its essence as the absence of information useful for task completion. Inspired by this, we come up with the more fundamental problem of Exploratory and Focused Manipulation (EFM). The proposed problem is about actively collecting information to complete challenging manipulation tasks that require exploration or focus. As an initial attempt to address this problem, we establish the EFM-10 benchmark that consists of 4 categories of tasks that align with our definition (10 tasks in total). We further come up with a Bimanual Active Perception (BAP) strategy, which leverages one arm to provide active vision and another arm to provide force sensing while manipulating. Based on this idea, we collect a dataset named BAPData for the tasks in EFM-10. With the dataset, we successfully verify the effectiveness of the BAP strategy in an imitation learning manner. We hope that the EFM-10 benchmark along with the BAP strategy can become a cornerstone that facilitates future research towards this direction. Project website: EFManipulation.github.io.