In the Wild Ungraspable Object Picking with Bimanual Nonprehensile Manipulation

TL;DR

This paper presents a bimanual nonprehensile manipulation approach enabling robots to pick ungraspable objects in cluttered grocery scenarios, achieving high success rates without hardware modifications.

Contribution

The work introduces a novel bimanual manipulation method for ungraspable objects, combining visual identification, nonprehensile nudging, and bimanual grasping in real-world grocery settings.

Findings

90% success rate in uncluttered scenes

67% success rate in cluttered scenes

Effective in real-world grocery store scenarios

Abstract

Picking diverse objects in the real world is a fundamental robotics skill. However, many objects in such settings are bulky, heavy, or irregularly shaped, making them ungraspable by conventional end effectors like suction grippers and parallel jaw grippers (PJGs). In this paper, we expand the range of pickable items without hardware modifications using bimanual nonprehensile manipulation. We focus on a grocery shopping scenario, where a bimanual mobile manipulator equipped with a suction gripper and a PJG is tasked with retrieving ungraspable items from tightly packed grocery shelves. From visual observations, our method first identifies optimal grasp points based on force closure and friction constraints. If the grasp points are occluded, a series of nonprehensile nudging motions are performed to clear the obstruction. A bimanual grasp utilizing contacts on the side of the end effectors is then executed to grasp the target item. In our replica grocery store, we achieved a 90% success rate over 102 trials in uncluttered scenes, and a 67% success rate over 45 trials in cluttered scenes. We also deployed our system to a real-world grocery store and successfully picked previously unseen items. Our results highlight the potential of bimanual nonprehensile manipulation for in-the-wild robotic picking tasks. A video summarizing this work can be found at youtu.be/g0hOrDuK8jM