Catch It! Learning to Catch in Flight with Mobile Dexterous Hands

TL;DR

This paper presents a two-stage reinforcement learning approach for a mobile robot with a dexterous hand to catch objects in flight, demonstrating high success rates in simulation and real-world tests.

Contribution

It introduces a novel two-stage RL framework for whole-body control of a mobile manipulator to catch flying objects, with training in simulation and successful real-world deployment.

Findings

Achieved about 80% success rate in simulation.

Demonstrated real-world catching of various objects.

Enhanced policy adaptivity through randomized training.

Abstract

Catching objects in flight (i.e., thrown objects) is a common daily skill for humans, yet it presents a significant challenge for robots. This task requires a robot with agile and accurate motion, a large spatial workspace, and the ability to interact with diverse objects. In this paper, we build a mobile manipulator composed of a mobile base, a 6-DoF arm, and a 12-DoF dexterous hand to tackle such a challenging task. We propose a two-stage reinforcement learning framework to efficiently train a whole-body-control catching policy for this high-DoF system in simulation. The objects' throwing configurations, shapes, and sizes are randomized during training to enhance policy adaptivity to various trajectories and object characteristics in flight. The results show that our trained policy catches diverse objects with randomly thrown trajectories, at a high success rate of about 80\% in simulation, with a significant improvement over the baselines. The policy trained in simulation can be directly deployed in the real world with onboard sensing and computation, which achieves catching sandbags in various shapes, randomly thrown by humans. Our project page is available at https://mobile-dex-catch.github.io/.