DeXtreme: Transfer of Agile In-hand Manipulation from Simulation to Reality
Ankur Handa, Arthur Allshire, Viktor Makoviychuk, Aleksei Petrenko,, Ritvik Singh, Jingzhou Liu, Denys Makoviichuk, Karl Van Wyk, Alexander, Zhurkevich, Balakumar Sundaralingam, Yashraj Narang, Jean-Francois Lafleche,, Dieter Fox, Gavriel State

TL;DR
This paper demonstrates successful transfer of deep reinforcement learning policies for dexterous in-hand manipulation from simulation to real-world robots, using techniques that enhance robustness and generalization across hardware and simulation environments.
Contribution
The authors introduce methods for training robust vision-based manipulation policies and pose estimators that transfer effectively from simulation to real robots, specifically with the Allegro Hand and Isaac Gym.
Findings
Vision policies outperform existing literature on reorientation tasks.
Policies are competitive with motion capture-based methods.
The approach enables sim-to-real transfer with affordable hardware.
Abstract
Recent work has demonstrated the ability of deep reinforcement learning (RL) algorithms to learn complex robotic behaviours in simulation, including in the domain of multi-fingered manipulation. However, such models can be challenging to transfer to the real world due to the gap between simulation and reality. In this paper, we present our techniques to train a) a policy that can perform robust dexterous manipulation on an anthropomorphic robot hand and b) a robust pose estimator suitable for providing reliable real-time information on the state of the object being manipulated. Our policies are trained to adapt to a wide range of conditions in simulation. Consequently, our vision-based policies significantly outperform the best vision policies in the literature on the same reorientation task and are competitive with policies that are given privileged state information via motion capture…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Code & Models
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsRobot Manipulation and Learning · Human Pose and Action Recognition · Human Motion and Animation
