Learning Dexterous Manipulation Skills from Imperfect Simulations

TL;DR

This paper introduces \\ours, a three-stage sim-to-real framework that improves dexterous manipulation by combining simplified simulation, tactile-enabled demonstrations, and behavior cloning, achieving robust real-world performance.

Contribution

The work presents a novel three-stage framework that enhances sim-to-real transfer for dexterous manipulation using tactile feedback and imitation learning.

Findings

High task progress ratios compared to direct sim-to-real transfer

Robust performance on unseen object shapes

Effective generalization to diverse geometries

Abstract

Reinforcement learning and sim-to-real transfer have made significant progress in dexterous manipulation. However, progress remains limited by the difficulty of simulating complex contact dynamics and multisensory signals, especially tactile feedback. In this work, we propose \ours, a sim-to-real framework that addresses these limitations and demonstrates its effectiveness on nut-bolt fastening and screwdriving with multi-fingered hands. The framework has three stages. First, we train reinforcement learning policies in simulation using simplified object models that lead to the emergence of correct finger gaits. We then use the learned policy as a skill primitive within a teleoperation system to collect real-world demonstrations that contain tactile and proprioceptive information. Finally, we train a behavior cloning policy that incorporates tactile sensing and show that it generalizes to nuts and screwdrivers with diverse geometries. Experiments across both tasks show high task progress ratios compared to direct sim-to-real transfer and robust performance even on unseen object shapes and under external perturbations. Videos and code are available on https://dexscrew.github.io.