VTAO-BiManip: Masked Visual-Tactile-Action Pre-training with Object Understanding for Bimanual Dexterous Manipulation

TL;DR

This paper presents VTAO-BiManip, a novel pretraining framework combining visual, tactile, and action data with object understanding, enabling more effective bimanual manipulation in robotics through curriculum RL and multimodal prediction.

Contribution

The paper introduces a new multimodal pretraining approach with a two-stage curriculum RL for improved bimanual manipulation, incorporating hand motion data and cross-modal regularization.

Findings

Achieves over 20% higher success rate than previous methods.

Effective in both simulated and real-world bottle-cap unscrewing tasks.

Enhances dual-hand coordination through multimodal pretraining.

Abstract

Bimanual dexterous manipulation remains significant challenges in robotics due to the high DoFs of each hand and their coordination. Existing single-hand manipulation techniques often leverage human demonstrations to guide RL methods but fail to generalize to complex bimanual tasks involving multiple sub-skills. In this paper, we introduce VTAO-BiManip, a novel framework that combines visual-tactile-action pretraining with object understanding to facilitate curriculum RL to enable human-like bimanual manipulation. We improve prior learning by incorporating hand motion data, providing more effective guidance for dual-hand coordination than binary tactile feedback. Our pretraining model predicts future actions as well as object pose and size using masked multimodal inputs, facilitating cross-modal regularization. To address the multi-skill learning challenge, we introduce a two-stage curriculum RL approach to stabilize training. We evaluate our method on a bottle-cap unscrewing task, demonstrating its effectiveness in both simulated and real-world environments. Our approach achieves a success rate that surpasses existing visual-tactile pretraining methods by over 20%.