DexSim2Real: Foundation Model-Guided Sim-to-Real Transfer for Generalizable Dexterous Manipulation

TL;DR

DexSim2Real introduces a novel framework leveraging foundation models for improved sim-to-real transfer in dexterous manipulation, combining visual realism, cross-modal policies, and curriculum learning.

Contribution

The paper presents an integrated approach using foundation models for domain randomization, visuo-tactile policy fusion, and task curriculum to enhance transferability across diverse manipulation tasks.

Findings

Achieves 78.2% success rate on real robots across six tasks.

Reduces sim-to-real gap to 8.3%, outperforming prior methods.

Demonstrates effectiveness of foundation model-guided domain randomization.

Abstract

Sim-to-real transfer remains a critical bottleneck for deploying dexterous manipulation policies learned in simulation to real-world robots. Existing approaches rely on manually designed domain randomization or task-specific adaptation, limiting their generalizability across diverse manipulation scenarios. We present DexSim2Real, an integrated framework that leverages vision-language foundation models to bridge the sim-to-real gap for dexterous manipulation. Our system combines three components: (1) Foundation Model-Guided Domain Randomization (FM-DR), which uses a vision-language model as a visual realism critic to optimize simulation parameters via closed-loop CMA-ES, complementing text-based approaches like DrEureka with direct visual feedback; (2) a Tactile-Visual Cross-Attention Policy (TVCAP) that adapts cross-attention visuo-tactile fusion to zero-shot sim-to-real RL; and (3) a Progressive Skill Curriculum (PSC) that builds on LLM-based task decomposition with a difficulty scheduler tailored to contact-rich dexterous tasks. Extensive experiments on six challenging manipulation tasks with blinded evaluation demonstrate that DexSim2Real achieves a 78.2% average real-world success rate, outperforming DrEureka and DeXtreme while reducing the sim-to-real performance gap to only 8.3%.