RoboPaint: From Human Demonstration to Any Robot and Any View

TL;DR

This paper introduces RoboPaint, a scalable pipeline that converts human demonstrations into robot training data for dexterous manipulation, bypassing the need for direct robot teleoperation and enabling effective policy learning.

Contribution

The authors present a novel real-sim-real data collection and editing pipeline that transforms human demonstrations into environment-specific robot training data, including a tactile-aware retargeting method and photorealistic simulation rendering.

Findings

84% success rate in 10 manipulation tasks

80% average success rate of policies trained on generated data

Efficient and cost-effective alternative to teleoperation

Abstract

Acquiring large-scale, high-fidelity robot demonstration data remains a critical bottleneck for scaling Vision-Language-Action (VLA) models in dexterous manipulation. We propose a Real-Sim-Real data collection and data editing pipeline that transforms human demonstrations into robot-executable, environment-specific training data without direct robot teleoperation. Standardized data collection rooms are built to capture multimodal human demonstrations (synchronized 3 RGB-D videos, 11 RGB videos, 29-DoF glove joint angles, and 14-channel tactile signals). Based on these human demonstrations, we introduce a tactile-aware retargeting method that maps human hand states to robot dex-hand states via geometry and force-guided optimization. Then the retargeted robot trajectories are rendered in a photorealistic Isaac Sim environment to build robot training data. Real world experiments have demonstrated: (1) The retargeted dex-hand trajectories achieve an 84\% success rate across 10 diverse object manipulation tasks. (2) VLA policies (Pi0.5) trained exclusively on our generated data achieve 80\% average success rate on three representative tasks, i.e., pick-and-place, pushing and pouring. To conclude, robot training data can be efficiently "painted" from human demonstrations using our real-sim-real data pipeline. We offer a scalable, cost-effective alternative to teleoperation with minimal performance loss for complex dexterous manipulation.