Real-to-Sim Grasp: Rethinking the Gap between Simulation and Real World in Grasp Detection

TL;DR

This paper introduces R2SGrasp, a novel real-to-sim framework for 6-DoF grasp detection that directly mitigates the simulation-to-real gap by real-time data repair and feature enhancement, improving real-world grasping performance.

Contribution

The work proposes a real-to-sim approach with data repair and feature enhancement modules, bypassing the noisy simulation-to-real adaptation, and constructs a large-scale simulated dataset for training.

Findings

R2SGrasp achieves strong real-world grasping performance.

The framework effectively mitigates camera noise in real data.

Large-scale dataset improves model generalization.

Abstract

For 6-DoF grasp detection, simulated data is expandable to train more powerful model, but it faces the challenge of the large gap between simulation and real world. Previous works bridge this gap with a sim-to-real way. However, this way explicitly or implicitly forces the simulated data to adapt to the noisy real data when training grasp detectors, where the positional drift and structural distortion within the camera noise will harm the grasp learning. In this work, we propose a Real-to-Sim framework for 6-DoF Grasp detection, named R2SGrasp, with the key insight of bridging this gap in a real-to-sim way, which directly bypasses the camera noise in grasp detector training through an inference-time real-to-sim adaption. To achieve this real-to-sim adaptation, our R2SGrasp designs the Real-to-Sim Data Repairer (R2SRepairer) to mitigate the camera noise of real depth maps in data-level, and the Real-to-Sim Feature Enhancer (R2SEnhancer) to enhance real features with precise simulated geometric primitives in feature-level. To endow our framework with the generalization ability, we construct a large-scale simulated dataset cost-efficiently to train our grasp detector, which includes 64,000 RGB-D images with 14.4 million grasp annotations. Sufficient experiments show that R2SGrasp is powerful and our real-to-sim perspective is effective. The real-world experiments further show great generalization ability of R2SGrasp. Project page is available on https://isee-laboratory.github.io/R2SGrasp.