RealD$^2$iff: Bridging Real-World Gap in Robot Manipulation via Depth Diffusion

TL;DR

This paper introduces RealD$^2$iff, a diffusion-based framework that synthesizes realistic noisy depth data from simulation, significantly enhancing zero-shot sim2real robot manipulation by bridging the visual gap caused by sensor noise.

Contribution

The work presents a hierarchical diffusion model with novel global and local noise modeling strategies, enabling realistic depth synthesis and zero-shot sim2real transfer in robotic manipulation.

Findings

Effective depth noise synthesis from simulation

Zero-shot sim2real robot manipulation achieved

No manual real sensor data collection needed

Abstract

Robot manipulation in the real world is fundamentally constrained by the visual sim2real gap, where depth observations collected in simulation fail to reflect the complex noise patterns inherent to real sensors. In this work, inspired by the denoising capability of diffusion models, we invert the conventional perspective and propose a clean-to-noisy paradigm that learns to synthesize noisy depth, thereby bridging the visual sim2real gap through purely simulation-driven robotic learning. Building on this idea, we introduce RealD$^2$iff, a hierarchical coarse-to-fine diffusion framework that decomposes depth noise into global structural distortions and fine-grained local perturbations. To enable progressive learning of these components, we further develop two complementary strategies: Frequency-Guided Supervision (FGS) for global structure modeling and Discrepancy-Guided Optimization (DGO) for localized refinement. To integrate RealD$^2$iff seamlessly into imitation learning, we construct a pipeline that spans six stages. We provide comprehensive empirical and experimental validation demonstrating the effectiveness of this paradigm. RealD$^2$iff enables two key applications: (1) generating real-world-like depth to construct clean-noisy paired datasets without manual sensor data collection. (2) Achieving zero-shot sim2real robot manipulation, substantially improving real-world performance without additional fine-tuning.