ReMAP-DP: Reprojected Multi-view Aligned PointMaps for Diffusion Policy

TL;DR

ReMAP-DP introduces a novel framework combining perspective reprojection and a dual-stream diffusion policy to enhance 3D spatial awareness in robot manipulation tasks, outperforming existing methods in simulation and real-world scenarios.

Contribution

It presents a new approach that fuses semantic and geometric features using a structure-aware dual-stream diffusion policy with perspective reprojection.

Findings

Achieves 59.3% success rate on RoboTwin 2.0, surpassing DP3 by 6.6%.

Improves performance by 28% on ManiSkill 3 Stack Cube task.

Demonstrates robustness and data efficiency in real-world high-precision manipulations.

Abstract

Generalist robot policies built upon 2D visual representations excel at semantic reasoning but inherently lack the explicit 3D spatial awareness required for high-precision tasks. Existing 3D integration methods struggle to bridge this gap due to the structural irregularity of sparse point clouds and the geometric distortion introduced by multi-view orthographic rendering. To overcome these barriers, we present ReMAP-DP, a novel framework synergizing standardized perspective reprojection with a structure-aware dual-stream diffusion policy. By coupling the re-projected views with pixel-aligned PointMaps, our dual-stream architecture leverages learnable modality embeddings to fuse frozen semantic features and explicit geometric descriptors, ensuring precise implicit patch-level alignment. Extensive experiments across simulation and real-world environments demonstrate ReMAP-DP's superior performance in diverse manipulation tasks. On RoboTwin 2.0, it attains a 59.3% average success rate, outperforming the DP3 baseline by +6.6%. On ManiSkill 3, our method yields a 28% improvement over DP3 on the geometrically challenging Stack Cube task. Furthermore, ReMAP-DP exhibits remarkable real-world robustness, executing high-precision and dynamic manipulations with superior data efficiency from only a handful of demonstrations. Project page is available at: https://icr-lab.github.io/ReMAP-DP/