R3DP: Real-Time 3D-Aware Policy for Embodied Manipulation

TL;DR

R3DP introduces a real-time 3D-aware manipulation policy that efficiently integrates large-scale 3D priors using asynchronous modules, significantly improving success rates and reducing inference time in embodied manipulation tasks.

Contribution

The paper presents R3DP, a novel system combining asynchronous modules and multi-view fusion to incorporate large 3D priors into real-time manipulation policies without latency penalties.

Findings

R3DP outperforms baselines with 32.9% and 51.4% higher success rates in different configurations.

It reduces inference time by 44.8% compared to naive integration methods.

The system effectively leverages temporal and multi-view data for improved manipulation performance.

Abstract

Embodied manipulation requires accurate 3D understanding of objects and their spatial relations to plan and execute contact-rich actions. While large-scale 3D vision models provide strong priors, their computational cost incurs prohibitive latency for real-time control. We propose Real-time 3D-aware Policy (R3DP), which integrates powerful 3D priors into manipulation policies without sacrificing real-time performance. A core innovation of R3DP is the asynchronous fast-slow collaboration module, which seamlessly integrates large-scale 3D priors into the policy without compromising real-time performance. The system maintains real-time efficiency by querying the pre-trained slow system (VGGT) only on sparse key frames, while simultaneously employing a lightweight Temporal Feature Prediction Network (TFPNet) to predict features for all intermediate frames. By leveraging historical data to exploit temporal correlations, TFPNet explicitly improves task success rates through consistent feature estimation. Additionally, to enable more effective multi-view fusion, we introduce a Multi-View Feature Fuser (MVFF) that aggregates features across views by explicitly incorporating camera intrinsics and extrinsics. R3DP offers a plug-and-play solution for integrating large models into real-time inference systems. We evaluate R3DP against multiple baselines across different visual configurations. R3DP effectively harnesses large-scale 3D priors to achieve superior results, outperforming single-view and multi-view DP by 32.9% and 51.4% in average success rate, respectively. Furthermore, by decoupling heavy 3D reasoning from policy execution, R3DP achieves a 44.8% reduction in inference time compared to a naive DP+VGGT integration.