PocketDP3: Efficient Pocket-Scale 3D Visuomotor Policy

TL;DR

PocketDP3 introduces a lightweight 3D diffusion policy with a novel MLP-Mixer based architecture, achieving state-of-the-art robotic manipulation performance with significantly fewer parameters and faster inference, suitable for real-time applications.

Contribution

The paper proposes a compact 3D diffusion policy architecture replacing heavy decoders with a lightweight Diffusion Mixer, enabling efficient, real-time robotic manipulation.

Findings

Achieves state-of-the-art results on three benchmarks.

Uses less than 1% of parameters compared to prior methods.

Supports two-step inference without performance loss.

Abstract

Recently, 3D vision-based diffusion policies have shown strong capability in learning complex robotic manipulation skills. However, a common architectural mismatch exists in these models: a tiny yet efficient point-cloud encoder is often paired with a massive decoder. Given a compact scene representation, we argue that this may lead to substantial parameter waste in the decoder. Motivated by this observation, we propose PocketDP3, a pocket-scale 3D diffusion policy that replaces the heavy conditional U-Net decoder used in prior methods with a lightweight Diffusion Mixer (DiM) built on MLP-Mixer blocks. This architecture enables efficient fusion across temporal and channel dimensions, significantly reducing model size. Notably, without any additional consistency distillation techniques, our method supports two-step inference without sacrificing performance, improving practicality for real-time deployment. Across three simulation benchmarks--RoboTwin2.0, Adroit, and MetaWorld--PocketDP3 achieves state-of-the-art performance with fewer than 1% of the parameters of prior methods, while also accelerating inference. Real-world experiments further demonstrate the practicality and transferability of our method in real-world settings. Code will be released.