LaMP: Learning Vision-Language-Action Policies with 3D Scene Flow as Latent Motion Prior

TL;DR

LaMP introduces a dual-expert framework embedding 3D scene flow as a latent motion prior to improve robotic manipulation, outperforming existing models especially under unfamiliar spatial dynamics.

Contribution

LaMP is the first to integrate dense 3D scene flow as a latent prior in a dual-expert vision-language-action model for robotics.

Findings

Outperforms baseline models on multiple benchmarks.

Achieves highest success rates with same training budgets.

Shows improved robustness under out-of-distribution perturbations.

Abstract

We introduce \textbf{LaMP}, a dual-expert Vision-Language-Action framework that embeds dense 3D scene flow as a latent motion prior for robotic manipulation. Existing VLA models regress actions directly from 2D semantic visual features, forcing them to learn complex 3D physical interactions implicitly. This implicit learning strategy degrades under unfamiliar spatial dynamics. LaMP addresses this limitation by aligning a flow-matching \emph{Motion Expert} with a policy-predicting \emph{Action Expert} through gated cross-attention. Specifically, the Motion Expert generates a one-step partially denoised 3D scene flow, and its hidden states condition the Action Expert without full multi-step reconstruction. We evaluate LaMP on the LIBERO, LIBERO-Plus, and SimplerEnv-WidowX simulation benchmarks as well as real-world experiments. LaMP consistently outperforms evaluated VLA baselines across LIBERO, LIBERO-Plus, and SimplerEnv-WidowX benchmarks, achieving the highest reported average success rates under the same training budgets. On LIBERO-Plus OOD perturbations, LaMP shows improved robustness with an average 9.7% gain over the strongest prior baseline. Our project page is available at https://summerwxk.github.io/lamp-project-page/.