VITA: Vision-to-Action Flow Matching Policy

TL;DR

VITA introduces a noise-free, conditioning-free flow matching policy that directly maps visual inputs to actions, significantly reducing inference time while maintaining or improving performance across various tasks.

Contribution

The paper presents VITA, a novel framework that eliminates the need for visual conditioning and iterative denoising in flow matching policies by using an action autoencoder and flow latent decoding.

Findings

VITA achieves 1.5x-2x faster inference than traditional methods.

VITA outperforms or matches state-of-the-art policies on multiple tasks.

The approach effectively bridges vision and action in a unified, efficient framework.

Abstract

Conventional flow matching and diffusion-based policies sample via iterative denoising from standard noise distributions (e.g., Gaussian), and require conditioning modules to repeatedly incorporate visual information during the generative process, incurring substantial time and memory overhead. To reduce the complexity, we develop VITA, VIsion-To-Action policy, a noise-free and conditioning-free flow matching policy learning framework that directly flows from visual representations to latent actions. Since the source of the flow is visually grounded, VITA eliminates the need for visual conditioning during generation. As expected, bridging vision and action is challenging, because actions are lower-dimensional, less structured, and sparser than visual representations; moreover, flow matching requires the source and target to have the same dimensionality. To overcome this, we introduce an action autoencoder that maps raw actions into a structured latent space aligned with visual latents, trained jointly with flow matching. To further prevent latent action space collapse during end-to-end training, we propose flow latent decoding, which anchors the latent generation process by backpropagating the action reconstruction loss through the flow matching ODE (ordinary differential equation) solving steps. We evaluate VITA on 9 simulation and 5 real-world tasks from ALOHA and Robomimic. VITA achieves 1.5x-2x faster inference compared to conventional methods with conditioning modules, while outperforming or matching state-of-the-art policies. Project page: https://ucd-dare.github.io/VITA/.