Mean Flow Policy with Instantaneous Velocity Constraint for One-step Action Generation

TL;DR

This paper introduces the mean velocity policy (MVP), a novel flow-based reinforcement learning policy that models mean velocity fields with an instantaneous velocity constraint, enabling fast one-step action generation and improved expressiveness.

Contribution

The paper proposes MVP, a new flow-based policy with an instantaneous velocity constraint, enhancing expressiveness and speed in one-step action generation for RL.

Findings

Achieves state-of-the-art success rates in robotic manipulation tasks.

Provides substantial speed improvements in training and inference.

Theoretically proves the effectiveness of the velocity constraint as a boundary condition.

Abstract

Learning expressive and efficient policy functions is a promising direction in reinforcement learning (RL). While flow-based policies have recently proven effective in modeling complex action distributions with a fast deterministic sampling process, they still face a trade-off between expressiveness and computational burden, which is typically controlled by the number of flow steps. In this work, we propose mean velocity policy (MVP), a new generative policy function that models the mean velocity field to achieve the fastest one-step action generation. To ensure its high expressiveness, an instantaneous velocity constraint (IVC) is introduced on the mean velocity field during training. We theoretically prove that this design explicitly serves as a crucial boundary condition, thereby improving learning accuracy and enhancing policy expressiveness. Empirically, our MVP achieves state-of-the-art success rates across several challenging robotic manipulation tasks from Robomimic and OGBench. It also delivers substantial improvements in training and inference speed over existing flow-based policy baselines.