$\pi_\texttt{RL}$: Online RL Fine-tuning for Flow-based Vision-Language-Action Models

TL;DR

This paper introduces $ exttt{pi}_ exttt{RL}$, a novel reinforcement learning framework for flow-based vision-language-action models, addressing intractable likelihoods with flow-noise and flow-SDE methods, leading to improved robotic task performance.

Contribution

The paper proposes two innovative techniques, flow-noise and flow-SDE, to enable RL fine-tuning of large-scale flow-based VLAs, overcoming likelihood intractability issues.

Findings

RL improves performance in diverse benchmarks.

Flow-noise enables exact likelihood computation.

Flow-SDE facilitates efficient exploration.

Abstract

Vision-Language-Action (VLA) models enable robots to understand and perform complex tasks from multimodal input. Although recent work explores using reinforcement learning (RL) to automate the laborious data collection process in scaling supervised fine-tuning (SFT), applying RL to large-scale flow-based VLAs (\eg, $\pi_0$, $\pi_{0.5}$) remains challenging due to intractable action log-likelihoods raised from flow matching. We address this challenge with $\pi_{\texttt{RL}}$, featuring two technical approaches: (1) \textbf{Flow-Noise} models the denoising process as a discrete-time MDP with a learnable noise network for exact log-likelihood computation. (2) \textbf{Flow-SDE} integrates denoising with agent-environment interaction, formulating a two-layer MDP that employs ODE-to-SDE conversion for efficient RL exploration. We evaluate $\pi_{\texttt{RL}}$ across various benchmarks, with experiments demonstrating that RL yields significant performance improvements in both in-distribution and out-of-distribution settings.