$\pi$-StepNFT: Wider Space Needs Finer Steps in Online RL for Flow-based VLAs

TL;DR

This paper introduces $ ext{π}$-StepNFT, a new online RL framework for flow-based VLAs that improves exploration and generalization by using step-wise guidance without likelihoods or auxiliary networks.

Contribution

It proposes a critic- and likelihood-free method that enables finer exploration in online RL for flow-based VLAs, enhancing robustness and out-of-distribution generalization.

Findings

Achieves competitive few-shot robustness on LIBERO.

Outperforms value-based baselines on ManiSkill in OOD scenarios.

Eliminates the need for auxiliary value networks.

Abstract

Flow-based vision-language-action (VLA) models excel in embodied control but suffer from intractable likelihoods during multi-step sampling, hindering online reinforcement learning. We propose \textbf{\textit{$\boldsymbol{\pi}$-StepNFT}} (Step-wise Negative-aware Fine-Tuning), a critic-and-likelihood-free framework that requires only a single forward pass per optimization step and eliminates auxiliary value networks. We identify that wider exploration spaces necessitate finer-grained, step-wise guidance for alignment. Empirically, $\pi$-StepNFT unlocks latent potential on LIBERO with competitive few-shot robustness. Moreover, it achieves superior generalization on ManiSkill, outperforming value-based baselines in OOD scenarios by preventing overfitting to multimodal features. This property offers a scalable solution promising for complex real-world applications.