ALOE: Action-Level Off-Policy Evaluation for Vision-Language-Action Model Post-Training

TL;DR

ALOE introduces an action-level off-policy evaluation method for vision-language-action models, enabling more effective reinforcement learning in real-world tasks by evaluating individual actions rather than entire trajectories.

Contribution

The paper proposes ALOE, a novel chunking-based temporal-difference bootstrapping framework for off-policy evaluation of individual actions in VLA systems, enhancing learning stability and efficiency.

Findings

Improves learning efficiency across diverse real-world tasks.

Supports stable policy improvement without sacrificing execution speed.

Effective in high-precision, long-horizon, and multi-object perception tasks.

Abstract

We study how to improve large foundation vision-language-action (VLA) systems through online reinforcement learning (RL) in real-world settings. Central to this process is the value function, which provides learning signals to guide VLA learning from experience. In practice, the value function is estimated from trajectory fragments collected from different data sources, including historical policies and intermittent human interventions. Estimating the value function of current behavior quality from the mixture data is inherently an off-policy evaluation problem. However, prior work often adopts conservative on-policy estimation for stability, which avoids direct evaluation of the current high-capacity policy and limits learning effectiveness. In this paper, we propose ALOE, an action-level off-policy evaluation framework for VLA post-training. ALOE applies chunking-based temporal-difference bootstrapping to evaluate individual action sequences instead of predicting final task outcomes. This design improves effective credit assignment to critical action chunks under sparse rewards and supports stable policy improvement. We evaluate our method on three real-world manipulation tasks, including smartphone packing as a high-precision task, laundry folding as a long-horizon deformable-object task, and bimanual pick-and-place involving multi-object perception. Across all tasks, ALOE improves learning efficiency without compromising execution speed, showing that off-policy RL can be reintroduced in a reliable manner for real-world VLA post-training. Videos and additional materials are available at our project website.