Learning while Deploying: Fleet-Scale Reinforcement Learning for Generalist Robot Policies

TL;DR

This paper introduces Learning While Deploying (LWD), a fleet-scale reinforcement learning framework that continually improves generalist robot policies through autonomous deployment, human intervention, and online learning across multiple real-world tasks.

Contribution

The paper presents a novel fleet-scale offline-to-online RL framework that enables continual policy improvement for generalist robots during deployment, combining value estimation and policy extraction techniques.

Findings

A single generalist policy achieved 95% success rate across tasks.

LWD improved performance on long-horizon manipulation tasks.

Fleet experience led to continuous policy enhancement.

Abstract

Generalist robot policies increasingly benefit from large-scale pretraining, but offline data alone is insufficient for robust real-world deployment. Deployed robots encounter distribution shifts, long-tail failures, task variations, and human correction opportunities that fixed demonstration datasets cannot fully capture. We present Learning While Deploying (LWD), a fleet-scale offline-to-online reinforcement learning framework for continual post-training of generalist Vision-Language-Action (VLA) policies. Starting from a pretrained VLA policy, LWD closes the loop between deployment, shared physical experience, policy improvement, and redeployment by using autonomous rollouts and human interventions collected across a robot fleet. To stabilize learning from heterogeneous, sparse-reward fleet data, LWD combines Distributional Implicit Value Learning (DIVL) for robust value estimation with Q-learning via Adjoint Matching (QAM) for policy extraction in flow-based VLA action generators. We validate LWD on a fleet of 16 dual-arm robots across eight real-world manipulation tasks, including semantic grocery restocking and 3--5 minute long-horizon tasks. A single generalist policy improves as fleet experience accumulates, reaching an average success rate of 95%, with the largest gains on long-horizon tasks.