Simple Recipe Works: Vision-Language-Action Models are Natural Continual Learners with Reinforcement Learning

TL;DR

This paper demonstrates that simple sequential fine-tuning with low-rank adaptation on large pretrained vision-language-action models can effectively enable continual learning in reinforcement learning tasks, challenging the need for complex strategies.

Contribution

It shows that straightforward sequential fine-tuning with LoRA is surprisingly effective for continual RL, outperforming more complex methods and offering new insights into lifelong learning with large models.

Findings

Seq. FT with LoRA achieves high plasticity and stability.

Simple method often outperforms complex continual learning strategies.

Robustness stems from synergy between pretrained models, LoRA, and on-policy RL.

Abstract

Continual Reinforcement Learning (CRL) for Vision-Language-Action (VLA) models is a promising direction toward self-improving embodied agents that can adapt in openended, evolving environments. However, conventional wisdom from continual learning suggests that naive Sequential Fine-Tuning (Seq. FT) leads to catastrophic forgetting, necessitating complex CRL strategies. In this work, we take a step back and conduct a systematic study of CRL for large pretrained VLAs across three models and five challenging lifelong RL benchmarks. We find that, contrary to established belief, simple Seq. FT with low-rank adaptation (LoRA) is remarkably strong: it achieves high plasticity, exhibits little to no forgetting, and retains strong zero-shot generalization, frequently outperforming more sophisticated CRL methods. Through detailed analysis, we show that this robustness arises from a synergy between the large pretrained model, parameter-efficient adaptation, and on-policy RL. Together, these components reshape the stability-plasticity trade-off, making continual adaptation both stable and scalable. Our results position Sequential Fine-Tuning as a powerful method for continual RL with VLAs and provide new insights into lifelong learning in the large model era. Code is available at github.com/UT-Austin-RobIn/continual-vla-rl.