Continually Evolving Skill Knowledge in Vision Language Action Model

TL;DR

Stellar VLA introduces a parameter-efficient continual learning framework for vision-language-action models, enabling knowledge accumulation and task specialization without increasing model size, validated on benchmarks and real-world platforms.

Contribution

It proposes a novel knowledge-driven continual learning approach with hierarchical variants that enhance scalability and knowledge transfer in VLA models.

Findings

Achieves strong performance with only 1% data replay on LIBERO benchmark.

Enables effective knowledge transfer in real-world dual-arm manipulation.

Visualizations show robust knowledge retention and task discovery.

Abstract

Vision-language-action (VLA) models show promising knowledge accumulation ability from pretraining, yet continual learning in VLA remains challenging, especially for efficient adaptation. Existing continual imitation learning (CIL) methods often rely on additional parameters or external modules, limiting scalability for large VLA models. We propose Stellar VLA, a knowledge-driven CIL framework without increasing network parameters. Two progressively extended variants are designed: T-Stellar for flat task-centric modeling and TS-Stellar for hierarchical task-skill structure. Stellar VLA enables self-evolving knowledge learning by jointly optimizing task representations and a learned knowledge space. We propose a knowledge-guided expert routing mechanism conditioned on knowledge relation and Top-K semantic embeddings, enabling task specialization without increasing model size. Experiments on the LIBERO benchmark show that Stellar VLAs achieve strong performance among both VLA and CIL baselines, using only 1 % data replay. Real-world evaluation on a dual-arm platform with distinct embodiment and scene configurations validates effective knowledge transfer. TS-Stellar excels in hierarchical manipulation, and visualizations reveal robust knowledge retention and task discovery. Project Website: https://stellarvla.github.io/