Analytic Task Scheduler: Recursive Least Squares Based Method for Continual Learning in Embodied Foundation Models

TL;DR

The paper introduces the Analytic Task Scheduler (ATS), a recursive least squares-based framework that enables continual learning in embodied foundation models, effectively preventing forgetting and improving task adaptability in robotic systems.

Contribution

It presents a novel recursive least squares-based scheduler that dynamically selects task-specific models, avoiding interference and forgetting in continual learning for embodied AI.

Findings

ATS demonstrates superior resistance to catastrophic forgetting.

The framework achieves accurate task recognition and model selection.

Validated on a real robot platform with strong adaptability.

Abstract

Embodied foundation models are crucial for Artificial Intelligence (AI) interacting with the physical world by integrating multi-modal inputs, such as proprioception, vision and language, to understand human intentions and generate actions to control robots. While these models demonstrate strong generalization and few-shot learning capabilities, they face significant challenges in continually acquiring new skills without forgetting previously learned skills, a problem known as catastrophic forgetting. To address this issue, we propose the Analytic Task Scheduler (ATS), a novel framework for continual learning in embodied foundation models. ATS consists of a task-specific model library, where each model is fine-tuned independently on a single task, and an analytic scheduler trained using recursive least squares (RLS) to learn the mapping between language instructions and task-specific models. This architecture enables accurate task recognition and dynamic model selection while fundamentally avoiding parameter interference across tasks. The scheduler updates its parameters incrementally using only statistics (autocorrelation and cross-correlation matrices), enabling forgetting-resistant learning without the need to revisit historical data. We validate ATS on a real-world robot platform (RM65B), demonstrating superior resistance to forgetting and strong adaptability to task variations. The results highlight ATS as an effective, scalable, and deployable solution for continual learning in embodied foundation models operating in complex, dynamic environments. Our code will be available at https://github.com/MIAA-Embodied-AI/AnalyticTaskScheduler