Hierarchical-Task-Aware Multi-modal Mixture of Incremental LoRA Experts for Embodied Continual Learning

TL;DR

This paper introduces a hierarchical framework and a novel method called Task-aware MoILE for embodied continual learning, enabling agents to learn high-level planning and actions while mitigating catastrophic forgetting.

Contribution

It proposes a hierarchical setup for embodied continual learning and a task-aware MoILE method that uses clustering and SVD to improve knowledge retention.

Findings

Reduces forgetting of old tasks compared to existing methods.

Effectively distinguishes and recognizes tasks using clustering.

Supports continuous learning of new tasks without significant performance loss.

Abstract

Previous continual learning setups for embodied intelligence focused on executing low-level actions based on human commands, neglecting the ability to learn high-level planning and multi-level knowledge. To address these issues, we propose the Hierarchical Embodied Continual Learning Setups (HEC) that divide the agent's continual learning process into two layers: high-level instructions and low-level actions, and define five embodied continual learning sub-setups. Building on these setups, we introduce the Task-aware Mixture of Incremental LoRA Experts (Task-aware MoILE) method. This approach achieves task recognition by clustering visual-text embeddings and uses both a task-level router and a token-level router to select the appropriate LoRA experts. To effectively address the issue of catastrophic forgetting, we apply Singular Value Decomposition (SVD) to the LoRA parameters obtained from prior tasks, preserving key components while orthogonally training the remaining parts. The experimental results show that our method stands out in reducing the forgetting of old tasks compared to other methods, effectively supporting agents in retaining prior knowledge while continuously learning new tasks.