DIMoE-Adapters: Dynamic Expert Evolution for Continual Learning in Vision-Language Models

TL;DR

DIMoE-Adapters introduces a dynamic expert evolution framework for continual learning in vision-language models, effectively balancing stability and plasticity across multiple domains.

Contribution

It proposes a novel dynamic expert evolution paradigm with SCEE and PGES components, enhancing adaptation and reducing catastrophic forgetting in multi-domain continual learning.

Findings

Outperforms state-of-the-art methods in various continual learning settings.

Improves plasticity by evolving a sparse expert pool.

Enhances stability across seen and unseen tasks.

Abstract

Continual learning enables vision-language models to accumulate knowledge and adapt to evolving tasks without retraining from scratch. However, in multi-domain task-incremental learning, large domain shifts intensify the stability-plasticity dilemma. Most existing methods rely on fixed architectures with statically allocated parameters, which limits adaptation to new domains and aggravates catastrophic forgetting. To address these challenges, we propose DIMoE-Adapters, a Dynamic Incremental Mixture-of-Experts Adapters framework that introduces a dynamic expert evolution paradigm to balance stability and plasticity. This paradigm is implemented through two collaborative components: Self-Calibrated Expert Evolution (SCEE) and Prototype-Guided Expert Selection (PGES). SCEE constructs and evolves a sparse expert pool through expert optimization dynamics, improving plasticity while reducing redundant capacity. PGES controls expert utilization based on the pool shaped by SCEE, improving stability across both previously encountered and unseen tasks. Extensive experiments show that DIMoE-Adapters outperforms previous state-of-the-art methods across various settings.