Incrementally Learning Multiple Diverse Data Domains via Multi-Source Dynamic Expansion Model

TL;DR

This paper introduces MSDEM, a novel continual learning framework that effectively integrates multiple data domains using dynamic expansion, attention, and routing mechanisms, achieving state-of-the-art results in complex multi-domain scenarios.

Contribution

The paper proposes MSDEM, a multi-source continual learning model with dynamic expansion, attention, and routing, enabling effective learning across diverse data domains.

Findings

Achieves state-of-the-art performance on multi-domain continual learning tasks.

Effectively reuses previous knowledge to improve generalization.

Demonstrates robustness in complex, multi-domain environments.

Abstract

Continual Learning seeks to develop a model capable of incrementally assimilating new information while retaining prior knowledge. However, current research predominantly addresses a straightforward learning context, wherein all data samples originate from a singular data domain. This paper shifts focus to a more complex and realistic learning environment, characterized by data samples sourced from multiple distinct domains. We tackle this intricate learning challenge by introducing a novel methodology, termed the Multi-Source Dynamic Expansion Model (MSDEM), which leverages various pre-trained models as backbones and progressively establishes new experts based on them to adapt to emerging tasks. Additionally, we propose an innovative dynamic expandable attention mechanism designed to selectively harness knowledge from multiple backbones, thereby accelerating the new task learning. Moreover, we introduce a dynamic graph weight router that strategically reuses all previously acquired parameters and representations for new task learning, maximizing the positive knowledge transfer effect, which further improves generalization performance. We conduct a comprehensive series of experiments, and the empirical findings indicate that our proposed approach achieves state-of-the-art performance.