Similarity-Aware Mixture-of-Experts for Data-Efficient Continual Learning

TL;DR

This paper introduces a similarity-aware mixture-of-experts framework for data-efficient continual learning, effectively handling limited data and overlapping tasks by leveraging task similarity and preventing negative transfer.

Contribution

It proposes an adaptive MoE approach with incremental global pooling and instance-wise prompt masking to improve knowledge transfer and task differentiation in continual learning.

Findings

Enhances sample efficiency across various data volumes.

Effectively manages task overlap and prevents negative transfer.

Broad applicability demonstrated in experiments.

Abstract

Machine learning models often need to adapt to new data after deployment due to structured or unstructured real-world dynamics. The Continual Learning (CL) framework enables continuous model adaptation, but most existing approaches either assume each task contains sufficiently many data samples or that the learning tasks are non-overlapping. In this paper, we address the more general setting where each task may have a limited dataset, and tasks may overlap in an arbitrary manner without a priori knowledge. This general setting is substantially more challenging for two reasons. On the one hand, data scarcity necessitates effective contextualization of general knowledge and efficient knowledge transfer across tasks. On the other hand, unstructured task overlapping can easily result in negative knowledge transfer. To address the above challenges, we propose an adaptive mixture-of-experts (MoE) framework over pre-trained models that progressively establishes similarity awareness among tasks. Our design contains two innovative algorithmic components: incremental global pooling and instance-wise prompt masking. The former mitigates prompt association noise through gradual prompt introduction over time. The latter decomposes incoming task samples into those aligning with current prompts (in-distribution) and those requiring new prompts (out-of-distribution). Together, our design strategically leverages potential task overlaps while actively preventing negative mutual interference in the presence of per-task data scarcity. Experiments across varying data volumes and inter-task similarity show that our method enhances sample efficiency and is broadly applicable.