Mixture of Experts Meets Prompt-Based Continual Learning

TL;DR

This paper reveals that the attention mechanism in pre-trained models functions as a mixture of experts, leading to a novel gating method called NoRGa that improves prompt-based continual learning.

Contribution

It provides a theoretical understanding of prompt effectiveness, introduces a new gating mechanism, and demonstrates improved continual learning performance.

Findings

Attention blocks encode mixture of experts architecture.

NoRGa improves continual learning performance.

Theoretical and empirical validation across benchmarks.

Abstract

Exploiting the power of pre-trained models, prompt-based approaches stand out compared to other continual learning solutions in effectively preventing catastrophic forgetting, even with very few learnable parameters and without the need for a memory buffer. While existing prompt-based continual learning methods excel in leveraging prompts for state-of-the-art performance, they often lack a theoretical explanation for the effectiveness of prompting. This paper conducts a theoretical analysis to unravel how prompts bestow such advantages in continual learning, thus offering a new perspective on prompt design. We first show that the attention block of pre-trained models like Vision Transformers inherently encodes a special mixture of experts architecture, characterized by linear experts and quadratic gating score functions. This realization drives us to provide a novel view on prefix tuning, reframing it as the addition of new task-specific experts, thereby inspiring the design of a novel gating mechanism termed Non-linear Residual Gates (NoRGa). Through the incorporation of non-linear activation and residual connection, NoRGa enhances continual learning performance while preserving parameter efficiency. The effectiveness of NoRGa is substantiated both theoretically and empirically across diverse benchmarks and pretraining paradigms. Our code is publicly available at https://github.com/Minhchuyentoancbn/MoE_PromptCL