PMoE: Progressive Mixture of Experts with Asymmetric Transformer for Continual Learning

TL;DR

The paper introduces PMoE, a novel model that reduces catastrophic forgetting in continual learning by using an asymmetric transformer architecture with progressive experts and an efficient routing mechanism.

Contribution

It proposes a new asymmetric transformer design with progressive experts and a specialized router to improve continual learning performance.

Findings

PMoE outperforms previous state-of-the-art methods on TRACE and language understanding datasets.

The asymmetric design effectively minimizes forgetting by separating general and new knowledge.

Progressive experts enhance the model's ability to incorporate new information without overwriting existing knowledge.

Abstract

Large Language Models (LLMs) encounter significant challenges in continual learning due to catastrophic forgetting, where new information overwrites previously acquired knowledge. This limitation leads to substantial environmental and economic waste. In this study, we introduce the PMoE, Progressive Mixture of Experts with Asymmetric Transformer, which aims to minimize forgetting by utilizing an asymmetric design with shallow layers dedicated to general knowledge and deep layers for new knowledge. PMoE incorporates progressively added experts in deep layers and a router that allocates new knowledge to the appropriate experts efficiently. The router, positioned adjacent to the deep layers, utilizes deep features aggregating consolidated information. This enables the router to perform efficiently, allocating new knowledge to the appropriate experts, which progressively increase in the deep layers. Extensive experiments on TRACE datasets and general language understanding datasets demonstrate that the proposed PMoE outperforms previous state-of-the-art approaches.