Neural Networks Remember More: The Power of Parameter Isolation and Combination

TL;DR

This paper introduces a novel continual learning method for language models that uses parameter isolation and combination to balance stability and plasticity, significantly reducing catastrophic forgetting.

Contribution

It proposes a new approach combining parameter isolation and task arithmetic to improve knowledge retention in continual language learning.

Findings

Outperforms existing state-of-the-art methods on benchmarks

Effectively mitigates catastrophic forgetting

Enhances model stability without sacrificing plasticity

Abstract

Catastrophic forgetting is a pervasive issue for pre-trained language models (PLMs) during continual learning, where models lose previously acquired knowledge when sequentially trained on a series of tasks. The model's ability to retain old tasks is referred to as stability, while its adaptability to new tasks is called plasticity. Therefore, the key to solving this problem is to find a trade-off between the plasticity and stability of the model. To address this issue, in this paper, we propose a novel method to achieve a balance between model stability and plasticity, thereby mitigating catastrophic forgetting. More specifically, our proposed approach leverages parameter isolation and a subsequent combination strategy. Initially, in the training stage, the model adapts to each downstream task via a parameter isolation method to prevent potential interference among different tasks. We then combine all trained parameters, which contain acquired knowledge, using the task arithmetic method and finally apply them to the backbone model. Empirical evaluations on continual language learning benchmarks substantiate the effectiveness of our approach, revealing a marked enhancement over existing state-of-the-art approaches.