Linked Adapters: Linking Past and Future to Present for Effective Continual Learning

TL;DR

Linked Adapters introduce a novel attention-based mechanism that enables effective knowledge transfer across task-specific adapters in continual learning, reducing catastrophic forgetting and improving performance on image classification tasks.

Contribution

The paper proposes Linked Adapters, a new method that facilitates backward and forward knowledge transfer among task-specific adapters using MLP-based attention in continual learning.

Findings

Significant performance improvement on diverse image classification datasets.

Effective mitigation of catastrophic forgetting in continual learning.

Demonstrated superiority over existing adapter-based approaches.

Abstract

Continual learning allows the system to learn and adapt to new tasks while retaining the knowledge acquired from previous tasks. However, deep learning models suffer from catastrophic forgetting of knowledge learned from earlier tasks while learning a new task. Moreover, retraining large models like transformers from scratch for every new task is costly. An effective approach to address continual learning is to use a large pre-trained model with task-specific adapters to adapt to the new tasks. Though this approach can mitigate catastrophic forgetting, they fail to transfer knowledge across tasks as each task is learning adapters separately. To address this, we propose a novel approach Linked Adapters that allows knowledge transfer through a weighted attention mechanism to other task-specific adapters. Linked adapters use a multi-layer perceptron (MLP) to model the attention weights, which overcomes the challenge of backward knowledge transfer in continual learning in addition to modeling the forward knowledge transfer. During inference, our proposed approach effectively leverages knowledge transfer through MLP-based attention weights across all the lateral task adapters. Through numerous experiments conducted on diverse image classification datasets, we effectively demonstrated the improvement in performance on the continual learning tasks using Linked Adapters.