Adapter-Enhanced Semantic Prompting for Continual Learning

TL;DR

This paper introduces Adapter-Enhanced Semantic Prompting (AESP), a lightweight continual learning framework that combines prompt tuning and adapters to mitigate catastrophic forgetting with efficient semantic-guided feature adaptation.

Contribution

It proposes a novel lightweight CL method integrating semantic prompts and adapters, with a matching mechanism for prompt selection, improving continual learning performance.

Findings

Achieves favorable performance across multiple CL datasets.

Reduces memory requirements compared to traditional methods.

Effectively mitigates catastrophic forgetting.

Abstract

Continual learning (CL) enables models to adapt to evolving data streams. A major challenge of CL is catastrophic forgetting, where new knowledge will overwrite previously acquired knowledge. Traditional methods usually retain the past data for replay or add additional branches in the model to learn new knowledge, which has high memory requirements. In this paper, we propose a novel lightweight CL framework, Adapter-Enhanced Semantic Prompting (AESP), which integrates prompt tuning and adapter techniques. Specifically, we design semantic-guided prompts to enhance the generalization ability of visual features and utilize adapters to efficiently fuse the semantic information, aiming to learn more adaptive features for the continual learning task. Furthermore, to choose the right task prompt for feature adaptation, we have developed a novel matching mechanism for prompt selection. Extensive experiments on three CL datasets demonstrate that our approach achieves favorable performance across multiple metrics, showing its potential for advancing CL.