PECTP: Parameter-Efficient Cross-Task Prompts for Incremental Vision Transformer

TL;DR

This paper introduces PECTP, a parameter-efficient prompt framework for incremental vision transformers that balances memory use and task performance by retaining cross-task prompts and classifier heads.

Contribution

The paper proposes a novel PECTP framework with PRM and HRM modules, enabling effective incremental learning with limited memory and improved generalization across tasks.

Findings

Outperforms existing prompt-based IL methods in accuracy.

Reduces memory requirements compared to prompt-extending methods.

Enhances cross-task generalization through prompt and classifier head retention.

Abstract

Incremental Learning (IL) aims to learn deep models on sequential tasks continually, where each new task includes a batch of new classes and deep models have no access to task-ID information at the inference time. Recent vast pre-trained models (PTMs) have achieved outstanding performance by prompt technique in practical IL without the old samples (rehearsal-free) and with a memory constraint (memory-constrained): Prompt-extending and Prompt-fixed methods. However, prompt-extending methods need a large memory buffer to maintain an ever-expanding prompt pool and meet an extra challenging prompt selection problem. Prompt-fixed methods only learn a single set of prompts on one of the incremental tasks and can not handle all the incremental tasks effectively. To achieve a good balance between the memory cost and the performance on all the tasks, we propose a Parameter-Efficient Cross-Task Prompt (PECTP) framework with Prompt Retention Module (PRM) and classifier Head Retention Module (HRM). To make the final learned prompts effective on all incremental tasks, PRM constrains the evolution of cross-task prompts' parameters from Outer Prompt Granularity and Inner Prompt Granularity. Besides, we employ HRM to inherit old knowledge in the previously learned classifier heads to facilitate the cross-task prompts' generalization ability. Extensive experiments show the effectiveness of our method. The source codes will be available at \url{https://github.com/RAIAN08/PECTP}.