ProTPS: Prototype-Guided Text Prompt Selection for Continual Learning

TL;DR

ProTPS introduces a prototype-guided text prompt selection method that enhances continual learning by promoting unique semantic prompts, demonstrated on diverse datasets including a new real-world Marine112 dataset.

Contribution

The paper proposes a novel prototype-guided approach for text prompt selection, improving class-specific prompt learning in continual learning scenarios.

Findings

ProTPS achieves performance close to upper bounds in class incremental and cross-dataset settings.

ProTPS outperforms recent state-of-the-art methods in various continual learning benchmarks.

Marine112 dataset presents new challenges for class and domain incremental learning.

Abstract

For continual learning, text-prompt-based methods leverage text encoders and learnable prompts to encode semantic features for sequentially arrived classes over time. A common challenge encountered by existing works is how to learn unique text prompts, which implicitly carry semantic information of new classes, so that the semantic features of newly arrived classes do not overlap with those of trained classes, thereby mitigating the catastrophic forgetting problem. To address this challenge, we propose a novel approach Prototype-guided Text Prompt Selection (ProTPS)'' to intentionally increase the training flexibility thus encouraging the learning of unique text prompts. Specifically, our ProTPS learns class-specific vision prototypes and text prompts. Vision prototypes guide the selection and learning of text prompts for each class. We first evaluate our ProTPS in both class incremental (CI) setting and cross-datasets continual (CDC) learning setting. Because our ProTPS achieves performance close to the upper bounds, we further collect a real-world dataset with 112 marine species collected over a span of six years, named Marine112, to bring new challenges to the community. Marine112 is authentically suited for the class and domain incremental (CDI) learning setting and is under natural long-tail distribution. The results under three settings show that our ProTPS performs favorably against the recent state-of-the-art methods. The implementation code and Marine112 dataset will be released upon the acceptance of our paper.