Boosting Open-Domain Continual Learning via Leveraging Intra-domain Category-aware Prototype

TL;DR

This paper introduces a prototype-based method for open-domain continual learning in vision-language models, improving task identification and domain knowledge retention to reduce forgetting and enhance zero-shot performance.

Contribution

It proposes a training-free Task-ID discriminator and intra-domain category-aware prototypes as domain priors, advancing ODCL by addressing task identification and domain knowledge preservation.

Findings

Achieved 2.37% and 1.14% improvements in class-incremental and task-incremental settings.

Effective on 11 datasets, demonstrating robustness and generalization.

Utilized prototypes as classifiers and domain priors to enhance ODCL performance.

Abstract

Despite recent progress in enhancing the efficacy of Open-Domain Continual Learning (ODCL) in Vision-Language Models (VLM), failing to (1) correctly identify the Task-ID of a test image and (2) use only the category set corresponding to the Task-ID, while preserving the knowledge related to each domain, cannot address the two primary challenges of ODCL: forgetting old knowledge and maintaining zero-shot capabilities, as well as the confusions caused by category-relatedness between domains. In this paper, we propose a simple yet effective solution: leveraging intra-domain category-aware prototypes for ODCL in CLIP (DPeCLIP), where the prototype is the key to bridging the above two processes. Concretely, we propose a training-free Task-ID discriminator method, by utilizing prototypes as classifiers for identifying Task-IDs. Furthermore, to maintain the knowledge corresponding to each domain, we incorporate intra-domain category-aware prototypes as domain prior prompts into the training process. Extensive experiments conducted on 11 different datasets demonstrate the effectiveness of our approach, achieving 2.37% and 1.14% average improvement in class-incremental and task-incremental settings, respectively.