Representation Calibration and Uncertainty Guidance for Class-Incremental Learning based on Vision Language Model

TL;DR

This paper introduces a novel vision-language model framework for class-incremental learning that uses representation calibration and uncertainty guidance to improve class separation and prediction accuracy across tasks.

Contribution

It proposes a new VLM-based continual learning framework with task-specific adapters, a cross-task representation calibration strategy, and an uncertainty-guided inference method.

Findings

Outperforms existing methods on multiple datasets.

Effectively reduces class confusion across tasks.

Improves class prediction accuracy.

Abstract

Class-incremental learning requires a learning system to continually learn knowledge of new classes and meanwhile try to preserve previously learned knowledge of old classes. As current state-of-the-art methods based on Vision-Language Models (VLMs) still suffer from the issue of differentiating classes across learning tasks. Here a novel VLM-based continual learning framework for image classification is proposed. In this framework, task-specific adapters are added to the pre-trained and frozen image encoder to learn new knowledge, and a novel cross-task representation calibration strategy based on a mixture of light-weight projectors is used to help better separate all learned classes in a unified feature space, alleviating class confusion across tasks. In addition, a novel inference strategy guided by prediction uncertainty is developed to more accurately select the most appropriate image feature for class prediction. Extensive experiments on multiple datasets under various settings demonstrate the superior performance of our method compared to existing ones.