Class-Incremental Learning with CLIP: Adaptive Representation Adjustment and Parameter Fusion

TL;DR

This paper introduces RAPF, a novel method for class-incremental learning with CLIP that adaptively adjusts representations and fuses parameters to reduce forgetting and improve performance on benchmarks.

Contribution

The paper proposes RAPF, a new approach that measures class influence, adjusts representations, and fuses parameters to enhance CLIP's incremental learning capabilities.

Findings

Achieves state-of-the-art results on multiple benchmarks.

Effectively mitigates forgetting of old classes.

Demonstrates superior performance over existing methods.

Abstract

Class-incremental learning is a challenging problem, where the goal is to train a model that can classify data from an increasing number of classes over time. With the advancement of vision-language pre-trained models such as CLIP, they demonstrate good generalization ability that allows them to excel in class-incremental learning with completely frozen parameters. However, further adaptation to downstream tasks by simply fine-tuning the model leads to severe forgetting. Most existing works with pre-trained models assume that the forgetting of old classes is uniform when the model acquires new knowledge. In this paper, we propose a method named Adaptive Representation Adjustment and Parameter Fusion (RAPF). During training for new data, we measure the influence of new classes on old ones and adjust the representations, using textual features. After training, we employ a decomposed parameter fusion to further mitigate forgetting during adapter module fine-tuning. Experiments on several conventional benchmarks show that our method achieves state-of-the-art results. Our code is available at \url{https://github.com/linlany/RAPF}.