Compensating Distribution Drifts in Class-incremental Learning of Pre-trained Vision Transformers

TL;DR

This paper introduces SLDC, a method to mitigate distribution drift in class-incremental learning with pre-trained vision transformers, significantly improving sequential fine-tuning performance by aligning feature distributions across tasks.

Contribution

The paper proposes a novel SLDC approach with linear and weakly nonlinear variants to address distribution drift in CIL of ViTs, enhancing sequential learning effectiveness.

Findings

SLDC improves classifier performance in CIL benchmarks.

Combining KD with SLDC achieves results comparable to joint training.

SLDC effectively aligns feature distributions across tasks.

Abstract

Recent advances have shown that sequential fine-tuning (SeqFT) of pre-trained vision transformers (ViTs), followed by classifier refinement using approximate distributions of class features, can be an effective strategy for class-incremental learning (CIL). However, this approach is susceptible to distribution drift, caused by the sequential optimization of shared backbone parameters. This results in a mismatch between the distributions of the previously learned classes and that of the updater model, ultimately degrading the effectiveness of classifier performance over time. To address this issue, we introduce a latent space transition operator and propose Sequential Learning with Drift Compensation (SLDC). SLDC aims to align feature distributions across tasks to mitigate the impact of drift. First, we present a linear variant of SLDC, which learns a linear operator by solving a regularized least-squares problem that maps features before and after fine-tuning. Next, we extend this with a weakly nonlinear SLDC variant, which assumes that the ideal transition operator lies between purely linear and fully nonlinear transformations. This is implemented using learnable, weakly nonlinear mappings that balance flexibility and generalization. To further reduce representation drift, we apply knowledge distillation (KD) in both algorithmic variants. Extensive experiments on standard CIL benchmarks demonstrate that SLDC significantly improves the performance of SeqFT. Notably, by combining KD to address representation drift with SLDC to compensate distribution drift, SeqFT achieves performance comparable to joint training across all evaluated datasets. Code: https://github.com/raoxuan98-hash/sldc.git.