FeTT: Continual Class Incremental Learning via Feature Transformation Tuning

TL;DR

This paper introduces FeTT, a feature transformation tuning method for continual class incremental learning that non-parametrically fine-tunes backbone features to prevent forgetting and improve performance without relying on previous task data.

Contribution

The paper proposes a novel FeTT model that fine-tunes features across tasks independently of previous data, addressing catastrophic forgetting and feature suppression issues in continual learning.

Findings

FeTT effectively reduces catastrophic forgetting in CL benchmarks.

Extended ensemble strategy with PTMs enhances performance.

FeTT operates independently of previous task data, improving flexibility.

Abstract

Continual learning (CL) aims to extend deep models from static and enclosed environments to dynamic and complex scenarios, enabling systems to continuously acquire new knowledge of novel categories without forgetting previously learned knowledge. Recent CL models have gradually shifted towards the utilization of pre-trained models (PTMs) with parameter-efficient fine-tuning (PEFT) strategies. However, continual fine-tuning still presents a serious challenge of catastrophic forgetting due to the absence of previous task data. Additionally, the fine-tune-then-frozen mechanism suffers from performance limitations due to feature channels suppression and insufficient training data in the first CL task. To this end, this paper proposes feature transformation tuning (FeTT) model to non-parametrically fine-tune backbone features across all tasks, which not only operates independently of CL training data but also smooths feature channels to prevent excessive suppression. Then, the extended ensemble strategy incorporating different PTMs with FeTT model facilitates further performance improvement. We further elaborate on the discussions of the fine-tune-then-frozen paradigm and the FeTT model from the perspectives of discrepancy in class marginal distributions and feature channels. Extensive experiments on CL benchmarks validate the effectiveness of our proposed method.