Navigating Semantic Drift in Task-Agnostic Class-Incremental Learning

TL;DR

This paper introduces a novel method for class-incremental learning that effectively calibrates semantic drift by compensating for mean shifts and covariance changes, improving model stability without task labels.

Contribution

It proposes a semantic drift calibration technique combining mean shift compensation, covariance calibration, and self-distillation to enhance task-agnostic class-incremental learning.

Findings

Significantly reduces feature distribution gap between tasks

Improves accuracy in class-incremental learning benchmarks

Effective in mitigating semantic drift without task labels

Abstract

Class-incremental learning (CIL) seeks to enable a model to sequentially learn new classes while retaining knowledge of previously learned ones. Balancing flexibility and stability remains a significant challenge, particularly when the task ID is unknown. To address this, our study reveals that the gap in feature distribution between novel and existing tasks is primarily driven by differences in mean and covariance moments. Building on this insight, we propose a novel semantic drift calibration method that incorporates mean shift compensation and covariance calibration. Specifically, we calculate each class's mean by averaging its sample embeddings and estimate task shifts using weighted embedding changes based on their proximity to the previous mean, effectively capturing mean shifts for all learned classes with each new task. We also apply Mahalanobis distance constraint for covariance calibration, aligning class-specific embedding covariances between old and current networks to mitigate the covariance shift. Additionally, we integrate a feature-level self-distillation approach to enhance generalization. Comprehensive experiments on commonly used datasets demonstrate the effectiveness of our approach. The source code is available at \href{https://github.com/fwu11/MACIL.git}{https://github.com/fwu11/MACIL.git}.