Constructing Enhanced Mutual Information for Online Class-Incremental Learning

TL;DR

This paper introduces an Enhanced Mutual Information (EMI) method for online class-incremental learning, improving knowledge retention and discrimination by decoupling and optimizing mutual information components.

Contribution

The paper proposes a novel EMI approach that considers diversity, representativeness, and separability of knowledge, addressing limitations of existing MI-based methods in OCIL.

Findings

EMI outperforms state-of-the-art methods on benchmark datasets.

Incorporating DMI, RMI, and SMI improves knowledge diversity and class separation.

Experimental results validate the effectiveness of the proposed approach.

Abstract

Online Class-Incremental continual Learning (OCIL) addresses the challenge of continuously learning from a single-channel data stream, adapting to new tasks while mitigating catastrophic forgetting. Recently, Mutual Information (MI)-based methods have shown promising performance in OCIL. However, existing MI-based methods treat various knowledge components in isolation, ignoring the knowledge confusion across tasks. This narrow focus on simple MI knowledge alignment may lead to old tasks being easily forgotten with the introduction of new tasks, risking the loss of common parts between past and present knowledge.To address this, we analyze the MI relationships from the perspectives of diversity, representativeness, and separability, and propose an Enhanced Mutual Information (EMI) method based on knwoledge decoupling. EMI consists of Diversity Mutual Information (DMI), Representativeness Mutual Information (RMI) and Separability Mutual Information (SMI). DMI diversifies intra-class sample features by considering the similarity relationships among inter-class sample features to enable the network to learn more general knowledge. RMI summarizes representative features for each category and aligns sample features with these representative features, making the intra-class sample distribution more compact. SMI establishes MI relationships for inter-class representative features, enhancing the stability of representative features while increasing the distinction between inter-class representative features, thus creating clear boundaries between class. Extensive experimental results on widely used benchmark datasets demonstrate the superior performance of EMI over state-of-the-art baseline methods.