Information-Theoretic Complementary Prompts for Improved Continual Text Classification

TL;DR

This paper introduces InfoComp, a novel continual text classification method that uses separate prompts for task-specific and task-invariant knowledge, leveraging information theory to improve learning without data replay.

Contribution

It proposes a dual-prompt framework inspired by human learning systems, explicitly learning shared and private knowledge to mitigate forgetting and enhance transfer in CTC.

Findings

Outperforms state-of-the-art methods on multiple benchmarks.

Effectively mitigates catastrophic forgetting without data replay.

Enhances forward knowledge transfer through shared representations.

Abstract

Continual Text Classification (CTC) aims to continuously classify new text data over time while minimizing catastrophic forgetting of previously acquired knowledge. However, existing methods often focus on task-specific knowledge, overlooking the importance of shared, task-agnostic knowledge. Inspired by the complementary learning systems theory, which posits that humans learn continually through the interaction of two systems -- the hippocampus, responsible for forming distinct representations of specific experiences, and the neocortex, which extracts more general and transferable representations from past experiences -- we introduce Information-Theoretic Complementary Prompts (InfoComp), a novel approach for CTC. InfoComp explicitly learns two distinct prompt spaces: P(rivate)-Prompt and S(hared)-Prompt. These respectively encode task-specific and task-invariant knowledge, enabling models to sequentially learn classification tasks without relying on data replay. To promote more informative prompt learning, InfoComp uses an information-theoretic framework that maximizes mutual information between different parameters (or encoded representations). Within this framework, we design two novel loss functions: (1) to strengthen the accumulation of task-specific knowledge in P-Prompt, effectively mitigating catastrophic forgetting, and (2) to enhance the retention of task-invariant knowledge in S-Prompt, improving forward knowledge transfer. Extensive experiments on diverse CTC benchmarks show that our approach outperforms previous state-of-the-art methods.