InTAct: Interval-based Task Activation Consolidation for Continual Learning

TL;DR

InTAct introduces an efficient interval-based method to preserve neural network functions during continual learning, reducing catastrophic forgetting by constraining neuron activation regions rather than high-dimensional weights.

Contribution

The paper proposes InTAct, a novel approach that enforces functional invariance at the neuron level using activation intervals, offering a computationally efficient alternative to weight-based constraints in continual learning.

Findings

Achieves state-of-the-art performance on benchmarks.

Provides mathematical guarantees of functional invariance.

Reduces computational cost compared to parameter constraints.

Abstract

Continual learning is a fundamental challenge in artificial intelligence that requires networks to acquire new knowledge while preserving previously learned representations. Despite the success of various approaches, most existing paradigms do not provide rigorous mathematical guarantees against catastrophic forgetting. Current methods that offer such guarantees primarily focus on analyzing the parameter space using \textit{interval arithmetic (IA)}, as seen in frameworks such as InterContiNet. However, restricting high-dimensional weight updates can be computationally expensive. In this work, we propose InTAct (Interval-based Task Activation Consolidation), a method that mitigates catastrophic forgetting by enforcing functional invariance at the neuron level. We identify specific activation intervals where previous tasks reside and constrain updates within these regions while allowing for flexible adaptation elsewhere. By ensuring that predictions remain stable within these nested activation intervals, we provide a tractable mathematical guarantee of functional invariance. We emphasize that regulating the activation space is significantly more efficient than parameter-based constraints, because the dimensionality of internal signals is much lower than that of the vast space of model weights. While our approach is architecture-agnostic and applicable to various continual learning settings, its integration with prompt-based methods enables it to achieve state-of-the-art performance on challenging benchmarks.