# Order parameters and phase transitions of continual learning in deep neural networks

## TL;DR

This paper introduces a theory for continual learning in deep neural networks, showing how task similarity and network depth affect learning and forgetting.

## Contribution

A statistical-mechanics theory that identifies order parameters and phase transitions in continual learning.

## Key findings

- Order parameters predict CL behaviors based on task similarity and network architecture.
- Increasing network depth reduces interference between tasks and lowers forgetting.
- Multihead CL shows phase transitions where performance drops sharply with low task similarity.

## Abstract

Continual learning (CL), the ability to learn new tasks without forgetting existing ones, is one of the greatest challenges in AI. Our work provides an analytically tractable theory that captures some key phenomena of CL in deep, wide neural networks. We highlight several “order parameters” that measure the similarity between tasks and show that they can be highly predictive of CL behaviors on classic benchmark tasks. Strikingly, we identify a set of phase transitions where the network’s CL ability changes abruptly with the order parameter. Our results provide quantitative understanding of how CL ability depends on task relations, network architectures, and learning procedures.

Continual learning (CL) enables animals to learn new tasks without erasing prior knowledge. CL in artificial neural networks (NNs) is challenging due to catastrophic forgetting, where new learning degrades performance on older tasks. While various techniques exist to mitigate forgetting, theoretical insights into when and why CL fails in NNs are lacking. Here, we present a statistical-mechanics theory of CL in deep, wide NNs, which characterizes the network’s input–output mapping as it learns a sequence of tasks. It gives rise to order parameters (OPs) that capture how task relations and network architecture influence forgetting and anterograde interference, as verified by numerical evaluations. For networks with a shared readout for all tasks (single-head CL), the relevant-feature and rule similarity between tasks, respectively measured by two OPs, are sufficient to predict a wide range of CL behaviors on classic benchmark tasks. In addition, the theory predicts that increasing the network depth can effectively reduce interference between tasks, thereby lowering forgetting. For networks with task-specific readouts (multihead CL), the theory identifies a phase transition where CL performance shifts dramatically as tasks become less similar, as measured by another task-similarity OP. While forgetting is relatively mild compared to single-head CL across all tasks, sufficiently low similarity leads to catastrophic anterograde interference, where the network retains old tasks and interpolates new training data perfectly but completely fails to generalize new learning. Our results delineate important factors affecting CL performance and offer theoretical insights into common heuristics for mitigation of forgetting.