Do Neural Networks Lose Plasticity in a Gradually Changing World?

TL;DR

This paper investigates how neural networks' plasticity is affected by gradual versus abrupt environmental changes, revealing that loss of plasticity is mainly due to abrupt task transitions and can be mitigated in gradual settings.

Contribution

The study introduces a framework for analyzing plasticity in gradually changing environments and demonstrates that gradual changes help preserve neural networks' learning ability.

Findings

Loss of plasticity is linked to abrupt task transitions.

Gradual environment changes mitigate plasticity loss.

Theoretical and empirical evidence supports gradual change benefits.

Abstract

Continual learning has become a trending topic in machine learning. Recent studies have discovered an interesting phenomenon called loss of plasticity, referring to neural networks gradually losing the ability to learn new tasks. However, existing plasticity research largely relies on contrived settings with abrupt task transitions, which often do not reflect real-world environments. In this paper, we propose to investigate a gradually changing environment, and we simulate this by input/output interpolation and task sampling. We perform theoretical and empirical analysis, showing that the loss of plasticity is an artifact of abrupt tasks changes in the environment and can be largely mitigated if the world changes gradually.