Reinitializing weights vs units for maintaining plasticity in neural networks

TL;DR

This paper compares reinitializing weights versus units to preserve neural network plasticity during continual learning, introducing a new selective weight reinitialization algorithm that outperforms previous methods in certain settings.

Contribution

The paper introduces a novel selective weight reinitialization algorithm and systematically compares it to existing reinitialization schemes in continual learning scenarios.

Findings

Reinitializing weights is more effective for small networks and those with layer normalization.

Reinitializing weights maintains plasticity across a broader range of settings.

Reinitializing units and weights are equally effective in large, normalization-free networks.

Abstract

Loss of plasticity is a phenomenon in which a neural network loses its ability to learn when trained for an extended time on non-stationary data. It is a crucial problem to overcome when designing systems that learn continually. An effective technique for preventing loss of plasticity is reinitializing parts of the network. In this paper, we compare two different reinitialization schemes: reinitializing units vs reinitializing weights. We propose a new algorithm, which we name \textit{selective weight reinitialization}, for reinitializing the least useful weights in a network. We compare our algorithm to continual backpropagation and ReDo, two previously proposed algorithms that reinitialize units in the network. Through our experiments in continual supervised learning problems, we identify two settings when reinitializing weights is more effective at maintaining plasticity than reinitializing units: (1) when the network has a small number of units and (2) when the network includes layer normalization. Conversely, reinitializing weights and units are equally effective at maintaining plasticity when the network is of sufficient size and does not include layer normalization. We found that reinitializing weights maintains plasticity in a wider variety of settings than reinitializing units.