Activation by Interval-wise Dropout: A Simple Way to Prevent Neural Networks from Plasticity Loss

TL;DR

This paper introduces AID, a novel dropout-based method that regularizes neural networks to prevent plasticity loss, thereby improving continual learning and reinforcement learning performance.

Contribution

AID is a new dropout-inspired technique that applies different dropout probabilities to preactivation intervals, effectively preventing plasticity loss in neural networks.

Findings

AID regularizes networks similar to deep linear models.

AID improves continual learning on CIFAR and TinyImageNet.

AID enhances reinforcement learning in Arcade Learning Environment.

Abstract

Plasticity loss, a critical challenge in neural network training, limits a model's ability to adapt to new tasks or shifts in data distribution. This paper introduces AID (Activation by Interval-wise Dropout), a novel method inspired by Dropout, designed to address plasticity loss. Unlike Dropout, AID generates subnetworks by applying Dropout with different probabilities on each preactivation interval. Theoretical analysis reveals that AID regularizes the network, promoting behavior analogous to that of deep linear networks, which do not suffer from plasticity loss. We validate the effectiveness of AID in maintaining plasticity across various benchmarks, including continual learning tasks on standard image classification datasets such as CIFAR10, CIFAR100, and TinyImageNet. Furthermore, we show that AID enhances reinforcement learning performance in the Arcade Learning Environment benchmark.