Fortuitous Forgetting in Connectionist Networks

TL;DR

This paper introduces a 'forget-and-relearn' paradigm that leverages selective forgetting to improve neural network training, unifying and enhancing existing iterative algorithms for better learning outcomes.

Contribution

It proposes a novel framework that uses targeted forgetting to shape learning trajectories and unifies various iterative training methods under this paradigm.

Findings

Selective forgetting removes undesirable information effectively.

The framework improves performance of existing algorithms.

Insights clarify neural network training dynamics.

Abstract

Forgetting is often seen as an unwanted characteristic in both human and machine learning. However, we propose that forgetting can in fact be favorable to learning. We introduce "forget-and-relearn" as a powerful paradigm for shaping the learning trajectories of artificial neural networks. In this process, the forgetting step selectively removes undesirable information from the model, and the relearning step reinforces features that are consistently useful under different conditions. The forget-and-relearn framework unifies many existing iterative training algorithms in the image classification and language emergence literature, and allows us to understand the success of these algorithms in terms of the disproportionate forgetting of undesirable information. We leverage this understanding to improve upon existing algorithms by designing more targeted forgetting operations. Insights from our analysis provide a coherent view on the dynamics of iterative training in neural networks and offer a clear path towards performance improvements.