Training neural networks with structured noise improves classification and generalization

TL;DR

This paper demonstrates that structured noise during neural network training enhances classification accuracy and generalization, approaching perfect memory retrieval even under high noise conditions.

Contribution

It introduces a method of adding structured noise to training data, significantly improving the performance of the training-with-noise algorithm and linking it to Hebbian Unlearning.

Findings

Structured noise improves memory retrieval and basin size.

Training-with-noise approaches Hebbian Unlearning at maximal noise.

Networks achieve near-perfect classification with structured noise.

Abstract

The beneficial role of noise-injection in learning is a consolidated concept in the field of artificial neural networks, suggesting that even biological systems might take advantage of similar mechanisms to optimize their performance. The training-with-noise algorithm proposed by Gardner and collaborators is an emblematic example of a noise-injection procedure in recurrent networks, which can be used to model biological neural systems. We show how adding structure to noisy training data can substantially improve the algorithm performance, allowing the network to approach perfect retrieval of the memories and wide basins of attraction, even in the scenario of maximal injected noise. We also prove that the so-called Hebbian Unlearning rule coincides with the training-with-noise algorithm when noise is maximal and data are stable fixed points of the network dynamics.