Regularization, early-stopping and dreaming: a Hopfield-like setup to address generalization and overfitting

TL;DR

This paper introduces a framework for optimizing attractor neural networks using regularization and early-stopping, linking unlearning protocols to generalization and overfitting control, supported by analytical and numerical results.

Contribution

It establishes a connection between unlearning protocols and regularization in neural networks, providing strategies to prevent overfitting through hyperparameter tuning.

Findings

Optimal neuron-interaction matrices relate to Hebbian kernels revised by unlearning.

Regularization hyperparameters and training time influence unlearning extent.

Numerical experiments reveal regimes of overfitting, failure, and success depending on dataset parameters.

Abstract

In this work we approach attractor neural networks from a machine learning perspective: we look for optimal network parameters by applying a gradient descent over a regularized loss function. Within this framework, the optimal neuron-interaction matrices turn out to be a class of matrices which correspond to Hebbian kernels revised by a reiterated unlearning protocol. Remarkably, the extent of such unlearning is proved to be related to the regularization hyperparameter of the loss function and to the training time. Thus, we can design strategies to avoid overfitting that are formulated in terms of regularization and early-stopping tuning. The generalization capabilities of these attractor networks are also investigated: analytical results are obtained for random synthetic datasets, next, the emerging picture is corroborated by numerical experiments that highlight the existence of several regimes (i.e., overfitting, failure and success) as the dataset parameters are varied.