Overparameterized Neural Networks Implement Associative Memory

TL;DR

This paper demonstrates that overparameterized neural networks naturally implement associative memory mechanisms, effectively storing and retrieving data and sequences through attractor dynamics, with theoretical proofs supporting these findings.

Contribution

It reveals that standard overparameterized neural networks inherently function as associative memory systems, both empirically and theoretically, for real-valued data.

Findings

Autoencoders store training samples as attractors.

Sequence encoding is more efficient than autoencoding for memory.

Theoretical proof of attractor storage for single-example autoencoders.

Abstract

Identifying computational mechanisms for memorization and retrieval of data is a long-standing problem at the intersection of machine learning and neuroscience. Our main finding is that standard overparameterized deep neural networks trained using standard optimization methods implement such a mechanism for real-valued data. Empirically, we show that: (1) overparameterized autoencoders store training samples as attractors, and thus, iterating the learned map leads to sample recovery; (2) the same mechanism allows for encoding sequences of examples, and serves as an even more efficient mechanism for memory than autoencoding. Theoretically, we prove that when trained on a single example, autoencoders store the example as an attractor. Lastly, by treating a sequence encoder as a composition of maps, we prove that sequence encoding provides a more efficient mechanism for memory than autoencoding.