Memory capacity of neural network models

TL;DR

This paper reviews theoretical models of neural network memory capacity, analyzing how factors like synapse complexity, sparseness, and memory correlations influence the number of memories that can be stored and retrieved.

Contribution

It provides a comprehensive overview of mathematical frameworks for understanding neural memory capacity, highlighting key factors affecting storage limits.

Findings

Memory capacity scales with neurons and synapses

Sparseness enhances memory storage

Synapse complexity impacts capacity

Abstract

Memory is a complex phenomenon that involves several distinct mechanisms. These mechanisms operate at different spatial and temporal levels. This chapter focuses on the theoretical framework and the mathematical models that have been developed to understand how these mechanisms are orchestrated to store, preserve and retrieve a large number of memories. In particular, this chapter reviews the theoretical studies on memory capacity, in which the investigators estimated how the number of storable memories scales with the number of neurons and synapses in the neural circuitry. The memory capacity depends on the complexity of the synapses, the sparseness of the representations, the spatial and temporal correlations between memories and the specific way memories are retrieved. Complexity is important when the synapses can only be modified with a limited precision, as in the case of biological synapses, and sparseness can greatly increase memory capacity and be particularly beneficial when memories are structured (correlated to each other). The theoretical tools discussed by this chapter can be harnessed to identify the important computational principles that underlie memory storage, preservation and retrieval and provide guidance in designing and interpreting memory experiments.