Optimal storage capacity of neural networks at finite temperatures

TL;DR

This paper extends Gardner's analysis to finite temperatures, calculating the optimal storage capacity of neural networks and revealing a phase transition in performance based on temperature and tolerance.

Contribution

It introduces a finite-temperature extension to Gardner's analysis, deriving the optimal storage capacity as a function of temperature and tolerance parameters.

Findings

Optimal capacity at zero temperature is 2, independent of tolerance.

Capacity increases with tolerance at finite temperatures.

Identifies a first-order transition in network performance at low temperatures.

Abstract

Gardner's analysis of the optimal storage capacity of neural networks is extended to study finite-temperature effects. The typical volume of the space of interactions is calculated for strongly-diluted networks as a function of the storage ratio $\alpha$, temperature $T$, and the tolerance parameter $m$, from which the optimal storage capacity $\alpha_c$ is obtained as a function of $T$ and $m$. At zero temperature it is found that $\alpha_c = 2$ regardless of $m$ while $\alpha_c$ in general increases with the tolerance at finite temperatures. We show how the best performance for given $\alpha$ and $T$ is obtained, which reveals a first-order transition from high-quality performance to low-quality one at low temperatures. An approximate criterion for recalling, which is valid near $m=1$, is also discussed.