Finite size scaling in neural networks

TL;DR

This paper shows that the capacity of neural networks to store patterns follows finite size scaling, enabling estimation of critical storage capacity from small system simulations, demonstrated on specific neural architectures.

Contribution

It introduces a finite size scaling method to estimate critical storage capacity in neural networks, applied to committee and parity machines.

Findings

Estimated critical capacity  and scaling exponent  for neural networks.

Validated finite size scaling approach with Gaussian patterns.

Analyzed networks with up to 5 hidden units.

Abstract

We demonstrate that the fraction of pattern sets that can be stored in single- and hidden-layer perceptrons exhibits finite size scaling. This feature allows to estimate the critical storage capacity \alpha_c from simulations of relatively small systems. We illustrate this approach by determining \alpha_c, together with the finite size scaling exponent \nu, for storing Gaussian patterns in committee and parity machines with binary couplings and up to K=5 hidden units.