Storage capacity of correlated perceptrons

TL;DR

This paper analyzes the storage capacity of correlated perceptrons, introducing a formalism to determine how output correlations affect the maximum number of random inputs they can handle.

Contribution

It presents a novel formalism for calculating storage capacity in correlated perceptrons and compares replica-symmetric results with properties of two-layer networks.

Findings

Correlations in hidden layers influence storage capacity.

Maximal input number depends on desired output correlations.

Results highlight the importance of correlations for neural network capacity.

Abstract

We consider an ensemble of $K$ single-layer perceptrons exposed to random inputs and investigate the conditions under which the couplings of these perceptrons can be chosen such that prescribed correlations between the outputs occur. A general formalism is introduced using a multi-perceptron costfunction that allows to determine the maximal number of random inputs as a function of the desired values of the correlations. Replica-symmetric results for $K=2$ and $K=3$ are compared with properties of two-layer networks of tree-structure and fixed Boolean function between hidden units and output. The results show which correlations in the hidden layer of multi-layer neural networks are crucial for the value of the storage capacity.