Correlations between hidden units in multilayer neural networks and replica symmetry breaking

TL;DR

This paper investigates how replica symmetry breaking affects the distribution of local fields and correlations in simple multilayer neural networks, revealing limited impact on correlations despite changes in storage capacity.

Contribution

It provides a detailed analysis of replica symmetry breaking effects on local field distributions and correlations in single hidden layer neural networks with specific architectures.

Findings

Replica symmetry breaking significantly alters storage capacity and local field distributions.

Correlation coefficients among hidden units are only minimally affected by replica symmetry breaking.

Numerical results are provided for specific network architectures like PARITY, COMMITTEE, and AND machines.

Abstract

We consider feed-forward neural networks with one hidden layer, tree architecture and a fixed hidden-to-output Boolean function. Focusing on the saturation limit of the storage problem the influence of replica symmetry breaking on the distribution of local fields at the hidden units is investigated. These field distributions determine the probability for finding a specific activation pattern of the hidden units as well as the corresponding correlation coefficients and therefore quantify the division of labor among the hidden units. We find that although modifying the storage capacity and the distribution of local fields markedly replica symmetry breaking has only a minor effect on the correlation coefficients. Detailed numerical results are provided for the PARITY, COMMITTEE and AND machines with K=3 hidden units and nonoverlapping receptive fields.