Correlation of internal representations in feed-forward neural networks

TL;DR

This paper investigates how internal representations in feed-forward neural networks develop correlations, providing a method to calculate these correlations and analyzing specific network models near saturation.

Contribution

It introduces a way to compute correlations of hidden node activities from joint probability distributions, enhancing understanding of neural network storage and generalization.

Findings

Correlations can be explicitly calculated for various network models.

Results are provided for parity, AND, and committee machines.

Analysis focuses on networks near saturation.

Abstract

Feed-forward multilayer neural networks implementing random input-output mappings develop characteristic correlations between the activity of their hidden nodes which are important for the understanding of the storage and generalization performance of the network. It is shown how these correlations can be calculated from the joint probability distribution of the aligning fields at the hidden units for arbitrary decoder function between hidden layer and output. Explicit results are given for the parity-, and-, and committee-machines with arbitrary number of hidden nodes near saturation.