UNREALIZABLE LEARNING IN FEEDFORWARD NEURAL NETWORKS

TL;DR

This paper uses statistical mechanics to analyze unrealizable generalization in large feed-forward neural networks, revealing phase transitions and the effects of noise on learning performance.

Contribution

It provides a detailed theoretical analysis of unrealizable learning in perceptrons and tree committee machines, including phase transitions and the impact of noise, using replica symmetry and symmetry breaking.

Findings

Identifies phase transition for low noise in perceptrons.

Shows transition disappears with increased noise.

Finds decay of generalization error as (ln α / α)^k for large α.

Abstract

Statistical mechanics is used to study unrealizable generalization in two large feed-forward neural networks with binary weights and output, a perceptron and a tree committee machine. The student is trained by a teacher being larger, i.e. having more units than the student. It is shown that this is the same as using training data corrupted by Gaussian noise. Each machine is considered in the high temperature limit and in the replica symmetric approximation as well as for one step of replica symmetry breaking. For the perceptron a phase transition is found for low noise. However the transition is not to optimal learning. If the noise is increased the transition disappears. In both cases $\epsilon _{g}$ will approach optimal performance with a $(\ln\alpha /\alpha)^k$ decay for large $\alpha$. For the tree committee machine noise in the input layer is studied, as well as noise in the hidden layer. If there is no noise in the input layer there is, in the case of one step of repl! ica symmetry breaking, a phase tra nsition to optimal learning at some finite $\alpha$ for all levels of noise in the hidden layer. When noise is added to the input layer the generalization behavior is similar to that of the perceptron. For one step of replica symmetry breaking, in the realizable limit, the values of the spinodal points found in this paper disagree with previously reported estimates \cite{seung1},\cite{schwarze1}. Here the value $\alpha _{sp} = 2.79$ is found for the tree committee machine and $\alpha _{sp} = 1.67$ for the perceptron.