Bounds on learning in polynomial time

TL;DR

This paper investigates the limits of polynomial-time learning in neural networks, analyzing capacity bounds and revealing discrepancies between theoretical predictions and empirical results, especially for the committee machine.

Contribution

It introduces new bounds on learning capacity for various neural architectures and clarifies the relationship between storage capacity and polynomial learning time.

Findings

Polynomial learning algorithms can be practical for certain network sizes.

Discrepancies exist between theoretical capacity predictions and empirical results for the committee machine.

New simulations reveal subtleties in defining learning capacity and time dependence.

Abstract

The performance of large neural networks can be judged not only by their storage capacity but also by the time required for learning. A polynomial learning algorithm with learning time $\sim N^2$ in a network with $N$ units might be practical whereas a learning time $\sim e^N$ would allow rather small networks only. The question of absolute storage capacity $\alpha_c$ and capacity for polynomial learning rules $\alpha_p$ is discussed for several feed-forward architectures, the perceptron, the binary perceptron, the committee machine and a perceptron with fixed weights in the first layer and adaptive weights in the second layer. The analysis is based partially on dynamic mean field theory which is valid for $N\to\infty$. Especially for the committee machine a value $\alpha_p$ considerably lower than the capacity predicted by replica theory or simulations is found. This discrepancy is resolved by new simulations investigating the learning time dependence and revealing subtleties in the definition of the capacity.