Storage capacity of a constructive learning algorithm

TL;DR

This paper derives bounds on the storage capacity of a constructive learning algorithm for the parity machine, using analytical methods to understand its limits in large data regimes.

Contribution

It provides the first analytical bounds on the storage capacity of the Tilinglike Learning Algorithm for the Parity Machine, including properties of biased perceptrons.

Findings

Lower bound proportional to number of hidden units

Upper bound close to previous predictions by Mitchinson and Durbin

Analytical characterization of perceptron with biased targets

Abstract

Upper and lower bounds for the typical storage capacity of a constructive algorithm, the Tilinglike Learning Algorithm for the Parity Machine [M. Biehl and M. Opper, Phys. Rev. A {\bf 44} 6888 (1991)], are determined in the asymptotic limit of large training set sizes. The properties of a perceptron with threshold, learning a training set of patterns having a biased distribution of targets, needed as an intermediate step in the capacity calculation, are determined analytically. The lower bound for the capacity, determined with a cavity method, is proportional to the number of hidden units. The upper bound, obtained with the hypothesis of replica symmetry, is close to the one predicted by Mitchinson and Durbin [Biol. Cyber. {\bf 60} 345 (1989)].