Ensemble learning of linear perceptron; Online learning theory

TL;DR

This paper analyzes the generalization error in ensemble learning of linear perceptrons within online learning, revealing how ensemble size and initial weight configurations affect performance.

Contribution

It provides exact calculations of generalization error for ensembles of linear perceptrons, highlighting the benefits of ensemble size and weight averaging strategies.

Findings

Ensemble of infinite perceptrons halves the error compared to a single perceptron.

Error decreases as O(1/K) with the number of perceptrons K.

Weighted averaging improves learning with inhomogeneous initial weights.

Abstract

Within the framework of on-line learning, we study the generalization error of an ensemble learning machine learning from a linear teacher perceptron. The generalization error achieved by an ensemble of linear perceptrons having homogeneous or inhomogeneous initial weight vectors is precisely calculated at the thermodynamic limit of a large number of input elements and shows rich behavior. Our main findings are as follows. For learning with homogeneous initial weight vectors, the generalization error using an infinite number of linear student perceptrons is equal to only half that of a single linear perceptron, and converges with that of the infinite case with O(1/K) for a finite number of K linear perceptrons. For learning with inhomogeneous initial weight vectors, it is advantageous to use an approach of weighted averaging over the output of the linear perceptrons, and we show the conditions under which the optimal weights are constant during the learning process. The optimal weights depend on only correlation of the initial weight vectors.