Online Learning with Ensembles

TL;DR

This paper analyzes the effectiveness of ensembles in supervised online learning, showing that ensembles can improve generalization for simple rules like perceptron but not for more optimized rules, due to an equivalence with single-student learning.

Contribution

It demonstrates the conditions under which ensembles improve learning and introduces a transform linking ensemble performance to single-student learning for any rule.

Findings

Ensembles improve perceptron learning performance.

No ensemble benefit for optimized learning rules.

Existence of a transform linking ensemble and single-student generalization behaviors.

Abstract

Supervised online learning with an ensemble of students randomized by the choice of initial conditions is analyzed. For the case of the perceptron learning rule, asymptotically the same improvement in the generalization error of the ensemble compared to the performance of a single student is found as in Gibbs learning. For more optimized learning rules, however, using an ensemble yields no improvement. This is explained by showing that for any learning rule $f$ a transform $\tilde{f}$ exists, such that a single student using $\tilde{f}$ has the same generalization behaviour as an ensemble of $f$-students.