Statistical Mechanics of On-line Ensemble Teacher Learning through a Novel Perceptron Learning Rule

TL;DR

This paper introduces a novel perceptron learning rule with a margin that bridges perceptron and Hebbian learning, demonstrating improved student performance in ensemble teacher learning scenarios.

Contribution

It proposes a new perceptron learning rule with a margin that continuously connects perceptron and Hebbian learning, enhancing ensemble learning performance.

Findings

Ensemble teacher learning benefits from a positive margin setting.

A larger margin amplifies the effect of ensemble teachers.

The proposed rule improves the student's approach to the true teacher.

Abstract

In ensemble teacher learning, ensemble teachers have only uncertain information about the true teacher, and this information is given by an ensemble consisting of an infinite number of ensemble teachers whose variety is sufficiently rich. In this learning, a student learns from an ensemble teacher that is iteratively selected randomly from a pool of many ensemble teachers. An interesting point of ensemble teacher learning is the asymptotic behavior of the student to approach the true teacher by learning from ensemble teachers. The student performance is improved by using the Hebbian learning rule in the learning. However, the perceptron learning rule cannot improve the student performance. On the other hand, we proposed a perceptron learning rule with a margin. This learning rule is identical to the perceptron learning rule when the margin is zero and identical to the Hebbian learning rule when the margin is infinity. Thus, this rule connects the perceptron learning rule and the Hebbian learning rule continuously through the size of the margin. Using this rule, we study changes in the learning behavior from the perceptron learning rule to the Hebbian learning rule by considering several margin sizes. From the results, we show that by setting a margin of kappa > 0, the effect of an ensemble appears and becomes significant when a larger margin kappa is used.