Statistical Mechanics of Nonlinear On-line Learning for Ensemble Teachers

TL;DR

This paper uses statistical mechanics to analyze the generalization performance of nonlinear perceptron models with ensemble teachers under on-line learning, revealing distinct behaviors for Hebbian and perceptron learning rules.

Contribution

It provides a comparative analysis of nonlinear versus linear models and distinguishes the behaviors of Hebbian and perceptron learning in ensemble teacher scenarios.

Findings

Hebbian learning leads to monotonic decrease in generalization error

The steady generalization error in Hebbian learning is independent of learning rate

Perceptron learning exhibits non-monotonic error dynamics

Abstract

We analyze the generalization performance of a student in a model composed of nonlinear perceptrons: a true teacher, ensemble teachers, and the student. We calculate the generalization error of the student analytically or numerically using statistical mechanics in the framework of on-line learning. We treat two well-known learning rules: Hebbian learning and perceptron learning. As a result, it is proven that the nonlinear model shows qualitatively different behaviors from the linear model. Moreover, it is clarified that Hebbian learning and perceptron learning show qualitatively different behaviors from each other. In Hebbian learning, we can analytically obtain the solutions. In this case, the generalization error monotonically decreases. The steady value of the generalization error is independent of the learning rate. The larger the number of teachers is and the more variety the ensemble teachers have, the smaller the generalization error is. In perceptron learning, we have to numerically obtain the solutions. In this case, the dynamical behaviors of the generalization error are non-monotonic. The smaller the learning rate is, the larger the number of teachers is; and the more variety the ensemble teachers have, the smaller the minimum value of the generalization error is.