Optimization of the Asymptotic Property of Mutual Learning Involving an Integration Mechanism of Ensemble Learning

TL;DR

This paper introduces an optimized mutual learning method that converges to the same generalization error as optimal ensemble learning by adjusting step sizes, enhancing online learning performance.

Contribution

It proposes a novel optimization of mutual learning step sizes to match the asymptotic performance of ensemble learning within an online learning framework.

Findings

Optimized mutual learning converges to the ensemble learning error.

The method improves generalization error without teacher involvement.

A relationship between step size and ensemble integration is established.

Abstract

We propose an optimization method of mutual learning which converges into the identical state of optimum ensemble learning within the framework of on-line learning, and have analyzed its asymptotic property through the statistical mechanics method.The proposed model consists of two learning steps: two students independently learn from a teacher, and then the students learn from each other through the mutual learning. In mutual learning, students learn from each other and the generalization error is improved even if the teacher has not taken part in the mutual learning. However, in the case of different initial overlaps(direction cosine) between teacher and students, a student with a larger initial overlap tends to have a larger generalization error than that of before the mutual learning. To overcome this problem, our proposed optimization method of mutual learning optimizes the step sizes of two students to minimize the asymptotic property of the generalization error. Consequently, the optimized mutual learning converges to a generalization error identical to that of the optimal ensemble learning. In addition, we show the relationship between the optimum step size of the mutual learning and the integration mechanism of the ensemble learning.