Can a student learn optimally from two different teachers?

TL;DR

This paper investigates how a student learning from an incorrect but related teacher in an online supervised setting experiences residual errors, and provides an approximation for this error validated by simulations.

Contribution

It introduces an analysis of residual errors when a student applies an optimal learning algorithm to a mismatched teacher, with validated approximations.

Findings

Optimal algorithms cause residual errors with wrong teachers.

Residual error approximation matches simulation results.

Mild conditions enable effective error estimation.

Abstract

We explore the effects of over-specificity in learning algorithms by investigating the behavior of a student, suited to learn optimally from a teacher $\mathbf{B}$, learning from a teacher $\mathbf{B}'\neq\mathbf{B}$. We only considered the supervised, on-line learning scenario with teachers selected from a particular family. We found that, in the general case, the application of the optimal algorithm to the wrong teacher produces a residual generalization error, even if the right teacher is harder. By imposing mild conditions to the learning algorithm form we obtained an approximation for the residual generalization error. Simulations carried in finite networks validate the estimate found.