Substitute Teacher Networks: Learning with Almost No Supervision

TL;DR

This paper introduces a scalable, low-supervision training method for neural networks, demonstrating its effectiveness in a baking task and challenging traditional supervised learning paradigms.

Contribution

It presents a novel almost no supervision training algorithm that is scalable and effective, with application to learning to bake and comprehensive quantitative analysis.

Findings

Outperforms current state-of-the-art in baking task

Highly scalable with many student networks

Provides rigorous quantitative validation

Abstract

Learning through experience is time-consuming, inefficient and often bad for your cortisol levels. To address this problem, a number of recently proposed teacher-student methods have demonstrated the benefits of private tuition, in which a single model learns from an ensemble of more experienced tutors. Unfortunately, the cost of such supervision restricts good representations to a privileged minority. Unsupervised learning can be used to lower tuition fees, but runs the risk of producing networks that require extracurriculum learning to strengthen their CVs and create their own LinkedIn profiles. Inspired by the logo on a promotional stress ball at a local recruitment fair, we make the following three contributions. First, we propose a novel almost no supervision training algorithm that is effective, yet highly scalable in the number of student networks being supervised, ensuring that education remains affordable. Second, we demonstrate our approach on a typical use case: learning to bake, developing a method that tastily surpasses the current state of the art. Finally, we provide a rigorous quantitive analysis of our method, proving that we have access to a calculator. Our work calls into question the long-held dogma that life is the best teacher.