Learning by Teaching, with Application to Neural Architecture Search

TL;DR

This paper introduces a novel learning framework inspired by human teaching, where a teacher model iteratively improves itself by teaching a student model, and applies it to neural architecture search across multiple datasets.

Contribution

The paper proposes the learning by teaching (LBT) framework, a new three-level optimization method for training models by iterative teaching and learning, applied to neural architecture search.

Findings

LBT improves neural architecture search performance.

The method achieves competitive results on CIFAR-10, CIFAR-100, and ImageNet.

LBT demonstrates effective model training through iterative teaching.

Abstract

In human learning, an effective skill in improving learning outcomes is learning by teaching: a learner deepens his/her understanding of a topic by teaching this topic to others. In this paper, we aim to borrow this teaching-driven learning methodology from humans and leverage it to train more performant machine learning models, by proposing a novel ML framework referred to as learning by teaching (LBT). In the LBT framework, a teacher model improves itself by teaching a student model to learn well. Specifically, the teacher creates a pseudo-labeled dataset and uses it to train a student model. Based on how the student performs on a validation dataset, the teacher re-learns its model and re-teaches the student until the student achieves great validation performance. Our framework is based on three-level optimization which contains three stages: teacher learns; teacher teaches student; teacher re-learns based on how well the student performs. A simple but efficient algorithm is developed to solve the three-level optimization problem. We apply LBT to search neural architectures on CIFAR-10, CIFAR-100, and ImageNet. The efficacy of our method is demonstrated in various experiments.