Learning from Mistakes -- A Framework for Neural Architecture Search

TL;DR

This paper introduces a novel machine learning framework inspired by human learning from mistakes, applying it to neural architecture search to improve model performance on standard datasets.

Contribution

The paper proposes the Learning From Mistakes (LFM) framework and develops an efficient algorithm, applying it to neural architecture search for the first time.

Findings

LFM improves neural architecture search results.

LFM achieves state-of-the-art performance on CIFAR-10, CIFAR-100, and ImageNet.

The framework effectively focuses learning on mistakes to enhance model accuracy.

Abstract

Learning from one's mistakes is an effective human learning technique where the learners focus more on the topics where mistakes were made, so as to deepen their understanding. In this paper, we investigate if this human learning strategy can be applied in machine learning. We propose a novel machine learning method called Learning From Mistakes (LFM), wherein the learner improves its ability to learn by focusing more on the mistakes during revision. We formulate LFM as a three-stage optimization problem: 1) learner learns; 2) learner re-learns focusing on the mistakes, and; 3) learner validates its learning. We develop an efficient algorithm to solve the LFM problem. We apply the LFM framework to neural architecture search on CIFAR-10, CIFAR-100, and Imagenet. Experimental results strongly demonstrate the effectiveness of our model.