Learning from Mistakes based on Class Weighting with Application to Neural Architecture Search

TL;DR

This paper introduces LFM-CW, a class weighting framework inspired by human mistake-driven learning, which improves neural architecture search by re-weighting classes based on performance, leading to lower error rates.

Contribution

The paper proposes a novel multi-level optimization framework, LFM-CW, that leverages class weighting based on validation loss to enhance neural architecture search.

Findings

Achieves lower error rates on CIFAR and ImageNet datasets.

Effective re-weighting improves model performance.

Outperforms baseline methods in neural architecture search.

Abstract

Learning from mistakes is an effective learning approach widely used in human learning, where a learner pays greater focus on mistakes to circumvent them in the future to improve the overall learning outcomes. In this work, we aim to investigate how effectively we can leverage this exceptional learning ability to improve machine learning models. We propose a simple and effective multi-level optimization framework called learning from mistakes using class weighting (LFM-CW), inspired by mistake-driven learning to train better machine learning models. In this formulation, the primary objective is to train a model to perform effectively on target tasks by using a re-weighting technique. We learn the class weights by minimizing the validation loss of the model and re-train the model with the synthetic data from the image generator weighted by class-wise performance and real data. We apply our LFM-CW framework with differential architecture search methods on image classification datasets such as CIFAR and ImageNet, where the results show that our proposed strategy achieves lower error rate than the baselines.