Small-Group Learning, with Application to Neural Architecture Search

TL;DR

This paper introduces small-group learning (SGL), a novel collaborative training framework for machine learning models inspired by human small-group learning, improving neural architecture search results on multiple datasets.

Contribution

We propose a new multi-level optimization framework for collaborative model training, applying it to neural architecture search to enhance model performance.

Findings

SGL improves neural architecture search outcomes on CIFAR-100, CIFAR-10, and ImageNet.

The framework effectively leverages model collaboration via pseudo-labeling.

Experimental results demonstrate the superiority of SGL over baseline methods.

Abstract

In human learning, an effective learning methodology is small-group learning: a small group of students work together towards the same learning objective, where they express their understanding of a topic to their peers, compare their ideas, and help each other to trouble-shoot problems. In this paper, we aim to investigate whether this human learning method can be borrowed to train better machine learning models, by developing a novel ML framework -- small-group learning (SGL). In our framework, a group of learners (ML models) with different model architectures collaboratively help each other to learn by leveraging their complementary advantages. Specifically, each learner uses its intermediately trained model to generate a pseudo-labeled dataset and re-trains its model using pseudo-labeled datasets generated by other learners. SGL is formulated as a multi-level optimization framework consisting of three learning stages: each learner trains a model independently and uses this model to perform pseudo-labeling; each learner trains another model using datasets pseudo-labeled by other learners; learners improve their architectures by minimizing validation losses. An efficient algorithm is developed to solve the multi-level optimization problem. We apply SGL for neural architecture search. Results on CIFAR-100, CIFAR-10, and ImageNet demonstrate the effectiveness of our method.