NAAP-440 Dataset and Baseline for Neural Architecture Accuracy Prediction

TL;DR

This paper introduces the NAAP-440 dataset and demonstrates that existing regression algorithms can accurately predict neural architecture accuracy early in training, significantly speeding up neural architecture search.

Contribution

The paper provides a new dataset and baseline method for predicting neural network accuracy from minimal training, aiding faster neural architecture search.

Findings

Regression algorithms can predict accuracy with minimal training epochs.

Predicted accuracy maintains correct ranking order with few violations.

The dataset and code are publicly available.

Abstract

Neural architecture search (NAS) has become a common approach to developing and discovering new neural architectures for different target platforms and purposes. However, scanning the search space is comprised of long training processes of many candidate architectures, which is costly in terms of computational resources and time. Regression algorithms are a common tool to predicting a candidate architecture's accuracy, which can dramatically accelerate the search procedure. We aim at proposing a new baseline that will support the development of regression algorithms that can predict an architecture's accuracy just from its scheme, or by only training it for a minimal number of epochs. Therefore, we introduce the NAAP-440 dataset of 440 neural architectures, which were trained on CIFAR10 using a fixed recipe. Our experiments indicate that by using off-the-shelf regression algorithms and running up to 10% of the training process, not only is it possible to predict an architecture's accuracy rather precisely, but that the values predicted for the architectures also maintain their accuracy order with a minimal number of monotonicity violations. This approach may serve as a powerful tool for accelerating NAS-based studies and thus dramatically increase their efficiency. The dataset and code used in the study have been made public.