NAS-Bench-101: Towards Reproducible Neural Architecture Search

TL;DR

NAS-Bench-101 provides a comprehensive, publicly available dataset of 423,000 neural architectures with pre-computed training results, enabling rapid evaluation and benchmarking of NAS algorithms without extensive computation.

Contribution

It introduces NAS-Bench-101, a large, reproducible dataset of neural architectures and their training results, facilitating research in NAS by reducing computational barriers.

Findings

Dataset contains over 5 million trained models.

Enables evaluation of NAS algorithms in milliseconds.

Useful for analyzing and benchmarking NAS methods.

Abstract

Recent advances in neural architecture search (NAS) demand tremendous computational resources, which makes it difficult to reproduce experiments and imposes a barrier-to-entry to researchers without access to large-scale computation. We aim to ameliorate these problems by introducing NAS-Bench-101, the first public architecture dataset for NAS research. To build NAS-Bench-101, we carefully constructed a compact, yet expressive, search space, exploiting graph isomorphisms to identify 423k unique convolutional architectures. We trained and evaluated all of these architectures multiple times on CIFAR-10 and compiled the results into a large dataset of over 5 million trained models. This allows researchers to evaluate the quality of a diverse range of models in milliseconds by querying the pre-computed dataset. We demonstrate its utility by analyzing the dataset as a whole and by benchmarking a range of architecture optimization algorithms.