NAS-Bench-1Shot1: Benchmarking and Dissecting One-shot Neural Architecture Search

TL;DR

This paper introduces a comprehensive benchmarking framework for one-shot neural architecture search (NAS), enabling systematic analysis of different methods, hyperparameter sensitivity, and comparison with blackbox optimizers, to deepen understanding of weight-sharing NAS algorithms.

Contribution

It presents a general framework and benchmarking platform for one-shot NAS, facilitating scientific analysis and comparison of various methods and hyperparameter effects.

Findings

State-of-the-art methods vary significantly in performance.

Hyperparameter tuning can notably improve NAS results.

Blackbox optimizers offer competitive alternatives.

Abstract

One-shot neural architecture search (NAS) has played a crucial role in making NAS methods computationally feasible in practice. Nevertheless, there is still a lack of understanding on how these weight-sharing algorithms exactly work due to the many factors controlling the dynamics of the process. In order to allow a scientific study of these components, we introduce a general framework for one-shot NAS that can be instantiated to many recently-introduced variants and introduce a general benchmarking framework that draws on the recent large-scale tabular benchmark NAS-Bench-101 for cheap anytime evaluations of one-shot NAS methods. To showcase the framework, we compare several state-of-the-art one-shot NAS methods, examine how sensitive they are to their hyperparameters and how they can be improved by tuning their hyperparameters, and compare their performance to that of blackbox optimizers for NAS-Bench-101.