Progressive Neural Architecture Search

TL;DR

This paper introduces a sequential model-based optimization approach for neural architecture search that significantly improves efficiency over previous reinforcement learning methods, leading to state-of-the-art CNN structures.

Contribution

It presents a novel SMBO-based method for neural architecture search that is more efficient and effective than existing RL and evolutionary approaches.

Findings

Up to 5x fewer models evaluated compared to RL methods

Achieves state-of-the-art accuracy on CIFAR-10 and ImageNet

8x faster in total compute time than previous methods

Abstract

We propose a new method for learning the structure of convolutional neural networks (CNNs) that is more efficient than recent state-of-the-art methods based on reinforcement learning and evolutionary algorithms. Our approach uses a sequential model-based optimization (SMBO) strategy, in which we search for structures in order of increasing complexity, while simultaneously learning a surrogate model to guide the search through structure space. Direct comparison under the same search space shows that our method is up to 5 times more efficient than the RL method of Zoph et al. (2018) in terms of number of models evaluated, and 8 times faster in terms of total compute. The structures we discover in this way achieve state of the art classification accuracies on CIFAR-10 and ImageNet.