Neural Architecture Search: Insights from 1000 Papers

TL;DR

This survey comprehensively reviews neural architecture search (NAS), highlighting its rapid development, key techniques, resources, and its surpassing of human-designed architectures in various deep learning tasks.

Contribution

It provides an organized taxonomy of NAS methods, search spaces, algorithms, and resources, offering a comprehensive guide to the rapidly evolving field.

Findings

NAS has outperformed human-designed architectures on many tasks.

Over 1000 papers on NAS have been published since 2020.

The survey organizes NAS research into search spaces, algorithms, and resources.

Abstract

In the past decade, advances in deep learning have resulted in breakthroughs in a variety of areas, including computer vision, natural language understanding, speech recognition, and reinforcement learning. Specialized, high-performing neural architectures are crucial to the success of deep learning in these areas. Neural architecture search (NAS), the process of automating the design of neural architectures for a given task, is an inevitable next step in automating machine learning and has already outpaced the best human-designed architectures on many tasks. In the past few years, research in NAS has been progressing rapidly, with over 1000 papers released since 2020 (Deng and Lindauer, 2021). In this survey, we provide an organized and comprehensive guide to neural architecture search. We give a taxonomy of search spaces, algorithms, and speedup techniques, and we discuss resources such as benchmarks, best practices, other surveys, and open-source libraries.